無細胞大規模 MIMO（Cell‑Free Massive MIMO）是一種將大量 分散式無線存取點（AP）與中央控制器結合的架構，透過多點聯合協 作為覆蓋區域內的用戶提供服務，可顯著提升系統的頻譜效率與覆蓋 均勻性。而編碼簿則是預先定義一組波束向量集合，各 AP 僅需在此 有限集合中選擇最適波束，透過簡化的波束搜尋與回報，減少系統開 銷並提高實時性與可靠性。本研究針對毫米波無細胞大規模 多輸入 多輸出系統，提出一種結合波束選擇與子載波配置的聯邦式深度強化 學習（Federated Deep Reinforcement Learning，FEDRL）框架，以最 小化總傳輸功率為優化目標，並同時滿足服務品質（QoS）要求，我 們的環境資訊包括AP對其服務用戶的路徑損耗和通道狀態資訊，獎 勵設定為傳輸功率限制減去實際的傳輸功率。AP 本地訓練結束後， 定期將網路參數透過 FedAvg 聚合至中央控制器，更新全域策略，再 下傳至各 AP，實現分散式且協調一致的學習過程。模擬結果顯示效 能比深度強化學習好，且與最佳解差異不大，並有運行時間上的優勢。;Cell‑Free Massive MIMO is an architecture that combines a large number of distributed access points (APs) with a central controller. Through multi‑point joint transmission, it serves all users within the coverage area, significantly improving both spectral efficiency and coverage uniformity. A codebook is a predefined set of beamforming vectors from which each AP selects the most suitable beam; by restricting beam selection to this finite set, beam search and feedback are simplified, system overhead is reduced, and real‑time performance and reliability are enhanced. In this study, we target a millimeter‑wave Cell‑Free MIMO OFDM system and propose a Federated Deep Reinforcement Learning (FEDRL) framework that jointly handles beam selection and OFDM subcarrier allocation. Our objective is to minimize total transmission power while satisfying quality‑of‑service (QoS) requirements. The state information consists of the path loss and channel state between each AP and its served users, and the reward is defined as the difference between a predefined power limit and the actual transmission power. After each AP completes its local training, network parameters are periodically aggregated at the central controller via FedAvg, the global policy is updated, and the new policy is redistributed to all APs—thus achieving a distributed yet coordinated learning process. Simulation results demonstrate that our FEDRL approach outperforms conventional deep reinforcement learning, closely approaches the optimal solution, and offers significant execution‑time advantages.
