在行動通訊領域中，無線電接取網路（Radio Access Network, RAN）的功耗占比最大，且隨著科技進步及應用需求的多樣化，對於更快速、大量資料傳輸的需求，將大幅提高無線網路的功耗。因此，無線網路的節能策略已成為近年來的重要研究課題，其中一項熱門的解決方案是基地台的休眠策略，其目的是透過在閒置或低流量時段將基地台切換至休眠模式，以降低功率消耗並提升能源使用效率。 隨著機器學習技術的快速發展，強化學習因為其適用於變化性大的環境中的特性，符合實際使用場景中用戶需求不斷變化的情況，因而被應用於基地台的開關決策中。本論文基於此概念，提出了一種適用於異質網路的多代理強化學習模型，在此模型中，每個小型基地台中具有一個獨立的代理，每個代理根據自身的局部觀測，做出開啟或關閉基地台的決策。做出開關決策後，透過負載平衡演算法來平衡所有開啟基地台中連線的用戶數量，根據本論文所參考的基地台功耗計算公式來看，基地台的連線用戶數量對於功耗有著重大影響，且會隨著連線數目增加呈現非線性上升，因此在開關的基礎上結合負載平衡可以進一步降低整體網路功耗。 本論文的實作概念為，在低連線需求時，基地台以休眠為主，而在高連線需求時，則透過負載平衡策略來有效降低功耗，並且不論連線需求的高低最小化斷線用戶裝置的數量。這種方法的好處是避免了單純的開關策略在高連線需求時節能效率降低的情況，在結果分析中，無論網路中用戶設備數量高低，本論文提出的策略均能保持約10%~15%功耗節省，而只犧牲10%左右的傳輸吞吐量。 ;In the field of mobile communications, the Radio Access Network (RAN) accounts for the largest proportion of power consumption. With technological advancements and the diversification of application demands, the increasing need for faster and higher-volume data transmission will significantly raise the power consumption of wireless networks. Therefore, energy-saving strategies for wireless networks have become a critical research focus in recent years. One widely studied solution is the base station sleep mode strategy, which aims to reduce power consumption and improve energy efficiency by switching base stations into sleep mode during idle or low-traffic periods. With the rapid development of machine learning techniques, reinforcement learning (RL) has gained attention due to its suitability for dynamic environments where user demands continuously change. RL has thus been applied to base station on/off decision-making. Based on this concept, this thesis proposes a multi-agent reinforcement learning (MARL) model tailored for heterogeneous networks. In this model, each small cell base station is controlled by an independent agent that makes decisions to turn the base station on or off based on its local observations. After the on/off decisions are made, a load balancing algorithm is applied to evenly distribute user connections among the active base stations. According to the base station power consumption formula referenced in this study, the number of connected users has a significant impact on power consumption, which increases nonlinearly with the number of connections. Therefore, combining load balancing with the on/off switching strategy can further reduce the overall network power consumption. The implementation concept of this thesis is to prioritize putting base stations into sleep mode during periods of low connection demand, while applying load balancing strategies under high demand to effectively reduce power consumption. Furthermore, the approach aims to minimize the number of disconnected user devices regardless of traffic load. This method overcomes the inefficiency of pure on/off strategies under heavy traffic conditions. Experimental results indicate that, regardless of the number of user devices in the network, the proposed strategy consistently achieves approximately 10% to 15% power savings while only sacrificing about 10% of transmission throughput.
