基於深度強化學習之低軌衛星下鏈通訊多波束追蹤設計與模擬

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：109

、訪客IP：3.133.139.164

姓名

林耕誼(Geng-Yi Lin) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

基於深度強化學習之低軌衛星下鏈通訊多波束追蹤設計與模擬
(Design and Simulation of Deep Reinforcement Learning-based Multi-beam Tracking for LEO Satellite Downlink Communications)

相關論文

★ 基於干擾對齊方法於多用戶多天線下之聯合預編碼器及解碼器設計	★ 應用壓縮感測技術於正交分頻多工系統之稀疏多路徑通道追蹤與通道估計方法
★ 應用於行動LTE 上鏈SC-FDMA 系統之通道等化與資源分配演算法	★ 以因子圖為基礎之感知無線電系統稀疏頻譜偵測
★ Sparse Spectrum Detection with Sub-blocks Partition for Cognitive Radio Systems	★ 中繼網路於多路徑通道環境下基於領航信號的通道估測方法研究
★ 基於代價賽局在裝置對裝置間通訊下之資源分配與使用者劃分	★ 應用於多用戶雙向中繼網路之聯合預編碼器及訊號對齊與天線選擇研究
★ 多用戶波束成型和機會式排程於透明階層式蜂巢式系統	★ 應用於能量採集中繼網路之最佳傳輸策略研究設計及模擬
★ 感知無線電中繼網路下使用能量採集的傳輸策略之設計與模擬	★ 以綠能為觀點的感知無線電下最佳傳輸策略的設計與模擬
★ 二使用者於能量採集網路架構之合作式傳輸策略設計及模擬	★ 基於Q-Learning之雙向能量採集通訊傳輸方法設計與模擬
★ 多輸入多輸出下同時訊息及能量傳輸系統之設計與模擬	★ 附無線充電裝置間通訊於蜂巢式系統之設計與模擬

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 ( 永不開放)

摘要(中)

6G是下一世代的行動通訊技術，它提供比當前5G更快的速度、更低的延遲和更高的頻譜效率，並支持更多設備和更廣泛的應用場景，其中低軌衛星通訊由於運行在地球軌道上，不像地面基礎設施會受到地理位置和地形的限制，因此可以實現全球性的通訊覆蓋，這使得低軌衛星通訊成為實現全球性物聯網的關鍵技術之一。
與高軌衛星通訊相比，低軌衛星通訊還具有低延遲及高可靠性等優點，這使得其成為實現高速數據傳輸、遠程操作和即時通訊等應用的理想選擇。然而，低軌衛星的移動速度非常快，為了應對高都卜勒效應所產生的通道變化，使用多輸入多輸出的波束成形技術進行的波束追蹤被視為有效抵抗快速衰減通道效應且具有高度波束靈活性的替代方案，但隨著波束數量以及用戶數量的增加，地面下鏈多用戶間的訊號干擾問題將更為嚴重。
本論文針對低軌衛星與地面用戶間的通訊服務進行設計，藉由波束成形技術設計多組波束，並結合數位預編碼方法進行波束權重係數設計，以抑制同區域地面用戶間的波束內干擾問題。而通過強化學習方法，吾人可根據低軌衛星軌跡資訊動態調整多波束角度追蹤策略，抑制不同區域地面用戶間造成的波束間干擾問題，提昇衛星通訊品質以達到最大總資料傳輸速率。數值模擬顯示，基於深度強化學習之低軌衛星下鏈通訊的傳輸吞吐量明顯優於本論文進行比較的其他三個演算法。

摘要(英)

6G is a next-generation mobile communication technology with faster speed, lower latency and higher spectral efficiency, supporting more devices and a wider range of application scenarios than the current 5G. Among them, LEO satellite communication can achieve global communication coverage due to its operation in earth orbit, unlike terrestrial infrastructure which is limited by geographical location and terrain, which makes LEO satellite communication one of the key technologies to realize global IoT.
Compared with high-orbit satellite communication, LEO satellite communication also has the advantages of low latency and high reliability, which makes it well suited for applications such as high-speed data transmission, remote operation, and real-time communication. However, LEO satellites move very fast, and in order to cope with the channel variations caused by high doppler effects, beam tracking with MIMO beamforming techniques is considered an option for high beam flexibility that effectively resists fast fading channel effects, but as the number of beams and users increases, the problem of signal interference among multiple users in the terrestrial downlink will become more severe.
In this paper, we design the communication service between LEO satellites and ground users by beamforming technique to design multiple beams and combine with digital pre-coding method to design beam weight coefficients to suppress the intra-beam interference problem between ground users in the same area. Through the reinforcement learning, we can dynamically adjust the multi-beam angle tracking strategy based on the LEO satellite trajectory information to minimize the interference between beams in different areas, suppress the inter-beam interference between ground users in different areas, and improve the quality of satellite communication to achieve the maximum total data transmission rate. Numerical simulations show that the transmission throughput of LEO satellite downlink communication based on deep reinforcement learning is significantly better than the other three algorithms compared in this paper.

關鍵字(中)

★ 低軌衛星
★ 強化學習
★ 深度神經網路
★ 波束追蹤

關鍵字(英)

★ Low Earth Orbit Satellite
★ Reinforcement Learning
★ Deep Neural Networks
★ Beam Tracking

論文目次

摘要 iii
Abstract iv
致謝 vi
目錄 vii
圖目錄 ix
表目錄 x
符號說明 xi
第一章緒論 1
1-1研究動機 1
1-2文獻探討 4
1.2.1基於強化學習的波束追蹤無線通訊系統文獻探討 4
1.2.2強化學習於衛星通訊設計相關應用 5
1-3論文貢獻 7
第二章背景理論介紹 8
2-1機器學習（Machine Learning） 8
2-2馬可夫決策過程（Markov Decision Processes） 9
2-3強化學習（Reinforcement Learning） 10
2-3-1 Q學習（Q-Learning） 12
2-3-2深度強化學習（Deep Reinforcement Learning） 13
第三章基於強化學習的低軌衛星下鏈通訊系統 16
3-1 低軌衛星通道模型 16
3-2 低軌衛星分波束多重接取系統模型 20
3-3 最佳化問題 21
3-4 基於Q學習的集中式低軌衛星下鏈通訊多波束追蹤設計 22
3-5 基於深度強化學習的集中式低軌衛星下鏈通訊多波束追蹤設計 27
第四章模擬結果 37
4-1 基於強化學習的集中式低軌衛星下鏈通訊多波束追蹤模擬結果 39
第五章結論 47
參考文獻 48
附錄A 53

參考文獻

[1] Y. Zhang, A. Liu, P. Li and S. Jiang, “Deep Learning (DL)-Based Channel Prediction and Hybrid Beamforming for LEO Satellite Massive MIMO System, ” IEEE Internet Things J., vol. 9, no. 23, pp. 23705-23715, Dec. 2022.
[2] M. Giordani and M. Zorzi, “Non-terrestrial Networks in the 6G Era: Challenges and Opportunities,” IEEE Network, vol. 35, no. 2, pp. 244-251, Mar. 2021.
[3] Z. Lin, M. Lin, J. Ouyang, W.-P. Zhu, A. D. Panagopoulos, and M.-S. Alouini, “Robust Secure Beamforming for Multibeam Satellite Communication Systems,” IEEE Trans. Veh. Technol., vol. 68, no. 6, pp. 6202-6206, Jun. 2019.
[4] S. Chen, S. Sun, G. Xu, X. Su, and Y. Cai, “Beam-Space Multiplexing: Practice, Theory, and Trends, From 4G TD-LTE, 5G, to 6G and Beyond,” IEEE Trans. Wireless Commun., vol. 27, no. 2, pp. 162-172, Apr. 2020.
[5] F. Fourati and M. -S. Alouini, “Artificial intelligence for satellite communication: A review,” Intelligent and Converged Networks, vol. 2, no. 3, pp. 213-243, Sep. 2021.
[6] N. Chand, P. Mishra, C. R. Krishna, E. S. Pilli, and M. C. Govil, “A comparative analysis of SVM and its stacking with other classification algorithm for intrusion detection,” in Proc. IEEE ICACCA, pp. 1-6, 2016.
[7] K. Y. Huang, L. C. Shen, K. J. Chen, and M. C. Huang, “Multilayer perceptron with genetic algorithm for well log data inversion,” in Proc. IEEE IGARSS, pp. 1544-1547, 2013.
[8] D. Ciregan, U. Meier, and J. Schmidhuber, “Multi-column deep neural networks for image classification,” in Proc. IEEE CVPR, pp. 3642-3649, 2012.
[9] Y. Li, “Deep reinforcement learning: an overview,” arXiv:1701.07274, 2017.
[10] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436-444, May 2015.
[11] M. Shinzaki, Y. Koda, K. Yamamoto, T. Nishio, M. Morikura, C.-H. Huang, Y. Shirato, and N. Kita, “Deep Reinforcement Learning-based Beam Tracking from mmWave Antennas Installed on Overhead Messenger Wires,” in Proc. IEEE VTC2020-Fall, pp. 1-6, 2020.
[12] S. Kim, G. Kwon, and H. Park, “Q-Learning-Based Low Complexity Beam Tracking for
mmWave Beamforming System,” in Proc. Int. Conf. Inf. Commun. Technol. Converg. (ICTC)., pp. 1451-1455, Oct. 2020.
[13] D. C. Araújo and A. L. F. de Almeida, “Beam Management Solution Using Q-Learning Framework,” in Proc. IEEE CAMSAP, pp. 594-598, 2019.
[14] H.-Chiang, K. -C. Chen, W. Rave, M. K. Marandi, and G. Fettweis, “Multi-UAV mmWave
Beam Tracking using Q-Learning and Interference Mitigation,” in Proc. IEEE ICC Workshops, pp. 1-7, 2020.
[15] K. Ma, D. He, H. Sun, Z. Wang and S. Chen, “Deep Learning Assisted Calibrated Beam Training for Millimeter-Wave Communication Systems,” IEEE Trans. Commun., vol. 69, no. 10, pp. 6706-6721, Oct. 2021.
[16] S. H. Lim, S. Kim, B. Shim and J. W. Choi, “Deep Learning-Based Beam Tracking for Millimeter-Wave Communications Under Mobility,” IEEE Trans. Commun., vol. 69, no. 11, pp. 7458-7469, Nov. 2021.
[17] J. Park, S. Hwang, H. Lee and I. Lee, “Deep Recurrent Q-Network Methods for mmWave Beam Tracking systems,” IEEE Trans. Veh. Technol., vol. 71, no. 12, pp. 13429-13434, Dec. 2022.
[18] H. -L. Chiang, K. -C. Chen, W. Rave, M. Khalili Marandi and G. Fettweis, “Machine-Learning Beam Tracking and Weight Optimization for mmWave Multi-UAV Links, ” IEEE Trans. Wireless Commun., vol. 20, no. 8, pp. 5481-5494, Aug. 2021.
[19] J. Liu, X. Li, T. Fan, S. Lv and M. Shi, “Model-Driven Deep Learning Assisted Beam Tracking for Millimeter-Wave Systems,” IEEE Commun. Lett., vol. 26, no. 10, pp. 2345-2349, Oct. 2022.
[20] H. -S. Ahn, O. Jung, S. Choi, J. -H. Son, D. Chung and G. Kim, “An Optimal Satellite Antenna Profile Using Reinforcement Learning,” IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 41, no. 3, pp. 393-406, May 2011.
[21] S. Chan, H. Lee, S. Kim and D. Oh, “Intelligent Low Complexity Resource Allocation Method for Integrated Satellite-Terrestrial Systems,” IEEE Commun. Lett., vol. 11, no. 5, pp. 1087-1091, May 2022.
[22] H. Tsuchida et al., “Efficient Power Control for Satellite-Borne Batteries Using Q-Learning in Low-Earth-Orbit Satellite Constellations,” IEEE Commun. Lett., vol. 9, no. 6, pp. 809-812, Jun. 2020.
[23] J. Huang, Y. Yang, L. Yin, D. He and Q. Yan, “Deep Reinforcement Learning-Based Power Allocation for Rate-Splitting Multiple Access in 6G LEO Satellite Communication System,” IEEE Commun. Lett., vol. 11, no. 10, pp. 2185-2189, Oct. 2022.
[24] X. Liu, H. Zhang, K. Long, A. Nallanathan and V. C. M. Leung, “Deep Dyna-Reinforcement Learning Based on Random Access Control in LEO Satellite IoT Networks,” IEEE Internet Things J., vol. 9, no. 16, pp. 14818-14828, Aug. 2022.
[25] D. Zhou, M. Sheng, Y. Wang, J. Li and Z. Han, “Machine Learning-Based Resource Allocation in Satellite Networks Supporting Internet of Remote Things,” IEEE Trans. Wireless Commun., vol. 20, no. 10, pp. 6606-6621, Oct. 2021.
[26] J. Yun, T. An, H. Jo, B. -J. Ku, D. Oh and C. Joo, “Dynamic Downlink Interference Management in LEO Satellite Networks Without Direct Communications,” IEEE Access, vol. 11, pp. 24137-24148, 2023.
[27] C. Jiang and X. Zhu, “Reinforcement Learning Based Capacity Management in Multi-Layer Satellite Networks,” IEEE Trans. Wireless Commun. vol. 19, no. 7, pp. 4685-4699, Jul. 2020.
[28] J. Zhao, F. Gao, Q. Wu, S. Jin, Y. Wu, and W. Jia, “Beam Tracking for UAV Mounted SatCom on-the-Move with Massive Antenna Array,” IEEE J. Sel. Areas Commun., vol. 36, no. 2, pp. 363-375, Feb. 2018.
[29] D. Cote, “Using machine learning in communication networks [Invited],” J. Opt. Commun. Netw., vol. 10, no. 10, pp. 400-409, Oct. 2018.
[30] Sutton, R.S., Barto, A.G. “Reinforcement Learning: An Introduction,” MIT Press, 1998.
[31] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
[32] Z. Lin, M. Lin, J. Wang, T. de Cola, and J. Wang, “Joint Beamforming and Power Allocation for Satellite-Terrestrial Integrated Networks with Non-Orthogonal Multiple Access,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 3, pp. 657-670, Jun. 2019.
[33] J. Wang, L. Wang and M. Xu, “Rain Attenuation Analysis of Ka Band Ship-borne Satellite Communication Station In Indian Ocean and Pacific Ocean,” 2020 IEEE 3rd International Conference on Information Communication and Signal Processing (ICICSP), Shanghai, China, pp. 385-388, 2020.
[34] S. Xia, Q. Jiang, C. Zou, and G. Li, “Beam Coverage Comparison of LEO Satellite Systems Based on User Diversification,” IEEE Access, Vol.7, pp. 181656-181667, Dec. 2019.
[35] H. -L. Chiang, K. -C. Chen, W. Rave, et al., “Machine-learning beam tracking and weight optimization for mmWave multi-UAV links,” IEEE Trans. Wireless Commun., vol.20, no.8, pp.5481–5494, 2021.
[36] M. Á. Vázquez, M. R. B. Shankar, C. I. Kourogiorgas, P.-D. Arapoglou, V. Icolari, and S. Chatzinotas, “Precoding, Scheduling, and Link Adaptation in Mobile Interactive Multibeam Satellite Systems,” IEEE J. Sel. Areas Commun., vol. 36, no. 5, pp. 971-980, May 2018.
[37] A. E. Alchalabi, S. Shirmohammadi, S. Mohammed, S. Stoian and K. Vijayasuganthan, “Fair Server Selection in Edge Computing With Q-Value-Normalized Action-Suppressed Quadruple Q-Learning,” IEEE Trans. AI, vol. 2, no. 6, pp. 519-527, Dec. 2021.
[38] K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” arXiv: 1502.01852v1, pp. 1-11, Feb. 2015.
[39] S. Ioffe, and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” arXiv: 1502.03167v3, pp. 1-11, Mar. 2015.
[40] D. Mishkin, and J. Matas, “All you need is a good init,” arXiv: 1511.06422v7, pp. 1-13, Feb. 2016.
[41] Andrej Karpathy’s blog, “Hacker’s guide to Neural Networks,” [Online]. Available: http://karpathy.github.io/neuralnets/.
[42] Frederik Kratzert’s blog, “Understanding the backward pass through Batch NormalizationLayer,”[Online].Available:http://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html.
[43] D. P. Kingma, and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv: 1412.6980v9, pp. 1-15, Jan. 2017.
[44] M. Abadi, A. Agarwal, and et al. “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv:1603.04467v2, pp. 1-19, Mar. 2016.

指導教授

古孟霖(Meng-Lin Ku)

審核日期

2023-7-4

推文