連網無人機路徑規劃與基地台連線策略之共同設計：使用模仿增強的深度強化學習方法

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：29

、訪客IP：3.133.149.168

姓名

黃大祐(Da-Yu Huang) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

連網無人機路徑規劃與基地台連線策略之共同設計：使用模仿增強的深度強化學習方法
(Joint Trajectory Design and BS Association for Cellular-Connected UAV: An Imitation Augmented Deep Reinforcement Learning Approach)

相關論文

★ 基於多代理人強化學習方法多架無人機自主追蹤之研究	★ 網路編碼於多架無人機網路以抵禦機器學習攻擊之研究
★ 使用多代理人強化學習於無線快取網路設計空中基地台三維路徑之研究	★ 空中無線感測：在無人機抖動環境下基於RSS的非接觸式人體偵測與定位
★ 協作式自編碼器嵌入最佳化方法於無人機群網路以接收訊號強度輔助定位之研究

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 (2024-9-1以後開放)

摘要(中)

在本論文中，我們考慮一個蜂窩網路連線無人機的路徑設計和基地
台連線策略問題。為了維持可靠連線需求，無人機在飛行期間必須與
蜂窩網路保持連線。由於地面基地台對無人機的天線增益會隨著無人
機位置而改變，因此無人機對基地台的連線策略應與飛行路徑共同考
慮，然而在現有文獻中尚未研究無人機路徑與連線策略的共同設計。
在本論文中，我們首先提出了共同規劃基地台連線策略和路徑設計的
問題，其目標為確保無人機與基地台在可容忍中斷連線的限制下最小
化任務完成時間。然後，我們提出一個深度學習框架解決該非凸最佳
化問題。對於無人機與基地台的連線策略，我們建構了一個在指定區
域內的信號強度無線電圖，用於訓練深度神經網路以近似無人機位置
和最佳連線基地台的非線性映射關係。為了解決基地台連線策略和路
徑設計共同決策而導致的高複雜性問題，我們提出了一種新穎的深度
強化學習方法來學習無人機最佳路徑，在該方法中無人機可以從過去
的良好經驗中學習策略。我們的結果顯示了在路徑長度方面，與現有
深度強化學習方法相比，所提出之改進方法更具有優勢。此外，使用
傳統選擇離無人機最近基地台的連線策略無法提供可靠的連線品質要
求，相較之下我們所提出的方法可以確保無人機在整條路徑上與基地
台保持可靠的通訊品質。

摘要(英)

This paper concerns the problem of trajectory design and base station (BS) association for cellular-connected unmanned aerial vehicles (UAVs).
To support safety-critical functions, one primary requirement for UAVs is to maintain reliable cellular connectivity at every time instant during the flight mission.
Since the antenna gain of a ground BS (GBS) changes with the position of the UAV, the UAV-GBS association strategy should be jointly considered with the trajectory design, which has not been studied in the prior arts.
In this paper, we first formulate the problem of joint BS association and trajectory design with the objective of minimizing the mission completion time under a connectivity outage constraint.
Then, a deep learning framework is proposed to solve the formulated non-convex optimization problem in a decoupled manner.
For the UAV-GBS association strategy, the signal strength radio map of a given area is constructed, which is used to train a deep neural network (DNN) to approximate the nonlinear mapping from the UAV position to the optimal GBS.
To tackle the high complexity due to the coupled decision variables of GBS association and UAV movement, a novel deep reinforcement learning (DRL) approach is developed to learn the optimal trajectory, in which the UAV can learn from its own past good experiences.
Our simulation results confirm the superiority of the proposed DRL approach compared to the conventional DRL approaches in terms of trajectory length.
Additionally, it is demonstrated that the nearest association scheme fails to provide reliable cellular connections, whereas our proposed approach can ensure strong connectivity with the GBS during the whole trajectory.

關鍵字(中)

★ 無人機
★ 蜂窩網路
★ 基地台連線
★ 路徑設計
★ 深度強化學習

關鍵字(英)

★ Unmanned aerial vehicle (UAV)
★ cellular networks
★ cell association
★ trajectory design
★ deep reinforcement learning

論文目次

論文摘要 ................................................................................................. i
Abstract .................................................................................................... iii
目錄............................................................................................................. vi
圖目錄.........................................................................................................vii
表目錄......................................................................................................... ix
一、緒論..................................................................................... 1
1.1 研究背景 . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 研究動機與目的 . . . . . . . . . . . . . . . . . . . . 2
1.3 論文架構 . . . . . . . . . . . . . . . . . . . . . . . . 3
二、文獻探討............................................................................. 4
2.1 飛行基地台之路徑設計 . . . . . . . . . . . . . . . . . 4
2.2 蜂窩網路連線之無人機路徑設計 . . . . . . . . . . . 4
2.3 綜合觀點 . . . . . . . . . . . . . . . . . . . . . . . . 5
三、系統模型和問題表述......................................................... 6
3.1 通道模型 . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 基地台與無人機天線模型 . . . . . . . . . . . . . . . 9
3.3 問題描述 . . . . . . . . . . . . . . . . . . . . . . . . 10
四、基於深度強化學習的演算法............................................. 16
4.1 強化學習模型 . . . . . . . . . . . . . . . . . . . . . . 16
4.1.1 狀態空間 . . . . . . . . . . . . . . . . . . . . . . . . 16
4.1.2 動作空間 . . . . . . . . . . . . . . . . . . . . . . . . 17
4.1.3 獎勵函數 . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 模仿增強的深度強化學習 . . . . . . . . . . . . . . . 20
4.3 用於飛行決策的雙深度Q網路 . . . . . . . . . . . . . 21
4.4 用於無人機與基地台連線策略的深度神經網路 . . . 22
4.5 學習演算法 . . . . . . . . . . . . . . . . . . . . . . . 23
4.6 演算法分析 . . . . . . . . . . . . . . . . . . . . . . . 25
五、模擬結果與分析................................................................. 28
5.1 路徑設計的模擬結果 . . . . . . . . . . . . . . . . . . 29
5.2 基地台連線的模擬結果 . . . . . . . . . . . . . . . . . 35
六、結論與貢獻......................................................................... 41
參考文獻..................................................................................................... 42
附錄一......................................................................................................... 49

參考文獻

[1] 3GPP TR 36.777, “Study on Enhanced LTE Support for Aerial Vehicles (Release 15),” Dec. 2017. [Online]. Available: http://www.3gpp.org/ftp//Specs/archive/36 series/36.777/
[2] Y. Zeng, J. Lyu, and R. Zhang, “Cellular-connected UAV: Potential, challenges, and promising technologies,” IEEE Wireless Communications, vol. 26, no. 1, pp. 120–127, 2019.
[3] 3GPP TR 22.825, “Study on Remote Identification of Unmanned Aerial Systems (Release 16),” Sep. 2018. [Online]. Available: https://www.3gpp.org/ftp/Specs/archive/22 series/22.825/
[4] 3GPP TR 22.829, “Enhancement for Unmanned Aerial Vehicles (Release 17),” Sep. 2019. [Online]. Available: https://www.3gpp.org/ftp/Specs/archive/22 series/22.829
[5] Y. J. Chen, D. K. Chang, and C. Zhang, “Autonomous tracking using a swarm of UAVs: A constrained multi-agent reinforcement learning approach,” IEEE Transactions on Vehicular Technology, pp. 1–1, 2020.
[6] H. Wu, X. Tao, N. Zhang, and X. Shen, “Cooperative UAV cluster- assisted terrestrial cellular networks for ubiquitous coverage,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 9, pp. 2045–2058, 2018.
[7] A. Fotouhi, H. Qiang, M. Ding, M. Hassan, L. G. Giordano, A. Garcia- Rodriguez, and J. Yuan, “Survey on UAV cellular communications: Practical aspects, standardization advancements, regulation, and security challenges,” IEEE Communications Surveys Tutorials, vol. 21, no. 4, pp. 3417–3442, 2019.
[8] N. Tafintsev, D. Moltchanov, M. Gerasimenko, M. Gapeyenko, J. Zhu, S. Yeh, N. Himayat, S. Andreev, Y. Koucheryavy, and M. Valkama, “Aerial access and backhaul in mmWave B5G systems: Performance dynamics and optimization,” IEEE Communications Magazine, vol. 58, no. 2, pp. 93–99, 2020.
[9] M. Chen, M. Mozaffari, W. Saad, C. Yin, M. Debbah, and C. S. Hong, “Caching in the sky: Proactive deployment of cache-enabled unmanned aerial vehicles for optimized quality-of-experience,” IEEE Journal on Selected Areas in Communications, vol. 35, no. 5, pp. 1046–1061, 2017.
[10] M. Chen, W. Saad, and C. Yin, “Liquid state machine learning for resource and cache management in LTE-U unmanned aerial vehicle (UAV) networks,” IEEE Transactions on Wireless Communications, vol. 18, no. 3, pp. 1504–1517, 2019.
[11] S. Chai and V. K. N. Lau, “Online trajectory and radio resource optimization of cache-enabled UAV wireless networks with content and energy recharging,” IEEE Transactions on Signal Processing, vol. 68, pp. 1286–1299, 2020.
[12] Y. J. Chen, K. M. Liao, M. L. Ku, and F. P. Tso, “Mobility-aware probabilistic caching in UAV-assisted wireless D2D networks,” in IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1–6.
[13] S. Hosseinalipour, A. Rahmati, and H. Dai, “Interference avoidance position planning in dual-hop and multi-hop UAV relay networks,” IEEE Transactions on Wireless Communications, pp. 1–1, 2020.
[14] X. Zhong, Y. Guo, N. Li, Y. Chen, and S. Li, “Deployment optimization of UAV relay for malfunctioning base station: Model-free approaches,” IEEE Transactions on Vehicular Technology, vol. 68, no. 12, pp. 11971–11984, 2019.
[15] C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 9, pp. 2059–2070, 2018.
[16] P. Yang, X. Cao, X. Xi, W. Du, Z. Xiao, and D. Wu, “Three- dimensional continuous movement control of drone cells for energy- efficient communication coverage,” IEEE Transactions on Vehicular Technology, vol. 68, no. 7, pp. 6535–6546, 2019.
[17] M. M. U. Chowdhury, S. J. Maeng, E. Bulut, and ¨ I. Güvenc ¸, “3D trajectory optimization in UAV-assisted cellular networks considering antenna radiation pattern and backhaul constraint,” IEEE Transactions on Aerospace and Electronic Systems, pp. 1–1, 2020.
[18] S. Zhang, Y. Zeng, and R. Zhang, “Cellular-enabled UAV communication: A connectivity-constrained trajectory optimization perspective,” IEEE Transactions on Communications, vol. 67, no. 3, pp. 2580–2604, 2019.
[19] B. Khamidehi and E. S. Sousa, “A double q-learning approach for navigation of aerial vehicles with connectivity constraint,” arXiv preprint arXiv:2002.10563, 2020.
[20] Y. J. Chen and D. Y. Huang, “Trajectory optimization for cellular- enabled UAV with connectivity outage constraint,” IEEE Access, vol. 8, pp. 29205–29218, 2020.
[21] H. Yang, J. Zhang, S. H. Song, and K. B. Lataief, “Connectivity- aware UAV path planning with aerial coverage maps,” in IEEE Wireless Communications and Networking Conference (WCNC), 2019, pp. 1–6.
[22] M. M. U. Chowdhury, W. Saad, and I. Guvenc, “Mobility management for cellular-connected UAVs: A learning-based approach,” arXiv preprint arXiv:2002.01546, 2020.
[23] Galkin, Boris and Amer, Ramy and Fonseca, Erika and DaSilva, Luiz A, “Intelligent UAV base station selection in urban environments: A supervised learning approach,” arXiv preprint arXiv:2003.01287, 2020.
[24] X. Huang, W. Xu, H. Shen, H. Zhang, and X. You, “Utility-energy efficiency oriented user association with power control in heterogeneous networks,” IEEE Wireless Communications Letters, vol. 7, no. 4, pp. 526–529, 2018.
[25] 3GPP TR 37.840, “Electromagnetic Compatibility (EMC) requirements for Active Antenna Array System (AAS) base station (Release 12),” Dec. 2013. [Online]. Available: https://www.3gpp.org/ftp/Specs/archive/37 - series/37.840/
[26] H. Wang, H. Zhao, J. Zhang, D. Ma, J. Li, and J. Wei, “Survey on unmanned aerial vehicle networks: A cyber physical system perspective,” IEEE Communications Surveys Tutorials, vol. 22, no. 2, pp. 1027–1070, 2020.
[27] N. Tang, H. Tang, B. Li, and X. Yuan, “Joint maneuver and beamwidth optimization for UAV-enabled multicasting,” IEEE Access, vol. 7, pp. 149503–149514, 2019.
[28] M. M. Azari, F. Rosas, K. Chen, and S. Pollin, “Ultra reliable UAV communication using altitude and cooperation diversity,” IEEE Transactions on Communications, vol. 66, no. 1, pp. 330–344, 2018.
[29] M. T. Dabiri, M. Rezaee, B. Maham, W. Saad, and C. S. Hong, “3D channel characterization and performance analysis of UAV-assisted millimeter wave links,” arXiv preprint arXiv:2003.00526, 2020.
[30] N. Senadhira, S. Durrani, X. Zhou, N. Yang, and M. Ding, “Uplink NOMA for cellular-connected UAV: Impact of UAV trajectories and altitude,” IEEE Transactions on Communications, vol. 68, no. 8, pp. 5242–5258, 2020.
[31] B. Galkin, J. Kibiłda, and L. A. DaSilva, “A stochastic model for UAV networks positioned above demand hotspots in urban environments,” IEEE Transactions on Vehicular Technology, vol. 68, no. 7, pp. 6985– 6996, 2019.
[32] Y. Zeng, X. Xu, S. Jin, and R. Zhang, “Simultaneous navigation and radio mapping for cellular-connected UAV with deep reinforcement learning,” arXiv preprint arXiv:2003.07574, 2020.
[33] C. Feng, M. Sun, and J. Zhang, “Reinforced deterministic and probabilistic load forecasting via q -learning dynamic model selection,” IEEE Transactions on Smart Grid, vol. 11, no. 2, pp. 1377–1386, 2020.
[34] A. M. Koushik, F. Hu, and S. Kumar, “Deep Q -learning-based node positioning for throughput-optimal communications in dynamic uav swarm network,” IEEE Transactions on Cognitive Communications and Networking, vol. 5, no. 3, pp. 554–566, 2019.
[35] J. Duan, D. Shi, R. Diao, H. Li, Z. Wang, B. Zhang, D. Bian, and Z. Yi, “Deep-reinforcement-learning-based autonomous voltage control for power grid operations,” IEEE Transactions on Power Systems, vol. 35, no. 1, pp. 814–817, 2020.
[36] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Thirtieth AAAI conference on artificial intelligence, 2016.
[37] Z. Ni, N. Malla, and X. Zhong, “Prioritizing useful experience replay for heuristic dynamic programming-based learning systems,” IEEE Transactions on Cybernetics, vol. 49, no. 11, pp. 3911–3922, 2019.
[38] D. Wang, W. Zhang, B. Song, X. Du, and M. Guizani, “Market- based model in CR-IoT: A Q-probabilistic multi-agent reinforcement learning approach,” IEEE Transactions on Cognitive Communications and Networking, vol. 6, no. 1, pp. 179–188, 2020.
[39] R. Jiao, X. Huang, X. Ma, L. Han, and W. Tian, “A model combining stacked auto encoder and back propagation algorithm for short-term wind power forecasting,” IEEE Access, vol. 6, pp. 17851–17858, 2018.
[40] R. Zhu, G. Jianxin, W. Feng, B. Lin, and Y. Huang, “Energy efficient transmission power control policy of the delay tolerable communication service,” IEEE Access, vol. 8, pp. 175815–175826, 2020.
[41] R. Li, Y. Zhao, C. Wang, X. Wang, V. C. M. Leung, X. Li, and T. Taleb, “Edge caching replacement optimization for D2D wireless networks via weighted distributed DQN,” in IEEE Wireless Communications and Networking Conference (WCNC), 2020, pp. 1–6.
[42] 3GPP TR 38.901, “Study on channel model for frequencies from 0.5 to 100 GHz (Release 15),” Jun. 2018. [Online]. Available: https://www.3gpp.org/ftp/Specs/archive/38 series/38.901/
[43] H. C. Nguyen, R. Amorim, J. Wigard, I. Z. Kovacs, and P. Mogensen, “Using LTE networks for UAV command and control link: A rural- area coverage analysis,” in IEEE 86th Vehicular Technology Conference (VTC-Fall), 2017, pp. 1–6.
[44] M. Alzenad, A. El-Keyi, F. Lagum, and H. Yanikomeroglu, “3-D placement of an unmanned aerial vehicle base station (UAV-BS) for energy-efficient maximal coverage,” IEEE Wireless Communications Letters, vol. 6, no. 4, pp. 434–437, 2017.

指導教授

陳昱嘉(Yu-Jia Chen)

審核日期

2020-12-3

推文