多無人機獵能通訊之優化及強化學習： 飛行軌跡、節點關聯及功率控制設計

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：40

、訪客IP：13.59.36.203

姓名

傅千維(Chien-Wei Fu) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

多無人機獵能通訊之優化及強化學習：飛行軌跡、節點關聯及功率控制設計
(Optimization and Reinforcement Learning for Multi-UAV Enabled Energy Harvesting Communications: UAV Trajectory, User Association and Power Control Designs)

相關論文

★ 基於干擾對齊方法於多用戶多天線下之聯合預編碼器及解碼器設計	★ 應用壓縮感測技術於正交分頻多工系統之稀疏多路徑通道追蹤與通道估計方法
★ 應用於行動LTE 上鏈SC-FDMA 系統之通道等化與資源分配演算法	★ 以因子圖為基礎之感知無線電系統稀疏頻譜偵測
★ Sparse Spectrum Detection with Sub-blocks Partition for Cognitive Radio Systems	★ 中繼網路於多路徑通道環境下基於領航信號的通道估測方法研究
★ 基於代價賽局在裝置對裝置間通訊下之資源分配與使用者劃分	★ 應用於多用戶雙向中繼網路之聯合預編碼器及訊號對齊與天線選擇研究
★ 多用戶波束成型和機會式排程於透明階層式蜂巢式系統	★ 應用於能量採集中繼網路之最佳傳輸策略研究設計及模擬
★ 感知無線電中繼網路下使用能量採集的傳輸策略之設計與模擬	★ 以綠能為觀點的感知無線電下最佳傳輸策略的設計與模擬
★ 二使用者於能量採集網路架構之合作式傳輸策略設計及模擬	★ 基於Q-Learning之雙向能量採集通訊傳輸方法設計與模擬
★ 多輸入多輸出下同時訊息及能量傳輸系統之設計與模擬	★ 附無線充電裝置間通訊於蜂巢式系統之設計與模擬

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 ( 永不開放)

摘要(中)

在物聯網時代，大量低功耗的無線通訊節點將會廣泛佈署，對於佈署在複雜、危險區域的節點，例如：沙漠、荒野、災難、戰場等，節點運作勢必依賴電池作為電力來源，而太陽能已被視為實現永久性無線通訊的有效方式。由於具備高機動性、靈活佈署以及成本低廉，無人機可以被靈活地調度從分佈廣泛的地面無線通訊節點收集感測數據，從而改善無線通訊的能源效率，然而無人機的飛行受限於本身搭載電池的電力限制，有效規劃無人機通訊的資源分配是一大設計挑戰。
本研究考慮多個獵能節點利用從太陽能收集的電量進行上鏈通訊傳輸資料至多台無人機，探討無人機飛行軌跡、無人機與節點通訊關聯以及功率控制策略，以有效管理多台無人機通訊環境下的同頻干擾，為確保公平性，採用最大化最差節點總資料傳輸率作為設計目標。此聯合設計是一個高度非凸問題，並且需要知道未來時間的瞬時獵能狀態和通道狀態資訊，然而這在現實環境中很難預測得知。為克服這些設計難題，本研究首先提出一種基於凸優化的離線方法，該方法僅利用統計平均的獵能狀態和通道狀態資訊，通過應用連續凸逼近和交替優化將問題轉化為三個凸子問題，進而求得無人機飛行軌跡、無人機與節點通訊關聯以及功率控制的離線策略。運用離線策略設計在線強化學習方法，根據即時環境資訊來改善系統效能，在離線優化的飛行路徑上規範多台無人機的飛行走廊，避免無人機進行不必要的飛行探索，藉此提高無人機於強化學習時的學習效率及效能。

摘要(英)

In the era of the Internet of Things (IoT), a large number of low-power wireless communication nodes will be widely deployed. For nodes deployed in complex and dangerous areas, e.g., deserts, wilderness, disasters, and battlefields, the operation mainly relies on batteries as the power source, and solar energy has been regarded as an effective way to achieve permanent wireless communications. Due to the advantages of high mobility, easy deployment, and low cost, unmanned aerial vehicles (UAVs) can be flexibly used to collect data from widely distributed ground wireless nodes, thus improving the energy efficiency of wireless communications. However, the flight of UAVs is limited by the power constraints of their own batteries, and it is an essential issue to appropriately design the resource allocation of UAV communications.
In this paper, we consider multiple solar-powered wireless nodes which utilize the harvested solar energy to transmit collected data to multiple UAVs in the uplink. In this context, we jointly design the UAV flight trajectory, UAV-node communication association, and uplink power control strategy to effectively use the harvested energy and manage the co-channel interference under a finite time horizon. To ensure the fairness of wireless nodes, the design goal is to maximize the worst sum rate among nodes. The joint design problem is highly non-convex and requires the causal (future) knowledge of the instantaneous energy harvesting information (EHI) and channel state information (CSI), which are difficult to predict in reality. To overcome these design challenges, we first propose an offline method based on convex optimization that only utilizes the average EHI and CSI and solve the problem via three convex sub-problems by applying successive convex approximation (SCA) and alternating optimization to find the offline strategy for UAV trajectory, UAV-node communication association, and uplink power control. Using the offline strategy, we further design an online reinforcement learning (RL) method to improve the system performance based on real-time environmental information. An idea of regulated flight corridors of multiple UAVs, based on the offline optimized flight paths, is proposed to avoid unnecessary flight exploration of UAVs and enables us to improve not only the learning efficiency but also the system performance, as compared with the conventional RL method.

關鍵字(中)

★ 無人機通訊
★ 獵能無線通訊
★ 軌跡設計
★ 通訊關聯
★ 功率控制
★ 凸優化
★ 強化學習

關鍵字(英)

★ Unmanned aerial vehicle (UAV) communication
★ energy harvesting (EH) communications
★ UAV trajectory
★ communication association
★ power control
★ convex optimization
★ reinforcement learning (RL)

論文目次

摘要 i
Abstract ii
致謝 iv
目錄 v
圖目錄 vii
表目錄 viii
符號說明 ix
第一章緒論 1
1-1 研究背景與動機 1
1-2 研究目的與問題 4
1-3 文獻探討 6
1.3.1多無人機於無線通訊系統之應用 6
1.3.2無人機於具有獵取再生能源功能之無線通訊系統應用 6
1.3.3強化學習於無線通訊系統及無人機應用 7
1-4 論文貢獻 8
第二章背景理論介紹 9
2-1 太陽能獵取模型( Energy Harvesting Model ) 9
2-2 強化學習( Reinforcement Learning ) 10
第三章多無人機系統之多獵能節點上鏈通訊 12
3-1 系統模型 12
3-2 最佳化問題 15
3-3 多無人機系統之多獵能節點上鏈通訊的離線凸優化設計 16
3.3.1 無人機與節點通訊關聯優化 18
3.3.2 無人機飛行軌跡優化 19
3.3.3 節點發射功率優化 28
3.3.4 飛行軌跡、節點關聯及功率控制聯合優化演算法 30
3.3.5 演算法收斂性 31
第四章基於強化學習之在線多無人機系統之多獵能節點上鏈通訊設計 33
4-1 離線凸優化輔助的強化學習(Convex-Assisted RL,CARL)設計 34
4-2 強化學習狀態轉變 37
4-3 多獵能節點上鏈通訊的傳統強化學習設計 38
4-4 獎勵集設計 39
第五章模擬結果 42
5-1 離線優化收斂圖 44
5-2 離線優化節點數量與節點電池容量效能圖 45
5-3 獎勵集設計方法比較 46
5-4 離線優化與在線優化比較 48
5-5 太陽能獵能時段比較 50
5-6 學習率比較 53
5-7 飛行走廊寬度比較 55
5-8 在線學習總次數比較 56
第六章結論 58
第七章附錄 59
附錄A 59
附錄B 61
附錄C 63
參考文獻 65

參考文獻

[1] Y. Zeng, Q. Wu, and R. Zhang, “Accessing from the sky: A tutorial on UAV communications for 5G and beyond,” Proc. IEEE, vol. 107, no. 12, pp. 2327–2375, Dec. 2019.
[2] US Department of Transportation, “Unmanned Aircraft System (UAS) Service Demand 2015–2035: Literature Review & Projections of Future Usage,” tech. rep., v.0.1, DOT-VNTSC-DoD-13-01, Sep. 2013.
[3] J. Lu, H. Okada, T. Itoh, R. Maeda and T. Harada, “Towards the world smallest wireless sensor nodes with low power consumption for ‘Green’ sensor networks,” in Proc. IEEE ICSENS, pp. 1–4, 2013,
[4] G. A. Akpakwu, B. J. Silva, G. P. Hancke and A. M. Abu-Mahfouz, “A Survey on 5G Networks for the Internet of Things: Communication Technologies and Challenges,” IEEE Access, vol. 6, pp. 3619–3647, 2018.
[5] L. Xie, J. Xu, and R. Zhang, “Throughput maximization for UAVenabled wireless powered communication networks,” IEEE Internet Things J., vol. 6, no. 2, pp. 1690–1703, Apr. 2019.
[6] F. Wu, D. Yang, L. Xiao, and L. Cuthbert, “Energy consumption and completion time tradeoff in rotary-wing UAV Enabled WPCN,” IEEE Access, vol. 7, pp. 79617–79635, Jun. 2019.
[7] J. Tang, J. Song, J. Ou, J. Luo, X. Zhang, and K.-K. Wong, “Minimum throughput maximization for multi-UAV enabled WPCN: A deep reinforcement learning method,” IEEE Access, vol. 8, pp. 9124–9132, Jan. 2020.
[8] J. Park, H. Lee, S. Eom, and I. Lee, “UAV-aided wireless powered communication networks: Trajectory optimization and resource allocation for minimum throughput maximization,” IEEE Access, vol. 7, pp. 134978–134991, 2019.
[9] G. Zhang, Q. Wu, M. Cui, and R. Zhang, “Securing UAV communications via joint trajectory and power control,” IEEE Trans. Wireless Commun., vol. 18, no. 2, pp. 1376–1389, Feb. 2019.
[10] Q. Wu, Y. Zeng, and R. Zhang, “Joint trajectory and communication design for UAV enabled multiple access,” in Proc. IEEE Global Commun. Conf., pp. 1–6, 2017.
[11] W. Mei, Q. Wu, and R. Zhang, “Cellular-connected UAV: Uplink association, power control and interference coordination,” IEEE Trans. Wireless Commun., vol. 18, no. 11, pp. 5380–5393, Nov. 2019.
[12] Keshav Singh, Meng-Lin Ku, Jia-Chin Lin and Tharmalingam Ratnarajah, “Toward Optimal Power Control and Transfer for Energy Harvesting Amplify-and-Forward Relay Networks,” IEEE Trans. Wireless Commun., Vol. 17, No. 8, pp. 4971–4986, Aug. 2018.
[13] Y. Sun, D. Xu, D. W. K. Ng, L. Dai, and R. Schober, “Optimal 3D-trajectory design and resource allocation for solar-powered UAV communication systems,” IEEE Trans. Commun., vol. 67, no. 6, pp. 4281–4298, Jun. 2019.
[14] Y. Che, Y. Lai, S. Luo, K. Wu and L. Duan, “UAV-Aided Information and Energy Transmissions for Cognitive and Sustainable 5G Networks,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1668–1683, Mar. 2021.
[15] Y. Lai, Y. L. Che, S. Luo and K. Wu, “Optimal Wireless Information and Energy Transmissions for UAV-Enabled Cognitive Communication Systems,” in Proc. IEEE Int. Conf. Commun. Syst. (ICCS), pp. 168–172, Dec. 2018.
[16] S. Salehi, J. Hassan, A. Bokani, S. A. Hoseini and S. S. Kanhere, “Poster Abstract: A QoS-aware, Energy-efficient Trajectory Optimization for UAV Base Stations using Q-Learning,” in 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pp. 329–330, 2020.
[17] S. A. Hoseini, J. Hassan, A. Bokani and S. S. Kanhere, “Trajectory Optimization of Flying Energy Sources using Q-Learning to Recharge Hotspot UAVs,” in IEEE INFOCOM Workshops, pp. 683–688, 2020.
[18] R. Chen, X. Li, Y. Sun, S. Li, and Z. Sun, “Multi-UAV coverage scheme for average capacity maximization,” IEEE Commun. Lett., vol. 24, no. 3, pp. 653–657, Mar. 2020.
[19] H. El Hammouti, M. Benjillali, B. Shihada, and M. Alouini, “Learn-as-you-fly: A distributed algorithm for joint 3D placement and user association in multi-UAVs networks,” IEEE Trans. Wireless Commun., vol. 18, no. 12, pp. 5831–5844, Dec. 2019.
[20] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Mobile unmanned aerial vehicles (UAVs) for energy-efficient Internet of Things communications,” IEEE Trans. Wireless Commun., vol. 16, no. 11, pp. 7574–7589, Nov. 2017.
[21] C. Zhan and Y. Zeng, “Aerial-ground cost tradeoff for multi-UAV enabled data collection in wireless sensor networks,” IEEE Trans. Commun., vol. 68, no. 3, pp. 1937–1950, Mar. 2020.
[22] Y. Hu, F. Zhang, T. Tian, and D. Ma, “Placement optimisation method for multi-UAV relay communication,” IET Commun., vol. 14, no. 6, pp. 1005–1015, Apr. 2020.
[23] X. Liu, Y. Liu, Y. Chen, and L. Hanzo, “Trajectory design and power control for multi-UAV assisted wireless networks: a machine learning approach,” IEEE Trans. Veh. Technol., vol. 68, no. 8, pp. 7957–7969, 2018.
[24] R. Chen, X. Li, Y. Sun, S. Li, and Z. Sun, “Multi-UAV coverage scheme for average capacity maximization,” IEEE Commun. Lett., vol. 24, no. 3, pp. 653–657, Mar. 2020.
[25] C. H. Liu, X. Ma, X. Gao, and J. Tang, “Distributed energy-efficient multi-uav navigation for long-term communication coverage by deep reinforcement learning,” IEEE Trans. Mobile Comput., vol. 19, no. 6, pp. 1274–1285, Jun. 2020.
[26] S. Zhang, H. Zhang, B. Di, and L. Song, “Cellular UAV-to-X communications: Design and optimization for multi-UAV networks,” IEEE Trans. Wireless Commun., vol. 18, no. 2, pp. 1346–1359, Feb. 2019.
[27] Q. Wang, W. Zhang, Y. Liu, and Y. Liu, “Multi-UAV dynamic wireless networking with deep reinforcement learning,” IEEE Commun. Lett., vol. 23, no. 12, pp. 2243–2246, Dec. 2019.
[28] Q. Wu, Y. Zeng, and R. Zhang, “Joint trajectory and communication design for multi-UAV enabled wireless networks,” IEEE Trans. Wireless Commun., vol. 17, no. 3, pp. 2109–2121, Mar. 2018.
[29] C. Shen, T. Chang, J. Gong, Y. Zeng, and R. Zhang, “Multi-UAV interference coordination via joint trajectory and power control,” IEEE Trans. Signal Process., vol. 68, pp. 843–858, 2020.
[30] L. Chiaraviglio, L. Amorosi, N. Blefari-Melazzi, P. Dell’Olmo, A. Lo Mastro, C. Natalino, and P. Monti, “Minimum cost design of cellular networks in rural areas with UAVs, optical rings, solar panels, and batteries,” IEEE Trans. Green Commun. Netw., vol. 3, no. 4, pp. 901–918, Dec. 2019.
[31] Dai R. “Path planning of solar-powered unmanned aerial vehicles at low altitude.” in Proc. IEEE MWSCAS, pp. 693–696, 2013.
[32] S. Hosseini, R. Dai, and M. Mesbahi, “Optimal path planning and power allocation for a long endurance solar-powered UAV,” in Proc. IEEE ACC, pp. 2588–2593, 2013.
[33] J. Bowman, J. Brooks, C. Lopez, A. Marcos-Martinez and A. Salman, “Secure Data Collection Using Autonomous Unmanned Aerial Vehicles,” in Proc. Syst. Inf. Eng. Design Symp. (SIEDS), pp. 1-6, 2020.
[34] X. Xu, Y. Zhao, L. Tao and Z. Xu, “Resource Allocation Strategy for Dual UAVs-Assisted MEC System with Hybrid Solar and RF Energy Harvesting,” in Proc. 3rd IEEE Int. Conf. Comput. Commun. Internet (ICCCI), pp. 52-57, 2021.
[35] B. Medepally and N. B. Mehta, “Voluntary energy harvesting relays and selection in cooperative wireless networks,” IEEE Trans. Wireless Commun., vol. 9, no. 11, pp. 3543–3553, Nov. 2010.
[36] C. K. Ho, P. D. Khoa, and P. C. Ming, “Markovian models for harvested energy in wireless communications,” in Proc. IEEE ICCS, pp. 311–315, 2010.
[37] M.-L. Ku, Y. Chen and K. J. Ray Liu, “Data-driven stochastic models and policies for energy harvesting sensor communications,” IEEE J. Sel. Areas Commun., vol. 33, no. 8, pp. 1505–1520, Aug. 2015.
[38] H.-H. Tsai, “Design and simulation of cooperative transmission policies for two-user energy harvesting networks”, Master Thesis, National Central University, 2015.
[39] L. Deng, G. Wu, J. Fu, Y. Zhang, and Y. Yang, “Joint resource allocation and trajectory control for UAV-enabled vehicular communications,” IEEE Access, vol. 7, pp. 132806–132815, 2019.
[40] C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach,” IEEE J. Sel. Areas Commun., vol. 36, no. 9, pp. 2059–2070, 2018.
[41] C. You and R. Zhang, “Hybrid Offline-Online Design for UAV-Enabled Data Harvesting in Probabilistic LoS Channels,” IEEE Trans. Wireless Commun., vol. 19, no. 6, pp. 3753–3768, Jun. 2020.
[42] National Renewable Energy Laboratory. (2012). Solar Radiation Resource Information. [Online]. Available: http://www.nrel.gov/rredc
[43] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Mach. Learn., vol. 8,no. 3, pp. 279–292, 1992.
[44] Al-Hourani, S. Kandeepan and S. Lardner, “Optimal LAP Altitude for Maximum Coverage,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp.569–572, Dec. 2014.
[45] J. Holis and P. Pechac, “Elevation dependent shadowing model for mobile communications via high altitude platforms in built-up areas,” IEEE Trans. Antennas Propag., vol. 56, no. 4, pp. 1078–1084, 2008.
[46] M. Grant and S. Boyd. (2016). CVX: MATLAB Software for Disciplined Convex Programming. [Online]. Available: http://cvxr.com/cvx
[47] U. F. Siddiqi, S. M. Sait and M. Uysal, “Deep Reinforcement Based Power Allocation for the Max-Min Optimization in Non-Orthogonal Multiple Access,” IEEE Access, vol. 8, pp. 211235–211247, 2020.

指導教授

古孟霖(Meng-Lin Ku)

審核日期

2021-8-13

推文