聯合無人機軌跡、用戶連結與充電功率之改良式強化學習於具有接收訊號強度指標輔助之無人機無線充電通訊

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：45

、訪客IP：18.118.4.64

姓名

雷惟婷(Wei-Ting Lei) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

聯合無人機軌跡、用戶連結與充電功率之改良式強化學習於具有接收訊號強度指標輔助之無人機無線充電通訊
(Improved Reinforcement Learning for Joint UAV Trajectory, User Association and Power Charging in RSSI-Assisted Wireless-Powered UAV Communications)

相關論文

★ 基於干擾對齊方法於多用戶多天線下之聯合預編碼器及解碼器設計	★ 應用壓縮感測技術於正交分頻多工系統之稀疏多路徑通道追蹤與通道估計方法
★ 應用於行動LTE 上鏈SC-FDMA 系統之通道等化與資源分配演算法	★ 以因子圖為基礎之感知無線電系統稀疏頻譜偵測
★ Sparse Spectrum Detection with Sub-blocks Partition for Cognitive Radio Systems	★ 中繼網路於多路徑通道環境下基於領航信號的通道估測方法研究
★ 基於代價賽局在裝置對裝置間通訊下之資源分配與使用者劃分	★ 應用於多用戶雙向中繼網路之聯合預編碼器及訊號對齊與天線選擇研究
★ 多用戶波束成型和機會式排程於透明階層式蜂巢式系統	★ 應用於能量採集中繼網路之最佳傳輸策略研究設計及模擬
★ 感知無線電中繼網路下使用能量採集的傳輸策略之設計與模擬	★ 以綠能為觀點的感知無線電下最佳傳輸策略的設計與模擬
★ 二使用者於能量採集網路架構之合作式傳輸策略設計及模擬	★ 基於Q-Learning之雙向能量採集通訊傳輸方法設計與模擬
★ 多輸入多輸出下同時訊息及能量傳輸系統之設計與模擬	★ 附無線充電裝置間通訊於蜂巢式系統之設計與模擬

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 ( 永不開放)

摘要(中)

在此篇論文中，吾人考慮以強化學習來研究搭載多天線的無人機在一無線充電網路(Wireless Powered Communication Network, WPCN)中的飛行之軌跡、用戶傳輸配置以及功率控制。無人機從充電站(power station)充滿電後出發，飛至裝置附近，接著利用無線電充電傳輸裝置(wireless powered transfer)為下行鏈路的裝置充電，裝置藉由獵取無人機所傳輸出來的射頻訊號從中獲取能量，並通過上行鏈路向無人機傳輸訊息。吾人考慮設計一電量有限的無人機之飛行軌跡、功率控制及用戶連結以最大化無人機所接受到的傳輸量。在此系統模型中，無人機能藉由裝置所回傳的接收能量強度得知通道情況，以符合實際情況。除此之外，為了確保無人機的服務品質，使得各裝置都傳輸資訊量相近，吾人另外設計了一個傳輸資訊量的閥值來限制各裝置的傳輸數量。強化學習是運算非凸性問題的一個強大工具，但它本身存在著維度問題，深度強化學習雖能避免維度問題，但難以解釋其中計算的機制。因此，吾人提出一改良式的強化學習，透過平均鄰近的過往所學習過的狀態值去估測當前狀態的期望數值，藉此簡易的方法來改善因維度過大，難以尋得最佳解的困境。由模擬結果可以證明此使用改良式強化學習能避免維度問題並且相較於常用的非線性估計方法複雜度來的更低。

摘要(英)

In this thesis, we investigate the trajectory design, user association, and power control of an unmanned aerial vehicle (UAV) in the wireless power communication network (WPCN). The UAV departs from the power station (PS) with a full battery and flies to the vicinity of each device to charge their battery then collect data from them. The UAV transfers the radio frequency (RF) signal to charge the devices in downlink and the devices harvest energy from the RF signal, then use the harvested energy to transmit data to the UAV in uplink. We jointly consider the trajectory design, power control, and user association of a battery-constrained UAV to maximize the system throughput. In this system model, the UAV can know the channel information through the received signal strength indicator (RSSI) to conform to the real-world situation. Besides, in order to guarantee the quality of service (QoS) so that all devices transmit similar amounts of information, we also design a threshold to limit the amount of the transmitted data from devices. Reinforcement learning (RL) is a strong tool to solve the non-convex problem, however, it suffers from the curse of dimensionality. Despite deep reinforcement learning (DRL) that can circumvent the curse of dimensionality, it is not interpretable. As the result of that, we propose an improved RL which estimates the expected value of the current state by averaging the Q-values of its neighboring states, using a simple method to deal with the dilemma that is too large to find the optimal solution. Numerical results demonstrate that the proposed algorithm can circumvent the curse of dimensionality, and the complexity is lower comparing to the existing non-linear approximation method.

關鍵字(中)

★ 無人機
★ 用戶配置
★ 軌跡設計
★ 功率控制
★ 無線充電通訊網路

關鍵字(英)

★ Unmanned aerial vehicle (UAV)nmanned aerial vehicle (UAV)
★ user association
★ wireless powered communication network (WPCN)

論文目次

摘要.........................i
Abstract....................ii
致謝........................iv
目錄.........................v
圖目錄......................vii
表目錄.....................viii
第一章緒論
1.1研究背景及動機...............1
1.2文獻探討....................3
1.3論文貢獻....................6
第二章系統模型介紹
2.1 無人機通訊系統.............7
2.2 馬可夫決策過程............10
2.3 強化學習介紹..............13
2.4 瓦片編碼..................15
第三章問題討論
3.1 系統模型.................17
3.2 κ-中斷容量...............19
3.3 最佳化問題...............21
3.4 公平性...................22
3.5 基於強化學習的軌跡設計....23
第四章應用聯合軌跡及用戶分配及功率充電之改良式強化學習
4.1 服務品質導向之強化學習 (QRL)....24
4.2 基於鄰近狀態平均的服務品質導向之強化學習(NAQRL)....28
4.3 基於瓦片編碼的服務品質導向之強化學習(TCQRL)....32
4.4 複雜度分析................35
第五章數據模擬
5.1 收斂圖...................37
5.2不同ε之比較................38
5.3 傳輸數據閥值設計不同之比較..39
5.4 裝置擺放位置不同之比較......40
5.5 上行傳輸時間不同之比較......41
5.6 NAQRL平均範圍不同之比較....42
5.7 TCQRL中瓦片層數比較........43
5.8 天線數量之比較..............44
5.9 不同演算法之比較............45
5.10 不同數量使用者之比較.......47
第六章結論....................49
第七章附錄
附錄A.........................50
附錄B..........................52
參考資料.......................53

參考文獻

[1] G. A. Akpakwu, B. J. Silva, G. P. Hancke and A. M. Abu-Mahfouz, "A Survey on 5G Networks for the Internet of Things: Communication Technologies and Challenges," in IEEE Access, vol. 6, pp. 3619-3647, 2018, doi: 10.1109/ACCESS.2017.2779844.
[2] S. Bi, Y. Zeng and R. Zhang, "Wireless powered communication networks: an overview," in IEEE Wireless Communications, vol. 23, no. 2, pp. 10-18, April 2016, doi: 10.1109/MWC.2016.7462480.
[3] A. V. Savkin and H. Huang, "Deployment of Unmanned Aerial Vehicle Base Stations for Optimal Quality of Coverage," in IEEE Wireless Communications Letters, vol. 8, no. 1, pp. 321-324, Feb. 2019, doi: 10.1109/LWC.2018.2872547.
[4] S. Yin, Z. Qu and L. Li, "Uplink Resource Allocation in Cellular Networks with Energy-Constrained UAV Relay," 2018 IEEE 87th Vehicular Technology Conference (VTC Spring), Porto, 2018, pp. 1-5, doi: 0.1109/VTCSpring.2018.8417737.
[5] S. Cho, K. Lee, B. Kang, K. Koo and I. Joe, "Weighted Harvest-Then-Transmit: UAV-Enabled Wireless Powered Communication Networks," in IEEE Access, vol. 6, pp. 72212-72224, 2018, doi: 10.1109/ACCESS.2018.2882128.
[6] L. Xie, J. Xu and R. Zhang, "Throughput Maximization for UAV-Enabled Wireless Powered Communication Networks - Invited Paper," 2018 IEEE 87th Vehicular Technology Conference (VTC Spring), Porto, 2018, pp. 1-7, doi: 10.1109/VTCSpring.2018.8417659.
[7] J. Park, H. Lee, S. Eom and I. Lee, "UAV-Aided Wireless Powered Communication Networks: Trajectory Optimization and Resource Allocation for Minimum Throughput Maximization," in IEEE Access, vol. 7, pp. 134978-134991, 2019, doi: 10.1109/ACCESS.2019.2941278.
[8] J. Tang, J. Song, J. Ou, J. Luo, X. Zhang and K. Wong, "Minimum Throughput Maximization for Multi-UAV Enabled WPCN: A Deep Reinforcement Learning Method," in IEEE Access, vol. 8, pp. 9124-9132, 2020, doi: 10.1109/ACCESS.2020.2964042.
[9] K. Li, W. Ni, E. Tovar and A. Jamalipour, "On-Board Deep Q-Network for UAV-Assisted Online Power Transfer and Data Collection," in IEEE Transactions on Vehicular Technology, vol. 68, no. 12, pp. 12215-12226, Dec. 2019, doi: 10.1109/TVT.2019.2945037.
[10] X. Liu, Y. Liu, Y. Chen and L. Hanzo, "Trajectory Design and Power Control for Multi-UAV Assisted Wireless Networks: A Machine Learning Approach," in IEEE Transactions on Vehicular Technology, vol. 68, no. 8, pp. 7957-7969, Aug. 2019, doi: 10.1109/TVT.2019.2920284.
[11] S. Zhang, Y. Zeng and R. Zhang, "Cellular-Enabled UAV Communication: A Connectivity-Constrained Trajectory Optimization Perspective," in IEEE Transactions on Communications, vol. 67, no. 3, pp. 2580-2604, March 2019, doi: 10.1109/TCOMM.2018.2880468.
[12] S. Zhang, H. Zhang, Q. He, K. Bian and L. Song, "Joint Trajectory and Power Optimization for UAV Relay Networks," in IEEE Communications Letters, vol. 22, no. 1, pp. 161-164, Jan. 2018, doi: 10.1109/LCOMM.2017.2763135.
[13] Y. Huang, J. Xu, L. Qiu and R. Zhang, "Cognitive UAV Communication via Joint Trajectory and Power Control," 2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), Kalamata, 2018, pp. 1-5, doi: 10.1109/SPAWC.2018.8446024.
[14] G. Zhang, Q. Wu, M. Cui and R. Zhang, "Securing UAV Communications via Joint Trajectory and Power Control," in IEEE Transactions on Wireless Communications, vol. 18, no. 2, pp. 1376-1389, Feb. 2019, doi: 10.1109/TWC.2019.2892461.
[15]H. Jung, K. Kim, J. Kim, O. Shin and Y. Shin, "A relay selection scheme using Q-learning algorithm in cooperative wireless communications," 2012 18th Asia-Pacific Conference on Communications (APCC), Jeju Island, 2012, pp. 7-11, doi: 10.1109/APCC.2012.6388091.
[16] C. H. Liu, Z. Chen, J. Tang, J. Xu and C. Piao, "Energy-Efficient UAV Control for Effective and Fair Communication Coverage: A Deep Reinforcement Learning Approach," in IEEE Journal on Selected Areas in Communications, vol. 36, no. 9, pp. 2059-2070, Sept. 2018, doi: 10.1109/JSAC.2018.2864373.
[17] J. P. Leite, P. H. P. de Carvalho and R. D. Vieira, "A flexible framework based on reinforcement learning for adaptive modulation and coding in OFDM wireless systems," 2012 IEEE Wireless Communications and Networking Conference (WCNC), Paris, France, 2012, pp. 809-814, doi: 10.1109/WCNC.2012.6214482.
[18] L. Deng, G. Wu, J. Fu, Y. Zhang and Y. Yang, "Joint Resource Allocation and Trajectory Control for UAV-Enabled Vehicular Communications," in IEEE Access, vol. 7, pp. 132806-132815, 2019, doi: 10.1109/ACCESS.2019.2941727.
[19] A. A. Khuwaja, Y. Chen, N. Zhao, M. Alouini and P. Dobbins, "A Survey of Channel Modeling for UAV Communications," in IEEE Communications Surveys & Tutorials, vol. 20, no. 4, pp. 2804-2821, Fourthquarter 2018, doi: 10.1109/COMST.2018.2856587.
[20] M. Mozaffari, W. Saad, M. Bennis, Y. Nam and M. Debbah, "A Tutorial on UAVs for Wireless Networks: Applications, Challenges, and Open Problems," in IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2334-2360, thirdquarter 2019, doi: 10.1109/COMST.2019.2902862.
[21] A. Al-Hourani, S. Kandeepan and S. Lardner, "Optimal LAP Altitude for Maximum Coverage," in IEEE Wireless Communications Letters, vol. 3, no. 6, pp. 569-572, Dec. 2014, doi: 10.1109/LWC.2014.2342736.
[22] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Second Edition — The MIT Press, Cambridge, MA, 2018
[23] Alexander A. Sherstov and Peter Stone. Function approximation via tile coding: Automating parameter choice. In J.-D. Zucker and I. Saitta, editors, SARA 2005, volume 3607 of Lecture Notes in Artificial Intelligence, pages 194--205, Berlin, 2005. Springer Verlag.
[24] Nadarajah, Saralees. “Exact distribution of the product of m gamma and n Pareto random variables.” J. Comput. Appl. Math. 235 (2011): 4496-4512.
[25] R. Jain, D. Chiu, and W. Hawe, ”A Quantitative Measure of Fairness and
Discrimination for Resource Allocation in Shared Computer Systems,”
DEC Research Report TR-301, Sept. 1984.
[26] Q. Liu, L. Shi, L. Sun, J. Li, M. Ding and F. Shu, "Path Planning for UAV-Mounted Mobile Edge Computing With Deep Reinforcement Learning," in IEEE Transactions on Vehicular Technology, vol. 69, no. 5, pp. 5723-5728, May 2020, doi: 10.1109/TVT.2020.2982508.
[27] M. S. Emigh, E. G. Kriminger, A. J. Brockmeier, J. C. Príncipe and P. M. Pardalos, "Reinforcement Learning in Video Games Using Nearest Neighbor Interpolation and Metric Learning," in IEEE Transactions on Computational Intelligence and AI in Games, vol. 8, no. 1, pp. 56-66, March 2016, doi: 10.1109/TCIAIG.2014.2369345.
[28] M. Ku, Y. Chen and K. J. R. Liu, "Data-Driven Stochastic Models and Policies for Energy Harvesting Sensor Communications," in IEEE Journal on Selected Areas in Communications, vol. 33, no. 8, pp. 1505-1520, Aug. 2015, doi: 10.1109/JSAC.2015.2391651

[29] W. Curran, R. Pocius and W. D. Smart, "Neural networks for incremental dimensionality reduced reinforcement learning," 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, 2017, pp. 1559-1565, doi: 10.1109/IROS.2017.8205962.
[30] Y. Zeng and R. Zhang, “Energy-Efficient UAV Communication With Trajectory Optimization,” in IEEE Transactions on Wireless Communications, vol. 16, no. 6, pp. 3747-3760, Jun. 2017.
[31] Y. Du, K. Wang, K. Yang and G. Zhang, "Energy-Efficient Resource Allocation in UAV Based MEC System for IoT Devices," 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates, 2018, pp. 1-6, doi: 10.1109/GLOCOM.2018.8647789.

指導教授

古孟霖(Meng-Lin Ku)

審核日期

2021-1-6

推文