| 摘要: | 本論文針對5G及其後的網絡中新型無線電（NR）車聯網（V2X）通信中的資源分配問題進行了研究。這些網絡的快速發展，特別是在3GPP Release 15及其後的版本中，迫切需要支持高數據率、高可靠性、頻譜效率和低延遲的關鍵安全消息和先進應用的強大解決方案。本文提出了一種基於多智能體強化學習（MARL）的可擴展多連接頻譜管理方法，擴展了NR V2X通信的 MAAC 模型。該方法在複雜、動態環境中提高了吞吐量、成功交付率和頻譜效率，同時減少了延遲和干擾。
我們提出的算法通過適應各種智能體(agent)來適應大規模系統，反映了V2X通信的特點。Transformer機制幫助模型泛化到不同環境，提升驗證能力和總效用。將網絡劃分為較小的協作區域可以減少狀態空間，通過合作和遷移學習展示了可擴展性，從而最大限度地減少決策延遲並有效處理動態變化。這些解決方案有效管理了V2X環境的複雜性，最小化通信開銷和決策複雜性，並利用共享觀察結果提高了大規模部署的實用性。
利用Transformer架構，本方法通過在小規模地圖上集中訓練並在具有不同數量智能體和路側單元（RSU）的不同大規模地圖上進行集中和分散測試來滿足可擴展性要求。此外，引入門控循環單元（GRU）層來支持智能體通信和優化系統性能。這種創新的組合提高了學習效率和決策能力，使智能體之間能夠進行協作學習和策略共享。
第一個研究問題探討了設計一種可擴展的V2X資源分配架構，以優化共享和動態環境中的系統吞吐量、頻譜效率和數據包傳送可靠性。為此，我們開發了一種部分可觀測網絡化馬爾可夫決策過程（MDP）模型，用於分佈式多連接管理。該模型考慮了V2I、V2S和V2V鏈路在資源限制下的整體系統吞吐量、效用、頻譜效率和數據包傳送可靠性。使用城市交通模擬（SUMO）平台進行的評估顯示，該模型顯著提高了吞吐量、成功交付率和頻譜效率，同時減少了延遲和干擾。
第二個研究問題探索了先進的優化和狀態估計技術，以確保動態V2X環境中的資源管理和高性能。這包括利用部分拉格朗日乘數將複雜的優化問題轉化為基於獎勵的系統，並實施具有額外預測層的Transformer基於狀態預測。這些技術能夠準確預測整個系統狀態，提高了在複雜和動態場景中的可擴展性和狀態估計能力。所提出的方法有效地管理了計算複雜性和內存需求，確保了資源管理的穩健性和高性能。
第三個研究問題探討了如何通過增強的智能體協作機制和衛星與車輛網絡的集成來提高V2X系統的網絡可靠性和擴展連接性。引入了創新的協作機制，使智能體能夠更有效地共享經驗和策略，從而提高了學習效率和決策質量。此外，整合衛星和車輛網絡提高了網絡可靠性並擴展了連接性，促進了跨多個區域的無縫通信。結果表明，這些機制顯著提升了V2X系統的性能和可靠性。
總之，本論文為V2X通信中的資源分配挑戰提供了綜合解決方案。通過整合衛星網絡、利用先進的Transformer架構和多智能體強化學習，所提出的方法顯著提升了V2X系統的性能和可靠性。此研究為V2X技術的未來發展奠定了基礎，有助於發展更加智能和高效的交通系統。;This dissertation addresses the significant challenges of resource allocation in New Radio (NR) Vehicle-to-Everything (V2X) communications within 5G and beyond networks. The rapid evolution of these networks, highlighted in 3GPP Release 15 and beyond, necessitates robust solutions to support critical safety messages and advanced applications that demand high data rates, reliability, spectrum efficiency, and low latency. The proposed solution is a scalable multi-connectivity spectrum management approach based on Multi-Agent Reinforcement Learning (MARL), extending the Multi-Agent Actor-Critic (MAAC) model for NR V2X communications. This approach enhances throughput, success delivery rates, and spectrum efficiency while reducing latency and interference in complex, dynamic environments.
Our proposed algorithms adapt to large-scale systems by accommodating various agents, reflecting the nature of V2X communication. Transformer mechanisms help generalize models to varying environments, improving validation capacity and total utility. Clustering the network into smaller collaboration regions reduces the state spaces, demonstrating scalability by minimizing decision delays and efficiently handling dynamic changes through cooperative and transfer learning. These solutions effectively manage the complexities of V2X environments, minimizing communication overhead and decision-making complexity and utilizing shared observations to enhance practicality in large-scale deployments.
Leveraging transformer architectures, this approach satisfies scalability requirements by training on a small-scale map in a centralized manner and testing on different large-scale maps with varying numbers of agents and Roadside Units (RSUs) in both centralized and decentralized manners. A Gated Recurrent Unit (GRU) layer is also introduced to support agent communication and optimize system performance. This innovative combination enhances learning efficiency and decision-making, enabling collaborative learning and policy sharing among agents.
The first research question addresses the design of a scalable V2X resource allocation architecture that optimizes system throughput, spectrum efficiency, and packet delivery reliability in shared and dynamic environments. This is achieved by developing a Partially Observable Networked Markov Decision Process (MDP) model for distributed multi-connectivity management. The model considers overall system throughput, utility, spectrum efficiency, and packet delivery reliability of V2I, V2S, and V2V links with resource limitations. Evaluations using the Simulation of Urban Mobility (SUMO) platform demonstrated that this model significantly improves throughput, success delivery rates, and spectrum efficiency while reducing latency and interference.
The second research question explores advanced optimization and state estimation techniques to ensure robust resource management and high performance in dynamic V2X environments. This involves utilizing Partial Lagrange multipliers to transform complex optimization problems into reward-based systems and implementing transformer-based state prediction with an additional prediction layer. These techniques accurately forecast the full system state, improving scalability and state estimation in complex and dynamic scenarios. The proposed methods efficiently manage computational complexity and memory requirements, ensuring robust resource management and high performance.
The third research question examines how enhanced agent collaboration mechanisms and the integration of satellite and vehicular networks can improve network reliability and expand connectivity in V2X systems. Innovative collaboration mechanisms are introduced, enabling agents to share experiences and policies more effectively, thus enhancing both learning efficiency and decision-making quality. Additionally, integrating satellite and vehicular networks improves network reliability and expands connectivity, facilitating seamless communication across diverse areas. The results show that these mechanisms significantly enhance the performance and reliability of V2X systems.
This dissertation provides a comprehensive solution to resource allocation challenges in V2X communications. By integrating satellite networks, utilizing advanced transformer architectures, and employing multi-agent reinforcement learning, the proposed approach significantly enhances the performance and reliability of V2X systems. This research lays the groundwork for future advancements in V2X technology, contributing to developing more intelligent and efficient transportation systems. |
