深度強化學習之Beyond 5G 輕量化排程器之研製

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：45

、訪客IP：3.142.171.112

姓名

莊于鋅(Yu-Hsin Chuang) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

深度強化學習之Beyond 5G 輕量化排程器之研製
(The Design and Implementation of DRL-based Lightweight Scheduler for Beyond 5G)

相關論文

★ 利用智慧天線系統實現精準室內定位技術	★ 電力線通訊之競爭存取與路由方法設計與實現
★ 設計與實作基於GRAPES函式庫之P2P即時串流系統	★ 利用離散餘弦基礎之聲音浮水印達到室內定位技術
★ 利用虛擬指紋建置法之智慧型天線系統實現精準室內定位技術	★ 即時影像串流自適應播放系統之研究
★ 利用模糊邏輯控制器於蜂巢式網路降低位置管理機制成本	★ 基於支持向量機及模糊推理之地震預警系統研製
★ 基於行動裝置之分散式多人會議系統	★ 以分群為基礎之3D無線與光學網路晶片頻道存取方法
★ 基於收前先聽LBR機制之授權型輔助接入LAA架構下於異質網路中暴露節點之研究	★ 支援跳頻之IEEE 802.15.4 ZigBee無線隨身網路機制設計與實現
★ 應用於IEEE 802.16行動無線都會網路省電模式參數設定之智慧策略	★ IEEE 802.15.4 ZigBee 無線隨身網路高效能路由演算法分析與設計
★ 應用於IEEE 802.16無線寬頻都會網路之具調適性自動重傳請求回報機制	★ 無線感測網路為基礎之空間平面圖自動建構之技術

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 (2025-7-28以後開放)

摘要(中)

隨著行動通訊快速發展，第三代合作夥伴計畫 (3GPP) 於第17 版標準 (Rel-17) 中，於
第五代行動通訊 (5G New Radio，NR) 要容納數以萬計之設備 (User Equipment，UE)，考量
未來行動通訊系統 (Beyond 5G 或 6G) 將同時支援更大量之終端設備，且服務模式由現行下
行主導 (DL dominant) 逐漸改變為上行主導 (UL dominant)，因此需更多頻譜資源傳送資料。
高頻段如FR2 符合頻寬需求。然而，高頻段之特性需使用更大子載波間距 (Sub-Carrier Spacing，
SCS) 抵抗相位噪聲（phase noise）。提高子載波間距也將造成每個時隙 (Slot) 時間縮短，進
一步壓縮資源排程器演算法所需運算時間之上限。若基站無法及時完成上下行資源排程，將
造成資源浪費，違背增加頻寬之初衷。
針對上行傳輸，此研究提出一種新穎之上行排程器—基於深度強化式學習之輕量化排程
器 (DRL-based Lightweight Scheduler，DRL-LS)，依據排隊中封包之資料特徵 (如剩餘延遲預
算 (Remaining Packet Delay Budget，R-PDB) 與服務等級等) 決定上行之優先等級。考慮排程
器演算法運算時間受限，資源排程器僅排程高優先等級封包；針對尚未排程之上行通道資源，
導入基於競爭 (Contention-based) 之通訊協定，使符合條件之上行封包 (亦即較不急迫之封包)
可於此資源內進行隨機競爭，競爭失敗之封包將於後續上行資源進行重傳。當上行封包隨時
間提高優先等級時，將改以傳統排程模式(亦即免競爭 (Contention-free) 模式) 進行排程，以
符合封包延遲預算。此 DRL-LS 透過 DRL 學習如何與通道環境互動於達成最高吞吐量與控
制排程器演算法運算時間之間取得平衡。
最後，為驗證DRL-LS與競爭上行行為之可行性，本研究將上述DRL-LS之代理人 (Agent)
部署於RIC 內，並且使用OpenAirInterface5G (OAI) 與Mosaic5G FlexRIC 開源軟體進行實際
驗證，觀察UEs 進行競爭之行為。

摘要(英)

With the rapid development of mobile communications, the 3rd Generation Partnership Project
(3GPP) in Release 17 specification has aimed to accommodate numerous User Equipments (UE) in
5G New Radio (NR) network. Considering the future mobile communication systems (beyond 5G or
6G), an even larger number of devices should be supported. Furthermore, the transmission behavior
is gradually shifting from downlink-dominant (DL dominant) to uplink-dominant (UL dominant).
This shift necessitates more frequency resources, namely bandwidth (BW), for UL data transmission.
Higher frequency bands, such as FR2, can fulfill the bandwidth requirements. However, the
characteristics of higher frequency bands require larger Sub-Carrier Spacing (SCS) to counteract
phase noise (PN). Increasing the SCS also leads to shorter slot duration, which further shortens the
computational time budget of scheduler. If the base station (gNB) fails to accomplish scheduling in
time, it will result in resource waste consequently.
For uplink transmission, this study proposes a novel uplink scheduler named the Deep
Reinforcement Learning-based Lightweight Scheduler (DRL-LS). The DRL-LS determines the
priority of UL data in the queue by its characteristic, such as the Remaining Packet Delay Budget (RPDB)
and the Quality of Service (QoS). Due to the limited time budget for scheduling, the scheduler
only schedules data of high priority in a contention-free manner. For those unscheduled UL data, a
contention-based (CB) UL transmission protocol is proposed, allowing eligible uplink data (i.e., less
urgent data) to access CB resources in a contention manner. Data packets that experience contention
failure will be retransmitted in subsequent UL resources. To meet the packet delay budget, the priority
of queueing UL data increases over time. Additionally, the DRL-LS would learn how to interact with
the environment to achieve a balance between the maximum throughput and media access protocol.
Finally, to validate the performance of the proposed DRL-LS, this study deploys the agent in the
RAN Intelligence Controller (RIC), which is built over the open platform including the
OpenAirInterface5G (OAI) and Mosaic5G FlexRIC open source software.

關鍵字(中)

★ 第五代行動通訊
★ 第六代行動通訊
★ 深度強化學習
★ 排程器
★ RIC
★ O-RAN

關鍵字(英)

★ Beyond 5G
★ DRL
★ Scheduler
★ RIC

論文目次

中文摘要 i
ABSTRACT ii
CONTENTS iii
LIST OF FIGURES iv
LIST OF TABLES vi
Chapter 1.INTRODUCTION 1
Chapter 2.BACKGROUND 4
2.1 Downlink Control Information (DCI) 4
2.2 PUSCH and UL-SCH Transmission 7
2.3 Scheduler Depictions 15
2.4 Remaining Packet Delay Budget (R-PDB) 18
2.5 Open Radio Access Network (O-RAN) 20
2.6 Deep Reinforcement Learning (DRL) 26
Chapter 3.RELATED WORKS 31
Chapter 4.DRL-based Lightweight Scheduler 33
4.1 System Architecture 33
4.2 Design of DRL-based LS (DRL-LS) 42
Chapter 5.SYSTEM EVALUATIONS AND IMPLEMENTATION 46
5.1 System Setup 46
5.2 Result Evaluation 50
5.3 The Implementation in OpenAirInterface5G and Mosaic5G FlexRIC 58
Chapter 6.CONCLUSION AND FUTURE WORK 68
REFERENCES 70
Appendix A 73

參考文獻

[1] 3GPP TS 23.501 “5G; NR; System architecture for the 5G System (5GS),” v17.7.0, Jan. 2023.
[2] 3GPP TS 38.101-1 “5G; NR; User Equipment (UE) radio transmission and reception; Part 1:
Range 1 Standalone,” v17.6.0, Aug. 2022.
[3] 3GPP TS 38.101-2 “5G; NR; User Equipment (UE) radio transmission and reception; Part 2:
Range 2 Standalone,” v17.8.0, Jan. 2023.
[4] 3GPP TS 38.211 “5G; NR; Physical channels and modulation,” v17.2.0, Jul. 2022.
[5] 3GPP TS 38.212 “5G; NR; Multiplexing and channel coding,” v17.2.0, Jul. 2022.
[6] 3GPP TS 38.213 “5G; NR; Physical layer procedures for control,” v17.2.0, Jul. 2022.
[7] 3GPP TS 38.214 “5G; NR; Physical layer procedures for data,” v17.2.0, Jul. 2022.
[8] 3GPP TS 38.300 “5G; NR; NR and NG-RAN Overall description; Stage-2,” v17.1.0, Aug. 2022.
[9] 3GPP TS 38.321 “5G; NR; Medium Access Control (MAC) protocol specification,” v17.1.0,
Aug. 2022.
[10] 3GPP TS 38.331 “5G; NR; Radio Resource Control (RRC); Protocol specification,” v17.1.0,
Aug. 2022.
[11] 3GPP TS 38.913 “5G; Study on scenarios and requirements for next generation access
technologies,” v17.0.0, May 2022.
[12] A. Mamane, M. Fattah, M. E. Ghazi, M. E. Bekkali, Y. Balboul and S. Mazer, "Scheduling
Algorithms for 5G Networks and Beyond: Classification and Survey," in IEEE Access, vol. 10,
pp. 51643-51661, 2022.
[13] S. Shakkottai and R. Srikant, ‘‘Scheduling real-time traffic with deadlines over a wireless
channel,’’ in Proc. 2nd ACM Int. Workshop Wireless Mobile Multimedia (WOWMOM). New
York, NY, USA: ACM Press, 1999, pp. 35–42. [Online]. Available:
http://portal.acm.org/citation.cfm?doid=313256.313273.
[14] X. Yuan and Z. Duan, ‘‘Fair Round-Robin: A low complexity packet scheduler with
proportional and worst-case fairness,’’ IEEE Trans. Comput., vol. 58, no. 3, pp. 365–379, Mar.
2009. [Online]. Available: http://ieeexplore.ieee.org/document/4626953/
[15] S. Ramabhadran and J. Pasquale, “The Stratified Round Robin Scheduler: Design, Analysis,
and Implementation,” IEEE/ACM Trans. Networking, vol. 14, no. 6, pp. 1362-1373, Dec. 2006.
[16] A. Silberschatz, P. B. Galvin, and G. Gagne, “Operating System Concepts,” nineth edition.
[17] F. Kelly, “Charging and rate control for elastic traffic,” Eur. Trans. Telecommun., vol. 8, pp.
33–37, Jan./Feb. 1997.
[18] 3GPP TSG-RAN WG2 Meeting #115 electronic, “Report of [AT115-e][042][eIAB] fairness,
latency and congestion (Interdigital),” Aug. 2021.
[19] R.S. Sutton and A.G. Barto, “Reinforcement learning: An introduction.,” Cambridge, MA,
USA: MIT press, 2018.
[20] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518,
no. 7540, pp. 529-533, Feb. 2015.
[21] I. Goodfellow, Y. Bengio, and A. Courville, “Deep learning,” Cambridge, MA, USA: MIT
press, 2016.
[22] O-RAN TS “O-RAN Architecture Description,” WG1, OAD-R003-v08.00.
[23] O-RAN TR “AI/ML workflow description and requirements,” WG2, AIML-v01.03.
[24] O-RAN TS “O-RAN Operations and Maintenance Interface Specification,” WG1, O1-
Interface.0-v04.00.
[25] O-RAN TS “Near-RT RIC Architecture,” WG3, RICARCH-R003-v04.00.
[26] O-RAN TS “E2 General Aspects and Principles,” WG3, E2GAP-R003-v03.00.
[27] R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman, “Network configuration protocol
(NETCONF),” Internet Requests for Comments, RFC Editor, RFC 6241, June 2011. [Online].
Available: http://www.rfc- editor.org/rfc/rfc6241.txt
[28] M. Polese, L. Bonati, S. D′Oro, S. Basagni, T. Melodia, “Understanding O-RAN: Architecture,
Interfaces, Algorithms, Security, and Research Challenges", arXiv:2202.01032 [cs.NI], Feb.
2022.
[29] K. Boutiba, M. Bagaa and A. Ksentini, “On using Deep Reinforcement Learning to reduce
Uplink Latency for uRLLC services," GLOBECOM 2022 - 2022 IEEE Global
Communications Conference, Rio de Janeiro, Brazil, pp. 407-412, 2022.
[30] Hossain, Q. C. Li, D. Ying, G. Wu and C. Xiong, “THz Channel Model for 6G
Communications,” 2021 IEEE 32nd Annual International Symposium on Personal, Indoor and
Mobile Radio Communications (PIMRC), Helsinki, Finland, pp. 1-7, 2021.
[31] Y. Xing and T. S. Rappaport, “Propagation Measurement System and Approach at 140 GHz-
Moving to 6G and Above 100 GHz,” 2018 IEEE Global Communications Conference
(GLOBECOM), Abu Dhabi, United Arab Emirates, pp. 1-6, 2018.
[32] S. Tarboush, H. Sarieddeen, M. -S. Alouini and T. Y. Al-Naffouri, “Single- Versus Multicarrier
Terahertz-Band Communications: A Comparative Study,” in IEEE Open Journal of the
Communications Society, vol. 3, pp. 1466-1486, 2022.
[33] T. S. Rappaport et al., “Wireless Communications and Applications Above 100 GHz:
Opportunities and Challenges for 6G and Beyond,” in IEEE Access, vol. 7, pp. 78729-78757,
2019.
[34] L. You, Q. Liao, N. Pappas and D. Yuan, “Resource Optimization with Flexible Numerology
and Frame Structure for Heterogeneous Services,” in IEEE Communications Letters, vol. 22,
no. 12, pp. 2579-2582, Dec. 2018.
[35] Y. Yu, T. Wang, and S. C. Liew, “Deep-Reinforcement Learning Multiple Access for
Heterogeneous Wireless Networks,” 2018 IEEE International Conference on Communications
(ICC), Kansas City, MO, USA, 2018.
[36] Y. Yu, S. C. Liew and T. Wang, “Non-Uniform Time-Step Deep Q-Network for Carrier-Sense
Multiple Access in Heterogeneous Wireless Networks,” in IEEE Transactions on Mobile
Computing, vol. 20, no. 9, pp. 2848-2861, 1 Sept. 2021.
[37] Y. Yu, S. C. Liew and T. Wang, “Multi-Agent Deep Reinforcement Learning Multiple Access
for Heterogeneous Wireless Networks with Imperfect Channels,” in IEEE Transactions on
Mobile Computing, vol. 21, no. 10, pp. 3718-3730, 1 Oct. 2022.
[38] M. Elsayem, H. Abou-zeid, A. Afana and S. Givigi, “Reinforcement Learning-based Dynamic
Resource Allocation for Grant-Free Access,” GLOBECOM 2022 - 2022 IEEE Global
Communications Conference, Rio de Janeiro, Brazil, pp. 1091-1096, 2022.
[39] K. Boutiba, M. Bagaa and A. Ksentini, “Radio Resource Management in Multi-numerology 5G
New Radio featuring Network Slicing,” ICC 2022 - IEEE International Conference on
Communications, Seoul, Korea, Republic of, pp. 359-364, 2022.
[40] Q. Zhou, X. Ye and L. Fu, “Deep Reinforcement Learning Based Scheduling Scheme for the
NR-U/WiGig Coexistence in Unlicensed mmWave Bands,” ICC 2022 - IEEE International
Conference on Communications, Seoul, Korea, Republic of, 2022.
[41] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3, pp. 279–292,
May 1992. [Online]. Available: https://doi.org/10.1007/BF00992698.
[42] OpenAir5GLAB, http://openairinterface.eurecom.fr
[43] OpenAirInterface5G, Gitlab Open Source, https://gitlab.eurecom.fr/oai/openairinterface5g
[44] FlexRIC, Gitlab Open Source, https://gitlab.eurecom.fr/mosaic5g/flexric
[45] S. Parkvall, E. Dahlman, A. Furuskar, and M. Frenne, “NR: The new 5G radio access
technology,” IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 24-30, Dec. 2017.
[46] T. H. Lee and A. Hajimiri, “Oscillator phase noise: a tutorial,” IEEE Journal of Solid-State
Circuits, vol. 35, no. 3 pp. 326-336, March 2000
[47] Robert Schmidt, “Slicing in heterogeneous software-defined radio access networks,”
Networking and Internet Architecture [cs.NI]. Sorbonne Université, 2021. English, NNT:
2021SORUS525, tel- 03783488.
[48] Ettus Research, USRP B210, https://www.ettus.com/all-products/ub210-kit/
[49] Yan Huang, Shaoran, Tom Hou, and Wenjing Lou, “GPU-Based Design to Achieve 100 ??
Scheduling for 5G NR,” Nvidia, Mar. 2019. [Online]. Available:
https://developer.nvidia.com/blog/gpu-based-100-μs-schedulingr-5g-nr/
[50] “White paper on 6G vision and Candidate Technologies,” IMT-2030 6G Promotion Group, Jun.
2021.

指導教授

許獻聰(Shiann-Tsong Sheu)

審核日期

2023-7-28

推文