A Study of Deep Reinforcement Learning on Mobile Traffic Forecasting and Offloading

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：20

、訪客IP：18.216.36.138

姓名

江九地(CHIANG CHIU-TI) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

(A Study of Deep Reinforcement Learning on Mobile Traffic Forecasting and Offloading)

相關論文

★ 基於馬賽克特性之低失真實體電路佈局保密技術	★ 多路徑傳輸控制協定下從無線區域網路到行動網路之無縫換手
★ 感知網路下具預算限制之異質性子頻段分配	★ 下行服務品質排程在多天線傳輸環境下的效能評估
★ 多路徑傳輸控制協定下之整合型壅塞及路徑控制	★ Opportunistic Scheduling for Multicast over Wireless Networks
★ 適用多用戶多輸出輸入系統之低複雜度比例公平性排程設計	★ 利用混合式天線分配之 LTE 異質網路 UE 與 MIMO 模式選擇
★ 基於有限預算標價式拍賣之異質性頻譜分配方法	★ 適用於 MTC 裝置 ID 共享情境之排程式分群方法
★ Efficient Two-Way Vertical Handover with Multipath TCP	★ 多路徑傳輸控制協定下可亂序傳輸之壅塞及排程控制
★ 移動網路下適用於閘道重置之群體換手機制	★ 使用率能小型基地台之拍賣是行動數據分流方法
★ 高速鐵路環境下之通道預測暨比例公平性排程設計	★ 用於行動網路效能評估之混合式物聯網流量產生器

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

隨著移動網路的爆炸性成長，現行的基地台架構將難以負擔未來的流量需求。其中一個解決方式是廣建小型基地台，將大型基地台的流量分流至小型基地台實現負載均衡。
然而，移動網路的流量需求是隨時間改變的，這將造成在不需要的時段開啟過多閒置的小型基地台，因此需要一套機制台控制小型基地台的部屬與使用。
在考量能源效率下，作者提出基於深度增強式學習並結合流量預測以預先反應的角度來解決小型基地台部署時的能源問題。
流量負載平衡框架由環境模型、流量模型和決策模型組成。
整個負載平衡環境是建立在多個大型基地台下覆蓋多個小型基地台。
流量預測模型會根據歷史的流量狀況預測下個時間流量的最大值、平均值和最小值達到多任務聯合學習的優勢。
作者調查並研究了深度學習的各種方法在流量預測上的表現，這些方法包括了 CNN、3D CNN 和 CNN與RNN的結合。
在決策模型上，根據流量預測模型的輸出結果，使用了深度增強式學習進行決策，要預先開啟多少數量的小型基地台才能達到最佳的能源使用效率。
最後，基於真實世界流量資料的實驗證明，隨著移動流量需求快速增長至超出一大型基地台的負荷，從流量預測的角度輔助決策可以更好的平衡大小型基地台間的負載。

摘要(英)

With the explosive growth in demand for mobile traffic, one of the promising solutions is to offload mobile traffic to small cells.
However, mobile traffic is time-varying which will cause large numbers of small cells were turned on at an unnecessary period.
In consideration of the energy problem, the author proposed a deep reinforcement learning based mobile offloading architecture with traffic prediction to solve the problem in a proactive manner.
The offloading architecture is composed of three components, environment, traffic prediction model, and decision model.
The environment comprises multiple macro cells with numerous small cells under their converge to offload mobile traffic.
The traffic prediction model is a multi-task learning architecture which can learn next epoch′s maximum, average, and minimum mobile traffic at the same time.
The author studied multiple popular deep learning approaches, including RNN, 3D CNN, and the combination of CNN and RNN and examined what kind of structure would obtain better prediction accuracy in time series data set, realistic telecommunication data.
And in the decision model, the author implemented a deep Q network which takes charge of how many small cells should be turned on among a macro cell according to the prediction result coming from the traffic prediction model.
The experiments were conducted on realistic mobile data to prove the mobile traffic prediction is beneficial to offloading policy when the traffic demand has skyrocketed.

關鍵字(中)

★ 深度增強式學習
★ 深度學習
★ 負載平衡
★ 流量預測

關鍵字(英)

★ Deep Learning
★ Deep Reinforcement Learning
★ Offloading
★ Mobile Traffic Forecasting

論文目次

1 Introduction 1
1.1 Motivation................................... 1
1.2 RelatedWork ................................. 2 1.2.1 MobileTrafficOffloading ...................... 2 1.2.2 MobileTrafficPrediction....................... 2
1.3 Contribution.................................. 4
2 Background 5
2.1 NeuralNetwork................................ 5 2.1.1 ConvolutionalNeuralNetwork.................... 6 2.1.2 RecurrentNeuralNetwork...................... 6
2.2 ReinforcementLearning ........................... 6 2.2.1 Q-Learning.............................. 7 2.2.2 DeepQNetwork(DQN)....................... 8
3 System Architecture 9
3.1 ArchitectureOverview ............................ 10
3.2 SystemModel................................. 11 3.2.1 CellularNetworkModel ....................... 11 3.2.2 LoadingRate............................. 11 3.2.3 EnergyEfficiency........................... 12 3.2.4 ProblemFormulation......................... 13
4 Data-Sets Description 15
4.1 TelecomItaliaDataSet............................ 15 4.2 BaseStationsDataSet ............................ 16
i
5 Mobile Traffic Prediction 18
5.1 Methodology ................................. 18
5.1.1 MultitaskLearning.......................... 18
5.1.2 RecurrentNeuralNetwork...................... 19
5.1.3 3DConvolutionalNeuralNetwork.................. 20
5.1.4 Combination of Convolutional and Recurrent Neural Networks . . . 20
5.2 PerformanceEvaluation............................ 21 5.2.1 ExperimentalSetup.......................... 21 5.2.2 ComparisonofForecastingApproaches . . . . . . . . . . . . . . . 22 5.2.3 ComparisonofMTLandSTL .................... 25 5.2.4 TheInfluenceofDataPreprocessing................. 26
6 Offloading Experiment 28
6.1 EnvironmentSetup .............................. 28
6.2 PerformanceComparison .......................... 28 6.2.1 PerformanceOverview........................ 29 6.2.2 EnergyEfficiencyComparison.................... 31 6.2.3 CellLoadingComparison ...................... 32
7 Conclusion Bibliography
35 36

參考文獻

[1] Andrej Karpathy, “Cs231n, convolutional neural networks for visual recognition.” [Online]. Available: http://cs231n.github.io/assets/nn1/neural net2.jpeg
[2] LISA lab, “Convolutional neural networks (lenet).” [Online]. Available: http: //deeplearning.net/tutorial/ images/mylenet.png
[3] Denny Britz , “Recurrent neural networks tutorial, part 1 – introduction to rnns.” [Online]. Available: http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/ 2015/09/rnn.jpg
[4] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1.
[5] A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver, “Massively Parallel Methods for Deep Reinforcement Learning,” arXiv, p. 14, 2015. [Online]. Available: http://arxiv.org/abs/1507.04296
[6] X. Chen, J. Wu, Y. Cai, H. Zhang, and T. Chen, “Energy-efficiency oriented traf- fic offloading in wireless networks: A brief survey and a learning approach for het- erogeneous cellular networks,” IEEE Journal on Selected Areas in Communications, vol. 33, no. 4, pp. 627–640, 2015.
[7] P. Sankt, “Lte networks: How far are the achievable capacities from the theoretical ones?” ICUMT2012, 2012.
[8] MARK HEATH, “Lte performance is good, but it’s not that good!” 2010. [Online]. Available: https://http://www.unwiredinsight.com/2010/lte-performance
[9] C. V. N. I. Cisco, “Global mobile data traffic forecast update, 2016–2021,” white paper, 2016.
[10] A. Aijaz, H. Aghvami, and M. Amani, “A survey on mobile data offloading: technical and business perspectives,” IEEE Wireless Communications, vol. 20, no. 2, pp. 104– 112, 2013.
[11] D. Lo ́pez-Pe ́rez, M. Ding, H. Claussen, and A. H. Jafari, “Towards 1 gbps/ue in cel- lular systems: Understanding ultra-dense small cell deployments,” IEEE Communi- cations Surveys & Tutorials, vol. 17, no. 4, pp. 2078–2101, 2015.
[12] X. Ge, S. Tu, G. Mao, C.-X. Wang, and T. Han, “5g ultra-dense cellular networks,” IEEE Wireless Communications, vol. 23, no. 1, pp. 72–79, 2016.
[13] J. Xu, J. Wang, Y. Zhu, Y. Yang, X. Zheng, S. Wang, L. Liu, K. Horneman, and Y. Teng, “Cooperative distributed optimization for the hyper-dense small cell deploy- ment,” IEEE Communications Magazine, vol. 52, no. 5, pp. 61–67, 2014.
[14] A. Imran and A. Zoha, “Challenges in 5G: how to empower SON with big data for enabling 5G,” IEEE Network, vol. 28, no. 6, pp. 27–33, Nov. 2014.
[15] M. Z. Shafiq, L. Ji, A. X. Liu, and J. Wang, “Characterizing and modeling internet traffic dynamics of cellular devices,” ACM SIGMETRICS Performance Evaluation Review, vol. 39, no. 1, p. 265, 2011.
[16] Y.Lv,Y.Duan,W.Kang,Z.Li,andF.-Y.Wang,“Trafficflowpredictionwithbigdata: a deep learning approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 16, no. 2, pp. 865–873, 2015.
[17] T. P. Oliveira, J. S. Barbar, and A. S. Soares, “Computer network traffic prediction: a comparison between traditional and deep learning neural networks,” International Journal of Big Data Intelligence, vol. 3, no. 1, pp. 28–37, 2016.
[18] “Multi-scale Internet traffic forecasting using neural networks and time series meth- ods,” Expert Systems, vol. 29, no. 2, pp. 143–155, 2012.
[19] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
[20] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level con- trol through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
[21] M. A. Marsan, L. Chiaraviglio, D. Ciullo, and M. Meo, “Multiple daily base station switch-offs in cellular networks,” in Communications and Electronics (ICCE), 2012 Fourth International Conference on. IEEE, 2012, pp. 245–250.
[22] C. Liu, B. Natarajan, and H. Xia, “Small cell base station sleep strategies for energy efficiency,” IEEE Transactions on Vehicular Technology, vol. 65, no. 3, pp. 1652– 1661, 2016.
[23] Y. S. Soh, T. Q. Quek, M. Kountouris, and H. Shin, “Energy efficient heterogeneous cellular networks,” IEEE Journal on Selected Areas in Communications, vol. 31, no. 5, pp. 840–850, 2013.
[24] V. A. Siris and M. Anagnostopoulou, “Performance and energy efficiency of mo- bile data offloading with mobility prediction and prefetching,” in World of Wireless, Mobile and Multimedia Networks (WoWMoM), 2013 IEEE 14th International Sym- posium and Workshops on a. IEEE, 2013, pp. 1–6.
[25] F. Xu, Y. Lin, J. Huang, D. Wu, H. Shi, J. Song, and Y. Li, “Big Data Driven Mobile Traffic Understanding and Forecasting: A Time Series Approach,” IEEE Transactions on Services Computing, vol. 9, no. 5, pp. 796–805, Sep. 2016.
[26] X. Zhou, Z. Zhao, R. Li, Y. Zhou, and H. Zhang, “The predictability of cellular net- works traffic,” 2012 International Symposium on Communications and Information Technologies, ISCIT 2012, pp. 973–978, 2012.
[27] R. Li, Z. Zhao, J. Zheng, Y. Chen, C. Mei, Y. Cai, and H. Zhang, “The Learning and Prediction of Application-level Traffic Data in Cellular Networks,” pp. 1–12, 2016. [Online]. Available: http://arxiv.org/abs/1606.04778
[28] W. Huang, G. Song, H. Hong, and K. Xie, “Deep architecture for traffic flow predic- tion: Deep belief networks with multitask learning,” IEEE Transactions on Intelligent Transportation Systems, vol. 15, no. 5, pp. 2191–2201, 2014.
[29] T. P. Oliveira, J. S. Barbar, and A. S. Soares, “Computer network traffic prediction: a comparison between traditional and deep learning neural networks,” International Journal of Big Data Intelligence, vol. 3, no. 1, p. 28, 2016. [Online]. Available: http://www.inderscience.com/link.php?id=73903
[30] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” California Univ San Diego La Jolla Inst for Cognitive Science, Tech. Rep., 1985.
[31] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, “Handwritten digit recognition with a back-propagation network,” in Advances in neural information processing systems, 1990, pp. 396–404.
[32] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279– 292, 1992.
[33] L.-J. Lin, “Reinforcement learning for robots using neural networks,” Carnegie- Mellon Univ Pittsburgh PA School of Computer Science, Tech. Rep., 1993.
[34] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning.” in AAAI, 2016, pp. 2094–2100.
[35] E. Bjo ̈rnson, L. Sanguinetti, J. Hoydis, and M. Debbah, “Designing multi-user mimo for energy efficiency: When is massive mimo the answer?” in Wireless Communica- tions and Networking Conference (WCNC), 2014 IEEE. IEEE, 2014, pp. 242–247.
[36] G. Barlacchi, M. De Nadai, R. Larcher, A. Casella, C. Chitic, G. Torrisi, F. Antonelli, A. Vespignani, A. Pentland, and B. Lepri, “A multi-source dataset of urban life in the city of Milan and the Province of Trentino.” Scientific data, vol. 2, p. 150055, 2015.
[37] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[38] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104– 3112.
[39] A.GravesandN.Jaitly,“Towardsend-to-endspeechrecognitionwithrecurrentneural networks,” in Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014, pp. 1764–1772.
[40] D.Tran,L.Bourdev,R.Fergus,L.Torresani,andM.Paluri,“Learningspatiotemporal features with 3d convolutional networks,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 4489–4497.
[41] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 1, pp. 221–231, 2013.
[42] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in The IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2015.
[43] K. Y. Chan, T. S. Dillon, J. Singh, and E. Chang, “Neural-Network-Based Models for Short-Term Traffic Flow Forecasting Using a Hybrid Exponential Smoothing and Levenberg–Marquardt Algorithm,” IEEE Transactions on Intelligent Transportation Systems, vol. 13, no. 2, pp. 644–654, Jun. 2012.

指導教授

黃志煒

審核日期

2018-1-23

推文