低硬體資源需求的CNN-XGB分類器設計

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：28

、訪客IP：18.227.0.21

姓名

陳勁為(Gin-Wei Chen) 查詢紙本館藏

畢業系所

軟體工程研究所

論文名稱

低硬體資源需求的CNN-XGB分類器設計
(A CNN-XGB classifier with low hardware resource requirement)

相關論文

★ 整合GRAFCET虛擬機器的智慧型控制器開發平台	★ 分散式工業電子看板網路系統設計與實作
★ 設計與實作一個基於雙攝影機視覺系統的雙點觸控螢幕	★ 智慧型機器人的嵌入式計算平台
★ 一個即時移動物偵測與追蹤的嵌入式系統	★ 一個固態硬碟的多處理器架構與分散式控制演算法
★ 基於立體視覺手勢辨識的人機互動系統	★ 整合仿生智慧行為控制的機器人系統晶片設計
★ 嵌入式無線影像感測網路的設計與實作	★ 以雙核心處理器為基礎之車牌辨識系統
★ 基於立體視覺的連續三維手勢辨識	★ 微型、超低功耗無線感測網路控制器設計與硬體實作
★ 串流影像之即時人臉偵測、追蹤與辨識─嵌入式系統設計	★ 一個快速立體視覺系統的嵌入式硬體設計
★ 即時連續影像接合系統設計與實作	★ 基於雙核心平台的嵌入式步態辨識系統

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 (2029-7-22以後開放)

摘要(中)

CNN-XGB架構結合了CNN的特徵提取和XGBoost的分類能力，在許多文獻中其性能優於單獨使用CNN或XGBoost。然而，過深的CNN會導致運算時間增加，為解決此問題，有學者剪去CNN尾端的部分層，試圖使XGBoost取代這些功能並提升效率，但也因此發現了模型性能降低的情形。本研究提出低硬體資源需求的CNN-XGB架構，與其他研究不同的地方在於我們減去了更多層的CNN神經層，並使用影像特徵算法如LBP、HOG輔助CNN，提供更多特徵資料給XGBoost分類器，讓CNN-XGB分類器的性能不會因為使用了深度剪枝的CNN而下降太多。在實驗設計中，我們會逐步減少CNN層數，觀察其效能和性能變化。此外，我們設計了一套自動化程式，可以將XGBoost模型從軟體端快速部署到硬體端。在實驗結果中，我們驗證了剪枝後的CNN雖然會導致CNN-XGB的辨識率下降1~5%，但運算時間和儲存資源分別可以降低10~25%與40~80%，在多模態CNN-XGB實驗中，使用多模態增強後，部分實驗結果顯示CNN-XGB的性能可以回升至與未剪枝前相同，同時保有低資源帶來的效能提升。而在XGB硬體化設計的實驗結果則驗證了XGBoost模型能成功部署在硬體端上，硬體化的XGBoost模型雖然辨識率下降1~6%，但運算速度可以相較軟體端提升至24到32倍。未來，期望能完成CNN部分的硬體化設計並接上本文設計好的XGBoost硬體化設計，讓本文提出的低資源需求的CNN-XGB分類器能夠完整在硬體端實現，並期望能在相關領域中有所貢獻。

摘要(英)

The CNN-XGB architecture combines the feature extraction capabilities of Convolutional Neural Networks (CNN) with the classification power of XGBoost. Many studies have shown that CNN-XGB outperforms using CNN or XGBoost alone. However, deep CNNs can lead to increased computation time. To address this issue, some researchers have pruned the tail end of the CNN layers, attempting to allow XGBoost to replace these functions. However, they have also found that this can lead to a decrease model’s performance. This study proposes a CNN-XGB architecture with low hardware resource requirement. Unlike other studies, we have reduced even more layers from the CNN and utilized image feature algorithms such as Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) to assist the CNN, providing more feature data to the XGBoost classifier. This approach aims to prevent significant performance drops despite using a deeply pruned CNN. In our experimental design, we gradually reduce the number of CNN layers and observe the changes in efficiency and performance. Additionally, we have developed an automated program to quickly deploy the XGBoost model from software to hardware. Our experimental results confirm that although pruning the CNN causes a 1-5% drop in the CNN-XGB recognition rate, computation time and storage resources can be reduced by 10-25% and 40-80%, respectively. In multimodal CNN-XGB experiments, using multimodal enhancement, some results show that the performance of CNN-XGB can recover to the level of the unpruned model while maintaining the efficiency gains brought by low resource usage. In experiments on the hardware implementation of XGBoost, results verify that the XGBoost model can be successfully deployed on hardware. Although the accuracy drops by 1-6%, the computation speed can increase by 24 to 32 times compared to the software implementation. In the future, we aim to complete the hardware design for the CNN part and connect it with the XGBoost hardware design developed in this study. This will enable the proposed low resource requirement CNN-XGB classifier to be fully implemented on hardware, contributing to advancements in the relevant fields.

關鍵字(中)

★ 深度卷積網路
★ 集成學習模型
★ 多決策樹硬體加速器
★ 硬體化設計

關鍵字(英)

★ CNN-XGB
★ XGBoost
★ FPGA

論文目次

摘要 i
誌謝 iii
圖目錄 vi
表目錄 ix
第一章、緒論 1
1.1 研究背景 1
1.2 研究目標 3
1.3 論文架構 3
第二章、文獻回顧 5
2.1 影像特徵擷取演算法 5
2.1.1 局部二值模式(LBP) 5
2.1.2 Hu 動差不變量 8
2.1.3 色彩特徵擷取 10
2.2 深度卷積網路 12
2.2.1 VGG 16
2.2.2 ResNet 17
2.3 集成學習模型 18
2.3.1 Random Forest 20
2.3.2 XGBoost 21
2.4 多決策樹硬體加速器 24
2.5 MIAT系統設計方法論 27
2.5.1 IDEF0階層式模組化設計 27
2.5.2 GRAFCET離散事件建模 29
第三章、低硬體資源需求的CNN-XGB分類器 32
3.1 低硬體資源需求CNN-XGB分類器 32
3.1.1 CNN模組 33
3.1.2 影像特徵演算法模組 34
3.1.3 XGBoost訓練模組 34
3.2 多模態CNN-XGB訓練模組系統架構 35
3.3 多模態CNN-XGB訓練模組離散事件建模 36
第四章、XGBoost硬體化設計 40
4.1 XGBoost硬體化設計 40
4.1.1 XGBoost再訓練模組 41
4.1.2 XGBoost硬體化工具 42
4.2 XGBoost硬體化設計系統架構 46
4.3 XGBoost硬體化設計離散事件建模 48
第五章、實驗 54
5.1 實驗環境 54
5.2 實驗資料集介紹 55
5.3 CNN-XGB實驗 57
5.3.1 CNN深度剪枝實驗 57
5.3.2 CNN-XGB多模態實驗 62
5.4 XGBoost硬體化設計實驗 65
5.4.1 再訓練後的性能比較 65
5.4.2 硬體化程式中的功能驗證 67
5.4.3 XGB硬體化設計合成與驗證 71
第六章、結論與未來展望 75
6.1 結論 75
6.2 未來展望 76
參考文獻 77

參考文獻

[1] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
[2] P. Chea and J. C. Mandell, "Current applications and future directions of deep learning in musculoskeletal radiology," Skeletal radiology, vol. 49, no. 2, pp. 183-197, 2020.
[3] X. Wu, D. Sahoo, and S. C. Hoi, "Recent advances in deep learning for object detection," Neurocomputing, vol. 396, pp. 39-64, 2020.
[4] S. Kuutti, R. Bowden, Y. Jin, P. Barber, and S. Fallah, "A survey of deep learning applications to autonomous vehicle control," IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 2, pp. 712-733, 2020.
[5] G. Yolcu, I. Oztel, S. Kazan, C. Oz, and F. Bunyak, "Deep learning-based face analysis system for monitoring customer interest," Journal of ambient intelligence and humanized computing, vol. 11, pp. 237-248, 2020.
[6] D. H. Hubel and T. N. Wiesel, "Receptive fields, binocular interaction and functional architecture in the cat′s visual cortex," The Journal of physiology, vol. 160, no. 1, p. 106, 1962.
[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, no. 6, pp. 84-90, 2017.
[8] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press, 2016.
[9] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, "Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions," Journal of big Data, vol. 8, pp. 1-74, 2021.
[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems, vol. 25, 2012.
[11] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[12] Kaggle : Your Home for Data Science. Available: https://www.kaggle.com
[13] R. Shyam, S. S. Ayachit, V. Patil, and A. Singh, "Competitive analysis of the top gradient boosting machine learning algorithms," in 2020 2nd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), pp. 191-196, 2020.
[14] B. V. Dasarathy and B. V. Sheela, "A composite classifier system design: Concepts and methodology," Proceedings of the IEEE, vol. 67, no. 5, pp. 708-713, 1979.
[15] R. E. Schapire, "The strength of weak learnability," Machine learning, vol. 5, pp. 197-227, 1990.
[16] L. Breiman, "Bagging predictors," Machine learning, vol. 24, pp. 123-140, 1996.
[17] L. Pang, J. Wang, L. Zhao, C. Wang, and H. Zhan, "A novel protein subcellular localization method with CNN-XGBoost model for Alzheimer′s disease," Frontiers in genetics, vol. 9, p. 751, 2019.
[18] R. H. Paradisa, D. Sarwinda, A. Bustamam, and T. Argyadiva, "Classification of diabetic retinopathy through deep feature extraction and classic machine learning approach," in 2020 3rd International Conference on Information and Communications Technology (ICOIACT), pp. 377-381, 2020.
[19] H. Nasiri and S. Hasani, "Automated detection of COVID-19 cases from chest X-ray images using deep neural network and XGBoost," Radiography, vol. 28, no. 3, pp. 732-738, 2022.
[20] W. Bakasa and S. Viriri, "Vgg16 feature extractor with extreme gradient boost classifier for pancreas cancer prediction," Journal of Imaging, vol. 9, no. 7, p. 138, 2023.
[21] A. Maleki, M. Raahemi, and H. Nasiri, "Breast cancer diagnosis from histopathology images using deep neural network and XGBoost," Biomedical Signal Processing and Control, vol. 86, p. 105152, 2023.
[22] M. Rahman, Y. Cao, X. Sun, B. Li, and Y. Hao, "Deep pre-trained networks as a feature extractor with XGBoost to detect tuberculosis from chest X-ray," Computers & Electrical Engineering, vol. 93, p. 107252, 2021.
[23] A. S. Syed, D. Sierra-Sosa, A. Kumar, and A. Elmaghraby, "A deep convolutional neural network-xgb for direction and severity aware fall detection and activity recognition," Sensors, vol. 22, no. 7, p. 2547, 2022.
[24] G. Baj, I. Gandin, A. Scagnetto, L. Bortolussi, C. Cappelletto, A. Di Lenarda, and G. Barbati, "Machine learning approaches for ECG-based models: discrimination and calibration for atrial fibrillation prediction," 2023.
[25] A. Nawaz, T. Ali, G. Mustafa, S. U. Rehman, and M. R. Rashid, "A novel technique for detecting electricity theft in secure smart grids using CNN and XG-boost," Intelligent Systems with Applications, vol. 17, p. 200168, 2023.
[26] H. Kode, K. Elleithy, and L. Almazedah, "Epileptic Seizure detection in EEG signals using Machine Learning and Deep Learning Techniques," IEEE Access, 2024.
[27] T. Obasi and M. O. Shafiq, "An experimental study of different machine and deep learning techniques for classification of encrypted network traffic," in 2020 IEEE International Conference on Big Data (Big Data), pp. 4690-4699, 2020.
[28] T. Ojala, M. Pietikainen, and D. Harwood, "Performance evaluation of texture measures with classification based on Kullback discrimination of distributions," in Proceedings of 12th international conference on pattern recognition, vol. 1, pp. 582-585, 1994.
[29] T. Ojala, M. Pietikainen, and T. Maenpaa, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns," IEEE Transactions on pattern analysis and machine intelligence, vol. 24, no. 7, pp. 971-987, 2002.
[30] T. Ojala, M. Pietikäinen, and D. Harwood, "A comparative study of texture measures with classification based on featured distributions," Pattern recognition, vol. 29, no. 1, pp. 51-59, 1996.
[31] M.-K. Hu, "Visual pattern recognition by moment invariants," IRE transactions on information theory, vol. 8, no. 2, pp. 179-187, 1962.
[32] H. Zhan and Y. Qi, "Chinese character image retrieval based on moment invariants and shape context," in 2015 IEEE International Conference on Computer and Communications (ICCC), pp. 146-150, 2015.
[33] G. A. Papakostas, V. G. Kaburlasos, and T. Pachidis, "Thermal infrared face recognition based on lattice computing (LC) techniques," in 2013 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1-6, 2013.
[34] S. Kahyaei and M.-S. Moin, "Robust matching of fingerprints using pseudo-Zernike moments," in 2016 4th International Conference on Control, Instrumentation, and Automation (ICCIA), pp. 116-120, 2016.
[35] B. Kaur, G. Joshi, and R. Vig, "Analysis of shape recognition capability of Krawtchouk moments," in International Conference on Computing, Communication & Automation, pp. 1085-1090, 2015.
[36] J. Hafner, H. S. Sawhney, W. Equitz, M. Flickner, and W. Niblack, "Efficient color histogram indexing for quadratic form distance functions," IEEE transactions on pattern analysis and machine intelligence, vol. 17, no. 7, pp. 729-736, 1995.
[37] J. R. Smith and S.-F. Chang, "VisualSEEk: a fully automated content-based image query system," in Proceedings of the fourth ACM international conference on Multimedia, pp. 87-98, 1997.
[38] D. Srivastava, R. Wadhvani, and M. Gyanchandani, "A review: color feature extraction methods for content based image retrieval," International Journal of Computational Engineering & Management, vol. 18, no. 3, pp. 9-13, 2015.
[39] A. R. Smith, "Color gamut transform pairs," ACM Siggraph Computer Graphics, vol. 12, no. 3, pp. 12-19, 1978.
[40] H. Qazanfari, H. Hassanpour, and K. Qazanfari, "Content-based image retrieval using HSV color space features," International Journal of Computer and Information Engineering, vol. 13, no. 10, pp. 533-541, 2019.
[41] M. Ansari and D. K. Singh, "Significance of color spaces and their selection for image processing: a survey," Recent Advances in Computer Science and Communications (Formerly: Recent Patents on Computer Science), vol. 15, no. 7, pp. 946-956, 2022.
[42] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[43] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
[44] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
[45] A. Canziani, A. Paszke, and E. Culurciello, "An analysis of deep neural network models for practical applications," arXiv preprint arXiv:1605.07678, 2016.
[46] A. H. Ribeiro, K. Tiels, L. A. Aguirre, and T. Schön, "Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness," in International conference on artificial intelligence and statistics, pp. 2370-2380, 2020.
[47] Y. Sun, Z. Li, X. Li, and J. Zhang, "Classifier selection and ensemble model for multi-class imbalance learning in education grants prediction," Applied Artificial Intelligence, vol. 35, no. 4, pp. 290-303, 2021.
[48] L. K. Hansen and P. Salamon, "Neural network ensembles," IEEE transactions on pattern analysis and machine intelligence, vol. 12, no. 10, pp. 993-1001, 1990.
[49] L. Breiman, "Random forests," Machine learning, vol. 45, pp. 5-32, 2001.
[50] N. C. Oza and K. Tumer, "Classifier ensembles: Select real-world applications," Information fusion, vol. 9, no. 1, pp. 4-20, 2008.
[51] T. Hastie, S. Rosset, J. Zhu, and H. Zou, "Multi-class adaboost," Statistics and its Interface, vol. 2, no. 3, pp. 349-360, 2009.
[52] T. Chen and C. Guestrin, "Xgboost: A scalable tree boosting system," in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
[53] P. D. Caie, N. Dimitriou, and O. Arandjelović, "Precision medicine in digital pathology via image analysis and machine learning," in Artificial intelligence and deep learning in pathology: Elsevier, pp. 149-173, 2021.
[54] L. Capitaine, R. Genuer, and R. Thiébaut, "Random forests for high-dimensional longitudinal data," Statistical methods in medical research, vol. 30, no. 1, pp. 166-184, 2021.
[55] I. D. Mienye and Y. Sun, "A survey of ensemble learning: Concepts, algorithms, applications, and prospects," IEEE Access, vol. 10, pp. 99129-99149, 2022.
[56] J. H. Friedman, "Greedy function approximation: a gradient boosting machine," Annals of statistics, pp. 1189-1232, 2001.
[57] C. Bentéjac, A. Csörgő, and G. Martínez-Muñoz, "A comparative analysis of gradient boosting algorithms," Artificial Intelligence Review, vol. 54, pp. 1937-1967, 2021.
[58] R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes, "Ensemble selection from libraries of models," in Proceedings of the twenty-first international conference on Machine learning, p. 18, 2004.
[59] Y. Zhang and A. Haghani, "A gradient boosting method to improve travel time prediction," Transportation Research Part C: Emerging Technologies, vol. 58, pp. 308-324, 2015.
[60] A. Alcolea and J. Resano, "FPGA accelerator for gradient boosting decision trees," in Electronics vol. 10, ed, p. 314, 2021.
[61] C.-H. Chen, M.-Y. Lin, and X.-C. Guo, "High-level modeling and synthesis of smart sensor networks for Industrial Internet of Things," Computers & Electrical Engineering, vol. 61, pp. 48-66, 2017.
[62] R. J. Mayer, "IDEF0 function modeling," A Reconstruction of the Original Air Force Wright Aeronautical Laboratory Technical Report, AFWAL-TR-81-4023 (The IDEF0 Yellow Book), Knowledge-Based System Inc, College Station, TX, 1992.

指導教授

陳慶瀚(Ching-Han Chen)

審核日期

2024-7-23

推文