基於區域卷積神經網路之第一人稱視角即時手指偵測

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：28

、訪客IP：18.191.150.207

姓名

陳永瀚(Yung-Han Chen) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

基於區域卷積神經網路之第一人稱視角即時手指偵測
(Egocentric-View Real-Time Fingertip Detection based on Regional Convolutional Neural Networks)

相關論文

★ 基於QT之跨平台無線心率分析系統實現	★ 網路電話之額外訊息傳輸機制
★ 針對與運動比賽精彩畫面相關串場效果之偵測	★ 植基於向量量化之視訊/影像內容驗證技術
★ 植基於串場效果偵測與內容分析之棒球比賽精華擷取系統	★ 以視覺特徵擷取為基礎之影像視訊內容認證技術
★ 使用動態背景補償以偵測與追蹤移動監控畫面之前景物	★ 應用於H.264/AVC視訊內容認證之適應式數位浮水印
★ 棒球比賽精華片段擷取分類系統	★ 利用H.264/AVC特徵之多攝影機即時追蹤系統
★ 利用隱式型態模式之高速公路前車偵測機制	★ 基於時間域與空間域特徵擷取之影片複製偵測機制
★ 結合數位浮水印與興趣區域位元率控制之車行視訊編碼	★ 應用於數位智權管理之H.264/AVC視訊加解密暨數位浮水印機制
★ 基於文字與主播偵測之新聞視訊分析系統	★ 植基於數位浮水印之H.264/AVC視訊內容驗證機制

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

本研究針對第一人稱視角 RGB 影像，進行手指指尖即時偵測，並依此於智慧型眼鏡中實作空中手寫輸入的應用。首先，我們以 Unity3D 建立訓練資料集，即利用 3D 手部模型合成於自然場景中以快速地產生大量且高品質的訓練影像與標記資料，同時避免人工標記所可能產生的誤差。我們討論如何讓人工合成影像更貼近實際影像，並利用包含調整背景複雜度、光線明亮度、色彩對比等方式產生多樣化的影像以增加模型的可靠度。接著，我們改良 Mask R-CNN 模型，藉由簡化特徵提取網路，以及改善網路模型對於偵測小物件的適應性，讓所提出的模型在精準度或速度上都為該領域最佳，在 640×480 的 RGB 影像上進行手指偵測，平均像素誤差僅 8.31 像素點，處理畫幀速度達到每秒 38.8 張。最後我們整合手指偵測網路模型於智慧型眼鏡中，以手指指尖移動軌跡作為手寫輸入，再利用 Google Input API 辨識文字以回傳候選字給智慧型眼鏡使用者選擇，建立適用於智慧型眼鏡的新互動輸入法。

摘要(英)

This research investigates real-time fingertip detection in RGB images/frames captured from such wearable devices as smart glasses. First, we established a synthetic dataset by using Unity3D and focused on the pointing gesture for egocentric view. The advantage of synthetic data is to avoid manual labeling errors and provide a large benchmark dataset with high quality. We discuss the dataset generation and how to produce the images in a natural way. Second, a modified Mask Regional Convolution Neural Network (Mask R-CNN) is proposed with one region-based CNN for hand detection and another three-layer CNN for locating the fingertip. We employ MobileNetV2 as the backbone network and simplify the number of bottleneck layers to avoid redundant features. Moreover, we improve the accuracy of detecting small objects by employing FPN and RoIAlign. We achieve fingertip detection with 25 milliseconds per frame for the 640×480 resolution by GPU and average 8.31 pixel errors. The processing speed is high enough to facilitate several interesting applications. One application is to trace the location of a user’s fingertip from first-person perspective to form writing trajectories. A text input mechanism for smart glasses can thus be implemented to enable a user to write letters/characters in air as the input and even interact with the system using simple gestures. Experimental results demonstrate the feasibility of this new text input methodology.

關鍵字(中)

★ 手指偵測
★ 智慧型眼鏡應用
★ 區域卷積神經網路
★ 空中手寫

關鍵字(英)

★ fingertip
★ smart glasses
★ region proposal network
★ air-writing

論文目次

第一章緒論 1
1.1. 研究動機 1
1.2. 研究貢獻 2
1.3. 論文架構 3
第二章相關研究 4
2.1. 傳統手指定位方法 4
2.2. 深度學習應用於物件偵測 5
2.2.1. R-CNN 5
2.2.2. Faster R-CNN 7
2.2.3. YOLOv3 9
2.2.4. Mask R-CNN 10
2.3. 深度學習應用於手指偵測 12
第三章提出方法 14
3.1. 特徵提取網路 18
3.2. 物件偵測網路架構設計 26
3.2.1. Feature Pyramid Network (FPN) 26
3.2.2. Region Proposal Network (RPN) 28
3.2.3. 3-layer CNN 29
3.3. 人工合成影像訓練資料集 30
第四章實驗結果 37
4.1. 開發環境 37
4.2. 手指偵測之網路訓練成果分析 37
4.2.1. 評估指標 40
4.2.2. 數據比較 41
第五章結論與未來展望 48
5.1. 結論 48
5.2. 未來展望 49
參考文獻 50

參考文獻

[1] K. He, G. Gkioxari, P. Dollár and R. Girshick, "Mask R-CNN," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 2980-2988.
[2] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature Pyramid Networks for Object Detection," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 936-944.
[3] S. Ren, K. He, R. Girshick and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017.
[4] M. de La Gorce, D. J. Fleet and N. Paragios, "Model-Based 3D Hand Pose Estimation from Monocular Video," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 9, pp. 1793-1805, Sept. 2011.
[5] P. Krejov and R. Bowden, "Multi-touchless: Real-time fingertip detection and tracking using geodesic maxima," 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, 2013, pp. 1-7.
[6] Liang, Hui & Yuan, Junsong & Thalmann, Daniel, “3D Fingertip and Palm Tracking in Depth Image Sequences,” MM 2012 - Proceedings of the 20th ACM International Conference on Multimedia, pp. 785-788, 2012
[7] Y. Cao, X. Niu and Y. Dou, "Region-based convolutional neural networks for object detection in very high resolution remote sensing images," 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Changsha, 2016, pp. 548-554.
[8] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 779-788.
[9] J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 6517-6525.
[10] J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv:1804.02767, April, 2018.
[11] Liu, Xiaorui; Huang, Yichao; Zhang, Xin; Jin, Lianwen, “Fingertip in the Eye: A cascaded CNN pipeline for the real-time fingertip detection in egocentric videos,” arXiv:1511.02282, November, 2015.
[12] Y. Huang, X. Liu, X. Zhang and L. Jin, "A Pointing Gesture Based Egocentric Interaction System: Dataset, Approach and Application," 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, 2016, pp. 370-377.
[13] Mukherjee, Sohom; Ahmed, Arif; Prosad Dogra, Debi; Kar, Samarjit; Pratim Roy, Partha, “Fingertip Detection and Tracking for Recognition of Air-Writing in Videos,” arXiv:1809.03016, September, 2018.
[14] Wei Liu and Dragomir Anguelov and Dumitru Erhan and Christian Szegedy and Scott E. Reed and Cheng-Yang Fu and Alexander C. Berg, “SSD: Single Shot MultiBox Detector,” ECCV, 2016.
[15] Simonyan, Karen & Zisserman, Andrew, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv 1409.1556, September, 2014.
[16] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 770-778.
[17] G. Huang, Z. Liu, L. v. d. Maaten and K. Q. Weinberger, "Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 2261-2269.
[18] C. Szegedy et al., "Going deeper with convolutions," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp. 1-9.
[19] F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 1800-1807.
[20] Howard, Andrew G.; Zhu, Menglong; Chen, Bo; Kalenichenko, Dmitry; Wang, Weijun; Weyand, Tobias; Andreetto, Marco; Adam, Hartwig, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, April, 2017.
[21] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 4510-4520.
[22] A. Gupta, A. Vedaldi and A. Zisserman, "Synthetic Data for Text Localisation in Natural Images," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 2315-2324.
[23] Zimmermann, Christian; Brox, Thomas, “Learning to Estimate 3D Hand Pose from Single RGB Images,” arXiv:1705.01389, May, 2017
[24] J. Zhang, J. Jiao, M. Chen, L. Qu, X. Xu and Q. Yang, "A hand pose tracking benchmark from stereo matching," 2017 IEEE International Conference on Image Processing (ICIP), Beijing, 2017, pp. 982-986.
[25] von Zitzewitz, Gustav, “Deep Learning and Real-Time Computer Vision for Mobile Platforms,” 2018.
[26] Yin Guobing, “Deep Learning: Separable Convolution,” https://yinguobing.com/separable-convolution
[27] Joyce Xu, “Deep Learning for Object Detection: A Comprehensive Review,” https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9
[28] Henriques, J. F., Caseiro, R., Martins, P., and Batista, J. “High-speed tracking with kernelized correlation filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):583–596, 2015.
[29] Kalal, Z., Mikolajczyk, K., and Matas, J. “Tracking-learning-detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1409–1422, 2012.
[30] Babenko, B., Yang, M.-H., and Belongie, S. “Robust object tracking with online multiple instance learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8):1619–1632, 2011.

指導教授

蘇柏齊(Po-Chyi Su)

審核日期

2019-8-6

推文