本研究針對第一人稱視角 RGB 影像，進行手指指尖即時偵測，並依此於智慧型眼鏡中實作空中手寫輸入的應用。首先，我們以 Unity3D 建立訓練資料集，即利用 3D 手部模型合成於自然場景中以快速地產生大量且高品質的訓練影像與標記資料，同時避免人工標記所可能產生的誤差。我們討論如何讓人工合成影像更貼近實際影像，並利用包含調整背景複雜度、光線明亮度、色彩對比等方式產生多樣化的影像以增加模型的可靠度。接著，我們改良 Mask R-CNN 模型，藉由簡化特徵提取網路，以及改善網路模型對於偵測小物件的適應性，讓所提出的模型在精準度或速度上都為該領域最佳，在 640×480 的 RGB 影像上進行手指偵測，平均像素誤差僅 8.31 像素點，處理畫幀速度達到每秒 38.8 張。最後我們整合手指偵測網路模型於智慧型眼鏡中，以手指指尖移動軌跡作為手寫輸入，再利用 Google Input API 辨識文字以回傳候選字給智慧型眼鏡使用者選擇，建立適用於智慧型眼鏡的新互動輸入法。;This research investigates real-time fingertip detection in RGB images/frames captured from such wearable devices as smart glasses. First, we established a synthetic dataset by using Unity3D and focused on the pointing gesture for egocentric view. The advantage of synthetic data is to avoid manual labeling errors and provide a large benchmark dataset with high quality. We discuss the dataset generation and how to produce the images in a natural way. Second, a modified Mask Regional Convolution Neural Network (Mask R-CNN) is proposed with one region-based CNN for hand detection and another three-layer CNN for locating the fingertip. We employ MobileNetV2 as the backbone network and simplify the number of bottleneck layers to avoid redundant features. Moreover, we improve the accuracy of detecting small objects by employing FPN and RoIAlign. We achieve fingertip detection with 25 milliseconds per frame for the 640×480 resolution by GPU and average 8.31 pixel errors. The processing speed is high enough to facilitate several interesting applications. One application is to trace the location of a user’s fingertip from first-person perspective to form writing trajectories. A text input mechanism for smart glasses can thus be implemented to enable a user to write letters/characters in air as the input and even interact with the system using simple gestures. Experimental results demonstrate the feasibility of this new text input methodology.
