近年來，深度學習的研究越來越廣泛，從基本的影像前處理、影像切割到物件辨識、語意分析等，逐漸取代傳統的影像處理演算法，傳統的手勢辨識演算法在複雜場景上需仰賴深度資訊才能正確辨識，且辨識效果不佳，而深度資訊需使用深度攝影機或雙CMOS攝影機來取得，對於一般使用者並不便利，因此本論文提出基於深度神經網路來進行手勢辨識之方法及其硬體架構設計，僅需單CMOS攝影機即可在複雜場景下辨識手勢，研究可分成兩個部分，一是神經網路模型之訓練，二是硬體架構之實現。在神經網路訓練部分，使用深度可分離卷積架構來建立神經網路模型，在訓練階段，模型區分為手部切割及手勢辨識，藉由先訓練手部切割子模型作為注意力模型，來輔助手勢辨識子模型之辨識率提升；在推理階段，僅需要使用手勢辨識及部分手部切割子模型即可進行手勢辨識，可避免使用全部模型來降低所需之權重參數及運算量。 硬體實現部分，我們設計深度卷積、逐點卷積、批量正規化及最大池化等模組來加速深度可分離卷積模型，並使用內部記憶體來暫存特徵資料，再透過DMA將資料傳送至外部記憶體儲存來減少內部記憶體使用量，同時為了減少權重參數及特徵資料所需之記憶體，資料皆使用定點數16bits來運算及儲存，本文亦規劃乒乓記憶體架構來最大化內部記憶體存取，來減少與外部記憶體存取之次數，全系統在Xilinx ZCU106開發板上實現，由CMOS攝影機將影像輸入至FPGA，手勢辨識完後將原始影像及辨識結果透過HDMI輸出並顯示於螢幕上，處理速度可以達到52.6FPS以及65.6 GOPS的計算量。 ;The research in deep learning has become extensively deep recently, such as image pre-processing, image segmentation, object recognition, semantic analysis, etc. Deep learning has gradually replaced the traditional algorithm. The traditional hand gesture recognition algorithm needs to depend on the depth information to recognize hand gesture correctly in complex backgrounds and yet the recognition rate is not good. The depth information needs to be obtained using a depth camera or a dual CMOS camera, which is not convenient for the common user due to its high price. Therefore, this paper proposes a method using deep neural network for hand gesture recognition and an implementation of its hardware architecture design. It only needs a single CMOS camera which can recognize hand gestures in complex background. The research can be divided into two parts; one is the design of neural network model and second is the implementation of the hardware architecture. In the neural network design part, a depthwise separable convolutional is used to establish a neural network model and the model can be divided into segmentation and classification. By training the segmentation model as the attention model, the recognition rate of the classification model is improved. In the inference stage, hand gesture recognition can be performed by only using the classification model and a part of segmentation model, which avoids the use of the whole model to reduce the amount of weights and calculation. In the hardware implementation part, this work designs depthwise convolution, pointwise convolution, batch normalization and max-pooling to accelerate the depthwise separable convolution. Design uses the on-chip memory to temporarily store the feature data and then transfers the data to the off-chip memory through DMA to reduce the on-chip memory usage. 16 bits fixed-point data is used for weights and feature data so that the memory size of weights and feature data can be reduced along with different calculations. This work also implements a ping-pong memory to maximize on-chip memory usage to reduce the access time to off-chip memory. The whole system is implemented on the Xilinx ZCU106 development board. The image is sent as input to the FPGA by the CMOS camera. After the gesture is recognized, the original image and the recognition result are outputted through the HDMI and displayed on the monitor. The implemented system can achieve the frame rate of 52.6 FPS and 65.6 GOPS.
