隨著時代演進，個人電腦到已發展到如今的智慧型手機裝置，人與電腦之間的溝通與交互也越來越重要。在多種形式的應用造成人們對於複雜應用之交互需求也越來越重視，進而發展出各式基於友善輸入之人機交互研究，而最常見的輸入方式既是使用手勢。因此，我們以局部及全局面觀來進行手勢辨識系統之相關研究。以局部層級之手指辨識為例，可進行指部追蹤與偵測，進而完成各項辨識挑戰，如：吉他撥弦演奏、布袋戲操偶及虛擬鍵盤打字，而上述這些手勢行為，可以透過有限狀態機之模型表示。透過結合傳統機於外觀辨識之手指追蹤方法，我們特別提出一個基於有限狀態機手勢辨識方法，並針對簡單的手勢範例做實驗，進行魯棒性之能力測試。在研究的實驗結果中，手勢辨識可以達到識別率的82％。 從全局的面觀來看，我們提出了在序列數據上使用3DCNN和LSTM進行基於深度學習的手勢識別的方法。在我們收集經由設計的數據集後，成功的在測試模型之階段，取得實時應用中的魯棒性。實驗結果證明，離線測試的準確率達到97％，實時應用程序的準確率達到92％。;Interaction between human and computer has become very important start from the first born of the personal computer to nowadays with smart phone devices. The demand of complex interaction in many form of application to be more natural lead the research on natural way of interaction design stood up. The common and most natural way to interact is using gesture. Thus in this work we study the gesture recognition system in local and global way. Local in the form of finger level gesture that connected to the finger detection and tracking. In this work, we are interested in solving the challenge of finger level gesture recognition on repeating-finite kind of gestures. For example guitar strumming, hand puppet actions, or virtual keyboard. This kind of gesture can be represented as the FSM model. By combining with fast but less accurate appearance-based method, we propose novel finger pose tracking using Finite State Model-based. To test the robustness of the proposed system we conduct the experiment on one simple repeating kind of gesture. The result able to reach 82% of recognition rate in the testing phase. The global way, we propose the deep learning based hand gesture recognition using 3DCNN and LSTM on the sequence data. We collected and design our own dataset to test the robustness of our model in real-time application. The result show that 97% accuracy rate on the offline testing then 92% accuracy on the real-time application.
