本論文提出一套結合手部動作的互動實驗室模擬系統，系統有兩個核心模組，分別是「手部動作辨識模組」和「互動實驗室模擬模組」，此系統讓使用者可以藉由肢體操作的方式達到具備真實操作感的實驗模擬效果。本論文所提出之「基於Kinect之互動實驗室」目前是針對高中之化學實驗所設計的，從許多基礎化學實驗中觀察出十種在做實驗時常會用到的手部動作，依此來構成此系統之動作單元。這十種動作單元可依活動範圍之大小分成兩類：大動作之動作單元與小動作之動作單元。其中，大動作指的是在操作設備時，會有較大範圍的移動變化之手部動作，至於小動作則是指活動範圍侷限於手掌部分變化的細部動作。 本論文利用Kinect感測器當作動作擷取器，提供相關手部資訊給「手部動作辨識模組」處理，然後此模組採用兩階段方式來進行動作辨識。首先，此模組會利用使用者的骨架資訊進行大動作的辨識；接著，會進行小動作的辨識，先萃取出本論文所提出之動作特徵後，再採用放射狀基底函數網路（radial basis function network簡稱RBFN）作為動作辨識單元。另外，「互動實驗室模擬模組」則提供簡單的互動介面，以結合「動作辨識模組」的功能來完成化學實驗的模擬，此外，本模組也設計出實驗課程的編寫介面，以提供老師未來可以很方便地進行課程的新增與修改。 本論文透過許多不同面向的實驗設計來驗證本系統之效能。首先會探討此系統的非特定使用者的推廣性，接著，會針對環境的變化來測試本系統的強健性 (robustness) 。在非特定使用者的推廣性的實驗方面，會將使用者分為特定使用者和非特定使用者兩群，將三位特定使用者的資料作為訓練RBF網路的樣本庫來源，然後，再以此訓練後之網路來測試另外八位使用者的辨識效果，以驗證本系統之非特定使用者的推廣性。至於本系統的環境變化強健性的測試，則是探討使用者的站立距離和朝向角度的不同是否會導致系統的辨識率有所變化。實驗結果顯示，特定使用者和非特定使用者在相同環境的辨識率皆為97%和96%，在不同環境下的條件下，特定使用者的辨識率90%~96之間，而非特定使用者辨識率為86%~91%之間。從結果可推論，不同的使用者在相同的環境下辨識率皆有96%以上；另外當相同使用者面對不同環境的時候，兩群使用者的辨識率便有較大的差異。故此系統對於不同使用者的推廣性是足夠的，但對於環境的強健性來說，辨識率最低會到86%。 This thesis presents a gesture-based interactive laboratory simulation system. The proposed system is consisted of two main modules. The first module is the “gesture recognition module” and the second one is the “interactive laboratory simulation module”. The proposed system provides users with a simulation environment to conduct experiments with the real sense of the operations via gestures. The current proposed “Kinect-based interactive laboratory” is developed to simulate the chemical experiments designed for students at senior high-schools. Ten basic gestures which were induced from many elementary chemical experiments consist of the main action units for the simulation system. These ten gestures can be categorized into two action units according to their size range of activities: the large-scale gesture action unit and the small-scale gesture action unit. While the large-scale gesture action unit refers to the gestures which will involve in a wide range of hand movements during the operations of experiment equipment, the small-scale gesture action unit refers to the gestures of which activity scopes are confined to the palm portion. In this thesis, a Kinect sensor is utilized as a motion capture device, providing relevant gesture information to the "gesture recognition module”. Then this module adopts a two-stage approach to gesture recognition. First of all, this module classifies large-scale gestures based on the skeleton information. It will extract salient gesture features proposed by this thesis from hand movements. The radial basis function network (RBFN) is adopted as the gesture recognition unit for classifying the small-scale gestures. In the following, the "Interactive laboratory simulation module" provides a simple interactive interface to combine the recognition results achieved by the "gesture recognition module" to complete the simulation of chemical experiments. In addition, an authoring tool is designed for teachers to allow them to be able to easily add and modify the system for the preparations of the future experiments designed by them. In this thesis, many different aspects of experiments were design to verify the performance of the system. This system will first discuss the generalization degree of the system to non-specific users and then the system robustness degree to the changes in the operating environment. In the experiments of non-specific users’ generalization testing, the users were divided into the specific user group consisted of 3 subjects and the non-specific user group consisted of 8 subjects. The data collected from specific user group was used to train RBF networks. Then data collected from non-specific user group were utilized to test the generalization performance of the trained RBF networks. As for the verification of the system robustness to the environmental changes, the experiments were designed to explore whether the user's different standing locations and the viewing angles will result in the changes of the recognition rates. Experimental results showed that under the same environment, the recognition rate could achieve at least 97% and 96% correct for the specific user and non-specific group. As for the specific users under the different environment, the recognition rate varied from 90% to 96%. For the non-specific user, the recognition rate varied from 85% to 91% due to the changes in either the locations or the viewing angles. Therefore, the influence of the changes in the operating environment was more apparent than the changes in the users.
