| dc.description.abstract | Nowadays new technologies of Human Computer Interaction (HCI) are being developed to deliver user′s command to the computer.
Developing natural and intuitive interaction techniques is an important goal in HCI.
Typically, users interact with the computer using mouse and keyboard.
Currently, the research is directed towards a new part of the interaction that provides a way to more natural, direct and effective communication.
Let users can interact with the computers through the hand, head, facial expressions, voice, and electromyography signal.
Interface based on interaction with hands is a natural and intuitive way to interact with the computers. Such an interface could be used for AR/VR environment.
This paper aims to apply the hand tracking and recognition used in virtual reality (VR) technology to a real-time application for the musical instrument, a virtual cello.
The proposed application can play the realistic sound of the musical instrument.
The user only needs to sit in front of the table and raises user’s hand to face the camera. The system will start playing the virtual cello after capturing the user′s hand gesture.
The program is flexible, the user can adjust parameters like key notes, chords, pitch up/down and tone mode.
We used Realsense and Myo sensor to capture the hand information of the user at the application.
The Realsense is responsible for hand tracking to trigger the sound.
The Myo sensor is responsible for hand gesture recognition to control MIDI functions.
However, we use the convolutional neural network (CNN) to recognize and analyze the four static hand gestures from surface electromyography.
In the system, we use OpenGL to draw our interface and display the 3D model in the screen, the OpenCV is helping us to process image.
We used Rtmidi library to generate the MIDI message and transfer to the digital audio workstation (DAW).
By using plugin virtual studio technology (VST) to make the musical instrument sound more realistic.
Although this system could not play the song which has a high speed, for slow songs, it is stable and could be used in a professional music performance.
In the experiment, we trained the convolutional neural network model with surface electromyography (sEMG)
The experimental results demonstrated that the accuracy is 94.3\% in the specific-user dataset and 86.1\% in general dataset.
Therefore, we also used dynamic time warping to evaluate the virtual cello.
This evaluation can be directly extended to measure the melodic similarity of generating the MIDI of virtual instrument and standard-MIDI. | en_US |