基於深度學習之虛擬吉他音樂演奏系統設計

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：11

、訪客IP：3.128.226.67

姓名

周恒瑋(Heng-Wei Zhou) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

基於深度學習之虛擬吉他音樂演奏系統設計
(The design of virtual guitar music performance system based on deep learning)

相關論文

★ 基於edX線上討論板社交關係之分組機制	★ 利用Kinect建置3D視覺化之Facebook互動系統
★ 利用 Kinect建置智慧型教室之評量系統	★ 基於行動裝置應用之智慧型都會區路徑規劃機制
★ 基於分析關鍵動量相關性之動態紋理轉換	★ 基於保護影像中直線結構的細縫裁減系統
★ 建基於開放式網路社群學習環境之社群推薦機制	★ 英語作為外語的互動式情境學習環境之系統設計
★ 基於膚色保存之情感色彩轉換機制	★ 一個用於虛擬鍵盤之手勢識別框架
★ 分數冪次型灰色生成預測模型誤差分析暨電腦工具箱之研發	★ 使用慣性傳感器構建即時人體骨架動作
★ 基於多台攝影機即時三維建模	★ 基於互補度與社群網路分析於基因演算法之分組機制
★ 即時手部追蹤之虛擬樂器演奏系統	★ 基於類神經網路之即時虛擬樂器演奏系統

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

人機互動之研究專注於研發使用者與電腦之溝通介面，而如何讓電腦藉由觀察與追蹤辨識演算法之設計來理解輔助裝置（諸如色彩／深度攝影機、智慧手環、傳感手套）之行為數據，為研究學者很重視的議題。近年來有越來越多體驗裝置之產品（如：虛擬頭盔、智慧眼鏡等）被廣泛應用於體感遊戲或是其他應用中，其中包含音樂演奏表演或是藝術表演，目的為讓玩家能夠在無實體硬體設備下（如：鋼琴、吉他等）以更友善且便利之方式演奏樂器，且讓觀眾能有更好的新科技視覺體驗。
先前人機互動之研究多以人體骨架之行為理解為主，適用於大動作之行為辨別，近年來各學者以機器學習與深度學習做為行為辨識主要之策略，其提升之準確率證明了機器學習與深度學習之可靠性。
為了能夠辨識細部的指尖指法，本論文提出一個以吉他為範例之吉他表演系統，該系統不僅能辨識及他左手之各和絃手勢，且能辨別右手之撥弦演奏行為。經由實驗證明，本系統不僅能被應用在吉他演奏表演系統，亦能應用在其他樂器演奏之表演系統（如：大提琴、小提琴、烏克麗麗等）。除此之外，本論文提出一個手勢辨識之評量機制，能夠用來評估模型與實時演奏之可靠性。

摘要(英)

Human-computer interaction (HCI) focused on developing the interfaces between users and computers in computer. Many researchers observe the ways in which user interact with computers and design tracking or recognizing mechanism that let computer realized the input command from user’s behavior by auxiliary sensors (e.g. camera, wisdom bracelet, sensing gloves). More and more experiential device (e.g. virtual reality headset, smart glasses) are widely used for users with somatosensory games but they are also used in other applications, including music or artistic performances, the purpose is to allow users to more conveniently control commands on a small number of devices and give audiences a new visual experience.
In the past, the researchers mainly focused on the observation and analysis of human skeleton movements by using the design of special algorithms, the computer can understand the limb behavior based on the human skeleton. In our studies. In recent years, more and more machine learning and deep learning approach are used in HCI related researches to prove their reliability. In order to be able to recognize the detailed fingering behavior of the finger, this thesis proposed a guitar playing system, which use deep learning as strategy to recognize the finger gesture between all guitar in left-hand, and picking behavior in right-hand. Also, a verification method for discriminating accuracy was proposed in this thesis, which can be used to prove the reliability of the guitar performance system. Experimental results prove that our system based on deep learning approach can effectively identify the fingering behavior, and also the performance system can be used for other musical instruments (e.g. cello, violin or ukulele).

關鍵字(中)

★ 人機互動介面
★ 深度學習
★ 虛擬樂器
★ 手部偵測

關鍵字(英)

★ Human–Computer Interaction
★ Deep Learning
★ Virtual Instrument
★ Hand Detection

論文目次

Contents
摘要 i
Abstract ii
Contents iii
List of Figures v
List of Tables viii
Chapter 1. Introduction 1
1.1 Background 1
1.2 Motivation 3
1.3 Thesis Organization 4
Chapter 2. Related work 5
2.1 Music Playing with HCI 5
2.2 Gesture Recognition using Deep Learning 6
2.3 Deep Learning with Gesture Recognition 11
2.4 Digital Music 14
Chapter 3. Proposed Framework 17
3.1 Guitar Structure and Action 19
3.2 Data Acquisition 20
3.3 Data Preprocess 24
3.4 Model Setup 26
3.4.1 Classification 26
3.4.2 Model Detail 27
3.5 Detection 31
Chapter 4. Experiment 34
4.1 Environment Setup 34
4.1.1 Camera 34
4.1.2 Hardware 38
4.1.3 Software 39
4.1.4 Data Recording Tool 42
4.2 Model Evaluation 45
4.2.1 Left-Hand Model Evaluation 45
4.2.2 Right-Hand Model Evaluation 49
4.3 Guitar Performance System 54
Chapter 5. Conclusion and Future Works 56
References 57

參考文獻

References
[1] G. Odowichuk, S. Trail, W. Page, W. Nie and P. Driessen, "Sensor fusion: Towards a fully expressive 3D music control interface," in Proceedings of 2011 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, Victoria, BC, Canada, 2011.
[2] L. S. Figueiredo, J. Teixeira, A. S. Cavalcanti, V. Teichrieb and J. Kelner, "An Open-Source Framework for Air Guitar Games," in VIII Brazilian Symposium on Games and Digital Entertainment, Rio de Janeiro, Brazil, 2009.
[3] Y. Che and Y. Qi, "Dynamic Projected Segmentation Networks For Hand Pose Estimation," in 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018.
[4] S. E. Wei, V. Ramakrishna, T. Kanade and Y. Sheikh, "Convolutional Pose Machines," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016.
[5] C. Zimmermann and T. Brox, "Learning to Estimate 3D Hand Pose from Single RGB Images," in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017.
[6] Y. Wang, C. Peng and Y. Liu, "Mask-Pose Cascaded CNN for 2D Hand Pose Estimation From Single Color Image," IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 11, pp. 3258-3268, 2018.
[7] S. Baek, K. I. Kim and T.-K. Kim, "Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, USA, 2019.
[8] Y. Che, Y. Song and Y. Qi, "A Novel Framework of Hand Localization and Hand Pose Estimation," in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, United Kingdom, 2019.
[9] L. Ge, H. Liang, J. Yuan and D. Thalmann, "Real-Time 3D Hand Pose Estimation with 3D Convolutional Neural Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 4, pp. 956 - 970, 1 4 2019.
[10] C. R. Naguri and R. C. Bunescu, "Recognition of Dynamic Hand Gestures from 3D Motion Data using LSTM and CNN architectures," in 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), Cancun, Mexico, 2018.
[11] R. Azad, M. Asadi-Aghbolaghi, S. Kasaei and S. Escalera, "Dynamic 3D Hand Gesture Recognition by Learning Weighted Depth Motion Maps," IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1729 - 1740, 12 7 2018.
[12] Y.-J. Son and O. Choi, "Image-based hand pose classification using faster R-CNN," in 2017 17th International Conference on Control, Automation and Systems (ICCAS), Jeju, South Korea, 2017.
[13] M. Abavisani, H. R. Vaezi Joze and V. M. Patel, "Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training".
[14] F. Wang, L. Kong, X. Zhang and H. Chen, "Gesture Recognition and Localization Using Convolutional Neural Network," in 2019 Chinese Control And Decision Conference (CCDC), Nanchang, China, China.
[15] N. Dhingra and A. Kunz, "Res3ATN -Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos," in 2019 International Conference on 3D Vision (3DV), Québec City, QC, Canada, Canada, 2019.
[16] O. Köpüklü, A. Gunduz, N. Kose and G. Rigoll, "Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks," in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France, France, 2019.
[17] "MIDI - Wikipedia," Wikipedia, [Online]. Available: https://en.wikipedia.org/wiki/MIDI.
[18] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, "Going deeper with convolutions," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015.
[19] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016.

指導教授

施國琛(Timothy K. Shih)

審核日期

2020-7-20

推文