基於深度學習之即時電吉他效果器模擬與虛擬實境空氣吉他系統;A Deep Learning-based Approach for a Black-box Real-time Guitar Amplifier Emulation and a VR-based Air Guitar System

NCU Institutional Repository > 資訊電機學院 > 資訊工程研究所 > 博碩士論文 > Item 987654321/93303

請使用永久網址來引用或連結此文件: http://ir.lib.ncu.edu.tw/handle/987654321/93303

題名:	基於深度學習之即時電吉他效果器模擬與虛擬實境空氣吉他系統;A Deep Learning-based Approach for a Black-box Real-time Guitar Amplifier Emulation and a VR-based Air Guitar System
作者:	林季劼;Lin, Ji-Jie
貢獻者:	資訊工程學系
關鍵詞:	深度學習;虛擬實境;虛擬樂器;效果器模擬;電腦視覺;音訊處理;Deep Learning;Virtual Reality;Virtual Instrument;Guitar Emplifier Emulation;Computer Vision;Audio Processing
日期:	2023-07-26
上傳時間:	2024-09-19 16:53:11 (UTC+8)
出版者:	國立中央大學
摘要:	近年來，虛擬實境的發展已成為大眾關注的焦點。隨著越來越多關於虛擬實境的產品和應用的出現，虛擬實境設備的效能不斷提升，成本也大幅下降，逐漸成為人手必備的設備。虛擬實境帶來的沉浸體驗不僅為使用者提供視覺上的享受，還提供了與傳統方式不同的互動方式。透過虛擬實境技術，使用者可以在虛擬環境中進行各種活動，如遊戲、會議、教育和醫療等。虛擬實境技術的重要性不斷增加，其中在虛擬樂器領域的應用也越來越廣泛，例如虛擬鋼琴和虛擬爵士鼓等。這些虛擬樂器的出現不僅讓使用者在虛擬環境中體驗彈奏樂器的樂趣，還降低了使用不同樂器的門檻，不受地點、時間、空間、設備和技術的限制，只要擁有虛擬實境設備，就能隨時隨地享受彈奏樂器的樂趣。因此，虛擬音樂會也受到越來越多的重視，如虛擬空間音場模擬和歷史演場會的3D重建等。然而，在過去的研究中，虛擬吉他多數在非虛擬實境的環境中進行，主要集中在對空氣吉他和弦的識別上。目前尚未有系統性地研究虛擬實境中的虛擬空氣吉他系統。而在商業化的虛擬吉他遊戲中，對手部姿勢的識別並不精準，僅能辨識手部是否彎曲和是否刷弦，無法準確辨識和弦和多樣的刷弦動作。因此，在本研究中，我們提出了一個虛擬空氣吉他系統，使用者只需透過虛擬實境設備即可彈奏吉他。我們利用深度學習模型的辨識能力和虛擬實境的視覺回饋優勢，能夠辨識高達30種和弦，並透過搖桿設備實現多種刷弦技巧。此外，本研究還運用了Black-box方式，透過結合WaveNet和FiLM，能夠模擬電吉他效果器在不同旋鈕值下的效果模擬，並且透過所提出的Knob Difference Loss，進一步提高模擬效果的準確率。在網路架構上，也提出的Kernel Dilation技巧，在不降低準確度的情況下，將先前研究中使用的WaveNet前饋速度提高了兩倍。使得在虛擬實境環境中的高效能運算情況下，能夠使用Intel7 11700 K處理器（於2021年發行）和NVDIA RTX 1060（於2016年發行），實現即時電吉他效果模擬。;In recent years, the development of virtual reality (VR) has become a focal point of public attention. With the emergence of more VR products and applications, the performance of VR devices has continuously improved, and the cost has significantly decreased, gradually becoming essential devices for individuals. VR provides an immersive experience that not only offers visual enjoyment to users but also provides interactive modes different from traditional methods. Through VR technology, users can engage in various activities in virtual environments, such as gaming, meetings, education, and healthcare. The significance of VR technology continues to increase, and its applications in the virtual instrument field, such as virtual pianos and virtual jazz drums, are becoming increasingly widespread. The emergence of these virtual instruments not only allows users to experience the pleasure of playing instruments in virtual environments but also lowers the barrier to learning different instruments. With VR devices, users can enjoy playing instruments anytime and anywhere, free from limitations of location, time, space, equipment, and technical expertise. Consequently, virtual concerts, including virtual spatial audio simulations and 3D reconstructions of historical performances, have gained more attention. However, in previous studies, virtual guitars were mostly conducted in non-VR environments, focusing primarily on the recognition of air guitar chords. There has been a lack of systematic research on virtual air guitar systems within VR. Moreover, commercial virtual guitar games currently have limited accuracy in recognizing hand gestures, only able to detect finger bending and simple strumming actions, but unable to accurately identify chords and various strumming techniques. Therefore, in this study, we propose a virtual air guitar system that allows users to play the guitar simply through VR devices. By leveraging the recognition capabilities of deep learning models and the visual feedback advantages of VR, our system can recognize up to 30 different chords and implement various strumming techniques using a joystick device. Furthermore, we apply a black-box approach by combining WaveNet and FiLM to simulate the effects of an electric guitar pedal at different knob settings. Additionally, we introduce a Knob Difference Loss to improve the accuracy of the simulated effects. In terms of the network architecture, we propose the Kernel Dilation technique, which doubles the forward speed of WaveNet used in previous studies without sacrificing accuracy. This enables real-time simulation of electric guitar effects even under high-performance computing VR environments, using an Intel 7 11700 K processor (released in 2021) and an NVIDIA RTX 1060 graphics card (released in 2016).
顯示於類別:	[資訊工程研究所] 博碩士論文

文件中的檔案:

檔案	描述	大小	格式	瀏覽次數
index.html		0Kb	HTML	16	檢視/開啟

在NCUIR中所有的資料項目都受到原著作權保護.

社群 sharing

資料載入中.....