探討動作觀察對於腦波人機介面於深度學習在元宇宙下的即時應用

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：115

、訪客IP：3.135.204.31

姓名

袁嘉浚(JIA-JUN YUAN) 查詢紙本館藏

畢業系所

電機工程學系

論文名稱

探討動作觀察對於腦波人機介面於深度學習在元宇宙下的即時應用
(Exploring the Real-time Applications of Action Observation in Brain-Computer Interface with Deep Learning in the Metaverse)

相關論文

★ 使用梳狀濾波器於相位編碼之穩態視覺誘發電位腦波人機介面	★ 應用電激發光元件於穩態視覺誘發電位之腦波人機介面判斷
★ 智慧型手機之即時生理顯示裝置研製	★ 多頻相位編碼之閃光視覺誘發電位驅動大腦人機介面
★ 以經驗模態分解法分析穩態視覺誘發電位之大腦人機界面	★ 利用經驗模態分解法萃取聽覺誘發腦磁波訊號
★ 明暗閃爍視覺誘發電位於遙控器之應用	★ 使用整體經驗模態分解法進行穩態視覺誘發電位腦波遙控車即時控制
★ 使用模糊理論於穩態視覺誘發之腦波人機介面判斷	★ 利用正向模型設計空間濾波器應用於視覺誘發電位之大腦人機介面之雜訊消除
★ 智慧型心電圖遠端監控系統	★ 使用隱馬可夫模型於穩態視覺誘發之腦波人機介面判斷與其腦波控制遙控車應用
★ 使用類神經網路於肢體肌電訊號進行人體關節角度預測	★ 使用等階集合法與影像不均勻度修正於手指靜脈血管影像切割
★ 應用小波編碼於多通道生理訊號傳輸	★ 結合高斯混合模型與最大期望值方法於相位編碼視覺腦波人機介面之目標偵測

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 (2026-6-30以後開放)

摘要(中)

在當今時代，腦機介面（Brain-Computer Interface, BCI）的研究正蓬勃發展，特別是在醫療康復、智能設備控制和娛樂等領域中展現出巨大的應用潛力。BCI的其中一個重要應用方向是想像運動（Motor Imagery, MI），透過與虛擬實境（Virtual Reality, VR）的互動中，同時動作觀察(Action Observation, AO)也被認為是想像運動訓練中重要的訓練策略。因此本篇論文結合了VR與Unity，設計了純MI與AO+MI兩種場景供受試者遊玩，並分析每位受試者在實驗過程中的拓樸圖變化。每位受試者需要在實驗中進行左手、右手運動想像以及休息狀態時，提供其腦電圖（EEG）訊號數據。本篇論文採用了Transformer架構結合EEGNet來建構模型。利用EEGNet的不同捲積層來獲取EEG訊號在時域與頻域的特徵，而Transformer因其自注意力機制（Self-Attention）對時序數據的強大適應性，在去除雜訊和提取關鍵特徵方面表現優異。Transformer能夠有效捕捉EEG訊號中的時序依賴性，從而提取出代表受試者想像運動的顯著特徵，進而進行三分類的判斷，即分辨出左手想像運動、右手想像運動以及休息狀態。同時實驗過程分為線下訓練與線上訓練，使模型可以持續的學習達到更高的正確率，在線下訓練中模型的準確率平均為74.88%，比單純使用EEGNet增加了6.13%，而在線上訓練階段受試者的遊戲表現進步了18.95%，證實了線下訓練與線上訓練的策略有助於受試者進行想像運動。

摘要(英)

In contemporary times, Brain-Computer Interface (BCI) research is flourishing, especially showcasing immense application potential in fields such as medical rehabilitation, intelligent device control, and entertainment. One significant application direction of BCI is Motor Imagery (MI). Through interaction with Virtual Reality (VR), Action Observation (AO) is also considered an important training strategy in Motor Imagery training. This paper integrates VR with Unity to design two scenarios for participants to engage in: pure MI and AO+MI. It also analyzes the topological map changes of each participant during the experimental process. Each participant needs to provide their electroencephalogram (EEG) signal data while performing left-hand, right-hand motor imagery, and resting states during the experiment.This paper adopts a model constructed by combining the Transformer architecture with EEGNet. EEGNet′s various convolutional layers are utilized to capture the features of EEG signals in both the time and frequency domains. Meanwhile, the Transformer, with its self-attention mechanism, exhibits excellent performance in noise removal and key feature extraction due to its strong adaptability to sequential data. The Transformer can effectively capture the temporal dependencies in EEG signals, thereby extracting significant features representing the participants′ motor imagery, leading to the classification of three states: left-hand motor imagery, right-hand motor imagery, and resting state.Additionally, the experimental process is divided into offline and online training, allowing the model to continuously learn and achieve higher accuracy. The average accuracy of the model in offline training is 74.88%, which is an improvement of 6.13% compared to using only EEGNet. During the online training phase, the participants′ game performance improved by 18.95%, demonstrating that the strategies of offline and online training are beneficial for participants in performing motor imagery.

關鍵字(中)

★ 腦機介面
★ 想像運動
★ 深度學習
★ 持續學習
★ 動作觀察
★ Transformer架構

關鍵字(英)

★ Brain-Computer Interface
★ Motor Imagery
★ Deep Learning
★ Continual Learning
★ Action Observation
★ Transformer

論文目次

第一章緒論 1
1-1研究動機與目標: 1
1-2 文獻探討: 4
第二章研究設計與方法 6
2-1實驗方法: 6
2-2 實驗流程 8
2-2-1線下訓練實驗流程 8
2-2-2 線上訓練實驗流程 12
2-3 資料預處理 14
2-4 模型架構 15
2-6-1 拓樸圖分析方法 22
2-6-2 韋爾奇分析方法 23
第三章實驗結果與討論 25
3-1 線下實驗結果 25
3-2 線上訓練實驗結果 49
3-2-1 線上訓練拓樸圖結果 49
第四章實驗結果與討論 59
第五章結論與未來展望 61
參考文獻: 62

參考文獻

[1] S. Saha et al., "Progress in brain computer interface: Challenges and opportunities," Frontiers in Systems Neuroscience, vol. 15, pp. 578875 %@ 1662-5137, 2021.
[2] L. A. Farwell, D. C. Richardson, G. M. Richardson, and J. J. Furedy, "Brain fingerprinting classification concealed information test detects US Navy military medical information with P300," Frontiers in neuroscience, vol. 8, pp. 410 %@ 1662-453X, 2014.
[3] M. Abu-Alqumsan and A. Peer, "Advancing the detection of steady-state visual evoked potentials in brain–computer interfaces," Journal of neural engineering, vol. 13, no. 3, p. 036005, 2016.
[4] A. Vourvopoulos et al., "Effects of a brain-computer interface with virtual reality (VR) neurofeedback: A pilot study in chronic stroke patients," Frontiers in human neuroscience, pp. 210 %@ 1662-5161, 2019.
[5] K. LaFleur, K. Cassady, A. Doud, K. Shades, E. Rogin, and B. He, "Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface," Journal of neural engineering, vol. 10, no. 4, p. 046003, 2013.
[6] H. Rajaguru and S. K. Prabhakar, "Performance analysis of local linear embedding (LLE) and Hessian LLE with Hybrid ABC-PSO for Epilepsy classification from EEG signals," 2018, pp. 1084-1088 %@ 1538624567: IEEE.
[7] H. Adeli and S. Ghosh-Dastidar, Automated EEG-based diagnosis of neurological disorders: Inventing the future of neurology. CRC press, 2010.
[8] Y.-E. Lee and S.-H. Lee, "EEG-transformer: Self-attention from transformer architecture for decoding EEG of imagined speech," in 2022 10th International Winter Conference on Brain-Computer Interface (BCI), 2022, pp. 1-4: IEEE.
[9] "EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces," Journal of neural engineering, vol. 15, no. 5, p. 056013, 2018.
[10] Y. Song, J. Liu, Q. Gao, and M. Liu, "A quadrotor helicopter control system based on Brain-computer interface," 2015, pp. 1478-1483 %@ 1479970980: IEEE.
[11] J.-H. Jeong, K.-T. Kim, D.-J. Kim, and S.-W. Lee, "Decoding of multi-directional reaching movements for EEG-based robot arm control," 2018, pp. 511-514 %@ 1538666502: IEEE.
[12] D. Kim and S.-B. Cho, "A brain-computer interface for shared vehicle control on TORCS car racing game," 2014, pp. 550-555 %@ 147995151X: IEEE.
[13] W.-Y. Hsu and Y.-W. Cheng, "EEG-Channel-Temporal-Spectral-Attention Correlation for Motor Imagery EEG Classification," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 1659-1669 %@ 1534-4320, 2023.
[14] S. J. M. Smith, "EEG in the diagnosis, classification, and management of patients with epilepsy," Journal of Neurology, Neurosurgery & Psychiatry, vol. 76, no. suppl 2, pp. ii2-ii7 %@ 0022-3050, 2005.
[15] D. S. Moschona, "An affective service based on multi-modal emotion recognition, using eeg enabled emotion tracking and speech emotion recognition," 2020, pp. 1-3 %@ 1728161649: IEEE.
[16] A. Qayyum, M. K. A. A. Khan, M. Mazher, and M. Suresh, "Classification of eeg learning and resting states using 1d-convolutional neural network for cognitive load assesment," 2018, pp. 1-5 %@ 1538691752: IEEE.
[17] T. Xu, R. Yin, L. Shu, and X. Xu, "Emotion recognition using frontal EEG in VR affective scenes," 2019, vol. 1, pp. 1-4 %@ 1538673959: IEEE.
[18] J.-P. Tauscher, F. W. Schottky, S. Grogorick, P. M. Bittner, M. Mustafa, and M. Magnor, "Immersive EEG: evaluating electroencephalography in virtual reality," 2019, pp. 1794-1800 %@ 1728113776: IEEE.
[19] J. A. Pineda, D. S. Silverman, A. Vankov, and J. Hestenes, "Learning to control brain rhythms: making a brain-computer interface possible," IEEE transactions on neural systems and rehabilitation engineering, vol. 11, no. 2, pp. 181-184 %@ 1534-4320, 2003.
[20] A. Vourvopoulos and S. Bermúdez i Badia, "Motor priming in virtual reality can augment motor-imagery training efficacy in restorative brain-computer interaction: a within-subject analysis," Journal of neuroengineering and rehabilitation, vol. 13, no. 1, pp. 1-14 %@ 1743-0003, 2016.
[21] G. Hou, H. Dong, and Y. Yang, "Developing a virtual reality game user experience test method based on EEG signals," 2017, pp. 227-231 %@ 1538609363: IEEE.
[22] R. Chaisaen et al., "Decoding EEG rhythms during action observation, motor imagery, and execution for standing and sitting," IEEE sensors journal, vol. 20, no. 22, pp. 13776-13786, 2020.
[23] S. Aggarwal and N. Chugh, "Signal processing techniques for motor imagery brain computer interface: A review," Array, vol. 1, pp. 100003 %@ 2590-0056, 2019.
[24] B. Zhang, H. Jiang, and L. Dong, "Classification of EEG signal by WT-CNN model in emotion recognition system," 2017, pp. 109-114 %@ 1538607719: IEEE.
[25] M. B. Dkhil, A. Wali, and A. M. Alimi, "Drowsy driver detection by EEG analysis using Fast Fourier Transform," 2015, pp. 313-318 %@ 1467387096: IEEE.
[26] P. K. Saha, M. A. Rahman, and M. N. Mollah, "Frequency domain approach in CSP based feature extraction for EEG signal classification," 2019, pp. 1-6 %@ 1538691116: IEEE.
[27] Y. Wang, B. Hong, X. Gao, and S. Gao, "Phase synchrony measurement in motor cortex for classifying single-trial EEG during motor imagery," 2006, pp. 75-78 %@ 1424400325: IEEE.
[28] A. Vaswani et al., "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[29] Y. Song, X. Jia, L. Yang, and L. Xie, "Transformer-based spatial-temporal feature learning for EEG decoding," arXiv preprint arXiv:2106.11170, 2021.
[30] Z. Wan, M. Li, S. Liu, J. Huang, H. Tan, and W. Duan, "EEGformer: A transformer–based brain activity classification method using EEG signal," Frontiers in Neuroscience, vol. 17, p. 1148855, 2023.
[31] T. Wang, S. U. Guan, K. L. Man, and T. O. Ting, "Time series classification for EEG eye state identification based on incremental attribute learning," 2014, pp. 158-161 %@ 147995277X: IEEE.
[32] Y. Shi, Y. Mi, J. Li, and W. Liu, "Concept-cognitive learning model for incremental concept learning," IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 2, pp. 809-821 %@ 2168-2216, 2018.
[33] C.-Y. Chiang, N.-F. Chang, T.-C. Chen, H.-H. Chen, and L.-G. Chen, "Seizure prediction based on classification of EEG synchronization patterns with on-line retraining and post-processing scheme," in 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2011, pp. 7564-7569: IEEE.
[34] Q.-T. Xu, J. Zhang, and Z.-H. Ling, "An End-to-End EEG Channel Selection Method with Residual Gumbel Softmax for Brain-Assisted Speech Enhancement," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 10131-10135: IEEE.
[35] S.-F. He, N.-N. Zhang, C.-M. Zhang, Z.-H. Fu, X.-L. Chen, and P. Xie, "Enhancement of motor cortex EEG during motor imagery: a visual feedback training study," in 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), 2019, pp. 1-5: IEEE.
[36] N. Syrov, L. Yakovlev, D. Bredichin, A. Miroshnikov, N. Shusharina, and A. Kaplan, "Activation of sensorimotor areas by passive observation of movements: a TMS-EEG study," in 2021 Third International Conference Neurotechnologies and Neurointerfaces (CNN), 2021, pp. 108-111: IEEE.
[37] 曹程富 and 李柏磊, "發展深度學習為基礎之即時腦波人機介面於元宇宙環境下的應用 / 曹程富著," Development and application of a real-time brain-computer interface based on deep learning in the metaverse environment, 撰者, 桃園市中壢區, 2023.
[38] M. Li and W. Chen, "FFT-based deep feature learning method for EEG classification," Biomedical Signal Processing and Control, vol. 66, p. 102492, 2021.
[39] P. Kant, S. H. Laskar, J. Hazarika, and R. Mahamune, "CWT based transfer learning for motor imagery classification for brain computer interfaces," Journal of Neuroscience Methods, vol. 345, p. 108886, 2020.
[40] K. K. Ang, Z. Y. Chin, H. Zhang, and C. Guan, "Filter bank common spatial pattern (FBCSP) in brain-computer interface," in 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), 2008, pp. 2390-2397: IEEE.
[41] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.
[42] M. M. Islam and M. M. H. Shuvo, "DenseNet based speech imagery EEG signal classification using Gramian Angular Field," in 2019 5th International Conference on Advances in Electrical Engineering (ICAEE), 2019, pp. 149-154: IEEE.
[43] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 2015, pp. 234-241: Springer.
[44] N. Mashhadi, A. Z. Khuzani, M. Heidari, and D. Khaledyan, "Deep learning denoising for EOG artifacts removal from EEG signals," in 2020 IEEE Global Humanitarian Technology Conference (GHTC), 2020, pp. 1-6: IEEE.
[45] 石曜嘉, "希爾伯特黃轉換應用於單筆腦電波訊號," 2009.
[46] L. Zhao and Y. He, "Power spectrum estimation of the welch method based on imagery EEG," Applied Mechanics and Materials, vol. 278, pp. 1260-1264, 2013.
[47] Y. Gao, X. Fu, T. Ouyang, and Y. Wang, "EEG-GCN: spatio-temporal and self-adaptive graph convolutional networks for single and multi-view EEG-based emotion recognition," IEEE Signal Processing Letters, vol. 29, pp. 1574-1578, 2022.
[48] L. Van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of machine learning research, vol. 9, no. 11, 2008.
[49] A. Li, Z. Wang, X. Zhao, T. Xu, T. Zhou, and H. Hu, "MDTL: a novel and model-agnostic transfer learning strategy for cross-subject motor imagery BCI," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 1743-1753, 2023.
[50] P. Cheng, P. Autthasan, B. Pijarana, E. Chuangsuwanich, and T. Wilaiprasitporn, "Towards asynchronous motor imagery-based brain-computer interfaces: a joint training scheme using deep learning," in TENCON 2018-2018 IEEE Region 10 Conference, 2018, pp. 1994-1998: IEEE.
[51] S. Jo and J. W. Choi, "Effective motor imagery training with visual feedback for non-invasive brain computer interface," in 2018 6th International Conference on Brain-Computer Interface (BCI), 2018, pp. 1-4: IEEE.
[52] A. Ravi, J. Tung, and N. Jiang, "Combined Action Observation, Motor Imagery and SSMVEP BCI Enhances Movement Related Cortical Potential," in 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), 2023, pp. 1-4: IEEE.
[53] L. Wang, P. Li, Y. Liu, and X. Bai, "Comparison of EEG Changes During Action Observation, Action Observation Combined with Motor Imagery and Motor Execution of Lower Limb," in 2017 International Conference on Computer Systems, Electronics and Control (ICCSEC), 2017, pp. 1075-1080: IEEE.

指導教授

李柏磊(BO-LEI LI)

審核日期

2024-7-26

推文