元宇宙中的虛擬互動:以跳舞為例

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：17

、訪客IP：3.15.190.49

姓名

吳翰鈞(Han-Chun Wu) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

元宇宙中的虛擬互動:以跳舞為例
(Virtual Interaction in Metaverse: Dacing as Example)

相關論文

★ 基於edX線上討論板社交關係之分組機制	★ 利用Kinect建置3D視覺化之Facebook互動系統
★ 利用 Kinect建置智慧型教室之評量系統	★ 基於行動裝置應用之智慧型都會區路徑規劃機制
★ 基於分析關鍵動量相關性之動態紋理轉換	★ 基於保護影像中直線結構的細縫裁減系統
★ 建基於開放式網路社群學習環境之社群推薦機制	★ 英語作為外語的互動式情境學習環境之系統設計
★ 基於膚色保存之情感色彩轉換機制	★ 一個用於虛擬鍵盤之手勢識別框架
★ 分數冪次型灰色生成預測模型誤差分析暨電腦工具箱之研發	★ 使用慣性傳感器構建即時人體骨架動作
★ 基於多台攝影機即時三維建模	★ 基於互補度與社群網路分析於基因演算法之分組機制
★ 即時手部追蹤之虛擬樂器演奏系統	★ 基於類神經網路之即時虛擬樂器演奏系統

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

近年來，虛擬現實的技術已經發展成熟，且已經被應用在各種不同的領域
上。使用者能夠藉由穿戴一些設備進入虛擬環境並和虛擬物件和其他使用者進
行互動。且因為 COVID-19，元宇宙也成為一個熱門的主題。元宇宙能夠幫助
被隔離的人們與其他人進行交流互動。但是，目前大多數的虛擬現實的設備是
手持的、穿戴式的或侵入式的。雖然這些設備能夠提供很好的性能，但它們可
能會讓一些使用者感到不舒服。
為了改善這個情況，在本篇論文中，我提出了一個結合了 MMpose、
Unity 遊戲引擎和反向動力學的系統，可以讓使用者控制他們的虛擬化身，在
虛擬環境中與虛擬物件和其他使用者互動與跳舞，且不需要手持、穿戴、或者
附加任何設備在他們身上。MMpose 是一個人體骨架偵測的模型，它可以藉由
單張的 RGB 輸入影像得到多人的 3D 人體關鍵點的資料，接著 Unity 接收這些
資料，最後再藉由反向動力學，使用者就可以控制他們的虛擬化身與虛擬環境
中的虛擬物件互動或者讓虛擬化身在虛擬環境中跳舞。這個方法讓使用者不必
再穿戴任何額外的裝置，只需要在相機前面做出相對應的姿勢，虛擬化身就可
以重現使用者的動作。

摘要(英)

In recent years, virtual reality has already fully developed and has
been used in many different fields. Users can use some devices to get into
virtual space to interact with virtual objects and other users. Metaverse is
also a popular topic because of COVID-19. It can help people who are
quarantined to communicate with other people. But, most devices are
handheld, wearable, or intrusive currently, and they may make some users
feel uncomfortable though their performance are very good.
To improve this situation, I propose a method that combine MMpose,
Unity engine, and inverse kinematics to let user can control an avatar to
interact with virtual objects and dance without holding, wearing, or
attaching anything to their body. MMpose can get multi-people 3D
keypoint data by single RGB image, then these data can be received in
Unity. Finally, combine these data and inverse kinematics, user can control
an avatar to interact with virtual objects or make avatar dance in Unity
virtual space. This method can make users control avatars and interact
with virtual objects, and they don’t need to wear or hold anything, only
need to move their bodies in front of the camera.

關鍵字(中)

★ 虛擬現實
★ 深度學習
★ 3D人體姿勢估計
★ 元宇宙

關鍵字(英)

★ Virtual Reality
★ Deep Learning
★ 3D Human Pose Estimation
★ Metaverse

論文目次

摘要.............................................................................................................................. ii
Abstract........................................................................................................................iii
List of figure.................................................................................................................vii
List of table .................................................................................................................. x
1. Introduction...............................................................................................................1
1.1 Background .....................................................................................................1
1.2 Motivation ......................................................................................................2
1.3 Thesis Organization.........................................................................................3
2. Related Work.............................................................................................................5
2.1 Virtual Reality .................................................................................................5
2.1.1 Devices and Sensors ............................................................................5
2.2 Metaverse .......................................................................................................6
2.3 Deep Learning..................................................................................................6
2.3.1 Convolutional Neural Network.............................................................7
2.4 Pose Estimation ..............................................................................................8
2.4.1 2D Pose Estimation...............................................................................8
2.4.2 3D Pose Estimation.............................................................................11
2.5 3D Human Body Model.................................................................................12
2.6 Kinematics.....................................................................................................13
2.6.1 Forward Kinematics............................................................................13
2.6.2 Inverse Kinematics..............................................................................15
2.7 Human-Object Interaction.............................................................................17
3. Primary Research.................................................................................................... 19
3.1 Unity..............................................................................................................19
3.2 Avatar.............................................................................................................19
3.3 Virtual Scene .................................................................................................20
3.4 Pose Estimation Model .................................................................................21
4. Proposed Framework............................................................................................. 23
4.1 Pose Estimation Model .................................................................................23
4.1.1 OpenPose with Zed ............................................................................23
4.1.2 MMpose.............................................................................................25
4.2 Interaction in Unity .......................................................................................26
4.2.1 Inverse Kinematics..............................................................................27
4.2.2 Interaction with Virtual Objects.........................................................32
4.2.3 Interaction between Avatars ..............................................................33
5. Experiment............................................................................................................. 35
5.1 Experiment Setup..........................................................................................35
5.1.1 Camera ...............................................................................................35
5.1.2 Hardware............................................................................................35
5.1.3 Software ............................................................................................36
5.2 System Evaluation..........................................................................................37
5.3 Game Process................................................................................................40
6. Conclusion and Future Work ................................................................................. 43
7. Reference ............................................................................................................... 45

參考文獻

[1] https://unity.com/ (Unity) [Online] [Accessed October 2020]
[2] https://github.com/CMU-Perceptual-Computing-Lab/openpose
(OpenPose open source) [Online] [Accessed October 2020]
[3] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh,
OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity
Fields. in: IEEE Transactions on Pattern Analysis and Machine Intelligence
2021
[4] https://www.stereolabs.com/zed-2/ (Zed-2) [Online] [Accessed 2021]
[5] https://github.com/open-mmlab/mmpose (MMpose open source)
[Online] [Accessed April 2022]
[6] https://tigercosmos.xyz/post/2020/05/ca/forward-kinematics-timewarping/ (Forward kinematics) [Online] [Accessed May 2022]
[7] https://tigercosmos.xyz/post/2020/06/ca/inverse-kinematics/ (Inverse
Kinematics) [Online] [Accessed May 2022]
[8] https://www.vive.com/tw/ (HTC vive) [Online] [Accessed June 2022]
[9] https://arvr.google.com/cardboard/ (Google cardboard) [Online]
[Accessed June 2022]
46
[10]https://medium.com/jameslearningnote/%E8%B3%87%E6%96%99%
E5%88%86%E6%9E%90-
%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92-%E7%AC%AC5-
1%E8%AC%9B-
%E5%8D%B7%E7%A9%8D%E7%A5%9E%E7%B6%93%E7%B6%B2%E7%B
5%A1%E4%BB%8B%E7%B4%B9-convolutional-neural-network4f8249d65d4f (CNN) [Online] [Accessed May 2022]
[11] https://www.intelrealsense.com/depth-camera-d435/ (Intel realsense
D435) [Online] [Accessed April 2022]
[12] https://smpl.is.tue.mpg.de/ (SMPL) [Online] [Accessed January 2022]
[13] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros.
Everybody Dance Now. In: 2019 IEEE/CVF International Conference on
Computer Vision (ICCV)
[14]https://assetstore.unity.com/packages/3d/characters/humanoids/hu
mans/audience-crowd-8563#publisher (audience) [Online] [Accessed
May 2022]
[15] https://assetstore.unity.com/packages/3d/props/interior/bar-chair106889#publisher (bar chair) [Online] [Accessed May 2022]
[16] https://assetstore.unity.com/packages/3d/props/apartment-kit-
47
124055#description (wine, wall, glass) [Online] [Accessed May 2022]
[17] Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. A
simple yet effective baseline for 3d human pose estimation. In: 2017 IEEE
International Conference on Computer Vision (ICCV)
[18] Sizhuo Zhang and Nanfeng Xiao. Detailed 3D Human Body
Reconstruction From a Single Image Based on Mesh Deformation. In: IEEE
Access.
[19] Georgia Gkioxari, Ross Girshick, Piotr Dollar, and Kaiming He.
Detecting and Recognizing Human-Object Interactions. In: 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition
[20] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu
Lu. HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D
Human Pose and Shape Estimation. In: 2021 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR)
[21] David C. Jeong, Jackie (Jingyi) Xu, and Lynn C. Miller. Inverse
Kinematics and Temporal Convolutional Networks for Sequential Pose
Analysis in VR. In: 2020 IEEE International Conference on Artificial
Intelligence and Virtual Reality (AIVR)
[22] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel
48
Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural Volumes: Learning
Dynamic Renderable Volumes from Images. In: Meta Research
[23] Sergey Prokudin, Michael J. Black, Javier Romero. SMPLpix: Neural
Avatars from 3D Human Models. In: 2021 IEEE Winter Conference on
Applications of Computer Vision (WACV)
[24] Jieqi Shi, Lingyun Xu, Peiliang Li , Xiaozhi Chen , and Shaojie Shen.
Temporal Point Cloud Completion With Pose Disturbance. In: IEEE
Robotics and Automation Letters
[25] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang,
Yan-Feng Wang, and Cewu Lu. Transferable Interactiveness Knowledge for
Human-Object Interaction Detection. In: 2019 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR)
[26] Zhijun Liang, Juan Rojas, Junfa Liu, and Yisheng Guan. VisualSemantic Graph Attention Networks for Human-Object Interaction
Detection. In: 2021 IEEE International Conference on Robotics and
Biomimetics (ROBIO)
[27] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard PonsMoll, and Michael J Black. Smpl: A skinned multi-person linear model. In:
ACM transactions on graphics (TOG), 2015.
49
[28] https://zh.wikipedia.org/zhtw/%E5%8D%B7%E7%A7%AF%E7%A5%9E%E7%BB%8F%E7%BD%91%E
7%BB%9C (CNN pooling layer) [Online] [Accessed May 2022]

指導教授

施國琛(Timothy K. Shih)

審核日期

2022-7-26

推文