| 摘要: | 最近因為新冠肺炎的出現，被隔離的人被孤立在不同的室內空間裡。世界社會正處於艱難的時期。為了能夠讓在室內的人與其他人擁有更多的互動，我們提出一個利用深度學習(AI)和影像處理技術來發展建立的一個可以讓在不同室內空間的人，可以正常地溝通互動玩遊戲的系統。其中處理的問題包括骨架資料的管理、虛擬物件的互動、在虛擬實境(VR)或是擴增實境(AR)中的互動控制、以及3D虛擬空間的建構。同時，我們專注於人與環境的互動，並且建構一個專屬迷你高爾夫球場空間的3D模型，讓多個使用者可以在此3D空間中生成並且互相互動。在3D迷你高爾夫球場的玩家，在單一RGB相機的限定視角下，重建出的虛擬人物會與使用者映照出相同的上半身姿勢，而此互動將以MediaPipe骨架辨識為核心進行。
除此之外，我們的互動模型還會考慮手部骨架和物件以及環境的關係，例如虛擬人物與高爾夫球球桿的互動，亦或是球桿與高爾夫球的互動，對於這些需要高精度互動功能，我們結合了模組功能中的手勢偵測和人體骨架偵測，追蹤手部骨架和脊椎骨架的節點並搭配IK (Inverse kinematics) 等演算法，將其投射到虛擬環境中，讓玩家無須控制器，只需簡單的手勢、或是一些直覺性的動作，就可以操控角色在元宇宙中活動，並以Mirror連線模組為輔助，建構出完整的高爾夫球世界，擁有更多的玩家對玩家、玩家對虛擬物品的互動，擴充元宇宙裡的社交性。;Recently, due to the COVID-19, quarantined people are isolated in different indoor spaces, the world is going through a difficult time. In order to allow people living indoors to have more interaction with other people, we propose a system that uses deep learning and image processing technology to develop and establish a system that allows people in different indoor spaces to communicate and play games normally. Problems solved include management of skeletal data, interaction of virtual objects, interactive control in virtual reality (VR) or augmented reality (AR), and Construction of 3D Virtual Space. At the same time, we focus on the interaction between people and the environment, and construct a 3D model of an exclusive mini golf course space, so that multiple users can generate and interact with each other in this 3D space. In a 3D mini golf course, under the limited viewing angle of a single RGB camera, the reconstructed avatar will reflect the same upper body posture as the user, and different 3D avatars can interact in the AR/VR space, and these interactions will be based on MediaPipe skeleton recognition as the core.
In addition to this, our interaction model also considers the relationship between the hand skeleton and objects and the environment, such as the interaction of an avatar with a golf club, or the interaction of a golf club with a golf ball. For these high-precision interactive functions, we combine gesture detection and human skeleton detection in the module function, track the nodes of the hand skeleton and the spine skeleton, and use algorithms such as IK (Inverse kinematics) to project them to the virtual space. Let the player do not need a controller, just simple gestures, or some intuitive movements, you can control the character to move in the metaverse. Using Mirror connection module as the support, to construct a complete mini golf world, and at the same time, there are more player-to-player and player-to-virtual item interactions, expanding the sociality in the metaverse. |
