| dc.description.abstract | Recently, due to the COVID-19, quarantined people are isolated in different indoor spaces, the world is going through a difficult time. In order to allow people living indoors to have more interaction with other people, we propose a system that uses deep learning and image processing technology to develop and establish a system that allows people in different indoor spaces to communicate and play games normally. Problems solved include management of skeletal data, interaction of virtual objects, interactive control in virtual reality (VR) or augmented reality (AR), and Construction of 3D Virtual Space. At the same time, we focus on the interaction between people and the environment, and construct a 3D model of an exclusive mini golf course space, so that multiple users can generate and interact with each other in this 3D space. In a 3D mini golf course, under the limited viewing angle of a single RGB camera, the reconstructed avatar will reflect the same upper body posture as the user, and different 3D avatars can interact in the AR/VR space, and these interactions will be based on MediaPipe skeleton recognition as the core.
In addition to this, our interaction model also considers the relationship between the hand skeleton and objects and the environment, such as the interaction of an avatar with a golf club, or the interaction of a golf club with a golf ball. For these high-precision interactive functions, we combine gesture detection and human skeleton detection in the module function, track the nodes of the hand skeleton and the spine skeleton, and use algorithms such as IK (Inverse kinematics) to project them to the virtual space. Let the player do not need a controller, just simple gestures, or some intuitive movements, you can control the character to move in the metaverse. Using Mirror connection module as the support, to construct a complete mini golf world, and at the same time, there are more player-to-player and player-to-virtual item interactions, expanding the sociality in the metaverse. | en_US |