ICACT20250008 Slide.20
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
This conclude my report.
Thank you for listening.
|
|
ICACT20250008 Slide.19
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
For future work, we hope that this system can become faster and more precise, so that when the user makes an action, the system can respond quickly and accurately.
|
|
ICACT20250008 Slide.18
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Conclusions
In this research can overcome the time and physical space limitations of traditional physical education and training. System users could carry out collaborative work or educational training in the digital twin virtual world anytime and anywhere. The proposed system can simulate various physical scenes and objects, and promote the possibility of collaborative cooperation through instant feedback and evaluation.
|
|
ICACT20250008 Slide.17
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
This is our demo video
|
|
ICACT20250008 Slide.16
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
The two images above illustrate that when we make a gesture showing the number 8, it will switch to the next virtual environment.
The two images below illustrate that when we make a gesture showing the number 6, it will switch to the previous virtual environment.
|
|
ICACT20250008 Slide.15
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
And the Results part
The picture shows the new application of digital twins based on the proposed system. Multiple users will locate at different viewpoints corresponding to the virtual 3D objects of digital twins. Hence, the location, distance, etc., of the virtual 3D objects would be different corresponding to the users’ viewpoints. Based on the gesture artificial intelligence recognition and relative position of 3D objects, the operation at one user side and the collaboration between multiple users could be still correctively presented.
|
|
ICACT20250008 Slide.14
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Final is the conclusion part.
|
|
ICACT20250008 Slide.13
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
When the gesture recognition module completes and transmits the recognition results, the result information is processed by Unity. Corresponding to different recognition results are represented by different operation command inputs. Therefore, unity can operate virtual objects according to the operation commands, including clicking, translation and movement. The results of these extended reality computing interactions will be rendered on the unity screen.
|
|
ICACT20250008 Slide.12
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
As shown in this picture.
The gesture artificial intelligence recognition results are obtained by capturing the operator's gestures through the camera. From the predefined observer angle, the recognized gestures, finger skeletons and other information can be rendering in the extended reality virtual space. The left side of the picture shows the perspective of the gesture operator. The left and right hands have been recognized and rendered by the MediaPipe hand recognition module as 3D virtual gestures and skeletons (purple line segments). The right side of the picture is the perspective facing the gesture operator. Hence, the virtual rendering result of the gesture appears in a shadow state (darker gray) instead of the reflection on the left (whiter and brighter). It shows that the 3D virtual gestures and skeletons drawn using the MediaPipe hand recognition module can be quickly integrated into the extended reality virtual environment. Users can be more intuitive and intuitive of manipulating virtual objects in extended reality.
|
|
ICACT20250008 Slide.11
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
In this research, in order to be able to create a virtual environment for different needs, we create rooms in Unity. When entering the room, a script connected to python will be triggered. The camera will be turned on by python and start recognition progress. At the same time, another script will receive the data transmitted by python. Next, the unity will capture the data of the corresponding scene.
|
|
ICACT20250008 Slide.10
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
where n indicates the total number of the feature points corresponding to the face mesh. In addition, when the value of i is n, the value of i+1 will be set as 1. Finally, the size or area of the user face can be evaluated and compared with the normal/average area or size of faces . In other words, comparing with the area or size between the measured face and average face, the possible distance from the camera to the face could be estimated. Following figure shows the evaluation results from the MediaPipe. When an user stand about 1 meter in front of the camera, the MediaPipe outputs the measured x, y, z, and the face size values of the same face.
|
|
ICACT20250008 Slide.09
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Considering being lightweight and quickly deployed on different types of hardware devices, this research uses Google MediaPipe as the basis for artificial intelligence recognition . Google MediaPipe is a development framework for processing machine learning applications. It can be applied to mobile devices such as PC, Android, IOS and utilize efficient management of resources to achieve low-latency performance. In this research, the hand recognition module was used to interact with the virtual environment. The perception pipeline can use MediaPipe to construct modular graphics, including model reasoning, media processing algorithms and data conversion. MediaPipe has many recognition modules. In this study, the hand recognition module was used to interact with the simulated environment.
|
|
ICACT20250008 Slide.08
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
And next is the method part.
|
|
ICACT20250008 Slide.07
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
the related works
YOLO
full name is you only look once
traditional methods like CNN, RNN is effective, but not good enough.
While these methods exhibit reasonably high accuracy,
they were also comes with complexity and slower inference times.
Over time, we have new solutions that could do the job quicker and more efficiently. that is YOLO.
YOLO is introduced in 2016 and it soon become the fastest methods,
Sacrifice a little bit of accuracy in exchange for extremely fast inference speed.
|
|
ICACT20250008 Slide.06
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
And the related works
To deploy the environment for more possible 3D virtual object designing,
Unity for the implementation of proposed, which is commonly used for 3D environment construction,is used.
Its functions provide the application of online or network GPU.
Unity can achieve outstanding results in color rendering through Web GPU.
When performing technical services and teaching training, the color distinction of the drawn extended reality virtual objects or environments will be more realistic. When considering various 3D models for teaching training or technical operations, there may be different situations or scripts.
Therefore, the database will be connected as a storage for storing data, including object parameters, restrictions and trigger conditions, etc.
In addition to using Unity to complete the 3D virtual environment and objects, the proposed system in this sresearch also needs to be able to provide different users with collaborative operations in different locations through the Internet.
|
|
ICACT20250008 Slide.05
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
And the problem part.
Due to the complexity of the 3D virtual object rendering, the assistance of GPU is needed. However, not every device at end user side could have the GPU progressing resources. Hence, if the GPU of the users’ devices is not available, the cloud server would render the virtual object in advance. According to the parallel processing and distributed system structure , the rendering work could be separated and distributed into individual GPU for image rendering based on the cloud computing environment. Finally, the rendered image could be delivered to the client users. In addition to the edge devices for parallel processing, the cloud computing platform could also provide the virtual machines with the assigned GPU for the image rendering. The remote client user can connect to the corresponding virtual machine for the rendered images.
|
|
ICACT20250008 Slide.04
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
In this research, the key differences compared to traditional VR and AR are the three points mentioned above. Since there is no need for controllers or electronic glasses, only a device capable of executing image recognition and a camera are required for this project, which makes our costs relatively low.
|
|
ICACT20250008 Slide.03
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
In our research, we developed a prototype of Extended Reality (XR). In recent years, artificial intelligence image recognition has become capable of recognizing gestures and locating users. Therefore, in this research, we use gestures to interact with the virtual environment.
|
|
ICACT20250008 Slide.02
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
First is the introduction.
|
|
ICACT20250008 Slide.01
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
this is our outline.
There are three parts in this report.
First is the introduction.
We will introduction of this report and what problem we have to solve.
And the related works.
Next is the methods part.
We will explain about Google MediaPipe, Unity, and how to integrate them together.
And show the results.
|
|
ICACT20250008 Slide.00
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Hello everyone my name is Cheng,Jay-Hong I come from National Formosa University
my report topic is Multi-Users Extended Reality Remote Collaborative Operation Application based on Digital Twins 3D Projecting
My advisor is Dr. Ming-Shen Jian
|
