 ICACT20260046 Slide.01
[Big Slide]
|
Chrome  Click!! |
 |
Hello everyone. Today, I am going to present our research titled “The Feasibility of Network Data Transmission Reducing Based on VLM.”
|
| ICACT20260046 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
In this presentation, we will introduce the background, the problem, our method, technical details, the system flow, our experiment results, and future applications.
|
| ICACT20260046 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
Digital Twin uses a physics model to connect the real and virtual world. When combined with XR and AI, many users can work together in the same virtual space but in different real environments. However, XR and Digital Twin systems need to send a large amount of video or image data. This creates heavy network traffic and causes delays.
Next will talk about the heavy network problem.
|
| ICACT20260046 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
The following are some of the issues contributing to network load.
Sending images takes a lot of bandwidth. This way may cause latency issues.
In real-time systems, even small delays will affect user experience. When devices send image files back and forth, the latency becomes very obvious. So, we need a way to reduce the size of transmitted data.
|
| ICACT20260046 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
Our solution has four steps. First, we use YOLO to detect objects in the image. Second, we build a data structure to store object attributes such as color, size, and position. Third, we use a Vision Language Model to create a relationship tree between each of the objects. Finally, we convert the image into simple natural-language sentences and send only text through to the network.
Next I`m going to explain why we use YOLO to detect objects.
|
| ICACT20260046 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
We chose YOLOv8 to be our detection training model
because it is a relatively mature training structure(architecture).
(In our research， we improved the YOLOv8 CSP structure to have better accuracy and stability.)
In the next section, we are going to discuss how we turn image content into text.
|
| ICACT20260046 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
The Vision Language Model reads the detected objects with YOLO and creates a text description.
And the visual encoder converts the image into LLM.
In the last, the language model then uses these LLM to write meaningful sentences. This is how we turn image content into text.
About to defend different detected objects’ weight question.
|
| ICACT20260046 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
We use the attention mechanism.
We apply different attention mechanisms so the model can focus [on important parts].(on the parts that we want.) |
| ICACT20260046 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
After using the attention mechanisms, we use the ViT-B/32 model to understand the image.
ViT-B/32 is a type of Vision Transformer. Then we check the result with OpenAI CLIP. CLIP does not need any extra training. It can directly compare our text with the image and tell us if the
description matches the picture.
|
| ICACT20260046 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
To produce better sentences, we also extract object attributes, including color, size, location, body part, and actions. More attributes lead to a richer and more accurate sentence.
|
| ICACT20260046 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
Building upon the previous stage, the human body is segmented into four regions. The system evaluates relative size, overlap,
and spatial distance between these regions to determine the appropriate textual attributes.
For example, when a person is wearing a hat, the system detects the head and the hat regions, evaluates their spatial overlap and relative position, and generates the sentence: “A person is wearing a hat.”
|
| ICACT20260046 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
This is our Flowchart. |
| ICACT20260046 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
Experimental Results
|
| ICACT20260046 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
We tested several images. Our system and CLIP both generated description sentences.
Then we used Adobe Firefly and DeeVid AI to rebuild images from the text.
This proves that text-based transmission is feasible.
|
| ICACT20260046 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
We tested several images. Our system and CLIP both generated description sentences.
Then we used Adobe Firefly and DeeVid AI to rebuild images from the text.
This proves that text-based transmission is feasible.
|
| ICACT20260046 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
We got the same result when we used another image to identify it.
|
| ICACT20260046 Slide.17
[Big Slide]
|
Chrome Click!! |
 |
We got the same result when we used another image to identify it.
|
| ICACT20260046 Slide.18
[Big Slide]
|
Chrome Click!! |
 |
Effectively reduces latency and bandwidth usage
Key visual information can be used to preserve and reconstruct an image
Only 1~1.25(%) of original image data size description
Text-based transmission instead of images
|
| ICACT20260046 Slide.19
[Big Slide]
|
Chrome Click!! |
 |
Thanks for the contribution and assistance of
Cloud Computing and Intelligent System Lab. in National Formosa University, Taiwan. |
| ICACT20260046 Slide.20
[Big Slide]
|
Chrome Click!! |
 |
Thanks for your listening |
