 ICACT20240034 Slide.00
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Hello everyone my name is Tsai,Ming-Hsun I come from National Formosa University
my report topic is Multiple Merged Structures Based on Image Recognition
for Converting Application of Natural Language Artificial Intelligence Service
My advisor is Dr. Jian,Ming-Shen Jian
|
| ICACT20240034 Slide.01
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And this is our outline.
There are three parts in this report.
First is the introduction.
We will introduction of this report and what problem we have to solve.
And the related works.
Next is the methods part.
We will talk about the YOLOv8 and the proposed system that we made and the datasets.
We will comparison the YOLO we use.
And show the results.
|
| ICACT20240034 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
First is the introduction.
|
| ICACT20240034 Slide.03
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In our research, we build a Data structure to make the recognition results to be objects.
and the other parts is focus on image to text, i2t, to make results can be converted into natural language.
|
| ICACT20240034 Slide.04
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And in this research, we will focus on three points.
Small models, Multiple merged models, orderly structured objects.
In recent years, AI services is more and more popular.
To support the AI services, huge hardware resources for complex computing are needed.
Big AI model offend need to use the official API or web-services provided by the company.
If we intend to use the custom version or deploy the model in our own system.
It will be more limitation.
The small models of AI service are more easily to manage and deploy.
So we choose to use numbers of small models and merged them together.
When the computing job request, it could be delivered to the corresponding model.
Through orderly structured objects, we can more clearly distinguish the relationship between objects, and it is also easier to convert into natural language.
|
| ICACT20240034 Slide.05
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And the problem part.
Because not everyone has powerful hardware and enough image resource to build a big datasets.
But in now, most of people has the cellphone and the small dataset can easily find on internet or some website. Like Roboflow.
So we want to build a i2t system that can run in limited hardware and use the small datasets we get in internet.
|
| ICACT20240034 Slide.06
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And the related works
In this report we use YOLO.
full name is you only look once.
traditional methods like CNN, RNN is effective, but not good enough.
While these models exhibit reasonably high accuracy,
they were also come with complexity and slower inference times.
Over time, we have new solutions that could do the job quicker and more efficiently. that is YOLO.
YOLO is introduced in 2016 and it soon become the fastest model,
Sacrifice a little bit of accuracy in exchange for extremely fast inference speed.
YOLO is the fastest than other one-state models.
in this research, we use YOLOv8, which is the state-of-the-art (SOTA) models.
It used CSPDarkNet as backbone.
And PAFPN as neck.
And head part it used YOLOv8 decoupled head.
It is faster and more accurate.
Because we need to recognize more and more detail, the traditional methods have to use more hardware cost.
So we choose to build small models base on YOLO and make each of model can be deploy decentralized.
|
| ICACT20240034 Slide.07
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And this is the architecture of the YOLOv8.
YOLOv8 follows the single-shot detection paradigm, it means that processes the entire image in a single forward pass to predict bounding boxes and class probabilities.
The YOLOv8 architecture comprises multiple backbone networks, like CSPDarkNet.
To providing users with flexibility based on computational resources and performance requirements.
These backbone networks are integrated with a neck and head architecture, enhancing feature extraction and bounding box prediction.
One of the notable features of YOLOv8 is its anchor-based approach to predicting bounding boxes.
YOLOv8 also introduces a novel ensemble approach, allowing users to combine different model variants to achieve superior performance.
|
| ICACT20240034 Slide.08
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And next is about Natural Language Processing.
Natural Language Processing (NLP) is a technology that enables computers to communicate with humans.
In NLP, Natural Language Generation (NLG) is a importance part.
it converts the results obtained by computers into human language,
allowing people to understand information that computer get from images.
In this research, we mainly using Natural Language Generation (NLG),
using GPT-3 as our tool to process results and transform them into natural language.
This enables us to present the information from images in a manner easily comprehensible to humans.
|
| ICACT20240034 Slide.09
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And next is the methods part. |
| ICACT20240034 Slide.10
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In our system, we add the SE module into YOLOv8.
The full name is Squeeze-and-Excitation Block.
It is a building block commonly used in convolutional neural networks to enhance feature representation.
That improved performance with small additional cost.
|
| ICACT20240034 Slide.11
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And the proposed System
This is our object structure,
using humans as an example.
It can be seen that the sub-objects owned by people may include glasses and clothing.
Attributes are like gender or height.
sub objects will also have their own properties.
Clothes has the color and material and glass has size and color.
And this would like a tree structure.
|
| ICACT20240034 Slide.12
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And this is the architecture of our system.
First starting the system, the input would be the pictures that need to be convert.
After inputting, the main objects will be recognized first.
The recognition results will be builds as data chain to save.
The determination of the objects class is makes by ourselves, and the importance of objects will be decided by size or proportion.
According to the object class that set in the advance, system will recognize if there have sub objects.
If it does, it will build a sub object data chain, and add to corresponding main data chain.
In each stage of image recognition would be different small model.
This multi-stage architecture make it can be deployed easier.
In the end, when system determines that there are no any other sub-objects,
system will merge all data chain together.
|
| ICACT20240034 Slide.13
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And this is the dataset we use.
We use three kinds of dataset to do the demo.
First is the part of COCO, we take 4000 of images to use.
And the second in the clothing dataset, we find a dataset that has 3600 images.
And final is the emotion dataset, it only has 400 images.
And all of our dataset are get from Roboflow.
|
| ICACT20240034 Slide.14
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And this is our training detail.
Our GPU is Quadro P2200.
And image size is 640.
Batch size is 16.
All of the datasets use same setting.
|
| ICACT20240034 Slide.15
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Final is the conclusion part.
|
| ICACT20240034 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The comparison.
We use yolov5 v8 and v8 add SE model to compare.
The Yolo v8 add SE model has better mAP.
So we decide to use the YOLOv8 add SE block in our research.
|
| ICACT20240034 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And the Results part
After image recognition after using our system, we can get a simple data structure chain.
When a new image comes, it will check the main object first, with own properties and sub-class object,
sub-class object will be the noun, the thing belong main object and properties will be adjective for main object.
For example, the no.1 main object is the person that in the middle.
the clothing is the sub-object and emotion is the properties.
Once we get the main object for the first segment of image recognition, we will send the results to the next independent model.
Each model is only responsible for a part of the recognition, allowing the model to focus more on a specific dataset, and recognize the sub-objects.
Models can run at the same time, it will reduce time cost.
|
| ICACT20240034 Slide.18
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
After that,
by merging multiple structures in the data structure chain to the main object, the suitable description can be given.
The results will be a group of simple words.
And it can be described as “a person who wears white shoes and black romper is happy” just use simple NLG program.
By use this kind of data structure chain, you can easily know the sub-object is belong to which main object.
For example, you can see the first person is the main object.
And it has two footwears which is white and the romper which is black and it is happy.
|
| ICACT20240034 Slide.19
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Conclusions
In this research, we offer a new way to convert image to text, by using Multiple objects structure like data chain and tree structure.
And each kind of the object structure is recognized by different independent modals.
base on that, we get more suitable text with more relevant and more orderly.
|
| ICACT20240034 Slide.20
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
And the Future work.
Our Future work is to include more datasets in our system, and deployment the system to the cloud.
We want to build a website that able to use people’s own datasets and modify our system to create their custom version.
|
| ICACT20240034 Slide.21
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This conclude my report.
Thank you for listening.
|
