ICACT20250006 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
That's all we have prepared today. Thank you for listening our presentation.
|
|
ICACT20250006 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Therefore, as a follow-up study, we plan to train a better text encoder and learn more Lora. Thank you.
|
|
ICACT20250006 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The limitations of the study were that our Korean text encoder was not as good as the existing English encoder, and the number of Lora created for K-Contents generation was insufficient.
|
|
ICACT20250006 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Our pipeline takes the generator from the Stable Diffusion 1.5 model and the image created based on the Korean Text Encoder and feeds it back into the Image Refiner in the SDXL model.
We use a Base Prompt such as “Realistic, Masterpiece,” etc. as the Text Condition for the SDXL model's Image Refiner, and output the final image after passing through the SDXL model's Refiner.
|
|
ICACT20250006 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This is the result of "female hanbok" and "gyonbokgung".
|
|
ICACT20250006 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This is the result of "Animation Character" and "male hanbok".
|
|
ICACT20250006 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The results are as follows.
For each trigger word or specific prompt, we were able to see good quality results.
|
|
ICACT20250006 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following is how to learn Lora specifically for K-Content.
First, we collected image data for various K-Contents.
We collected ink paintings and animations from AIhub, and collected hanbok and Gyeongbokgung Palace data through web data crawling.
The numbers and details of the datasets can be found in the table below.
Then, we preprocessed the images into squares. The images were cropped twice at random locations to extract a total of two images from one image.
|
|
ICACT20250006 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here’s a comparison of the three image generation methods we explored.
First, SD1.5 is very fast, taking just 2.94 seconds per inference, with low VRAM usage at 6 GB. However, the image quality is lower, especially for details like hands and faces.
Next, SDXL with translation takes 15.56 seconds, requiring 24 GB of VRAM. While slower and resource-intensive, it produces high-quality images, reflecting prompts and details like hands and faces very well.
Finally, the SD1.5 with SDXL Refiner strikes a balance. It’s faster than SDXL alone, at 4.53 seconds, with moderate VRAM usage of 10 GB. It also delivers excellent detail, particularly in hands and faces.
Each method has its trade-offs in speed, resource usage, and quality.
|
|
ICACT20250006 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Finally, we optimized the pipeline by combining the strengths of the two previous methods.
We imported only SDXL's Refiner and applied it to the results from Stable Diffusion 1.5.
This allowed us to have moderate ram usage and inference time.
|
|
ICACT20250006 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In contrast, using Stable Diffusion XL model with Translation model has the advantage of generating high resolution and well made images. Also, this method has disadvantages of memory usage and expensive generating time.
|
|
ICACT20250006 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Let's take a closer look at the three methods.
The first method, using Single Stable Diffusion 1.5 model has the advantage of fast image generation and low memory usage. However, there are disadvantages of image quality and generating poor hands and faces.
|
|
ICACT20250006 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Let me explain the image generation methods we tried.
First, we used the Stable Diffusion 1.5 model on its own. This was a simple and stable approach.
Next, we worked with the SDXL model. Since it had difficulty processing Korean prompts, we translated them into English using a translation model before inputting them. This helped us overcome the language limitation.
Lastly, we generated images with SD1.5 and refined them using the SDXL Refiner to enhance their quality.
These three methods helped us explore different ways to optimize image generation.
|
|
ICACT20250006 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Also, Our model even performed well enough to compare with the results of the English encoder.
|
|
ICACT20250006 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This is the experimental result for the korean text encoder.
The experiment was conducted using random prompts generated by GPT.
You can see that our model preforms better then previous study, Bingsu/koclip.
|
|
ICACT20250006 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
After collecting datasets, we trained CLIP Encoder with the Korean datasets.
We used a method in the paper "Making monolingual sentence embeddings multilingual using knowledge distillation, 2020".
For the training, used hyperparameter is shown in the slide.
|
|
ICACT20250006 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
First, let's talk about the korean text encoder implementation.
We collected three datasets named “Korean-English Translation Corpus Dataset” from the AI Hub.
And merged the three datasets to create 4.3 million Korean-English corpus pairs.
The resulting preprocessed corpus pair CSV file was 1.2 gigabytes.
|
|
ICACT20250006 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
There has been some related research. Diffusion-Based Text to Image Model had SD1.5, SDXL, etc. For K-Contents Image Generation, there was a Low Rank Adaptation method that trained only some parameters. For learning Korean Text Encoder, there were Koclip on github, bingsu/koclip on huggingface.
|
|
ICACT20250006 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
With the recent boom of generative AI models like ChatGPT, Text2Image models have gained significant attention. Simultaneously, the global demand for Korean culture, or 'K-Contents,' is increasing. In this context, our research aims to build a pipeline that generates high-quality, Korean-specific images using Korean text prompts.
|
|
ICACT20250006 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
My name is Seungeun Kang, and I am here to present our research titled 'Research on Building K-Contents Specialized Text2Image Pipeline'. This study focuses on building a unique pipeline for generating Korean culture-specific images using text prompts.
|
