 ICACT20260070 Slide.01
[Big Slide]
|
Chrome  Click!! |
 |
this research addresses the challenge of identifying AI-generated content. As digital authenticity becomes increasingly compromised by advanced generative models, the study explores how leveraging large multimodal models can enhance both detection accuracy and the interpretability necessary for forensic analysis . |
| ICACT20260070 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
The primary motivation for this research stems from the ubiquity of Generative AI, now utilized by 83% of creative professionals and widely adopted in personal contexts . While beneficial, this technology accelerates the dissemination of misinformation, as illustrated by the viral "Himalayan Pagoda Blossoms" case shown here . Although claimed to be a rare natural phenomenon, tools like Hive Moderation confirmed the image to be 99.7% AI-generated. This example demonstrates how highly realistic fake content can mislead the public, necessitating robust detection tools . |
| ICACT20260070 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
Beyond viral images, Generative AI poses severe security risks, including fraud and deepfakes . A notable incident occurred in February 2024, involving a finance clerk scammed out of 25 million dollars following a video call with a deepfake CFO and colleagues . This incident demonstrates that AI-generated content represents a significant cybersecurity threat rather than merely an aesthetic issue. Consequently, effective systems must not only detect fake images but also provide interpretability to explain "generation traces," thereby helping users understand why an image is flagged . |
| ICACT20260070 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
This study presents several key contributions. First, the GIDLMM framework is introduced, combining a detection module based on CLIP and LoRA with an interpretation module that explains visual artifacts . Second, the method demonstrates high efficiency, achieving effective detection using only 1,000 training samples and fine-tuning a small fraction of parameters. Third, a novel "AI Judges" prompt mechanism is proposed to select optimal image descriptions without the high computational cost of traditional methods, improving both quality and efficiency for real-world applications . |
| ICACT20260070 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
The research builds upon the CLIP architecture, a multimodal model pre-trained on 400 million image-text pairs and known for strong zero-shot capabilities . While existing methods like "Linear Probing" or "Adapter Networks" utilize CLIP for detection, these approaches typically focus solely on binary classification- determining if an image is real or fake without providing an explanation . This work addresses that gap by enhancing CLIPs semantic understanding through generated descriptions and adding an interpretability layer to identify specific generation traces . |
| ICACT20260070 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
The GIDLMM system architecture consists of four distinct stages as illustrated in Figure 3.1 . First, the Data Collection stage utilizes the GenImage dataset. Second, the Data Processing stage enhances CLIPs detection capabilities by employing Large Multimodal Models, such as LLaMA and ChatGPT, to generate detailed image descriptions . Third, the Detection Model is trained to classify images utilizing these enhanced features. Finally, the Truthfulness Prediction stage assesses system performance using Accuracy and AUC metrics . |
| ICACT20260070 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
The GenImage dataset, a large-scale benchmark based on ImageNet-1k, was selected for this study . Unlike prior datasets focusing on specific deepfakes like faces, GenImage includes diverse categories generated by eight different models, ranging from GANs like BigGAN to diffusion models like Stable Diffusion and Midjourney . The dataset maintains a balanced distribution of real and fake images with high resolutions, providing an ideal foundation for training a robust model capable of generalizing across various generative techniques. |
| ICACT20260070 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
In the Data Processing stage, multiple Large Multimodal Models- including ChatGPT, LLaMA 3.2, and LLaVA- are employed to generate text descriptions for the training images. These descriptions serve as augmented data to align image and text representations within the CLIP feature space . To ensure reliability and prevent hallucinations, a voting mechanism is implemented where a description generated by one model is verified by three others . Only descriptions passing this majority vote are utilized, ensuring the training data remains factual and consistent . |
| ICACT20260070 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
Three prompt strategies were examined. The first, "Simple Descriptive Prompts," explicitly directs the model to identify clues such as unnatural textures or lighting . However, the t-SNE visualization in Figure 3.4 demonstrates that feature vectors for real and fake image descriptions overlap significantly using this method . This lack of separation indicates that simple prompts fail to produce sufficiently discriminative semantic features, making effective differentiation between real and AI-generated images difficult based on this strategy alone . |
| ICACT20260070 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
To address the limitations of simple prompts, the "Self-Consistency" technique was applied . This approach involves querying the LMM three times for each image and employing a voting process to select the most consistent description . As indicated by the t-SNE visualization in Figure 3.7, this method successfully separates real and AI-generated descriptions within the feature space . While producing high-quality data, the requirement for multiple queries and voting rounds significantly increases computational cost and latency . |
| ICACT20260070 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
The "AI Judges" mechanism is proposed to balance quality and efficiency. Instead of multiple query rounds, three descriptions are generated in a single session, and virtual judges such as a "Lighting Judge" or "Clarity Judge" evaluate them based on specific criteria . This approach simulates diverse perspectives to select the best description instantly. As shown in Figure, the method achieves effective feature separation similar to self-consistency but with significantly lower computational overhead, making it practical for real-time applications . |
| ICACT20260070 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
For the detection module, the CLIP ViT-L/14 model functions as the feature extractor, with its pre-trained weights frozen to retain general semantic knowledge . Low-Rank Adaptation (LoRA) modules are inserted into the transformer blocks, specifically training only the rank-8 matrices . This enables fine-tuning using just 0.32% of the total parameters approximately 1.35 million out of 400 million . A contrastive loss function aligns the image and text features, followed by an MLP classifier to determine the authenticity of the image. |
| ICACT20260070 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
Upon classifying an image as fake, the system passes it to the generation trace interpretation module. This module employs a frozen CLIP encoder with LoRA to extract visual features, which are then projected via an MLP to align with the language models space . These features are fed into a decoder-only LLM (LLaMA 3.1), also fine-tuned with LoRA . Trained via supervised learning on ground truth descriptions, this module generates text explaining specific artifacts, such as unnatural lighting or pixelation, thereby reducing hallucinations and enhancing trust . |
| ICACT20260070 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
Experiments utilized an Intel i9-12900 processor and an NVIDIA GeForce RTX 4070 Ti SUPER GPU with 16 GB of VRAM. This configuration was selected to manage the memory requirements of the large multimodal models and the quantization processes necessary for the interpretation module . PyTorch and the Hugging Face PEFT library were employed for efficient fine-tuning . The development environment operated across both Windows and Linux to accommodate specific quantization tools required for the LLM training . |
| ICACT20260070 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
The detection module was evaluated using descriptions generated by various LMMs, including ChatGPT, LLaMA 3.2, Gemma 3, and LLaVA. The results in Table 4.6 indicate that regardless of the model utilized, the average accuracy consistently exceeds 89%, with LLaMA 3.2 achieving the highest accuracy at 89.59% . This demonstrates that the GIDLMM framework generalizes effectively across different multimodal architectures . Additionally, detection performance on advanced generators like Midjourney and Stable Diffusion proved particularly strong, with AUC scores exceeding 97%. |
| ICACT20260070 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
The impact of different LoRA rank settings 8, 16, 32, and 64 on model performance was analyzed . Results indicate that increasing the rank does not necessarily yield better performance. With the AI Judges mechanism, a rank of 8 achieved the highest average accuracy of 89.25%. As the rank increased, performance displayed a slight downward trend. This suggests that for a dataset size of 1,000 images, a rank of 8 provides sufficient trainable parameters without overfitting, confirming that high performance is achievable with minimal computational cost . |
| ICACT20260070 Slide.17
[Big Slide]
|
Chrome Click!! |
 |
A comparison of the three prompt designs shows that the "Simple Descriptive Prompt" and the proposed "AI Judges" mechanism achieved accuracies of approximately 89.5% and 89.3%, respectively. The "Self-Consistency" method performed the lowest at 87.51%. This lower performance is attributed to the tendency of the Self-Consistency method to generate longer, more complex responses that exceed CLIPs 77-token input limit, leading to truncation and information loss . This validates that simpler and more efficient prompting strategies better suit the constraints of the CLIP model architecture |
| ICACT20260070 Slide.18
[Big Slide]
|
Chrome Click!! |
 |
GIDLMM was compared against the state-of-the-art "CLIPping" model on GAN datasets. While the CLIPping "Adapter Network" performs slightly better when trained on massive datasets of 100,000 images, GIDLMM remains highly competitive using only 1,000 training samples . Crucially, when compared to the CLIPping "Linear Probing" method, GIDLMM significantly outperforms it across all dataset sizes, achieving much higher average accuracy . This highlights the models superior data efficiency and ability to learn effective features even in low-resource scenarios. |
| ICACT20260070 Slide.19
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
On datasets from commercial tools such as Midjourney, Adobe Firefly, and DALL-E 3, GIDLMM demonstrates a distinct advantage. Even when trained on just 1,000 images, the model achieves an average accuracy of 94.87%, significantly outperforming the CLIPping model, which only reached 89.63% even with 100,000 training samples. This represents a performance improvement of nearly 14%. These results demonstrate that GIDLMM is exceptionally effective at detecting high-quality images from modern commercial generators without requiring massive training datasets. |
| ICACT20260070 Slide.20
[Big Slide]
|
Chrome Click!! |
 |
A distinguishing feature of GIDLMM is the ability to explain detection decisions . For instance, when analyzing a Stable Diffusion image of a spider, the model identified "unnatural textures" and noted that the "body is overly smooth" with "legs too uniform". Similarly, for a Midjourney image of a dog, it pointed out "unnatural lighting on the dogs face," described as an "overly bright and uniform glow". These specific, semantic descriptions extend beyond simple binary labels, offering users concrete reasons to verify the detection result. |
| ICACT20260070 Slide.21
[Big Slide]
|
Chrome Click!! |
 |
Continuing with interpretability, the model successfully analyzed a Wukong-generated image, noting "repetitive pasta" and a "lack of realistic texture". For a GLIDE-generated monkey, it highlighted that the "fur lacks depth and dimension" and flagged "visible pixelation". These examples confirm that the interpretation module can effectively detect and describe generation traces across various types of generators, providing transparency that is crucial for forensic analysis and combating misinformation . |
| ICACT20260070 Slide.22
[Big Slide]
|
Chrome Click!! |
 |
An analysis of cases where the model incorrectly classified real images as AI-generated was also conducted. In one instance, the model erroneously claimed a real image displayed "unnatural textures" and "jagged edges" on leaves. While these were classification errors, the generated explanations remain valuable as they allow human reviewers to verify specific claims—checking if the leaves truly appear jagged or if the texture is organic. This transparency facilitates easier debugging and human verification compared to opaque "black box" decisions. |
| ICACT20260070 Slide.23
[Big Slide]
|
Chrome Click!! |
 |
In conclusion, the proposed GIDLMM framework successfully utilizes Large Multimodal Models to enhance AI-generated image detection. The system achieved an average accuracy of 89% and an AUC of 95% while training with only 1,000 samples and modifying less than 0.34% of the parameters . The proposed "AI Judges" mechanism proved to be both efficient and effective. Furthermore, the model outperforms state-of-the-art approaches on commercial datasets by over 13% and uniquely provides interpretable explanations for decisions, addressing the critical need for trust in AI detection |
| ICACT20260070 Slide.24
[Big Slide]
|
Chrome Click!! |
 |
Several promising directions for future research have been identified. Plans include incorporating newer LMMs such as LLaMA 4 or Mistral to further improve dataset quality and diversity . Future work will also explore larger feature extraction architectures, like ConvNext-Large, to enhance cross-modal alignment. Finally, the interpretation module is intended to evolve from generating static single-round descriptions to supporting multi-turn interactive dialogue . This would allow users to ask follow-up questions about specific image regions, making the system a more powerful tool for analysis. |
