 ICACT20240342 Slide.00
[Big Slide]
|
Chrome  Click!! |
 |
Hello everyone, I’m Liujing, a PhD student from the University of Chinese Academy of Sciences. It is my privilege to be here today to share a presentation on Utterance-Level Incongruity Learning Network for Multimodal Sarcasm Detection.
|
| ICACT20240342 Slide.01
[Big Slide]
|
Chrome Click!! |
 |
Here, I will provide a comprehensive overview of our work, focusing on the following aspects: introduction, contributions, design, experiments and Conclusion
|
| ICACT20240342 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
To begin with, what exactly is sarcasm? Sarcasm involves expressing a meaning that is often the opposite of its literal sense. It frequently employs positive language to convey negative implications.
Consequently, in the initial stages, we focused on identifying sarcasm through the detection of textual inconsistencies.
However, more and more social media platforms allow users to post multi- modal comments such as videos, which bring both opportunities and challenges to multi-modal sarcasm detection.
|
| ICACT20240342 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
So, how can we more efficiently detect sarcasm meanings from videos? Our work will make the following three contributions:
Firstly, We propose a novel utterance-level incongruity learning network for multimodal sarcasm detection. The core module is multimodal utterance-level attention (M-ULA) which relies on utterance-level to extract unimodal and multimodal features and effectively explore the complex interrelationships present in multimodal data.
Secondly, To capture the incongruity information, we design an incongruity learning network (ILN) which consists of primary modality choosing, cross attention mechanism and self-attention mechanism to explore sufficient incongruity information.
Lastly, we conducted extensive experiments and comparisons on a public datasets to demonstrate the superiority of the proposed method. Additionally, we conducted ablation experiments, which demonstrated the rationality and necessity of the proposed method.
|
| ICACT20240342 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
Here, this is the overview of our model.
|
| ICACT20240342 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
The module in the bottom is the Utterance-Level Features Extraction Layer which include unimodal feature extractions, alignment and multimodal feature extraction.
|
| ICACT20240342 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
For the unimodal feature extractions, we employ a pre-trained BERT-base model to encode the input textual tokens into word embedding, and take advantage of Long Short-Term Memory(LSTM) to extract audio and vision features.
Next for the alignment part, we deploy a 1-dimensional convolution layer and a scale layer to align the dimension of feature vectors space.
And for the Utterance-Level Feature Representations, we employ utterance-level attention(ULA) to learn the distinctive properties of unimodal modality, as for the detailed information, refer to the next page.
|
| ICACT20240342 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
Let‘s take a closer look at how multimodal feature extraction module works.
We take three token-level unimodal features as input and gain their utterance-level features represent, and concatenate these six features to obtain the fused features.
Then we use the attention mechanism to obtain attention score and updated multimodal feature $X_{m}^{\prime}$
Eventually, we split the interactive features into two sets and passing through several linear layers, we obtain the multimodal features representation.
|
| ICACT20240342 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
Here, the module in the medium is the Incongruity Learning Network (ILN) Layer which can obtain inconsistent features between multimodal. including primary modality selecting, incongruity leaning by cross-attention and self-attention.
|
| ICACT20240342 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
First, to assess the modality inconsistency, it is essential to identify the primary modality, we automatically assign weight values based on the varying contributions of each modality for sarcasm detection, with a higher contribution leading to a higher weight value. We update the weight values in each training batch while maintaining the sum of all trainable weight values equal to 1. We obtain the primary modality P and auxiliary modalities A_1, A_2, A_3.
|
| ICACT20240342 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
Then, In order to obtain incongruity from primary modality and auxiliary modalities, we leverage the cross-attention(CMA) mechanism to acquire interacted auxiliary modalities A_1 tilde，A_2 tilde and A_3 tilde。
Subsequently, we can obtain the interacted primary modality representations.
|
| ICACT20240342 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
Last, for the top of this layer, we concatenate interacted primary modality, and then we employ a self-attention(SA) mechanism to identify its salient components as the ultimate primary modality representation.
In the end, we obtain the multimodal inconsistency representation by concatenating auxiliary modalities.
|
| ICACT20240342 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
Through previous research， we can obtain the multimodal inconsistency representation, here, prediction layer is to detect dataset whether sarcasm.
|
| ICACT20240342 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
Therefore, the ultimate utterance-level incongruity representation is passed through a fully connected layer with a softmax activation function to produce a probability distribution y in the multimodal sarcasm decision space.
|
| ICACT20240342 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
Moreover, We evaluate our proposed modal on a publicly available multi-modal sarcasm detection benchmark dataset MUStARD published by Castro.
Detailed statistics of the experimental dataset are provided in the table.
|
| ICACT20240342 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
The comparative evaluation experiment results of our proposed model and all baselines are shown in this Table. We can draw the following conclusions:
First, our proposed ULIL outperforms existing baselines on all metrics, confirming the effectiveness of our model in multimodal sarcasm detection.
Second, all of the trimodal sarcasm detection approaches are superior to just text modality detection methods, demonstrating the complementarity of text, image, and audio modalities, thereby enhancing the performance of sarcasm detection.
Third, all models exhibit lower performance in the speaker-independent setup compared to the speaker-dependent one. This is attributed to the absence of speaker overlap between the training and testing sets in the speaker-independent configuration, making it more challenging than the speaker-dependent setting.
Fourthly, in both the speaker-dependent and speaker-independent settings, our model ULIL achieves an F1-score value that is 2\% and 2.8\% higher than the most recent baseline IWAN, respectively.
|
| ICACT20240342 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
To further analyze our proposed model and assess the necessity of the three modalities, we conduct ablation experiments on the dataset with different modalities removed. As text modality is the most fundamental, we separately removed audio and vision modalities and recorded the experimental results in Table. Removing either audio or vision modality leads to a performance drop, indicating the crucial importance of these two modality features for the sarcasm detection task. Audio and vision modality features complement text features, enabling a more comprehensive capture of inconsistent features.
|
| ICACT20240342 Slide.17
[Big Slide]
|
Chrome Click!! |
 |
On the other hand, to further analyze the effectiveness of the main modules in ULIL, we removed the primary modules, the multimodal utterance-level attention module and the incongruity representation module. According to the results in Table, it is evident that the performance deteriorates when the utterance-level attention module is not used, underscoring its capability to capture crucial sentiment information.
|
| ICACT20240342 Slide.18
[Big Slide]
|
Chrome Click!! |
 |
Okay, in this paper, we investigate a comprehensive and robust approach that employs an Utterance-level Incongruity Learning Network (ULIL) to address multimodal sarcasm detection tasks. Unlike previous works, we extract utterance-level unimodal features and multimodal features fused by a multimodal utterance-level attention. Additionally, we explore an incongruity learning network for sarcasm detection. Specifically, considering the sarcasm utterances process incongruous characteristics, we deploy a cross-attention and self-attention mechanism to acquire incongruities between primary modality and auxiliary modalities. Our proposed model has been experimentally tested on a public benchmark dataset, demonstrating a significant performance improvement compared to a series of baselines for the multimodal sarcasm detection task.
|
| ICACT20240342 Slide.19
[Big Slide]
|
Chrome Click!! |
 |
Thanks for your listening!
|
