Now 5097 visitors
Today:0 Yesterday:0
Total: 147118 212S 50P 4R
2026-10-20, Week 43
Member Login
TACT Journal Homepage
Call for Paper
Committee
Find My Paper
Author Homepage
Statistics & History
Welcome Message
Paper Procedure
FAQ
Paper Archives
Outstanding Papers
Program & Proceedings
Presentation Platform
Hotel & Travel Info
Photo Gallery
Scheduler Login
Seminar
Archives Login
Sponsors


















Presentation Assistant Go to [Q&A Session]
Paper Number
Paper Title
Keyword
ICACT20260151 Slide.01        [Big Slide]      [YouTube] Chrome Click!!
Hello everyone. My name is Hong-Quan Do, from FPT University, Vietnam. It is my honor to present our research at this year’s ICACT conference, titled “Multi-View Facial Expression Recognition with AU-Guided Heatmaps.” This work was conducted in collaboration with Dr. Hoang Van Thanh and Professor Tu Minh Phuong.
ICACT20260151 Slide.02        [Big Slide] Chrome Click!!
The ability to recognize and interpret human facial expressions represents one of the most fundamental aspects of human social interaction and emotional communication. Facial Expression Recognition (FER) is fundamental to affective computing and human-computer interaction, enabling systems to infer emotional states from facial cues. It takes images or videos as input and subsequently classifies expressions into one of seven basic and universal emotions: Anger, Disgust, Fear, Happiness, Neutral, Sadness, and Surprise.
ICACT20260151 Slide.03        [Big Slide] Chrome Click!!
The advancement of deep learning techniques has revolutionized the field of computer vision, and FER has benefited substantially from these developments. Deep learning approaches for facial expression recognition have demonstrated remarkable progress, primarily leveraging convolutional neural networks and their variants to learn hierarchical feature representations directly from raw facial. However, these current deep learning approaches rely solely on raw facial images and standard augmentation but lack explicit incorporation of facial anatomical knowledge. This forces models to discover expression-discriminative regions from scratch rather than leveraging established facial behavior research.
ICACT20260151 Slide.04        [Big Slide] Chrome Click!!
The key contributions of this work can be summarized as follows: First, we propose MV-FER, a unified training paradigm that synthesizes three data views: original images, augmented flipped images with random erasing for view consistency, and AU-guided heatmaps for attention alignment. Second, we introduce a method for generating AU-guided heatmaps from facial landmarks and FACS theory, which serve as a signal to guide the spatial attention mechanisms of deep networks toward expression-based regions. We demonstrate consistent performance improvements over state-of-the-art methods on benchmark dataset RAF-DB, with ablation studies confirming the progressive gains from each view component.
ICACT20260151 Slide.05        [Big Slide] Chrome Click!!
Recent advances in Facial Expression Recognition (FER) have increasingly incorporated domain-specific knowledge to enhance model performance. This section reviews key developments in multi-view learning approach, and facial landmark utilization that form the foundation for our work.
ICACT20260151 Slide.06        [Big Slide] Chrome Click!!
Recent studies have increasingly adopted multi-view learning paradigms. In this approach, multiple augmented versions of the same input image—obtained through transformations including horizontal flipping, random cropping, affine warping, or color space modifications—are treated as distinct yet semantically consistent views. Joint learning across these complementary views enhances model robustness and encourages invariance to nuisance factors. However, conventional multi-view augmentation strategies remain generic in nature, as they can be applied to any image recognition task without exploiting the unique anatomical structure of human faces. In contrast, FER presents a distinctive opportunity to leverage domain-specific priors derived from established psychological and anatomical frameworks. Specifically, Action Units (AUs), as formally defined by the Facial Action Coding System (FACS) developed by Ekman and Friesen in 1978, describe localized regions of facial muscle activity that exhibit strong correlations with emotional expressions. Prior researches in the field of computer vision, such as Martinez et al. in 2019 and Belharbi et al. in 2024, have also established comprehensive mappings between facial expressions and their corresponding action units.
ICACT20260151 Slide.07        [Big Slide] Chrome Click!!
Facial landmark detection, which identifies and localizes predefined keypoints on human faces, has become instrumental in numerous face analysis applications. In FER, researchers have leveraged facial landmarks as informative structural features to complement raw image data. Jung et al. in 2017 pioneered this direction by introducing a dual-network CNN architecture that processes both facial images and landmarks through weighted summation. Building on this foundation, Meng et al. in 2022 proposed a dual-branch network combining CNN and Graph Convolutional Networks (GCN) Beyond direct landmark utilization, recent research has explored heatmap-based approaches to guide model attention toward expression-relevant facial regions. Notably, in previous work, we addressed the question of how to effectively represent facial landmark data: whether raw coordinate positions or heatmap-based representations provide better inputs for neural networks, and how to identify optimal fusion strategies for combining heatmap-based features with image-based features. These key developments in multi-view learning approach, and facial landmark utilization that form the foundation for this work where we propose a multi-view framework to improve FER performance. It is named as MV-FER.
ICACT20260151 Slide.08        [Big Slide] Chrome Click!!
We propose a multi-view framework to improve FER performance. It is named as MV-FER.
ICACT20260151 Slide.09        [Big Slide] Chrome Click!!
The proposed multi-view method, MV-FER, is depicted in Figure 5. MV-FER utilizes three complementary views: original facial images and horizontally flipped augmented images as dual input branches, while AU-guided heatmaps—derived from facial landmarks and FACS theory—to guide attention toward expression-critical regions including the eyes, brows, and mouth areas. The framework employs a unified training paradigm ensuring consistent representation learning across both input views while aligning learned attention with expression-relevant facial regions. This integration of multi-view learning and attention alignment enables our proposed method, named MV-FER, to achieve enhanced facial expression recognition performance. During training, the pair (x_1, x_2) is fed into a shared backbone network and Classifier head to produce class predictions and feature representations, while x_3 is used for attention alignment. During the validation and inference phases, only the original view of each image in the test set is employed. Each image is fed into the trained backbone network to generate class logits, which are subsequently passed through a softmax layer to obtain a probability distribution over the seven expression categories: surprise, fear, disgust, happiness, sadness, anger, and neutral. The final prediction is determined by selecting the class with the highest probability.
ICACT20260151 Slide.10        [Big Slide] Chrome Click!!
As mentioned, instead of using landmark-derived heatmaps as direct input features, we generate AU-guided heatmaps that serve purely as supervisory signals to guide attention learning—essentially teaching the network where to focus on expression-critical regions. The Facial Action Coding System (FACS) by Ekman in 1978 provides an anatomically grounded framework for decomposing facial expressions into Action Units (AUs)—elementary muscle movements that form the basis of emotional displays. Each basic emotion is characterized by a distinctive subset of AUs localized to specific facial regions, enabling systematic computational analysis of expressions. This established knowledge enables us to construct AU-guided heatmaps that direct attention toward expression-relevant areas.
ICACT20260151 Slide.11        [Big Slide] Chrome Click!!
Figure 6 illustrates our pipeline of generating AU-Guided heatmaps. The process initiates with facial landmark detection on the input image, providing precise spatial coordinates for key facial features. These detected landmarks are subsequently mapped to anatomically meaningful facial regions that correspond to expression-critical areas as defined by FACS theory. The systematic construction of expression-specific heatmaps is enabled through the integration of expert knowledge via pre-established lookup tables (Tables I and II). These tables provide comprehensive mappings between facial expressions and their corresponding Action Units, as well as the spatial relationships between AUs and specific facial regions.
ICACT20260151 Slide.12        [Big Slide] Chrome Click!!
We demonstrate consistent performance improvements over state-of-the-art methods on benchmark dataset RAF-DB, with ablation studies confirming the progressive gains from each view component.
ICACT20260151 Slide.13        [Big Slide] Chrome Click!!
We evaluate the effectiveness of our proposed method on the Real-world Affective Faces Database (RAF-DB), a widely adopted benchmark for facial expression recognition. The dataset has been carefully curated to cover diverse conditions, including variations in age, gender, ethnicity, head pose, illumination, and occlusion, making it highly representative of real-world FER challenges. It is also important to note that the dataset is highly imbalanced, with an imbalance ratio of 16.78, reflecting substantial variation in class frequencies. In our experimental evaluation, we employed ResNet as the backbone network, following established practices in recent facial expression recognition literature to ensure fair comparison with state-of-the-art methods that utilize similar pre-training strategies. At a first glance, Figure 7 depicts the training and validation performance of the proposed model across 100 epochs. The validation curves closely follow the trends observed in the training curves. Leveraging pre-trained weights, the model converges rapidly, with training loss nearing zero by epoch 30. Validation curves stabilize after ~20 epochs, but accuracy remains competitive (around 88%), demonstrating effective generalization to unseen data.
ICACT20260151 Slide.14        [Big Slide] Chrome Click!!
We benchmark MV-FER against state-of-the-art methods on the RAF-DB dataset using the evaluation setup described above. MV-FER is assessed in three variants, demonstrating progressive improvements through the integration of augmented views and AU-guided heatmap views. The quantitative results are reported in Table III. Our full three-view MV-FER framework achieves an accuracy of 89.54%, exceeding the recent strong published works: Guided Interpretable FER (88.95%) by +0.59 percentage points and EAC (88.75%) by +0.79 percentage points. Because these comparisons use the same ResNet-50 backbone as several competitors, the gains can be attributed to our multi-view learning design rather than architectural capacity. Notably, the two-view variant (without any landmark information) already attains 88.75%, matching the best landmark-free baseline (EAC, 88.75\%). When landmarks are introduced only as structured supervision (not as input features), MV-FER rises to 89.54%, outperforming all landmark-based methods, including pACNN, RAN, OADN, FERGCN, and Guided Interpretable FER.
ICACT20260151 Slide.15        [Big Slide] Chrome Click!!
To demonstrate the effectiveness of each component in our framework, we conducted ablation experiments comparing three variants of our approach: (1) single-view baseline using only original images, (2) two-view variant incorporating original and augmented images, and (3) the full MV-FER framework including AU-guided heatmaps. The results reveal substantial progressive improvements: the single-view baseline achieves 85.43% accuracy, while the addition of the augmented view (horizontally flipped images) dramatically improves performance to 88.75% (+3.32 percentage points). The incorporation of AU-guided heatmaps in the full MV-FER framework further enhances accuracy to 89.54% (+0.79 percentage points), representing a cumulative improvement of +4.11 percentage points over the single-view baseline. Ablation studies reveal that the incorporation of flipped views contributes the majority of the performance improvement, while AU-guided supervision provides consistent additional enhancement.
ICACT20260151 Slide.16        [Big Slide] Chrome Click!!
Finally, an examination of the class-wise expression accuracy reveals that the proposed model achieves strong recognition performance for several categories, most notably happiness (95.27%), neutral (90.44%), and sadness (88.08%). In contrast, fear (60.81%) remains the most challenging category, largely due to its similarity with expressions such as surprise and sadness. This outcome aligns with prior findings in the FER literature, which highlight the inherent difficulty of distinguishing subtle expressions like fear and disgust from neighboring affective states.
ICACT20260151 Slide.17        [Big Slide] Chrome Click!!
We introduced MV-FER, a novel multi view facial expression recognition framework that fundamentally differs from existing approaches in its systematic integration of domain-specific knowledge with multi-view learning paradigms. Instead of consuming AU heatmaps as inputs, we recast them as structured supervisory signals that align network attention with expression-discriminative regions, while a dual-branch architecture processes each image and its horizontal flip to enforce flip-semantic consistency. Empirically, MV-FER achieves 89.54% accuracy on RAF-DB, surpassing recent strong published works that employ the same ResNet backbone, thereby demonstrating that performance gains arise from our multi-view modeling approach rather than architectural advantages. Future work includes the development of self-supervised cross-view alignment mechanisms that could eliminate landmark dependencies while preserving the benefits of structured anatomical guidance.
ICACT20260151 Slide.18        [Big Slide] Chrome Click!!
That concludes my presentation. Thank you for your attention. I hope you found it informative, and I would be happy to answer and discuss any questions you may have. Thanks again.