 ICACT20230035 Slide.00
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Hello everybody, my name is Qiang Wu from Teikyo University. Today I am pleased to share our team's research about a modified self-cure network for facial expression recognition under face masks that we called SCN-SAM. |
| ICACT20230035 Slide.01
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This slide shows the flow of the presentation. First, we introduce the research background and related studies. Next, we explain how to apply a mask to an unmasked face image, which we call the mask-wearing process. Then, we describe our proposed method and experimental results. Finally, a summary of this study is presented. |
| ICACT20230035 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Facial expression recognition is one of the research themes of computer vision and is widely used in society. For example, it can detect dangerous driving by measuring changes in the driver's facial expressions. However, according to the spread of COVID-19, it is difficult to read facial expressions because wearing masks has become routine. So, in this research, we aim to realize facial expression recognition of masked face images and improve recognition accuracy. |
| ICACT20230035 Slide.03
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The next section describes the SCN model of related studies. Currently, there are various research methods for facial expression recognition. Considering cost and convenience, we used deep learning for this research. This time, I conducted research based on this SCN model. Because as shown in Table 1, the SCN model has higher recognition performance than conventional facial expression recognition methods for unmasked face images. And in general facial recognition problems, blurred images, or incorrect annotations of images in large-scale image datasets can make the model’s training difficult. SCN can effectively suppress the over-fitting of the network to images with uncertain labeling in large-scale facial expression datasets. SCN model may be helpful in this research because masked face images may lose some facial features and thus increase their noisy labels. |
| ICACT20230035 Slide.04
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Since there are not many existing mask-wearing facial expression image datasets, it is necessary to mask the unmasked facial expression images in advance. So we use the collected datasets to create mask-wearing images. The flow of the mask-wearing process is shown in Figure 2. First, the unmasked face images from the dataset are input into the Face Alignment Network. Face Alignment Network is a network based on the Hourglass network. It can also detect 68 landmarks from a face image, as shown in Figure 2. In this experiment, we will use the landmarks marked in red which are 14 landmarks out of 68 landmarks, to mask the face images. |
| ICACT20230035 Slide.05
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Regarding datasets, there are several datasets on facial expression recognition. We used the two most cited datasets in this experiment, FERPlus, and RAF-DB. According to table 2, the image size in FERPlus is 48*48, and the image size in RAF-DB is 100*100. And the contents of both datasets are the same, seven types of facial expression images. |
| ICACT20230035 Slide.06
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The results of the mask-wearing process are shown in Table 3. The red numbers indicate the percentage of images each class cannot process. Overall, the mask-wearing process failed for about 3% of the images in FERPlus and about 1% in RAF-DB. The number of images that fail to be processed is small and should not affect the entire dataset. |
| ICACT20230035 Slide.07
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Here are some samples of an image that we cannot process this time. The FAN model is difficult to detect landmarks because most images are in profile, and the faces are not visible through glasses or hands. |
| ICACT20230035 Slide.08
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Next, we conducted a facial expression recognition experiment to see if SCN can recognize facial expressions using masked images. Table 4 shows the results of the experiments with the SCN model as a recognizer using two datasets. For FERPlus, 87.5% of the face images were recognized without masks, and 75.2% were recognized after masking. In the case of RAF-DB, 87.2% of the face images were recognized without masks, and 78.9% were recognized after masking. So, conventional methods can achieve facial expression recognition even after mask-wearing. Moreover, the mask-wearing process lost accuracy of around ten percentage points, even in images where the lower half of the face is not visible. |
| ICACT20230035 Slide.09
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Figure 4 shows our proposed SCN-SAM model. The following is a brief description of SCN-SAM. |
| ICACT20230035 Slide.10
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
First, a mask is applied to the unmasked face image using the mask-wearing process, as shown in Figure 5. We then use the backbone CNN network to extract the deep features of the masked facial images. The CNN backbone used in this study is ResNet-18. |
| ICACT20230035 Slide.11
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Next, each feature is entered into the attention importance weighting module. This module consists of a fully-connected layer and a sigmoid activation function. It can calculate the importance of samples in model training. The higher the uncertainty of the sample, the lower the importance weight will be. So this module can use the sigmoid function to calculate the importance weight of each extracted deep feature. |
| ICACT20230035 Slide.12
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We then average the obtained batch of importance weights in the self-adjustment module. We called the weighting mean θ. θ is used as a discriminant threshold, and when the weight is more significant than the threshold, it is classified as a high importance weight, λ_HW. Otherwise, the weight is classified as a low importance weight, λ_LW. So the self-adjustment module can classify high and low weights according to the importance weights inputted to the module. |
| ICACT20230035 Slide.13
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In the importance weight regularization module, each batch of samples is automatically divided into two groups, the high-importance group and the low-importance group. We count the data volume of incoming high-weight and low-weight samples separately. We can compute the model's training loss ℒ_𝐼𝑊𝑅 by computing the resulting mean and the trainable parameter μ. |
| ICACT20230035 Slide.14
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
High importance weights can make the model pay more attention to the training of these samples. However, to effectively utilize the samples of the low-importance group, the relabeling module compares the maximum predicted probability to the probability of a given label if the maximum predicted probability is higher than the given label with a threshold value. The sample is updated to a new label. So, it can fix the labels of images with low importance weights. |
| ICACT20230035 Slide.15
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The most significant difference between the SCN-SAM model and the SCN model is whether it has a SAM module. Because the importance weight is determined according to the image’s uncertainty, the weight of each image is not the same. SAM can classify high and low weights according to the importance weights inputted to the module.So it can mine high-importance-weight samples in training samples and change the proportion of high- and low-importance-weight samples so that the model can pay more attention to learning the sample features of high importance weight. The increase in blurred images and the influence of ambiguous annotations due to the presence of masks causes the number δ of highly-importance samples to become very unstable during training instead remaining at a constant value. In model training, SCN-SAM mines high-importance samples and classifies high and low-importance samples so that the δ value is no longer constant, thus solving this problem well. |
| ICACT20230035 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This is the experimental result of the facial expression recognition of the proposed model using masked face images. In order to show that our proposed model SCN-SAM can effectively suppress the uncertainty of facial expressions caused by masks, we choose ResNet-18 as the baseline and the backbone of SCN-SAM and compare it to SCN. As shown in Figure 11, the SCN-SAM outperforms the baseline and SCN consistently. In the evaluation performed on the masked RAF-DB dataset, SCN does not suppress the uncertainty caused by the presence of masks well, and the performance is almost the same as the baseline model. However, our improved model, SCN-SAM, can suppress the influence of uncertainty caused by masks. The accuracy of the masked RAF-DB is 0.53% higher than that of SCN. The experimental results on the masked FERPlus are even more apparent. In the masked FERPlus, the SCN shows a negative performance, slightly worse than the accuracy of the baseline model. With the help of our proposed SAM module, the accuracy of SCN-SAM is improved by 2.28%. |
| ICACT20230035 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Some of the recognition results are visualized to make the experimental results easier to understand. As shown in Figure 12, the labels at the top are the ground-truth labels of the image, while the labels at the bottom are the predicted labels. SCN-SAM was able to correctly recognize images that SCN recognized incorrectly. |
| ICACT20230035 Slide.18
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In order to verify whether the model also works in the annotation ambiguity of unmasked facial expression datasets, we conduct the following experiments. The annotation ambiguity in the dataset is mainly manifested in the uncertain annotation of the annotator caused by the annotation error and the uncertainty of the image. We quantitatively evaluate the improvement of SCN-SAM against the mislabeled annotations on RAF-DB. In detail, we randomly select a part of the training samples and modify the labels to other categories. As summarized in Table 5, our proposed SCN-SAM achieves the best results after adding 10%, 20%, or 30% synthetic noise samples to the training samples. |
| ICACT20230035 Slide.19
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In order to further verify the effectiveness of the SAM module, we also performed ablation experiments on the FERPlus dataset. The experimental results are summarized in Table 6. Under different ratios of noise data, the experimental group with the SAM module can obtain the best performance. Therefore, focusing on the high importance samples during the training phase is essential. |
| ICACT20230035 Slide.20
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In this research work, to effectively suppress the problem of annotation ambiguity of images of people wearing masks, we propose SCN-SAM and design a self-adjustment module. This module can mine high-importance-weight samples in training samples and change the proportion of high- and low-importance-weight samples so that the model can pay more attention to learning the sample features of high importance weight. Experiments on masked datasets and unmasked synthetic ambiguous datasets show that our SCN-SAM can suppress the uncertainty in recognizing masked facial expressions and suppress annotation ambiguity in the facial expression datasets without masks. In the future, we will use SCN-SAM to investigate further different computer vision fields to improve the model’s generality. In addition to facial expression recognition, there are many other fields in computer vision where ambiguous annotation is a problem, such as object detection or face recognition. We believe that the proposed method will also be effective in these fields. After completing the improvement of the model’s generality, we will extend the model to real life to develop the application. |
| ICACT20230035 Slide.21
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Thank you for your attention. |
