ICACT20210175 Slide.40
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20210175 Slide.39
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20210175 Slide.38
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20210175 Slide.37
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Thank you all for your attention. If you have any question, it's my pleasure to answer.
|
|
ICACT20210175 Slide.36
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Although some of in-text citations have been provided, I highly recommend you could see the full paper for gaining all of the helpful references that we have used.
|
|
ICACT20210175 Slide.35
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To conclude our presentation, I draw here some other related problems in solving FER. -Intensity of Facial Expression -Scene Complexity -Image Resolution -Databases -Analyze different subjects: male/female, young/old, ethnicity -Real-time whole-face analysis of facial expression dynamics
|
|
ICACT20210175 Slide.34
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Till now, 1050 images were collected in the real condition with different illumination, pose and resolution, with and without glasses from 150 volunteers living in Hanoi. In 2017, we used a subset of 354 images, performed our experiment that uses a density-based clustering with side information and active learning as we named MCSSDBS to group facial expression images based own its predicted emotion. The result was published in Intelligent Data Analysis, 2019.
|
|
ICACT20210175 Slide.33
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Vietnam has made a progress in building an e-government. Vietnam ranked 89th of 193 countries and territories on the e-government development index (EGDI). However, Vietnam E-government has not yet delivered on all its potential opportunities
We are now working on the direction of how to evaluate citizens’ feedbacks. Hanoi's authority has installed camera surveillance systems mostly on one-stop shop. It is now able to record photos through camera systems and then extract facial parts to evaluate how the citizens feel about the service.
Since 2017, our own ITI facial data set has been collected with the aim to serve automated facial image analysis and synthesis and for perceptual studies
|
|
ICACT20210175 Slide.32
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
From those comparison results, we can conclude some statements that:
-Most of the cases, deep learning-based methods are capable of learning facial characteristics and improving facial expression recognition.
-Traditional machine learning-based approaches can nicely solve the FER problem on JAFFE and CK+ which are small datasets captured in laboratory conditions. Here, both Haar and HoG are easy to use, fast to train and bought up good results - 98.90% (Haar features) and 99.79% (HoG features) corresponding on JAFFE and CK+.
-Evaluations in FER2013 demonstrate that deep learning-based methods are the best candidate to address the problem in real conditions with different illumination, pose and resolution.
-Data augmentation is applied in most deep learning methods training FER2013. The more data it has, the better performance it can obtain.
-Hybrid and ensemble learning has been attracting more and more researchers since they did help in improving the model accuracy.
|
|
ICACT20210175 Slide.31
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Finally, those noticeable results on FER2013 dataset.
|
|
ICACT20210175 Slide.30
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Secondly, some of the remarkable experimental results on CK+ dataset.
|
|
ICACT20210175 Slide.29
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here we perform some of the aforementioned methods on the three benchmark datasets to illustrate the performance of existing state-of-art FER approaches as shown in the next three slides.
First, on JAFFE dataset.
|
|
ICACT20210175 Slide.28
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Ensemble methods are made using various grouping techniques such as bagging or boosting to train multiple but normally homogeneous models and combine their predictions.
|
|
ICACT20210175 Slide.27
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The current advances in hardware mostly GPUs is accelerating deep learning. We are currently beholding the trend of hybrid or ensemble learning methods. Both methods make use of the information fusion concept but in slightly different ways.
Hybrid models are made through integration of completely different, heterogeneous machine learning approaches, for example traditional machine learning features (or called handcrafted features) and deep learning features.
|
|
ICACT20210175 Slide.26
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Inspired by the great success of DAE based deep network architecture in facial expression recognition [49], its variations have also been developed.
Deep sparse autoencoders have been stacked into a deep structure (DSAE) and proposed to solve the problem [50]. Different from autoencoders, sparse autoencoders have a greater number of hidden nodes than the number of input nodes. A sparsity penalty is applied on the hidden layer in addition to the reconstruction error in order to prevent overfitting.
|
|
ICACT20210175 Slide.25
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
A deep autoencoder is trained in the same way as a single-layer neural network. It is composed of two, symmetrical deep-belief networks. One network for encoding and another for decoding.
The encoder layer compresses the input data while the decoder layer does the reverse to produce the uncompressed version of the data.
The actual use of autoencoders is for determining a compressed version of the input data with the lowest amount of loss in data. Important data is not lost but the overall size of the data is reduced significantly.
|
|
ICACT20210175 Slide.24
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Motivated by the success of DBN, Ping Liu et al., 2014 [48] proposed Boosted Deep Belief Network (BDBN) to perform the three stages in a unified loopy framework.
A BDBN consists of multiple DBNs, each of which is composed of multiple layers and intends to learn a hierarchical feature representation corresponding to image patches extracted at a specific location.
|
|
ICACT20210175 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
DBN proposed by Hinton et al. [43] is a graphical model that uses probabilities and unsupervised learning to extract a deep hierarchical representation of the training data.
Its network is like a stack of Restricted Boltzmann Machines (RBMs) in the middle, and the last layer is a classifier. The nodes in each layer of a DBN are connected to all the nodes in the previous and subsequent layer.
Different from CNN, DBN works globally and each layer will learn the entire input and is regulated in order. It makes the optimal choice at each layer in the sequence, eventually finding a global optimum
|
|
ICACT20210175 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Based on the foundation architecture of the CNN, some variations have also been applied to solve the problem of Facial Expression Recognition, including Deep convolutional neural network (DCNN) – a feedforward neural network that applies back propagation algorithms to adjust the parameters (weights and biases), AlexNet, Inception, Residual Neural Network (Resnet), and 2-Channel CNN where one channel is a standard CNN network and another is trained as a Convolutional Autoencoder, or in which each channel trains different facial parts.
However, having a large number of parameters is the main problem of those networks. This makes them impracticable to be trained on small datasets. Besides, they also require long training/test time and large memory cost.
|
|
ICACT20210175 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here is an illustration of a typical CNN architecture showing how they are connected from input to output.
It consists of three types of heterogeneous layers: convolutional layers, pooling layers, and fully connected layers.
The first part consists of Convolutional and max-pooling layers which act as the feature extractor. The second part consists of the fully connected layer which performs non-linear transformations of the extracted features and acts as the classifier.
|
|
ICACT20210175 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Now we explore the typical framework of a deep learning approach.
One of the key challenges in computer vision is to deal with the variance of data in the real world. It can be said that Deep learning networks are designed to cope with this problem.
As being mentioned, Unlike a traditional machine learning-based method, a deep learning method makes all these hand-crafted features obsolete, as it defines the features space itself.
In our work, we introduce the principles of convolutional neural networks, deep belief networks, and deep auto encoders.
|
|
ICACT20210175 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Finally, these handcrafted features are fed as an input to the classification. Many classification methods have been employed in the FER systems, such as Linear Discriminant Analysis (LDA), Support vector machine (SVM), Random forest, AdaBoost, Decision Tree, Naïve Bayes, and Extreme Learning Machine (ELM) that is a type of single hidden-layer feed-forward neural networks (SLFN).
|
|
ICACT20210175 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Geometric-based representation considers the shape information (in-transient features, e.g., facial points or locations of eyebrows, eyes, the mouth, the nose) explicitly and ignores the texture. Most geometric feature-based approaches use the active appearance model (AAM) or its variations, to track a dense set of facial points.
Appearance-based representation, on the other hand, uses the intensity values or the pixels that refer to the textural changes, such as wrinkles and furrows. The techniques adopted as an Appearance-based include Local binary pattern (LBP), scale invariant feature transform (SIFT), histograms of oriented gradients (HOG), Gabor wavelet representation, and principal component analysis (PCA).
|
|
ICACT20210175 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The process of feature extraction mostly requires domain level expertise, that is reason why we call these obtained features by this way are “handcrafted” features.
The facial features can be extracted from the entire face or particular facial regions, and categorized into two types, i.e. geometric-based (or shape-based) and appearance-based.
|
|
ICACT20210175 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The next step is to extract the information about the encountered facial expression in an automatic way. Feature extraction is finding and depicting of positive features of concern within an image.
|
|
ICACT20210175 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The aim of Normalization steps is to obtain images in a normalized intensity, uniform size, and shape. Thereby, the performance of the FER system can be improved.
This phase includes different tasks such as orientation normalization, image scaling, contrast adjustment, and additional enhancement to improve the expression frames.
|
|
ICACT20210175 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
After the presence of a face has been detected in the observed scene, normalization can be carried out.
|
|
ICACT20210175 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
One of the problems that any algorithm solving Face Acquisitions should be considered is Head Pose Estimation.
|
|
ICACT20210175 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Face Acquisitions a process of localizing and extracting the face region from the background. The Viola-Jones algorithm, 2001, was designed for frontal faces. is a widely used mechanism for facial detection.
The Viola and Jones algorithm uses Haar-like features to detect faces.
Given an image, the algorithm looks at many smaller sub-regions and tries to find a face by looking for specific features in each sub-region. It needs to check many different positions and scales because an image can contain many faces of various sizes.
|
|
ICACT20210175 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In a traditional machine learning-based approach, facial expression recognition can be divided into four major steps: (1) face acquisition stage to automatically find the face region for the input images; (2) normalization of intensity, uniform size and shape; (3) facial data extraction and representation-extracting and representing the information about the encountered facial expression in an automatic way; and (4) facial expression recognition step that classifies the features extracted in the appropriate expressions.
|
|
ICACT20210175 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In the next few slides, we will explain the frameworks of the three FER solutions, but now, I'd like to shortly highlight the main difference between a traditional machine learning approach and a deep learning approach.
Unlike traditional machine learning and computer vision approaches where features are defined by hand, a deep learning method learns to extract the features itself directly from the training database using iterative algorithms like gradient descent
|
|
ICACT20210175 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In our evaluation comparisons, we use the performance measurement, accuracy, for each referred experiment. Accuracy can be defined as the sum of correct decisions divided by the total number of classifications.
However, it is worth noting that accuracy alone does not tell the full story when you are working with a class-imbalanced data set.
|
|
ICACT20210175 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here we show some publicly available datasets that are widely used in our reviewed papers. Approach to collect FER-related data was mostly through the images captured in the laboratory, such as JAFFE and CK+, in which volunteers make corresponding expressions under particular instructions. However, since 2013, emotion recognition competitions have collected large-scale and unconstrained datasets, for example, FER2013 queried automatically by the Google image search API. This implicitly promotes the transition of FER from lab-controlled to real-world scenarios.
|
|
ICACT20210175 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In detail, our main contributions include:
Firstly, we comprehensively highlight the differences and similarities of the three FER approaches:
-Traditional machine learning – based approaches
-Deep learning – based approaches
-Hybrid and Ensemble learning
It is worth noting that several surveys on automatic expression analysis have been published in recent years, but those study focus either on traditional machine learning-based methods, or recently deep learning-based methods. So, different from these surveys, in this work, we conduct more specific and detailed research from 2014 to 2020 on widely used datasets to clarify all the three FER types.
Last but not least, we discuss our current application scenarios in Vietnam.
|
|
ICACT20210175 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
When studying the problem of FER, you either want to recognize a small set of facial expression that is the main objective of message-based approach or recognize facial actions, for example, eyebrow raise, frown, so on and so forth in case of a sign-based approach.
In this work, we only focus on the former approach.
|
|
ICACT20210175 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In 1971, Paul Ekman et al.’s facial expression work gave the correlation between a person’s emotional state & psychological state, which described the FACS.
Next in 1992, he defined 7 face expressions: Happy, Neutral, Angry, Disgust, Fear, Sad, and Surprise. They are named as 7 universal emotions and are used by most researchers.
|
|
ICACT20210175 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Back to the year 1872, Darwin published “The Expression of the Emotions in Man and Animals”, in which he argued that all humans, and even other animals, show emotion through remarkably similar behaviours. Darwin treated the emotions as separate discrete entities, or modules, such as anger, fear, disgust, and so on.
|
|
ICACT20210175 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We cannot read the mind, but our face reflects somehow our feeling, our expression. Expression critically influences all aspects of our lives, from how we live, work, and to the decisions that we make.
Facial expression recognition (FER) provides the machine a way of sensing emotions that can be considered one of the mostly used artificial intelligence and pattern analysis applications.
|
|
ICACT20210175 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Our presentation today includes 8 parts. I first introduce the problem of Facial Expression Recognition in part 1. Second, I highlight our main contributions. After that, some of the widely used datasets in Facial Expression Recognition will be shown. There are 3 types of FER systems, one based on traditional machine learning approaches, another based on deep learning approaches, and the last one based on hybrid & ensemble learning. We explain differences and similarities of the three approaches above then provide experimental comparisons of researches from 2014 to 2020.
Finally, we described its current applications in our country - Vietnam.
|
|
ICACT20210175 Slide.01
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Hi everybody, my name is Quan, from the Information Technology Institute, Vietnam National University. For ICACT 2021, I am here to present our work - a survey of Facial Expression Recognition.
|
![[Big slide for presentation]](/upload/2021/0175/Presentation/40/Slide40.jpg){kind=link}
![[Big slide for presentation]](/upload/2021/0175/Presentation/39/Slide40.jpg){kind=link}
![[Big slide for presentation]](/upload/2021/0175/Presentation/38/Slide40.jpg){kind=link}