 ICACT20250190 Slide.01
[Big Slide]
|
Chrome  Click!! |
 |
Hello everybody, my name is Omur Fatmanur Erzurumluoglu. Our presentation today is about a real-time emotion recognition system using fog computing. In this study, we focused on two parts: emotion recognition and fog computing for real-time applications. |
| ICACT20250190 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
Emotion recognition is a prominent research subject in the fields of affective computing and human-computer interaction. To address the emotion recognition problem, input signals can be categorized into 4 types. The physiological signals are considered more reliable than facial, gesture, and speech signals because they are less susceptible to subjective control by individuals. Additionally, using speech signals is often ineffective in noisy environments, and language barriers further limit its practicality. Also, camera-based methods raise significant security concerns, particularly in systems that involve continuous indoor monitoring, where there is a high risk of privacy violations. Furthermore, video data demands more storage and processing resources compared to physiological signals. Therefore, physiological data not only provides a more objective and consistent basis for emotion recognition but is also more secure and efficient to handle.
|
| ICACT20250190 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
There are several uses for emotion recognition to monitor multi-user emotional states simultaneously in real-time.
For smart homes, emotion recognition may be integrated to provide automation for family members by modifying lighting, music, and other environmental elements according to the overall emotional states of the households.
In professional environments, emotion recognition systems can optimize productivity, collaboration, and employee well-being. Real-time feedback on employees’ emotional states can help managers allocate tasks better, ensuring individuals work on assignments aligned with their emotional readiness and energy levels.
In educational institutions, emotion recognition can support personalized learning and improve student engagement. Teachers can gain a better understanding of class-wide or individual emotional responses to lessons to tailor their teaching strategies. Emotion detection can aid in identifying students struggling with emotional challenges, so counselors or teachers can provide appropriate support.
In the entertainment industry, real-time emotion recognition can revolutionize user experiences and content delivery. Emotion detection can enable games to adapt their difficulty levels or challenges based on the player’s emotional responses to create an immersive experience. Performers, streamers, or event organizers can use emotion recognition to gauge audience reactions and adjust their performances accordingly.
In the healthcare sector, emotion recognition systems can enhance patient care, monitoring, and therapy outcomes. These systems can continuously track the emotional states of patients to provide real-time insights for diagnosing depression, anxiety, or PTSD. Emotion detection can help medical staff obtain patients’ pain, discomfort, or disgust during treatments to improve patient care and satisfaction. For therapy sessions, emotion recognition can help therapists better understand their patients' progress and emotional responses, leading to more effective treatment plans. |
| ICACT20250190 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
Now, I am moving on to the second part of the study, which is fog computing. Fog computing is a decentralized computing infrastructure where data and processing are distributed between the cloud and edge devices closer to the edge layer to improve efficiency and reduce latency. Fog architecture provides low latency, low bandwidth usage, energy-efficient solutions for real-time problems, and supports user mobility. |
| ICACT20250190 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
The purpose of the study is to implement real-time emotion recognition using IOT. The user or database will continuously send physiological signals to the computing system, which will then return the predicted emotion to the receiver user. |
| ICACT20250190 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
For this system, low latency, computational power, energy resources, service management, and scalability are focus areas. |
| ICACT20250190 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
In this study, we aim to create real-time emotion recognition from a physiological signals system by using fog computing.
The challenge for success is to find the balance between high emotion recognition accuracy and high transmission performance. |
| ICACT20250190 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
There are studies focused on real-time emotion recognition in IoT applications. These studies used facial and physiological signals as data sources. And, cloud, edge, and fog computing were used. Among them, only Ejaz and Shah’s study used fog computing for real-time emotion recognition but using facial signals, so the proposed system emerges as an open research area. |
| ICACT20250190 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
This is the list of contributions of the fog computing-based real-time emotion recognition system. This study is the first study that performs real-time emotion recognition from physiological signals using fog computing architecture. It was evaluated with 11 metrics such as accuracy, latency, precision, and so on.
|
| ICACT20250190 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
The pipeline of the system is shown in the figure. In the network, there is a client and computation unit. Red arrows show the request and response flow in the network and black arrows show the data flow in the computation unit.
The client user or database will send EEG and ECG signals to the computation unit. When the computation unit receives the signals, they will pass through preprocessing, and feature extraction operations. After that extracted features will be fed into a pre-trained model to predict the user’s emotion. Finally, the predicted emotion is sent back to the client. For different user cases, the receiver can be adjusted to deliver the predicted emotion to another user or system based on application requirements.
|
| ICACT20250190 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
The DREAMER dataset was used. It includes 14 channels of EEG and 2 channels of ECG signal data which were gathered while the 22 participants were subject to audio and video stimuli. In total 16 simultaneous channel signals exist. The dataset used 9 emotions, but for this study, we focused on 4 emotions which are amusement, calmness, disgust, and surprise. This selection ensures that the study addresses a wide spectrum of emotions, from high-arousal (surprise) to low-arousal (calmness), and from positive (amusement) to negative (disgust) as shown in Figure.
To the volunteers, 8 film clips were shown. These videos in 4 selected emotion categories were divided into two sets containing an equal number of videos showing four emotions. One set was used for model training, and the other set was used for model testing. This approach ensures that the same signal is not segmented and is used for both training and testing the model. |
| ICACT20250190 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
ECG and EEG signals were segmented into 3-second length moving windows, and the window was shifted by 1 second as shown in the figure. |
| ICACT20250190 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
The ECG data was down-sampled from 256 Hz to 128 Hz, whereas the frequency of EEG signals is already 128 Hz. In order to eliminate the information that the windows belong to the same signal, Z-score normalization was applied to each window.
The figure displays raw and normalized ECG and EEG signals of the first 3 seconds of the first participant's amusement trial.
|
| ICACT20250190 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
After that, 36 statistical features were extracted as shown in Table. Features obtained across simultaneous windows from ECG and EEG signals were concatenated using feature-level fusion. Since 36 features were derived from 16 simultaneous windows, a feature vector with 576 features was obtained. |
| ICACT20250190 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
In the figure, the overall data flow is shown. The first 8 film clips were split into training and test sets. After that, ECG and EEG signals were segmented into 3-second signals. After, the preprocessing operation was applied. On this data, two separate datasets were obtained from time-based and frequency-based feature extraction operations. |
| ICACT20250190 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
Random forest ML model were employed to predict emotions from ECG and EEG signals. The model was trained once offline. The same model was used in real-time. |
| ICACT20250190 Slide.17
[Big Slide]
|
Chrome Click!! |
 |
In fog computing, the role of the cloud is to provide centralized management in the proposed system. It was not chosen as the primary computation unit due to high bandwidth requirements, security concerns, and potential latency issues while it can perform complex computations.
The role of the broker is to manage communication between devices in the network. The broker coordinates the distribution of tasks in the fog nodes to optimize performance and reduce latency while ensuring efficient resource allocation.
The role of the worker is to manage data processing tasks at the edge of the network. The worker handles ECG and EEG signal transmission and performs emotion recognition. That eliminates the need for data to be transmitted to the cloud, which minimizes latency and improves response times.
The role of the client is to generate and send data to the fog nodes for processing. Clients are typically end-user devices or sensors that collect data. For this study, the client device is responsible for sending ECG and EEG signals from the DREAMER dataset at a frequency of 32 Hz. |
| ICACT20250190 Slide.18
[Big Slide]
|
Chrome Click!! |
 |
Then, extracted features are applied to the model to estimate the user’s present emotion in real-time. A worker is the main computation unit. Cloud, broker, and worker nodes within the architecture are able to perform computational tasks as a computation unit, so computation tasks can be carried out on cloud and broker nodes when needed instead of worker nodes. |
| ICACT20250190 Slide.19
[Big Slide]
|
Chrome Click!! |
 |
The computing system of fog architecture is responsible for executing the tasks requested by users. As shown in the figure, initially, the computing system receives raw physiological signals from the user or database, then handles the processing of signals by applying preprocessing, and feature extraction operations. |
| ICACT20250190 Slide.20
[Big Slide]
|
Chrome Click!! |
 |
As I mentioned before, 8 film clips for 4 emotions were split into training and test sets. Sliding window, preprocessing, and feature extraction operations were applied. Instance numbers of data in the training and test sets corresponding to the emotions of participants are shown in the table.
|
| ICACT20250190 Slide.21
[Big Slide]
|
Chrome Click!! |
 |
The metrics used in the evaluation of the proposed system's recognition performance are accuracy, precision, recall, and F1 score.
|
| ICACT20250190 Slide.22
[Big Slide]
|
Chrome Click!! |
 |
When the random forest model was trained and tested with 576 features. We achieved an accuracy of 80%, a recall of 80%, a precision of 84%, and an F1 score of 80%.
|
| ICACT20250190 Slide.23
[Big Slide]
|
Chrome Click!! |
 |
The table shows the comparison between existing studies that perform discrete emotion analysis using ML models and our study. Our study outperforms existing studies that utilize ML models for 4 discrete emotion recognition from ECG and EEG signals in terms of accuracy. |
| ICACT20250190 Slide.24
[Big Slide]
|
Chrome Click!! |
 |
As a cloud device, Digital Ocean was used.
For a broker node, Raspberry Pi 4 model b was used.
As a worker node, Raspberry Pi 4 model b was used.
For the Wi-Fi access point, an Infinity 401 modem was used.
As a client device, an Asus Rog Strix G15 computer was used. |
| ICACT20250190 Slide.25
[Big Slide]
|
Chrome Click!! |
 |
The performance of the architecture was evaluated through a system transmitting recorded physiological signals in the DREAMER dataset from the client. The mock user sent 32 requests per second, with 31 of them streaming ECG and EEG data to the computation unit and adding this data to a buffer. On the last request, the computation unit processed the accumulated data in the buffer. |
| ICACT20250190 Slide.26
[Big Slide]
|
Chrome Click!! |
 |
The transmission performance is measured using the metrics below.
Latency is the amount of time that elapses between a client sending data and the data arriving in the computation unit.
Queuing delay is the amount of time that a request remains in the queue before execution
Execution time is the total duration for which the emotion recognition pipeline steps are sequentially applied to incoming requests
Response time is the total amount of time between sending the emotion recognition requests and receiving the response.
Jitter represents the variation in total response time.
Resource usage represents CPU and memory usage data for the respective computation units. Elevated resource utilization may result in heightened system latency and diminished service capacity for concurrent users.
Network bandwidth usage represents the total of incoming and outgoing data. The total network bandwidth usage for different computation units was considered. |
| ICACT20250190 Slide.27
[Big Slide]
|
Chrome Click!! |
 |
To examine the impact of the number of features calculated on the system's response time and to determine the final system configuration based on accuracy and total response time, the system was tested with varying numbers of features.
The Mean Decrease in Impurity (MDI) feature selection method was used in the training dataset. After applying MDI, the contribution rates of the features were obtained and sorted in descending order.
After sorting the features in order of importance, the number of features used was selected from 4 to 576 in increments of 4.
|
| ICACT20250190 Slide.28
[Big Slide]
|
Chrome Click!! |
 |
The plot illustrates the relationship between response time and accuracy as the number of features processed in the worker which is the primary device for computational tasks. As the number of features increases, response time exhibits an upward trend because of the computing load of features. Accuracy, depicted on the secondary y-axis, peaks at 84.21% with 104 features. The stability that occurs around 82% suggests that the increase in the number of features does not significantly impact accuracy. However, a slight decline is observed after 200 features indicating potential overfitting or redundancy in the feature set.
In this study, the main goal was not only to obtain a high-performance model but also to design a real-time emotion recognition system by taking into account all evaluation criteria. Therefore, instead of the model with 84% accuracy achieved with 104 features and an emotion response time of 78.16 ms, a model with 24 features achieving 80% accuracy and a response time of 45.01 ms, half the duration of the best model, was chosen to get optimum results for real-time emotion recognition system. Although there is a 5% loss in accuracy, the system operates approximately with half the response time. The evaluation criteria results are based on tests conducted on the system with 24 features.
|
| ICACT20250190 Slide.29
[Big Slide]
|
Chrome Click!! |
 |
Once the optimal number of features was determined, the system was first evaluated using a single-worker network for a single-user scenario. This system was evaluated on latency, queuing delay, execution time, response time, jitter, resource usage, and network bandwidth usage.
|
| ICACT20250190 Slide.30
[Big Slide]
|
Chrome Click!! |
 |
The figure gives the mean latency for different computing devices. The data shows that the cloud node had the highest mean latency at 32.2 ms due to its multi-hop connection and longer data travel distances. In contrast, the edge device demonstrates the lowest mean latency at 1.6 ms because the data source also performs as a computational unit. The worker and broker nodes show mean latencies of 4.7 and 4.8 ms respectively. These results show the potential of edge device and fog computing architecture to significantly reduce latency, making them more suitable for real-time emotion recognition applications that require swift and responsive data processing.
|
| ICACT20250190 Slide.31
[Big Slide]
|
Chrome Click!! |
 |
The mean queuing delay results for different computing devices are presented in the figure. The data shows that the edge device has a significantly higher mean queuing delay of 13.7 ms, compared to the other devices, which all exhibit mean queuing delays below 1 ms. This difference shows that the client device demonstrates substantial queuing delays due to its inherent network characteristics or load-handling capacity. In contrast, worker, broker, and cloud nodes demonstrate comparable and minimal queuing delays, indicating more efficient queuing processes. Worker and broker nodes show slightly better performance compared to the cloud node.
|
| ICACT20250190 Slide.32
[Big Slide]
|
Chrome Click!! |
 |
The mean execution time results for various computing architectures are displayed in the figure. As expected, the worker and broker nodes exhibit similar and the highest mean execution times of 34.7 ms and 35.3 ms, respectively because they have the same processor power. On the other hand, the cloud and edge nodes show lower mean execution times, with the cloud node at 8.9 ms and the edge node at 5.7 ms as they have more powerful processors.
|
| ICACT20250190 Slide.33
[Big Slide]
|
Chrome Click!! |
 |
The figure provides the total response time in different devices. Worker device has a lower response time than broker and cloud devices, indicating that workers perform their duty both adequately and effectively in the fog architecture. The cloud component has the highest response time at 73.3 ms, followed by the worker and broker devices at 43.3 ms and 43.9 ms, respectively. The edge device exhibits the lowest mean response time at 22.5 ms. Although the edge device has the lowest mean response time, it is not suitable for our real-time emotion recognition system which demands extensive and simultaneous multi-user interactions and integrations. Also, because the system needs continuous battery-powered operations, it can result in increased battery drain in edge devices. Furthermore, our system requires centralized data analysis to ensure consistency and compliance with regulatory standards, which is more effective on fog computing systems rather than on edge devices.
|
| ICACT20250190 Slide.34
[Big Slide]
|
Chrome Click!! |
 |
In the figure, the results indicate that the cloud node demonstrates the highest jitter at 36.6 ms, suggesting a significant variability in latency. The worker and broker nodes show comparable jitter levels of 17.4 ms and 16.8 ms, respectively, indicating more stable and consistent network performance compared to the cloud node. |
| ICACT20250190 Slide.35
[Big Slide]
|
Chrome Click!! |
 |
The analysis of mean bandwidth usage across different computing layers is revealed in the figure. The worker, broker, and cloud layers exhibit mean bandwidths of 373.9 kbps, 392.9 kbps, and 384.2 kbps respectively, indicating relatively high and consistent network usage. In contrast, the edge layer demonstrates a dramatically lower mean bandwidth of 8.5 kbps. |
| ICACT20250190 Slide.36
[Big Slide]
|
Chrome Click!! |
 |
The figure shows the CPU and memory usage results of computing service devices. The worker and broker nodes exhibit similar and balanced resource usage, with CPU usage at 6.7% and memory usage at 7.2% because they have the same hardware system. The CPU usage of cloud nodes is similar to broker and worker devices with 6.2% although it is relatively more powerful and is optimized for computation. And, cloud node shows a slightly higher memory usage at 12.2%. Moreover, the edge device stands out with minimal CPU usage at just 0.4%, but significantly higher memory usage at 41.9%.
|
| ICACT20250190 Slide.37
[Big Slide]
|
Chrome Click!! |
 |
The experimental results showed that a worker node can efficiently handle computational tasks with low latency, queueing delay, and most importantly total response time. Every second, the system uses the 3-second data in the buffer to recognize the current emotion of the user and then sends the predicted emotion to the user. This procedure takes approximately 43 ms, including a latency of 4.7 ms and an execution time of 34.7 ms. In that case, a single worker device can serve approximately 20 users. Also, the system can be scaled to serve more users by adding additional worker devices. |
| ICACT20250190 Slide.38
[Big Slide]
|
Chrome Click!! |
 |
We developed a real-time emotion recognition system based on physiological signals utilizing fog computing architecture. The system is suited for environments where the emotional data of users can be processed in places such as educational institutions, workplaces, smart homes, and healthcare facilities. Random forest machine learning model was used on the DREAMER dataset. 84.21% accuracy was achieved with 104 features but considering resource utilization and time performance, the system has been configured to calculate 24 features in 43 ms with 79.70% accuracy.
|
| ICACT20250190 Slide.39
[Big Slide]
|
Chrome Click!! |
 |
There are several unexplored research directions related to this study, optimizing computational resources, expanding the dataset and emotion classes, conducting real-world trials to assess the practical usability of the proposed system, and implementing deep learning models within the fog computing framework with larger datasets. Additionally, the integration of textile-based sensors could significantly enhance the system's practicality and comfort. Because these sensors are lightweight, breathable, and adaptable to daily routines, using these sensors may encourage users to consistently use them. |
| ICACT20250190 Slide.40
[Big Slide]
|
Chrome Click!! |
 |
Thank you for your attention. Is there any question or comment? |
