 ICACT20230071 Slide.00
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Smart Mirror Activated by User’s Face Recognition with Simulation of Artificial Intelligence Classifier Jixin Liu* , Xing Wen* , Juhui He* , Hongmei Li* , Peixin Chen* , Zhongyong Wang* , Jingquan Chen* , Bing-Yuh Lu* , Weijun Lai* , Cancheng Huang* *Faculty of Automation, Guangdong University of Petrochemical Technology, Maoming City, Guangdong, China franklinlu888@outlook.com |
| ICACT20230071 Slide.01
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Introduction
Face recognition
Smart mirror
OpenCV
Our proposal
|
| ICACT20230071 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Introduction
As a popular identity authentication technology in recent years, face recognition captures the typical features of the face and compares it with the typical feature data in the database, calculates the similarity of feature points, and finally draws a judgment.
Compared with the other biometric technologies, face recognition has the advantages of simple operation and no touch to avoid infections.
|
| ICACT20230071 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
Introduction
Yu et al. [1] presented a smart mirror, LUX, which employed deep learning model for Korean text sentiment analysis to detect user’s mental comfort and response interactively.
Ghazal et al. [2] proposed that a smart mirror which displayed a variety of information services and regular updates social media and news outlets was controlled by smart phone in ambient Internet of Things (IoT) environments.
Yusri et al. [3] indicated that Smart Mirror which allows users to access information such as time and date, weather, warning, traffic, and location map and also control the lights in the house is a concept of smart home-based IoT.
|
| ICACT20230071 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
Introduction
For technological challenge, many researchers developed the smart mirrors using Raspberry Pi for the central control [5-9].
[5] Sun, Y., Geng, L., & Dan, K. X., “Design of smart mirror based on Raspberry Pi.” In Proc. 2018 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), pp. 77-80, 2015.
[6] Jin, K., Deng, X., Huang, Z., & Chen, S., “Design of the smart mirror based on raspberry PI.” In Proc. 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pp. 1919-1923, 2018.
[7] Ok, F., Can, M., Üçgün, H., & Yüzgeç, U., “Smart mirror applications with raspberry PI.” In Proc. 2017 International Conference on Computer Science and Engineering (UBMK), pp. 94-98. 2017.
[8] Akshaya, R., Raj, N. N., & Gowri, S., “Smart mirror-digital magazine for university implemented using raspberry PI.” In Proc. 2018 International Conference on Emerging Trends and Innovations In Engineering And Technological Research (ICETIETR), pp. 1-4. 2018.
[9] Kumbhar, P. Y., Mulla, A., Kanagi, P., & Shah, R., “Smart mirror using Raspberry PI.” International Journal for Research in Emerging Science and Technology, vol. 5, no. 4, pp. 2349-2610, 2018.
|
| ICACT20230071 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
Introduction
In this study, the central controller is Loongson 1C which is a new microcontroller designed and fabricated in China. The developing planform is Loongson 1C Zhilong development board. The information of daily life such as weather, temperature, business schedule or touring information can show on the smart mirror.
Furthermore, some of the imaging processing computation were implemented in this study.
|
| ICACT20230071 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
Methods
System structure
Microcontroller
ESP8266
Face recognition
ESP 32
Adaptive Neuro Fuzzy Inference System
(ANFIS)
|
| ICACT20230071 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
Methods
The technical requirements of this smart mirror were as follows:
1. Response messages depending on individual users.
2. Face only without any other operations.
3. Avoid mistaking the users’ faces.
4. Convenient APP control.
For achieving the technical requirements, the system was designed by the following parts.
|
| ICACT20230071 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
A. System structure
The system block diagram is presented in Fig. 1 which
displayed 5 parts included APP of smart phone,
microcontrollers, cloud, camera for face recognition, and
display and mirror. The APP serving as a manager of the user
was connected with the central controller by Bluetooth
communication. The central controller was connected with
ESP8266 for WIFI link to communicate the data with the
cloud. The camera was a part of the OpenMV module which
functioned as the face recognition processor to acquisition the
features of the user faces for identification and connected with
central controller for data processing of features. After all, the
data and images projected onto the screen and mirror. |
| ICACT20230071 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
Methods - Microcontroller
This smart mirror system employed the Loongson 1C as the central controller which contains a floating-point processing unit to enhance the system's floating-point data processing capability. 1C's memory interface supports multiple types of memory, allowing flexible system design. Support 8-bit SLC NAND or MLC NAND FLASH, provide high-capacity storage expansion interface.
|
| ICACT20230071 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
Methods – ESP 8266
The ESP8266 WIFI module can be used independently or as a slave to run on other host MCUs. ESP8266WIFI module integrates antenna switch, RF circuit, power amplifier, low noise amplifier, filter and power management module for minimal external circuitry and minimizes the size of the PCB. At the same time, the ESP can quickly switch between sleep/wake modes to achieve energy saving, and with the adaptive radio frequency adjustment with low power operation, it can achieve low power consumption while the device communicates with the Internet.
|
| ICACT20230071 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
D. Face recognition
The data processes of the face recognition were shown in
Fig. 2. After start and initialization of the system, the smart
mirror detects the human face. If the mirror acquisitions the
human face, it takes a picture of the face. If it is not a human
face, the system keeps in this loop. If the mirror take the face
of user who has finished the registration, the features of the
user’s face sends to central controller to display the personal
messages onto the mirror else the process goes to step 6 for
saving the features of the new user’s face. Next, the features
of the user’s face sends to central controller to display the
personal messages onto the mirror then the process goes to
step 3.
The face recognition algorithm of OpenMV is to calculate
the eigenvalues of photos, and compare the eigenvalues of the
photos in the database to obtain the differences of the
eigenvalues. The photo of user’s face in the database with the
minimal compared error is the result of the recognition. |
| ICACT20230071 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
Methods – ESP 32
ESP32 integrates complete transmit/receive RF functions such as antenna switch, RF balun, power amplifier, filter, power management module and advanced self-calibration circuit. The self-calibration circuit realizes dynamic automatic adjustment to eliminate the defects of the external circuit.
The ESP32 comes with two 32-bit, LX6 CPUs clocked at up to 240MHz. ESP32 supports the Bluetooth 4.2 standard and integrates Bluetooth/BLE radio frequency and low power technology. It can meet the communication requirements between APP and Loongson and reduce the overall power consumption.
|
| ICACT20230071 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
Methods – ANFIS
ANFIS’s architecture proposed by Kosko et al [17,18] is computed using the methods of both artificial neural network and fuzzy logic. This architecture combines the machine learning of artificial neural network and the if-then rules of Fuzzy logic.
The peak values of histogram of the errors of eigenvalues were simulated as the input of ANFIS system and the true (0) or false (100) values were the given training results. The simulation was computed in MatLab users’ development environment.
|
| ICACT20230071 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
Results
APP
Face recognition
Smart mirror
|
| ICACT20230071 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
A. APP
The demonstration of the operations of APP was presented
in Fig. 3. The functions of the APP included the managements
of devices and users. The device management supported the
searching of devices with Bluetooth communication and
identified the functioned devices. The connections between
APP and devices showed on the screen of the smart phone.
The user management can add, delete and update the
information of users. It can assign the user ID to the users. |
| ICACT20230071 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
. Face recognition
The error distributions of eigenvalues in 120 times of tests
of face recognition (a) in the data base, and (b) not in the data
base were shown in Fig. 4 where X-axis denoted as the error
of eigenvalue and Y-axis denoted as the ratio of the deviation.
In (a), the mean of the error of eigenvalue (μ) was 5422.52
and the standard deviation (σ) was 2019.6. However, in (b),
μwas 9439.85 andσ was 1750.4. The better threshold was
7200. For strangers of this smart mirror, the successful rate of
strangers (SRS) was 100%. For the user, SRS = 90% and
accuracy of user’s recognition (SRU) was 100%. |
| ICACT20230071 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
For the auto classification, the simulated performance of
ANFIS classifier was presented in Fig. 5. The better threshold
was 7000 to 7500 of errors. The classification was established
by the scores (S). If S>30, the recognition is fault. |
| ICACT20230071 Slide.18
[Big Slide]
|
Chrome Click!! |
 |
C. The smart mirror
At all, the smart mirror was demonstrated in Fig. 6. This
system develops a face recognition information reminder
system. This system uses the Loongson 1C Zhilong
development board as the main control chip and runs the RTThread real-time operating system. With the operating system,
it possesses the characteristics of high development efficiency
and strong system expansibility. The server built with Django
provides a stable response service for the system's information
transmission. This system mainly uses the Loongson 1C
Zhilong development board, ESP8266, SSD1306 display
screen, and the server side is implemented by Django. |
| ICACT20230071 Slide.19
[Big Slide]
|
Chrome Click!! |
 |
Discussion |
| ICACT20230071 Slide.20
[Big Slide]
|
Chrome Click!! |
 |
Discussion
This design meets the requirements of the design scheme to a certain extent, and the design purpose is to realize a personal information reminder system with simple operation and strong scalability. After testing, the system has achieved the expected effect, is convenient in operation, and has strong expansibility.
The possible improvements of the smart mirror were as follows:
(1) Using the algorithm of convolutional neural network to realize face recognition in the embedded system can achieve higher speed and accuracy [15, 16];
(2) Improve the circuit system of the system, and design the power supply circuit to supply power for OpenMV and Zhilong development boards.
|
| ICACT20230071 Slide.21
[Big Slide]
|
Chrome Click!! |
 |
Discussion
The ANFIS classifier was with 2 rules and 80 epoch number in this study. We have tried a test of 4 rules and 40 epoch number. However, the results of the 2 rules and 80 epoch number presented a better performance in classifications. For the two-state results of recognition, the 2 rules Fuzzy logic match the reality.
|
| ICACT20230071 Slide.22
[Big Slide]
|
Chrome Click!! |
 |
Conclusion |
| ICACT20230071 Slide.23
[Big Slide]
|
Chrome Click!! |
 |
Conclusion
The study completes the following tasks:
(1) Complete the construction of the Loongson 1C Zhilong development board hardware platform, RT-Thread development platform, and Django development platform, complete the clarification of the serial port between the Zhilong development board and other hardware devices, and determine the device-related communication technology;
(2) Carry out the architecture design of the system, modular decomposition for each function, the server function realization, and making a detailed design and implementation plan for the system hardware part;
(3) Using Visual Studio Code to program thread communication and synchronization code, server information analysis code, IIC control display code, as well as OpenMV IDE to program face recognition code, face detection code, UART communication code. In addition, the Pycharm IDE has been employed for server code to implement server request response.
|
| ICACT20230071 Slide.24
[Big Slide]
|
Chrome Click!! |
 |
Conclusion
The COVID-19 pandemic is still threatening human beings. A smart mirror with the function of face recognition activation is a non-touching solution for avoiding the infections to support an idea for elevating human health.
|
| ICACT20230071 Slide.25
[Big Slide]
|
Chrome Click!! |
 |
The authors thank for the support from Guangdong University of Petrochemical Technology, Guangdong, China
(Project number: 702/519244)
|
| ICACT20230071 Slide.26
[Big Slide]
|
Chrome Click!! |
 |
Thank you! |
