Overview

Emotion is an important form of expression in our body, and how to effectively identify and judge accurate emotions is also a key prerequisite for achieving emotion regulation in our projects. There are various ways to achieve emotion recognition, because emotion will be reflected in all aspects of a person, such as expression, speech, heart rate, etc.With the current more developed artificial intelligence technology, emotion recognition technology based on expressions, language, etc. has been more mature, but the effect of emotion recognition for some special cases, such as fake expressions, is still poor. Although micro-expression recognition technology has developed to a certain extent in recent years, due to the lack of training data, single and algorithm defects, most recognition effects or generalization performance are poor and lack practical value; and heart rate, as a physiological signal in itself, is different from expression and language, which is not easy for the brain to control actively. And it has been shown earlier that heart rate can be influenced by emotion [1], but the use of heart rate for emotion recognition is influenced by factors such as physical movement and is closely related to the physical indicators of the subject.

Based on the above analysis, in order to achieve more accurate emotion recognition, heart rate and expression are a pair of complementary indicators, and relatively easy to obtain and process. To achieve multimode emotion recognition based on heart rate and expression, for expression recognition, artificial intelligence Convolutional Neural Networks can be used to make judgments; while for heart rate, our product targets a relatively static office population, and algorithms need to be established to judge emotional fluctuations and be able to interact with users to obtain user information.

1. Algorithms and Models

1.Emotion recognition

(1) First, we call the laptop's own camera for user expression capture (after user authorization) considering the positioning office scenario and user hardware burden

(2) Then, the image containing the face will be transformed into a 48*48 matrix as the convolutional neural network input

(3) The algorithm convolutional neural network is used to compare the combined scores of different expressions of the database material

(4) The expressions with the highest scores will be displayed and combined with the bracelet results

2.Heart rate

According to Yang et al.'s study [3], a model that responds to the relationship between heart rate and expression was obtained, and the effect of exercise on heart rate was considered. However, since the users oriented in our project are basically engaged in relatively static and smooth activities such as indoor offices, the parameters related to exercise in the model are set to 0.

First, consider the heart rate required to maintain the functioning of the human body :

Where

and are the minimum and maximum heart rate of the human body, respectively, and is the parameter reflecting the movement, which we set to 0, so we have the following equation:
is related to the person's age with the relationship
is related to the overall cardiovascular profile of the person and is affected differently for biological males and biological females
For biological males:
For biological women:
where is the overall cardiovascular status, Knowing the above information and the heart rate of the user at this moment, it is possible to predict the heart rate at the next point in time.

where a is a constant, a=0.0003, Write which is the difference between the actual heart rate and the predicted heart rate Then writeFinally, the mental arousal level (MAL) can be calculated to reflect the presence or absence of mood swings

The MAL range is between -1 and 1, and in the absence of mood swings, the range is typically between -0.3 and 0.5 again

Based on this model, in addition to the user's heart rate, we also need to obtain three user personal information: age, gender and minimum heart rate or overall cardiovascular profile. Implementation of multimodality

When MAL exceeds the normal range, the emotion of face emotion recognition is output; otherwise, no emotion is output.

2. Realization Process

Entering the program will pop up a help window, prompting the user to enter the range of parameters to be obtained

After closing the help window, it will enter the main window, and users can still click the help button in the bottom right corner of the main window to get help

A warning window will pop up when the user enters parameters for or with errors

In the Data Security and Privacy section: When the user enters the parameters correctly, clicking the Start button will bring up the Privacy Statement button, asking the user to confirm permission to use the camera and stating that the information entered will be encrypted and will not be saved

After the user accepts, the recognition program will start running, the main window will display the heart rate (the program currently uses random numbers for heart rate), and at the same time the camera window and the window showing the real-time MAL will pop up, and the recognized emotions will be displayed in the MAL window

3. Privacy and Security

1.Permission Acquisition Statement

Camera permission is the most sensitive part of the whole program involving privacy protection . Although this is only a local application, information will not be directly leaked to others through the Internet, but may cause the user's personal space to cause discomfort, so in the part of the program that needs to use the camera, we will inform the user, and apply for permission.

2.Personal Information Protection

Since the model needs to obtain users' personal information, for the privacy protection of users' personal information, the program parameters are basically transmitted in a closed-loop, and basically there is no situation that they will be listened to by others. Nevertheless, during parameter transmission, we still encrypt the user parameters, using a 1024-bit RSA encryption system, and do not save the public and private keys locally, and automatically destroy them at the end of the program run, thus achieving tighter privacy protection.

3.Legality of input

Considering that the user may try to make the program run buggy, the range of each parameter is strictly required, and a warning pop-up will appear if the user's input is out of range or if the input is not a number. Also, to prevent user "injection", we use the literal_eval() function of the ast library in the final parameter passing to prevent possible misuse.

4. Reference

[1]Anne-Marie Brouwer, Elsbeth van Dam, Jan B. F. van Erp , Derek P. Spangler and Justin R. Brooks: Improving Real-Life Estimates of Emotion Based on Heart Rate: A Perspective on Taking Metabolic Heart Rate Into Account, 2018

[2]I Goodfellow, D Erhan, PL Carrier, A Courville, M Mirza, B Hamner, W Cukierski, Y Tang, DH Lee, Y Zhou, C Ramaiah, F Feng, R Li, X Wang, D Athanasakis, J Shawe-Taylor, M Milakov, J Park, R Ionescu, M Popescu, C Grozea, J Bergstra, J Xie, L Romaszko, B Xu, Z Chuang, and Y. Bengio: Challenges in Representation Learning: A report on three machine learning contests., 2013

[3]Zhaofang Yanga, Wenyan Jiab, Guangyuan Liuc, Mingui Sun: Quantifying mental arousal levels in daily living using additional heart rate, 2017