A Lightweight Method for Almost Real-Time Recognition of Emotions through Facial Geometry with Normalized Landmarks

Abstract

Recognizing emotion is an intrinsic and fundamental part of human social interactions and is closely tied to behaviors that produce distinct facial patterns. Facial expressions indicate emotional context, so scientific, artistic, medical, and commercial interest in the field has increased, driving the development of computational techniques that can recognize emotion automatically. Although current methods provide satisfactory results, challenges persist, particularly challenges regarding the variable patterns of facial shapes and the response time achieved with low computational resources. For instance, some applications requiring instantaneous emotion recognition with high accuracy and low latency may be limited by the processing power, specially in the case of mobile devices. Here, we present a practical and simple method called REFEL (Recognizing Emotions through Facial Expression and Landmark Normalization), designed to identify facial expressions and human emotions in digital images. This method addresses sequential steps that reduce sample variability such as anatomical, scale factor and geometric variations and performs reductions of color, brightness and others in preprocessing tasks. REFEL normalizes facial fiducial points, commonly referred to as landmarks, and allows fine-tuning of informative aspects delineating facial patterns. Using landmark positions makes the process of recognizing emotions more reliable. REFEL also exploits classifiers explicitly tailored to identify facial emotions accurately. As in the case of related works, we employed Machine Learning algorithms, to achieve average accuracy higher than 90% for emotion recognition, when we applied REFEL before classification. We have experimented REFEL with various datasets including facial images that consider racial, age and gender factors as well as facial rotation. In this study, we also compared emotion classification without grouping emotions and with two emotion groups (Fear-Surprise and Anger-Disgust). Analysis of the ROC curves revealed that grouping emotions led to a slight improvement in the average performance of the REFEL method, with a 3% increase in accuracy. Our method represents an enhanced approach in terms of hit rate and response time, generates resilient outcomes, and relies less on the training set and classifier architecture. Furthermore, REFEL performs well, almost in real time, lowers the processing costs inherent to training, and is particularly suited to devices with limited processing capabilities, like cell phones. Emotion recognition methods usually have almost minimal real-time delay, which enables the system to react fast but not necessarily instantaneously. With REFEL, we hope to improve computational synthesis techniques and resources, and to help robust and motivating assistive technologies to advance. As future efforts, we intend to consider 3D images and videos.