Open Access iconOpen Access



Negative Emotions Sensitive Humanoid Robot with Attention-Enhanced Facial Expression Recognition Network

Rongrong Ni1, Xiaofeng Liu1,*, Yizhou Chen1, Xu Zhou1, Huili Cai1, Loo Chu Kiong2

1 College of IoT Engineering, Hohai University, Changzhou, 213100, China
2 Faculty of Computer Science & Information Technology, Universiti Malaya, Kuala Lumpur, 50603, Malaysia

* Corresponding Author: Xiaofeng Liu. Email: email

Intelligent Automation & Soft Computing 2022, 34(1), 149-164.


Lonely older adults and persons restricted in movements are apt to cause negative emotions, which is harmful to their mental health. A humanoid robot with audiovisual interactions is presented, which can correspondingly output positive facial expressions to relieve human's negative facial expressions. The negative emotions are identified through an attention-enhanced facial expression recognition (FER) network. The network is firstly trained on MMEW macro-and micro-expression databases to discover expression-related features. Then, macro-expression recognition tasks are performed by fine-tuning the trained models on several benchmarking FER databases, including CK+ and Oulu-CASIA. A transformer network is introduced to process the sequential features engineered by the FER network and output a final stable control order. This order is used to control the robot's facial motor units to generate different expressions, e.g., a smile expression. Evaluations on benchmarking databases verify that the proposed method can precisely recognize facial expressions. The joint modulation with the humanoid robot proves that the robot can respond effectively to the user's negative emotions.


Cite This Article

R. Ni, X. Liu, Y. Chen, X. Zhou, H. Cai et al., "Negative emotions sensitive humanoid robot with attention-enhanced facial expression recognition network," Intelligent Automation & Soft Computing, vol. 34, no.1, pp. 149–164, 2022.

cc This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 1275


  • 727


  • 0


Share Link