Open Access
ARTICLE
Facial Expression Recognition Based on the Fusion of Infrared and Visible Image
Jiancheng Zou1, Jiaxin Li1,*, Juncun Wei1, Zhengzheng Li1, Xin Yang2
1 North China University of Technology, Beijing, 100144, China
2 Department of Computer Science, Middle Tennessee State University, Murfreesboro, 37132, USA
* Corresponding Author: Jiaxin Li. Email:
Journal on Artificial Intelligence 2021, 3(3), 123-134. https://doi.org/10.32604/jai.2021.027069
Received 10 January 2022; Accepted 14 January 2022; Issue published 25 January 2022
Abstract
Facial expression recognition is a research hot spot in the fields of
computer vision and pattern recognition. However, the existing facial expression
recognition models are mainly concentrated in the visible light environment. They
have insufficient generalization ability and low recognition accuracy, and are
vulnerable to environmental changes such as illumination and distance. In order
to solve these problems, we combine the advantages of the infrared and visible
images captured simultaneously by array equipment our developed with two
infrared and two visible lens, so that the fused image not only has the texture
information of visible image, but also has the contrast information of infrared
image. On the other hand, we improved the WGAN by adding SSIM and LBP loss
functions to ensure the structural similarity between the fused image and infrared
image, and also the texture similarity between the fused image and visible image
respectively. Finally, a facial expression recognition model Pyconv-SE18 with
pyramid convolution and attention mechanism module is designed to extract the
important feature information of facial expression in multiple scales. We add
cosine distance loss function to reduce the feature difference within the class.
Experiment results show that the robustness of expression recognition algorithm
to illumination is improved based on the fused images. The accuracy of this model
on FER2013 and CK+ public data sets are 69.3% and 94.6%, respectively.
Keywords
Cite This Article
J. Zou, J. Li, J. Wei, Z. Li and X. Yang, "Facial expression recognition based on the fusion of infrared and visible image,"
Journal on Artificial Intelligence, vol. 3, no.3, pp. 123–134, 2021. https://doi.org/10.32604/jai.2021.027069