Enhancing Facial Emotion Recognition Using DCNN through Effective Extraction for High-Level Features

1  Introduction

In the area of facial expression recognition (FER), the identification and classification of seven universal emotional states, surprise, disgust, fear, happiness, neutrality, anger, and contempt, are critical [1,2]. Over the past ten years, there has been a lot of research done on facial expression recognition, and its applications, such as friendly robots and avatar animation, have grown. Since people differ greatly in how they display their emotions, it is difficult for machine learning techniques to recognize facial expressions [3,4]. When analyzing diverse individuals, changes in brightness, background, and pose, due to variances in shape and ethnicity, among other factors, are emphasized even in images of the same person with the same facial expression [5,6] to automate the process. This process starts with face detection, passes through feature extraction, and ends with facial emotion recognition. These three crucial processes comprise the core technique of automated facial emotion recognition and its applications, as depicted by Fig. 1 below.

Figure 1: The principal procedures of automated face expression recognition.

The first step is face collection, where high-quality photos are taken; the second is crucial face data extraction and representation, where pertinent facial features are extracted from the raw data; and the third is recognition, where these features are subjected to complex algorithms to detect and classify facial expressions [7]. Generally speaking, there are two primary methods in the field of facial feature extraction: the handcrafted methods, which focus on manually chosen features and patterns, and the geometric or appearance feature-based methods, which focus on significant facial features [8–10]. On the other hand, the automatic facial expression analysis’s enormous promise was proven in a variety of fields (see Fig. 2), including clinical psychology, neurology, human-computer interactions, and even advanced applications like robots, telemedicine, criminal investigation, lie detection, etc. [11,12].

Figure 2: Facial emotion recognition application scenarios.

Overall, studying a range of variables is necessary for the effective classification of emotions, including the common-sense emotions of happiness and anger in different degrees. While they come naturally to humans, facial recognition software has found difficulty and expense in detecting the characteristics and intensities [13]. The availability of both small- and large-scale datasets and the progress made in deep learning techniques have propelled recent advances in face recognition. In terms of accuracy and performance, deep face recognition systems, which comprise face detection, face alignment, and face representation, have advanced significantly. These systems use convolutional neural networks (CNNs) to recognize faces, align them with a canonical view, and extract typical features for identification. Nevertheless, position, illumination, emotion, aging, and a lack of training data in face recognition remain major obstacles to capturing high-level features for these systems [11]. To overcome these shortcomings, academics have concentrated on introducing new, sophisticated models in the interim. This has led to the emergence of virtual sample methods and generic learning methods that make use of virtual images, more multi-sample generic sets, and better loss functions or network architectures to improve recognition performance [14,15].

On the other hand, facial emotion detection can be challenging due to various factors such as 3D face posture, illumination, position, high noise, opacity, and varying lighting conditions, as shown in Fig. 3 [6]. In emotion recognition applications, artificial neural networks (ANN) and deep neural networks (DNN) tend to perform better than classical machine learning methods because they achieve higher accuracy [16,17]. Further, the continuously presented research in the literature, which is likewise an extension of these neural network models, also outperforms the deep learning models.

Figure 3: Facial emotion detection challenge examples.

This work is therefore intended to be completed in conjunction with the literature to address some of the FER works’ limitations. To improve the performance of emotion facial recognition, our work compares traditional and improved machine/deep learning models based on accuracy, precision, and f-score [18,19]. According to the literature, there are numerous operational problems with image identification from a technical standpoint. These problems include those of reliability, efficiency, and the resolution of difficult or specially designed solutions [20,21]. Meanwhile, to handle unconstrained face detection, pose estimation, landmark localization, and recognition, deep learning has also been used to improve face-recognition skills [22,23]. Presently, convolutional neural networks (CNNs), a type of deep learning algorithm, are widely used in state-of-the-art techniques for facial emotion recognition. These CNNs do a commendable job of accurately identifying emotions by extracting hierarchical features from facial images. In computer vision applications, including object recognition, object tracking, image classification and segmentation, and, most importantly, facial emotion recognition, CNN is the most widely employed deep learning model. According to Karnati et al. (2023) [24], this is because the CNN model has proven to be an especially effective model for accurately identifying and understanding facial expressions over time, supporting applications in cognitive computing, computer-human interaction, emotional recognition, and other fields. To achieve reliable emotion detection across various datasets, a CNN-based DCNN architecture has been developed and fine-tuned in conjunction with RandomOverSampler to handle class imbalance and regularization techniques to address the overfitting and vanishing gradient problem. As demonstrated in the methodology section, the deep architecture has been trained to precisely extract the most important high-level features, even in the presence of class imbalance. In a nutshell, this study aims to contribute the following:

1.    Developing and fine-tuning a deep CNN model to increase the accuracy by effectively capturing the discriminative facial features for emotion classification.

2.    Training DCNN to automatically learn high-level hierarchical features from image data, as it can be trained to handle variations in facial expressions, lighting conditions, orientation, etc.

3.    Handling class imbalance by applying the oversampling strictly to the training set, ensuring that data leakage is avoided, and improving model reliability

4.    Evaluating the model using one-holdout validation, 5-fold cross-validation, and cross-dataset testing, providing a thorough assessment of intra-dataset consistency and inter-dataset generalization.

5.    Performing extensive comparisons against classical machine learning models, state-of-the-art baselines, and most recent approaches to show the effectiveness of the proposed method.

2  Literature Review

Over the past few decades, a significant amount of work has been put into FER using both deep learning and conventional machine learning techniques. However, because there are so many variables to consider, including head pose, lighting, and facial features, the FER remains a difficult task [25]. This problem can be solved in a variety of ways, but the majority of them need a sizable amount of training data [26]. Emotions are generally represented as discrete entities with labels in categorical representations. According to Yalçin & Alisawi (2024) [27], the six primary emotions, anger, disgust, fear, happiness, sadness, and surprise, are distinguished by unique universal signals and physiologies. Despite the progress in the field, the literature has been developing AI-based tools and models to enhance FER-based technologies’ accuracy. For example, Han (2026) [28] proposed an artificial intelligence technique based on the Xception–ResNet–Senet (Squeeze-and-Excitation Network) network module of convolutional channel attention to increase the accuracy of face recognition. Initially, the Xception network employed a deep separable convolutional network. The output of the separable convolutional network was then supplemented with the Senet network’s attention module, and the output channel’s weight was assigned based on its significance. According to the experimental results, the model was competitive with its competitors. However, besides having inadequate preprocessing for FER-2013 and RAF-DB, Xception’s gradient vanishing problem was not effectively solved, considering the results’ inconsistency across datasets.

Kırbız (2025) in [3] proposed a deep learning framework for automatic facial emotion based on DCNN. Despite the increase in dataset size as authors merge all three datasets, the minority classes still suffer from an insufficient number of training samples, leading to data imbalance, which was minimized by the augmentation technique and random weighted sampling. Experimental results demonstrated that the proposed method could recognize the seven basic emotions with competitive accuracy on the merged dataset. Nevertheless, the model is complex and thus prone to overfitting. Further, the validation process is missing, and thus, no fine-tuning process was introduced, suggesting customizing the model on each dataset. Similarly, CK+, KDEF, and FER-2013 were used in [5] to perform image classification based on the facial emotion recognition concept. This research study offered a comprehensive analysis involving 68 and 98 landmarks with different FER deep models, examining how landmarking and different network architectures contribute to emotion recognition accuracy. Besides having a complex model, its accuracy is still in need of further enhancement. Ly et al. (2025) [7] proposed a novel hybrid approach that integrates Graph Attention Networks (GAT) for geometry-aware feature extraction and Residual Masking Networks (RMN) for pixel-level feature refinement. Experimental results on benchmark datasets, including CK+, JAFFE, RAF-DB, and FER-2013, demonstrated the model’s competitiveness. However, besides being a highly inefficient model, the fine-tuning process is missing, suggesting customizing the model for each dataset.

Driven by the fact that existing CNN models still have limitations in terms of feature representation and recognition accuracy for facial expressions, Zhang et al. (2025) [8] proposed CAGNet, a combination of convolutional block attention modules and global average pooling modules. Additionally, Batch Normalization (BN) layers and Dropout techniques were incorporated to improve model stability and generalization. CAGNet was evaluated on two standard datasets, FER-2013 and CK+, and the experimental results demonstrated that the model is competitive. Nonetheless, besides being an inefficient model, the model’s validation phase is absent, casting doubt over the model’s reproduction. Chen et al. (2025) [9] introduced a facial expression recognition method utilizing an enhanced ConvNeXt network. To enhance feature information extraction, the authors integrated the SENET attention mechanism into the ConvNeXt block. Experimental results showed that the improved ConvNeXt network achieved higher accuracy compared to other deep learning models. While ConvNeXt combines the efficiency of CNN and the global modeling ability of the transformer, it comes with computational costs, specifically when it is combined with the SENET attention mechanism. Veerappan et al. (2024) adopted the VGGNet architecture with extensive hyper-parameter tuning, experimentation, and different optimization techniques. However, the model was evaluated over a single dataset, limiting the model’s generalization [29].

On the other extreme, a lightweight attentional network incorporating multi-scale feature fusion (LANMSFF) was introduced in [30] to address the computational costs. The proposed approach achieved results comparable to SOTA (state-of-the-art) methods in terms of parameter counts and robustness to pose variation. Besides being an inefficient model, accuracy was used as the decisive metric despite having imbalanced data, casting doubts over the model’s reliability. Gupta et al. (2024) presented a ResNet-50-based model using facial emotions to detect the real-time engagement of online learners [31]. However, according to the results secured, the model does not use transfer learning effectively, as accuracy was not competitive compared with the SOTA models. Further, the fine-tuning process is missing, making the work irreproducible. To improve the accuracy of facial emotion recognition using FER-2013, Xu & Yan (2024) proposed an ensemble of mini-Xception models [14], where all weighted confidence scores are incorporated into the recognition mechanism. However, only one model was used for the comparison, which is the conventional mini-Xception model. Besides the absence of data sampling and hyperparameter processes, the model was customized for the FER-2013 dataset. The model was examined over a single dataset, limiting its generalization power. An EfficientNet model was presented in [32] as an efficient model for facial emotion recognition using FER-2013. However, the efficiency of the model comes at the expense of its low accuracy. Further, the model was examined on the FER-2013 dataset as if it were customized for it, limiting its generalization power.

Meanwhile, Tshibangu & Tapamo (2024) proposed a ConvNet model to extract features using the local binary patterns (LBP), region-based oriented FAST, and rotational BRIEF (ORB) [33]. However, because neither the experimental setup nor the image processing methods were mentioned, the model description lacked adequate depth. There was no explanation of the validation procedure. Also, despite the assertion that JAFFE was used in the evaluation research, results were not provided. Lastly, no comparative analysis was conducted, not even against a single competitor. Likewise, CNN models constructed using two methods, transfer learning (fine-tuned) with pre-trained models of Inception-V3 and MobileNet-V2, were proposed in [34] as a solution to the inadequate tuning of the model’s hyperparameters. However, even with the use of transfer learning and pretrained models, the models’ performance was not competitive when compared to SOTA models, particularly on JAFFE. A self-cure relation network (SCRNet), a metric-based few-shot model that is resistant to label noise, was proposed in [35]. The work’s drawback is the somewhat low accuracy of the suggested system. In addition to appearing difficult, the model was inefficient. The authors state that during the meta-training stage, a class prototype kept in external memory allows SCRNet, their suggested approach, to provide corrected labels to noisy data. A residual network topology with a multi-headed channel attention (MCA) module was developed by Shen & Xu (2023) [36]. While the convolutional layer parameters were pre-trained using the migration learning method, the ResNet18 backbone network was then incorporated with the developed MCA module. The multi-head structure concentrated more on the local characteristics of the images, while the attention mechanism employed different coefficients or weights to suppress irrelevant information and highlight relevant information. According to experimental results, the model achieved recognition results with accuracy rates of 72.7%, 98.8%, and 93.33% in FER-2013, CK+, and JAFFE datasets, respectively.

Anwarul et al. (2023) presented a hybrid model of the pre-trained models using ensemble learning (HE-CNN) to enhance the overall accuracy [37]. The datasets used were not widely used by the most recent and well-established models. Further, the experimental setup missed a lot of information, like image pre-processing and data sampling details. Białek et al. (2023) improved CNNs’ structure for image classification [38]. Several models are proposed, both custom and transfer learning types. Furthermore, combinations of them as ensembles, alongside various methods of dataset modification, are presented. The results on the FER-2013 dataset showed the model’s competitiveness. However, even if the model had an acceptable accuracy, it is still in need of improvement. A multimodal approach based on deep learning for facial identification behind a mask using verbal and facial expressions was presented by Shahzad et al. in 2023 [39]. A dataset of voices and faces was used to train the multimodal model: the CREMA-D and TESS datasets for vocal expressions and the M-LFW dataset for the masked dataset. The voice data was transformed into images by taking a spectrogram in order to homogenize the data. The multimodal model was then trained using the dataset. The findings of the experiment show that the suggested multimodal algorithm works better than unimodal techniques and other cutting-edge deep neural network models.

The convolutional neural networks were used to propose an extended approach for face emotion detection in [40]. The model used the expressive vector (EV) to identify the five distinct types of typical facial expressions. Every iteration of the double-level CNN results in different weights and exponent values for the final perception layer. The experiments showed that the proposed model was competitive. However, the accuracy was moderately low, and no SOTA models were compared.

A hybrid deep learning architecture called ExpressNet-MoE was proposed [41], which combines a CNN with a mixture-of-experts (MoE) framework. To increase generalization and flexibility across various datasets, the suggested model started by choosing the most pertinent expert networks for each input. BlazeFace was used for face detection and alignment in the preprocessing of facial images. Each dataset was split into training and testing groups using stratified sampling to preserve class distribution. By using multi-scale feature extraction to capture both global and local facial information, this model improved emotion recognition accuracy. Nevertheless, besides its lower accuracy, because it incorporates multiple CNN-based feature extractors, a MoE module for adaptive feature selection, and a residual network backbone for deep feature learning, it is an effective and intricate model. Debnath et al. (2022) [42] introduced the feature fusion strategy, complementary features from several models or descriptors, to increase robustness. Nevertheless, the combined model was complicated, its validation was not scalable, and the feature fusion process was not thoroughly described to ensure replication. Meanwhile, Kedari et al. (2021) provided a summary of current Facial Emotion Recognition (FER) stages, techniques, and datasets [43]. Then, using FER-2013 and CK+, the authors enhanced the CNN to detect the fundamental human emotions. The results showed that the model was competitive with its rivals, especially CK+. However, as demonstrated by the FER-2013 results, the CNN architecture was not deep enough to capture the most intricate and informative features. Furthermore, the fine-tuning process appeared to be done systematically over each dataset.

Niu et al. (2021) combined the local binary patterns (LBP) with the oriented FAST and rotated BRIEF (ORB) to extract features from facial expressions to create a combined algorithm [44]. Support vector machines (SVMs) classified the combined features to facilitate recognition. Several difficult databases, including the MMI database, the Japanese Female Facial Expressions database (JAFFE), and the Cohn-Kanade database (CK+), were used to assess the suggested approach. The suggested paradigm proved effective, according to experimental data involving seven different emotion states: neutral, joy, sadness, surprise, anger, fear, and disgust. The model was complex, and the results were not competitive compared to some SOTA models. Shuvo et al. (2021) presented an automated convolutional neural network-based method to identify the mental state of a person based on the features surrounding their eyes [45]. Only the eyes and their surrounding tissues, the glabella, brow, and upper and lower eyelids, were taken into account in this study to identify the emotional state. To improve accuracy, the dataset was trained using pre-learned weights from the ImageNet dataset, and the prediction was analyzed using a confusion matrix. Nevertheless, besides the fact that the dataset is rarely used in the literature, the upper and lower eyelids, the glabella, and the brow were the only anatomical features that the authors took into account. Pranav et al. (2020) developed a two-layer convolutional neural network model for specific use to enhance the FER accuracy [46]. Jain et al. (2019) used a deep CNN to classify each image into one of six face emotions (surprise, fear, happiness, anger, disgust, and sadness) to increase the accuracy [47]. However, it had a very complex design, making it time-inefficient. Further, the model in [47] lacks generalization. For instance, with CK+, the number of training samples was limited. Also, no details are provided for how the authors did the hyper-parameter process (no mention of the parameter values) or the validation phase, making the reproduction process complicated.

Zhang et al. (2019) proposed a spatial-temporal recurrent neural network to improve facial recognition ability by combining the learning of two distinct data sources into a spatial-temporal dependency model [48]. Similar to Jain’s works, it had a very complex design and was thus time-inefficient. Only accuracy was used for evaluation against the competitors. Besides using accuracy as a sole metric for evaluation against competitors, details of the hybrid optimization process are missing. Finally, Lopes et al. (2017) suggested using a CNN in conjunction with particular image preprocessing techniques to improve face expression recognition [49]. However, the model was only competitive over CK+. Besides missing the eye detection technique, the proposed model was seen as not being able to handle the variation between the expression classes, which is why it gets lower accuracy than expected in JAFFE and BU-3DFE. After looking into each of these efforts, we find that the majority of them share common problems, like the lack of a fine-tuning procedure, which is essential for reproducing and validating the work, as it is completely connected with establishing the model’s hyper-parameters. Furthermore, the majority of them continue to suffer from inefficiency because their models are either heavyweight models with various parameters that take up time and memory or sophisticated models that are prone to overfitting. Furthermore, a sizable portion of them were assessed using a single dataset, which restricted the model’s capacity for generalization. Last but not least, even with these advancements, the test accuracy still needs improvement.

3  Methodology

3.1 Motivations

Even while there has been unquestionable progress in developing FER models with respectable accuracy, our careful analysis, as shown in Table 1, indicates that there is still much space for improvement. Our claim is supported by several factors, including the inadequacy of parameter tuning procedures [7,31,33,42]; the complexity of the model, which makes it extremely prone to overfitting risk and thus having conflicting results across different datasets [3,5]; the failure to distinguish the most informative features [9,49]; and ineffective preprocessing that can capture local and global high-level features [2,46]. Thus, the goal of this study is to develop and fine-tune a deep CNN model that is trained to manage changes in face expressions, illumination, orientation, and other factors and that is capable of efficiently learning high-level features from image data.

3.2 Qualitative Interpretation of Learned Facial Regions

In the following, the facial representations, which are implicitly emphasized by the proposed DCNN during feature learning across the CK+, JAFFE, and FER-2013 datasets, are qualitatively analyzed. The convolutional and pooling operations employed in the network progressively learned hierarchical facial representations automatically during end-to-end training. Through layered convolutions, batch normalization, nonlinear activations, and pooling operations, the network gradually captures discriminative facial characteristics associated with emotional expressions. Lower convolutional layers primarily capture broad intensity variations and edge-related patterns, while deeper layers encode more subtle and localized facial characteristics, including wrinkles, eyebrow tension, eyelid movement, and lip contours. The following analysis represents theoretically motivated inferences drawn from established CNN literature, rather than empirically verified region activations. Therefore, the following analysis should be interpreted as a qualitative explanation of the facial regions implicitly learned by the DCNN, rather than an explicit computational region-selection process. First, Tables 1–3 present a qualitative interpretation of the eye and eyebrow regions implicitly emphasized by the proposed DCNN across the three datasets, along with representative facial expression samples for visual reference.

•   The initial convolutional layers (64 filters with 5 × 5 kernels) are expected to capture large-scale contrast variations, including raised eyebrows, furrowed brows, and widened eyes. Batch normalization and ELU (Exponential Linear Unit) activation functions enhance subtle facial variations, such as vertical wrinkles between the eyebrows and eyelid tension, while max-pooling preserves the most discriminative feature responses.

•   The incorporation of dropout (40%) following the pooling operation encourages the model to generalize by preventing over-reliance on subject-specific facial patterns.

Second, Tables 4–6 provide a qualitative analysis of the mouth-related facial characteristics hypothesized to be learned by the deeper convolutional layers.

•   The intermediate and deeper convolutional layers (128 and 256 filters with 3 × 3 kernels) are hypothesized to encode finer mouth-related characteristics, including lip curvature, lip tightening, and mouth opening patterns.

•   Repeated normalization and nonlinear activation operations are designed to improve the sensitivity of the network toward subtle expression-related variations around the lips and mouth corners, while progressive dropout rates ranging from 40% to 60% promote robustness across different facial identities and expression intensities.

Third, Tables 7–9 summarize how the learned representations from different facial regions are implicitly integrated to form a global facial expression representation.

•   Following the flattening operation, the dense layer (128 units) combines features learned from different facial regions into a unified expression representation.

•   The final dropout layer (60%) reduces over-reliance on any single facial characteristic and encourages the network to learn coordinated expression patterns distributed across the entire face.

Overall, the proposed DCNN does not explicitly isolate facial regions; rather, it implicitly learns discriminative facial representations through hierarchical feature extraction during training.

3.3 Proposed Model

The model is a Sequential Deep Convolutional Neural Network (DCNN) optimized for facial emotion recognition, which classifies facial expressions into seven emotion categories: anger, disgust, fear, happiness, sadness, surprise, and contempt. The architecture comprises three convolutional blocks followed by fully connected layers, designed to hierarchically extract spatial features and generalize across diverse facial expressions, as depicted in Tables 10 and 11 and Fig. 4.

Figure 4: Proposed DCNN.

The architectural and regularization parameters were determined through a systematic fine-tuning process, described as follows. (1) Conv2D Layers: Multiple convolutional blocks were employed, starting with 64 filters, then 128, and finally 256 filters (See Table 10). Each convolutional layer utilized a 5 × 5 kernel for the first block and 3 × 3 kernels for subsequent blocks, with ELU (Exponential Linear Unit) activation and the ‘he_normal’ kernel initializer as given in Eq. (1).

ELU(x)={x, ifx>0;α(ex−1), ifx≤0}(1)

The ELU activation function helps alleviate the dying ReLU problem and promotes faster learning and better generalization by allowing negative output values, thereby pushing mean activations closer to zero. The hyperparameter α is set to 1. Batch Normalization was applied after each convolutional layer. (2) MaxPooling2D Layers: A MaxPooling2D layer with pool_size = (2, 2) was added after each convolutional block to progressively reduce spatial dimensions, decrease computational load, and introduce a degree of translation invariance. (3) Batch Normalization: Batch Normalization was applied after every Conv2D layer across all three convolutional blocks to stabilize the learning process, accelerate convergence, and provide an additional regularization effect. (4) Dropout: Dropout layers were incorporated after each pooling layer with rates of 0.4, 0.4, and 0.5 for the first, second, and third convolutional blocks, respectively, and a rate of 0.6 was applied before the final output layer. This progressive increase in dropout rate applies stronger regularization to the higher-level features and the dense layer, which are generally more susceptible to overfitting. (5) Flatten Layer: A Flatten layer was applied to convert the two-dimensional feature maps into a one-dimensional vector, serving as input to the fully connected layers. (6) Dense Layer: A fully connected Dense layer with 128 units and ELU activation was employed, followed by Batch Normalization and a Dropout layer with a rate of 0.6, prior to the final classification layer. (7) Loss Function: Categorical cross-entropy was used as the loss function, defined as L = −Σ yi log(pi), where yi is the true label in one-hot encoded form and pi is the predicted probability for class i. (8) Output Layer: A Dense layer with 7 units and softmax activation was used as the output layer, corresponding to the seven target emotion classes. (9) Regularization: Batch Normalization was applied after every convolutional layer and after the dense hidden layer to stabilize training, accelerate convergence, and provide a regularization effect throughout the network. (10) Class Imbalance Mitigation: Oversampling was combined with dropout-based regularization to address class imbalance in the training data. This technique was applied exclusively to the training set to prevent data leakage into the validation or test partitions. (11) Vanishing Gradient Mitigation: The proposed architecture comprises six convolutional layers. The risk of vanishing gradients in deeper layers was mitigated through the use of ELU activations and Batch Normalization after each convolutional layer, both of which contribute to maintaining healthier gradient flow during backpropagation. Furthermore, the proposed DCNN incorporates the following design principles: (1) Progressive Feature Learning: The architecture progressively increases the number of filters across deeper convolutional layers (64 → 128 → 256), as illustrated in Table 10. This design enables the network to learn low-level features such as edges and intensity variations in earlier layers and progressively more abstract and discriminative representations in deeper layers. (2) Max Pooling for Spatial Downsampling: MaxPooling2D with pool_size = (2, 2) is applied after each convolutional block to effectively downsample the feature maps, reducing computational complexity while preserving the most discriminative spatial responses. (3) Dropout Placement Strategy: Dropout layers were positioned after each pooling layer and after the fully connected dense layer. This placement targets the two stages in the network most prone to overfitting, the transition between convolutional and dense representations, and ensures that the model learns robust, distributed feature representations rather than relying on any single activation pathway.

3.4 Data Pre-Processing

The primary goal of this procedure is to transform the raw image data into a structured and normalized format, suitable for training the developed deep convolutional neural network. The CK+ dataset, for example, contains images of individuals displaying various facial expressions. To effectively utilize this data, several preprocessing techniques are employed. These include setting random seeds for reproducibility, defining image parameters, iterating through emotion-labeled directories to load and process images, applying oversampling for the training set to address class imbalance, and finally, splitting the data for model training and evaluation. Each step is carefully chosen to prepare the dataset for optimal model performance in recognizing seven distinct facial expressions: anger, contempt, disgust, fear, happy, sadness, and surprise. Specifically speaking, the initial step in the preprocessing pipeline involves importing necessary Python libraries that provide functionalities for numerical operations, file system interaction, data manipulation, handling imbalanced datasets, etc. The steps are done as follows (See Figs. 5 and 6):

1.    Data: Image data, after being loaded and processed, was converted into NumPy arrays (img_data = np.array (img_data_list, dtype = ‘float32’)) for efficient numerical operations. In Python, img_data = np.array (img_data_list, dtype = ‘float32’) is the list of image arrays (img_data_list), which is converted into a single NumPy array named img_data. The dtype = ‘float32’ argument ensures the pixel values are stored as floating-point numbers, which is also beneficial for numerical stability and precision during computations, especially normalization. For FER-2013, the dataset is organized as image files stored in directories by emotion class. Pandas is used to construct a DataFrame containing image file paths and their corresponding labels by scanning the directory structure.

Images were loaded directly from disk during training using Keras’ ImageDataGenerator, which handles preprocessing operations such as rescaling and data augmentation. The generator read images from their file paths, converted them into numerical arrays, and resized them to the required input shape (48 × 48). This approach eliminated the need for manual pixel processing, as all image loading and transformation steps were handled dynamically during model training.

2.   Normalization: pixel values in img_data were normalized by dividing each pixel value by 255.0. Since grayscale images typically have pixel values ranging from 0 (black) to 255 (white), this operation scales the values to the range [0, 1]. Normalization often helps in faster convergence and stable training of neural networks. After that, reshaping was done using NumPy arrays, which were reshaped to fit the input requirements of the CNN, img_data = img_data.reshape(−1, img_rows, img_cols, num_channel).

3.   Label handling: labels were converted to a NumPy array (labels = np.array(labels)). The labels = []: An empty Python list was initialized to store the corresponding emotion labels for each image. In Python, labels = np.array(labels) as the list of integer labels (labels) was converted into a NumPy array. Then, to ensure the reproducibility of our experimental results and the stability of the data split, a fixed random state of 42 was applied so that np.random.seed(42). This statement initialized NumPy’s pseudo-random number generator with a seed value of 42.

4.   Emotions Category Mapping: emotion category mapping was done so that emotions = {‘anger’: 0, ‘contempt’: 1, ‘disgust’: 2, ‘fear’: 3, ‘happy’: 4, ‘sadness’: 5, ‘surprise’: 6}: A dictionary was created to map emotion category names to integer labels. This numerical encoding was necessary for training machine learning models.

5.   Image Processing: cv2 (OpenCV) was used as a library for computer vision tasks. It was used here for reading and resizing images. Then, image reading was done using cv2.imread(os.path.join(data_path, dataset, img), cv2.IMREAD_GRAYSCALE) loaded images in grayscale format. After that, image resizing was done using cv2.resize(input_img, (img_rows, img_cols)), which resized each image to a uniform dimension of 48 × 48 pixels.

6.   Image parameters: img_rows, img_cols = 48, 48. These variables defined the target dimensions to which all images will be resized. Each image was a 48 × 48 pixel grid. Then, num_channel = 1 to indicate that the images were processed in grayscale.

7.   Data Partitioning: The next step was to partition the processed dataset into training, validation, and test sets. This was crucial for training the model, tuning its hyperparameters, and evaluating its generalization performance on unseen data. A split ratio of approximately 80% for training, 10% for validation, and 10% for testing was performed. In Python, sklearn.model_selection.train_test_split was used as a function from the scikit-learn library to split arrays or matrices into random train and test subsets. This function was used twice: first, to separate a test set (x_train_val, x_test, y_train_val, y_test = train_test_split(...)) and then to separate a validation set from the remaining data (x_train, x_val, y_train, y_val = train_test_split(...)). Tensorflow (as tf) was used as an open-source platform for machine learning, particularly deep learning. After that, tensorflow.keras.utils.to_categorical was used as a utility function from Keras (part of TensorFlow) to convert class vectors (integers) to binary class matrices (one-hot encoding). Y_train_res = the resampled train set; y_train_cat = to_categorical (y_train_res, num_classes = len(emotions)); y_val_cat = to_categorical (y_val, num_classes = len(emotions)); y_test_cat = to_categorical(y_test, num_classes = len(emotions)) converted the integer labels into a one-hot encoded format suitable for categorical cross-entropy loss functions in neural networks. On the other hand, for subject-independent evaluation, all experiments on CK+ and JAFFE were conducted using subject-independent splits, where identities in the training, validation, and test sets were mutually exclusive. This protocol ensured that the model was evaluated on entirely unseen identities, making the reported results a measure of generalization rather than memorization.

8.   Class imbalance: the imblearn.over_sampling.RandomOverSampler was used as a technique from the imbalanced-learn library to perform random oversampling and was applied for the training set only. It works by randomly picking samples from the minority classes with replacement until the desired class distribution is achieved. RandomOverSampler was used to address class imbalance in the dataset by oversampling the minority emotion classes to match the majority class. The fit_resample method was applied to the flattened train_image data and labels (x_train, y_train). Before applying oversampling, oversampler expects the data to be in 2D, so our 4D image data (x_train) was flattened into a 2D array (x_train_flat). Each image, originally (48, 48, 1), became a 1D vector of 48 ∗ 48 ∗ 1 = 2304 pixel values. x_train_flat = x_train.reshape (x_train.shape [0], −1) ros = RandomOverSampler(random_state = 42); x_train_res, y_train_res = ros.fit_resample(x_train_flat, y_train); x_train.shape [0] is the number of samples, and −1 infers the product of the remaining dimensions.

Figure 5: Methodology detailed steps.

Figure 6: Proposed FER methodology.

To validate the proposed model, common “widely used” datasets are utilized, which are known to contain problems such as imbalance, irrelevant images, and misclassified images. These datasets are JAFFE [50], CK+, and FER-2013. Fig. 7 draws randomly selected samples taken from these datasets.

Figure 7: Randomly selected samples per emotion class—CK+, JAFFE, and FER-2013.

CK+ dataset: it consists of facial images, 981 images that have been resized to 48 × 48 pixels and converted to grayscale. The dataset contains seven emotion classes, labeled as follows: 0: anger, 1: contempt, 2: disgust, 3: fear, 4: happy, 5: sadness, and 6: surprise. Emotions are distributed over splits as follows: Training set [0: anger (111 samples), 1: contempt (42 samples), 2: disgust (135 samples), 3: fear (57 samples), 4: happy (168 samples), 5: sadness (66 samples), 6: surprise (198 samples)]. Validation set [0: anger (12 samples), 1: contempt (6 samples), 2: disgust (24 samples),3: fear (9 samples), 4: happy (18 samples), 5: sadness (9 samples), 6: surprise (24 samples)]. Test set [0: anger (12 samples), 1: contempt (6 samples), 2: disgust (18 samples), 3: fear (9 samples), 4: happy (21 samples), 5: sadness (9 samples), 6: surprise (27 samples)].

JAFFE dataset: it has 213 images of 10 distinct Japanese women with a range of facial expressions. 60 annotators annotated the images with average semantic scores for each of the seven facial expressions that each subject was asked to make (anger, disgust, fear, happiness, sadness, surprise, and neutral). Emotions are distributed over splits as follows: Training set [0: Angry (24 samples), 1: Disgust (23 samples), 2: Fear (26 samples), 3: Happy (24 samples), 4: Neutral (24 samples), 5: Sad (23 samples), 6: Surprise (24 samples)]. Validation set [0: Angry (2 samples), 1: Disgust (4 samples), 2: Fear (3 samples), 3: Happy (3 samples), 4: Neutral (3 samples), 5: Sad (3 samples), 6: Surprise (3 samples)]. Test set [0: Angry (3 samples), 1: Disgust (2 samples), 2: Fear (3 samples), 3: Happy (4 samples), 4: Neutral (3 samples), 5: Sad (4 samples), 6: Surprise (3 samples)].

FER-2013 dataset: The dataset consists of approximately 30,000 grayscale facial images of size 48 × 48 pixels, representing seven facial expressions. The emotion classes used in this study are: Angry, Disgust, Fear, Happy, Neutral, Sad, and Surprise. The dataset is organized into train/test/val, which are distributed as follows: Training set [Anger (3314 samples), Disgust (3366 samples), Fear (3346 samples), Happy (3333 samples), Neutral (3346 samples), Sad (3192 samples), Surprise (3321 samples)]. Validation set [Anger (414 samples), Disgust (421 samples), Fear (418 samples), Happy (417 samples), Neutral (418 samples), Sad (399 samples), Surprise (415 samples)]. Test set [Anger (415 samples), Disgust (421 samples), Fear (418 samples), Happy (417 samples), Neutral (418 samples), Sad (399 samples), Surprise (415 samples)]. To improve model generalization, data augmentation was applied to the training set using TensorFlow’s ImageDataGenerator, including transformations such as rotation, shifting, shearing, and horizontal flipping. Additionally, to address class imbalance, RandomOverSampler was applied only to the training set, ensuring balanced class representation without affecting validation and test sets.

3.5 Evaluation Metrics

The efficacy of the suggested designs relative to their competitors were assessed through the application of standard assessment metrics. We offered the performance assessments of our proposed models for the training, validation, and test sets. Furthermore, we evaluated different machine learning (ML) approaches against our proposed model. Besides confusion matrices, we compared our model against SOTA models in the literature. The evaluation metrics included accuracy, precision, recall, F1, and ROC.

4  Experimental Results

This section draws detailed results of the developed DCNN over each dataset in three phases: training, validation, and testing. It is worth indicating that the bold font across all the following tables expresses the best performance. For all experiments and model training, we utilized a computational instance equipped with an NVIDIA Tesla P100 GPU (16 GB HBM2) for accelerated processing. This was supported by an Intel Xeon CPU (typically 2 cores) and approximately 13–16 GB of system RAM, all operating on a Linux-based operating system.

4.1 Convergence Analysis

The accuracy-loss figure for each dataset’s training and validation curves is shown below (see Figs. 8–10). The training and validation losses are represented by the loss plot, whereas the accuracy plot shows the training accuracy and validation accuracy. The model was trained for a fixed number of 150 epochs across all datasets, as this choice was guided by the accuracy and loss curves, which indicated stable convergence. All models were trained across all datasets using the Adam optimizer with a learning rate of 0.001 (β1 = 0.9, β2 = 0.999, ε = 1e−7), categorical cross-entropy loss, and a batch size of 32.

Figure 8: CK+ accuracy vs. loss for train and validation.

Figure 9: JAFFE accuracy vs. loss for train and validation.

Figure 10: FER-2013 accuracy vs. loss for train and validation.

CK+: As drawn in Fig. 8, at Epoch 1, the model starts with an accuracy of 30.70% and a relatively high loss of 2.4124, which reflects the expected behavior of an untrained network. However, the learning progression is very rapid. By Epoch 2, accuracy increases significantly to over 72%, and by Epochs 3–5, it already exceeds 83% to 92%. This shows that the model is quickly capturing important patterns in the dataset. By around Epoch 10, training accuracy reaches approximately 97%, and shortly after (Epoch 11–15), it approaches 99%–100%. From this point onward, the model maintains near-perfect accuracy throughout the remaining epochs, with the loss reducing to very small values close to zero. Overall, the training phase demonstrates fast convergence and strong learning capacity, with the model effectively fitting the training data within a short number of epochs. The validation performance also improves rapidly alongside the training metrics. At Epoch 1, validation accuracy begins at 61.22%, which already indicates that the model is capturing useful patterns early in training. As training progresses, by Epoch 6–8, validation accuracy increases to approximately 88%–96%, and around Epoch 10, it reaches 99%. From Epoch 11 onward, validation accuracy frequently reaches 100%, with very low validation loss values. Throughout the training process, validation accuracy remains consistently high, with only minor fluctuations (typically between 97% and 100%). The validation loss also stays very low, indicating confident and accurate predictions.

JAFFE: The model demonstrates effective convergence on the JAFFE dataset in Fig. 9. Both training and validation metrics showed substantial improvement in the early epochs. The model’s training accuracy begins at approximately 31.58% in the first epoch, signifying the initial phase of learning from a state of low prior knowledge. The progression goes as follows: The accuracy shows a steady improvement, rising to 50.53% by the end of epoch 2. By epoch 10, the training accuracy significantly increases to 88.42%. The model continues to refine its learning, achieving 93.68% accuracy by epoch 12. Towards the later stages of training, the accuracy approaches perfect scores, reaching 100.00% at epoch 48, and concluding at 100% at epoch 150. The training loss starts at 1.8799 in the first epoch and demonstrates a consistent and sharp decrease. It drops to 1.3305 by epoch 2 and further to 0.3863 by epoch 10, and the decrease continues, reaching 0.2116 by epoch 12. The training loss eventually stabilizes at a very low value, around 0.0070 by the final epoch (epoch 150). In turn, for the validation phase, the validation accuracy also shows rapid improvement, generally tracking the training accuracy after the initial epochs, with some fluctuations. It starts at 11.76% in the first epoch, and at the end, validation accuracy remains high, hitting 88.4% at the final epoch. The validation loss demonstrates a significant decrease after the initial epochs. It starts at 6.7973 in the first epoch, and at the end, validation loss remains low, 0.4880, in the final epoch. On the other hand, the test accuracy, evaluated on a separate 15% of the data not seen during training or validation, reached 95.12%.

FER-2013: As shown in Fig. 10, based on the training results, the model follows a clear, steady learning pattern, improving consistently from the very beginning to the final epochs. At the start (Epoch 1), the model performs quite poorly, with an accuracy of about 19.10% and a high loss of 2.5197.

This is expected because, at this stage, the network has not yet learned any meaningful patterns and is essentially making near-random predictions. As training progresses into the early phase (around Epochs 2–10), there is a sharp improvement. Accuracy quickly rises above 60%, showing that the model is beginning to capture basic visual features such as edges, shapes, and simple facial structures. Between Epoch 10 and Epoch 30, the model continues to improve steadily, reaching around 77% accuracy. This stage reflects the transition from basic feature detection to learning more meaningful emotional patterns in the data. In the mid-training phase (around Epoch 30–80), performance becomes more stable, with accuracy gradually increasing from about 78% to 83%. The loss also consistently decreases, indicating that the model is refining its understanding and making fewer mistakes. Towards the later stages (Epoch 80–150), the improvements become smaller but more consistent. The model eventually reaches around 87% training accuracy, with loss dropping to approximately 0.36. At this point, learning has mostly stabilized, and the model has formed strong internal representations of the dataset. At the beginning, validation accuracy starts at about 31.43% (Epoch 1). While not very high, it is better than random guessing and improves quickly in the next few epochs. By Epochs 6–10, validation accuracy jumps significantly to around 65%–70%, indicating that the model is not just memorizing the training data but is already learning patterns that generalize to unseen samples. From Epoch 15 onwards, validation accuracy consistently stays above 70%, and by Epoch 20–30, it reaches around 77%–81%. This indicates solid generalization performance. During the mid to late training stages (Epoch 30–150), validation accuracy becomes relatively stable, mostly fluctuating between 80% and 85%, with occasional peaks slightly above this range. Overall, the validation curve shows good stability with only minor fluctuations, which is typical due to batch variations and dataset diversity. Importantly, there is no large gap between training and validation performance.

4.2 Results

Starting with the CK+ dataset, the proposed DCNN has demonstrated optimal performance (accuracy, precision, recall, F1, ROC, and G-means) over CK+, achieving 100% across each metric. To evaluate the model performance against its baselines, the confusion matrices are presented along with accuracy, precision, recall, F1 score, and ROC curve. Figs. 11–14 draw the confusion matrices of all training, validation, and testing accuracy as well as F1, showing the optimality of DCNN over CK+.

Figure 11: CK+ confusion matrix–training accuracy.

Figure 12: CK+ confusion matrix–validation accuracy.

Figure 13: CK+ confusion matrix–test accuracy.

Figure 14: CK+ confusion matrix–F1.

On the other hand, Tables 12–14 provide the detailed class-wise results of the DCNN model, broken down by each individual emotion class on the CK+ dataset. This detailed view allows us to understand how well the DCNN model recognizes and distinguishes between different emotional expressions. Further, Fig. 15 shows the ROC curve demonstrating an optimal performance for the model with this critical metric. The ROC curve is plotted to assess the trade-off between true positive rate and false positive rate. With ROC, our model is able to produce the most efficient results, showing its power to address the trade-off between emotion recognition and the false alarm rate.

Figure 15: CK+ ROC curve.

The proposed DCNN also demonstrates effective performance in terms of all metrics over the JAFFE and FER-2013 datasets, achieving an accuracy of 0.952 and 0.844, respectively. Figs. 16–25 draw the confusion matrices of all training, validation, and testing accuracy and F1 and ROC curves, showing a competitive performance over JAFFE and FER-2013, respectively.

Figure 16: JAFFE confusion matrix–training accuracy.

Figure 17: JAFFE confusion matrix–validation accuracy.

Figure 18: JAFFE confusion matrix–test accuracy.

Figure 19: JAFFE confusion matrix–F1.

Figure 20: FER-2013 confusion matrix–training accuracy.

Figure 21: FER2013 confusion matrix–validation accuracy.

Figure 22: FER2013 confusion matrix–test accuracy.

Figure 23: FER2013 confusion matrix–F1.

Figure 24: JAFFE ROC curve.

Figure 25: F-2013 ROC curve.

Tables 15–19 provide a comprehensive evaluation of the DCNN model’s performance (CLASS-WISE results), broken down by each emotion class on the JAFFE and FER-2013 datasets. This detailed view allows us to understand how well the DCNN model recognizes and distinguishes between different emotional expressions. The proposed model, which is the optimal situation offered by the DCNN at a particular false alarm probability, achieves the highest emotion recognition probability to identify the false alarm overall, since the ROC curve is a plot of emotion recognition probability vs. false alarm probability, with 1.00 and 0.998 on CK+ and JAFFE.

Across all datasets, it is worth indicating that the ROC-AUC (trade-off between sensitivity and specificity) score of the proposed model has been highly competitive, showing how well the DCNN is able to distinguish the positive and negative classes. With a value of 100 over CK+, the model achieved perfect performance.

4.3 Proposed DCNN vs. Machine Learning Models

In this section, all results across datasets are summarized in Tables 20 and 21 for both effectiveness and efficiency metrics. As drawn in Table 18, the results express a clear dominance for the proposed DCNN over its baseline ML models in terms of all efficacy metrics. Our model has seen ideal conditions, in which facial recognition has perfect accuracy across CK+ and FER-2013 in particular. However, unfortunately, our model lags behind all its rivals in terms of efficiency metrics.

5  Discussion

As can be seen from the methodology and results sections above, high-level hierarchical features were extracted from input images using the DCNN to consistently and gradually extract increasingly abstract representations. It has been demonstrated that the DCNN model can reliably capture both high-level features and the spatial relationships and overall context of the image. Effective feature extraction with varying filter widths was made possible by the exact use of the optimized deep convolutional layers to extract both local and global data. The findings obtained indicate that facial expression identification benefits from this architecture’s ability to dynamically learn and represent complex patterns and features. In the following, the results of the DCNN equipped with and without oversampling are shown to demonstrate how the oversampling strategy can address class imbalance and improve the accuracy of the model (See Table 20).

Tables 20 and 22 present the effectiveness results, demonstrating the relative effectiveness of the proposed model compared to its bare model, which did not address class imbalance. Note that by “bare-DCNN”, we mean the proposed DCNN without the oversampling method. As shown in Table 23, the proposed DCNN demonstrated competitive efficacy throughout JAFFE and FER-2013, except CK+, where both models performed similarly. On the other hand, while Table 21 shows the efficiency of the proposed DCNN vs. all classical models, Table 23 shows the efficiency of DCNN vs. Bare-DCNN, showing that Bare-DCNN is much faster than DCNN in regard to the training time, validation time, and total run time as well. Meanwhile, no significant changes have been noted regarding other efficiency metrics such as memory usage, model size, and number of parameters. This suggests that the outstanding performance of DCNN with the oversampling technique comes at the expense of its efficiency as provided by the Bare-DCNN. In summary, it is noted that the developed tuned DCNN has been struggling in the efficiency area, even if it has better competitiveness when compared to its baseline machine learning models and state-of-the-art works. The model’s run time, memory use, and number of parameters have all been inefficient. This is because the deep layers employed in DCNN are designed with a primary focus on increasing the accuracy of the FER task.

To put it another way, the goal of DCNN design was to increase FER accuracy rather than the run time and parameter count. To balance the trade-off between efficacy and efficiency, we intend to present a lightweight model in the future.

5.1 Cross-Validation and Cross-Datasets Evaluation Using Image-Level Split

In this sub-section, both cross-validation and cross-dataset evaluation results are provided to offer a comprehensive and reliable assessment of model performance. Together, these evaluation strategies offer a more trustworthy and holistic validation of the proposed approach. While the cross-validation results are provided in Tables 24–26 for all three datasets under consideration, the detailed results across all folds, including ROC curves, are given in Appendix A. The results presented in Tables 24 and 25 report the aggregated 5-fold cross-validation performance of the proposed DCNN on the CK+ and JAFFE, respectively. As observed, like one-holdout validation, the model achieves perfect scores across all evaluation metrics, including precision, recall, F1-score, accuracy, ROC-AUC, and G-Mean for every emotion class.

This consistent performance indicates that the proposed model is highly effective in learning discriminative facial features within the CK+ dataset. The perfect precision and recall values across all classes suggest that the model is capable of correctly identifying all instances without false positives or false negatives, leading to optimal F1-scores and accuracy. Similarly, the ROC-AUC values of 1.0 across all classes confirm perfect separability between emotion classes, while the G-Mean values further demonstrate balanced classification performance without bias toward any specific class. On the other hand, such results can be attributed to the controlled nature of the CK+ dataset, which consists of well-aligned facial expressions with limited variability in pose, illumination, and background. This makes it comparatively less challenging than more unconstrained datasets like FER-2013, as given in Table 26. Additionally, the use of a DCNN combined with careful fine-tuning and appropriate handling of class imbalance contributes to the model’s ability to fully capture the underlying patterns in the data. The cross-dataset results are also provided to better assess the model’s generalization capability in real-world environments.

Table 25 presents the aggregated 5-fold cross-validation results of the proposed DCNN on the JAFFE dataset, demonstrating a strong overall performance across all emotion classes, with high precision, recall, and F1-scores, as well as consistently high ROC-AUC values (≥0.98), indicating effective class separability. This suggests that the model is able to handle the characteristics of the JAFFE dataset, which consists of posed facial expressions captured under controlled conditions from a small, homogeneous group of subjects. The limited variability in terms of pose, illumination, and demographic diversity simplifies the classification task, contributing to the high performance observed across most classes. In particular, happy, neutral, and angry achieve the highest scores, reflecting the model’s ability to learn clear and well-defined expression patterns. However, performance variations are still evident. The fear class shows comparatively lower recall and F1-score, suggesting confusion with visually similar expressions, while disgust and sadness exhibit moderate performance. These challenges can be attributed to subtle inter-class differences and the limited number of samples available for certain expressions, which is a known limitation of JAFFE.

The results presented in Table 26 summarize the aggregated 5-fold cross-validation performance on FER-2013. Overall, the model demonstrates strong discriminative capability across all emotion classes, as evidenced by consistently high ROC-AUC values exceeding 0.95 and even reaching 0.99. This indicates that the model is still effective in distinguishing between different facial expressions despite the inherent noise and variability of the dataset. A closer examination of class-wise metrics reveals that disgust, fear, surprise, and anger achieve the highest performance, with F1-scores ranging from approximately 0.79 to 0.93 and high G-Mean values, reflecting balanced sensitivity and specificity. In particular, disgust attains the best overall performance, which is notable given its typically limited representation in FER-2013. These results suggest that the model is capable of learning discriminative features even for less frequent classes. In contrast, neutral and sad expressions exhibit comparatively lower performance, with F1-scores around 0.69–0.71 and reduced G-Mean values. This can be attributed to the subtle nature of these expressions and their significant overlap with other classes, leading to increased misclassification. Additionally, the happy class shows high precision but relatively lower recall, indicating that while predictions are accurate when made, some instances are not detected. It is also important to note the gap between ROC-AUC and F1-scores for certain classes. While ROC-AUC values remain consistently high, the comparatively lower F1-scores suggest that the model’s decision boundaries are not equally optimal across all classes. This highlights the impact of intra-class variability within the dataset. However, the observed performance variations across classes emphasize the need for further improvements in the future to handle subtle and overlapping expressions.

The cross-dataset evaluation is provided in Tables 27–29, where a trained model must handle new users, devices, or conditions that differ from the training data. It is worth indicating that oversampling was applied to the training set used to train the model in this scenario. As given in Table 27, since not all emotion classes are shared across datasets (e.g., CK+ includes “contempt”), samples belonging to classes presented in both CK+ and JAFFE were only retained, excluding any unique or missing labels like “contempt” or “neutral.” After that, each image was converted to grayscale, resized to 48 × 48, oversampled, and normalized to the [0, 1] range to match the input shape expected by the JAFFE-trained model. Finally, the model predicted probability distributions for all 7 JAFFE classes. These were filtered to match only the shared class indices between CK+ and JAFFE (see Table 28).

A primary hurdle in cross-dataset FER is the lack of label uniformity across benchmarks. To facilitate a rigorous comparison between CK+, JAFFE, and FER-2013, a Shared-Label Inference Protocol was utilized. This approach relies on two main pillars: label alignment and probabilistic filtering. First, the disparate annotation schemes were addressed through Semantic Mapping. While JAFFE and FER-2013 used the standard “Anger” label, other external datasets or metadata might have used terms like “Rage” or “AN”; the proposed DCNN unified these into a single Anger category to ensure compatibility. Because these benchmarks do not share an identical class list, a Dynamic Intersection was built for each evaluation pair. For instance, while FER-2013 and JAFFE support a 7-class spectrum (including “Neutral”), the CK+ training set lacks a “Neutral” category. Conversely, CK+ includes “Contempt”, which is absent in the others. To maintain a consistent baseline, these unique classes were excluded, resulting in a shared 6-class evaluation. Second, Probabilistic Post-Processing was employed during inference to handle varying label distributions without retraining the model. Then, a raw probability distribution was generated across all original training classes while ignoring the values for classes absent in the target dataset. By renormalizing the remaining probabilities, the final prediction was determined via a selection from the shared labels only. This ensures the model is not penalized for failing to predict classes it was never trained on, such as “Neutral” in the case of CK+, nor is the accuracy skewed by categories that do not exist in the target set. Ultimately, this framework preserves the integrity of original annotations while enabling a fair, head-to-head comparison.

As given in Table 29, the model was trained on the CK+ dataset, which includes a series of posed facial expressions captured from multiple subjects. It contains images of 7 primary emotions, including the CK+ specific class “Contempt.” It is worth noting that “Neutral” is not a major category in CK+ emotion classification, and “Contempt” is often not present in other datasets like JAFFE or FER-2013.

5.2 Subject-Level Split Evaluation

The image-level split evaluation has been complemented with the subject-independent evaluation on CK+ and JAFFE datasets, whose class distributions are drawn in Tables 30 and 31. To evaluate the proposed model’s ability to effectively generalize to unseen individuals, the Subject-Independent Stratified Split strategy was implemented. This methodology ensures that the model was tested on subjects it had never encountered during training or validation, thereby preventing identity leakage and providing a realistic estimate of performance on new populations. In this protocol, subjects were strictly separated across splits, ensuring evaluation on unseen identities.

The partitioning process was governed by two simultaneous constraints: (1) Group Constraint: All images belonging to a specific Subject_ID (extracted from the file prefix) were kept together in a single fold. (2) Stratification Constraint: The distribution of the seven emotion classes in CK+ (Anger, Contempt, Disgust, Fear, Happy, Sadness, Surprise) and the seven emotion classes in JAFFE (Angry, Disgust, Fear, Happy, Neutral, Sad, Surprise) were maintained across all subsets to ensure class balance. The 10-fold configuration was employed to achieve an approximate 80/10/10 distribution for training, validation, and testing, respectively. The data was partitioned as follows: Training Set (80%): Comprising 8 out of 10 folds, used for model weight optimization. Validation Set (10%): Comprising 1 fold (the fold was automatically chosen, not manually), which was used for hyperparameter tuning and to prevent overfitting. Test Set (10%): Comprising 1 fold (the fold was automatically chosen, not manually), reserved strictly for the performance evaluation. The “automatically chosen” means that the fold was not manually selected. Instead, this line of code performs the task of obtaining the test indices (test_idx = next(sgkf.split(img_data, labels, groups = groups))). First, the dataset was shuffled in a controlled way (because of random_state = 20). Then, the data was split into 10 equal folds, while considering that each class is fairly represented (stratified), and samples from the same person are not mixed between training and testing. After these 10 folds were created, Python prepared all of them in the background to automatically select the folds for each set. It is worth indicating that both manual selection-based results are also drawn vs. the automatically selected one to draw a fair and comprehensive comparison between both options.

To rigorously confirm the integrity of the split, a set-theoretic intersection analysis was performed. This formal disjointness ensures that the model learns general facial features associated with emotions rather than memorizing the specific physiological characteristics of individual participants. Let S_train, S_val, and S_test represent the sets of unique subject identities in each partition. We verified that: S_train∩S_val=∅, S_train∩S_test=∅, and S_val∩S_test=∅. While Tables 30 and 31 below show the final distributions, Tables 32–35 and Figs. 26–35 draw the results for one-holdout validation. Moreover, both Tables 36 and 37 provide the aggregated performance for 5-fold CV. It is worth noting that all detailed results are given in Appendix A.

Figure 26: CK+ accuracy vs. loss for train and validation.

Figure 27: JAFFE accuracy vs. loss for train and validation.

Figure 28: CK+ accuracy confusion matrix–auto option.

Figure 29: CK+ ROC confusion matrix–auto option.

Figure 30: CK+ accuracy confusion matrix–manual option.

Figure 31: CK+ ROC confusion matrix–manual option.

Figure 32: JAFFE accuracy confusion matrix–auto option.

Figure 33: JAFFE ROC confusion matrix–auto option.

Figure 34: JAFFE accuracy confusion matrix–manual option.

Figure 35: JAFFE ROC confusion matrix–manual option.

The accuracy vs. loss figure for each dataset’s training and validation curves is shown below (refer to Figs. 26 and 27). The number of epochs is 150 across both datasets.

For CK+, the model demonstrates a steep and efficient learning trajectory during the initial epochs, quickly transitioning from low-level feature extraction to high-confidence classification. At Epoch 1, the model begins with a low baseline accuracy of 28.34% and a high loss of 2.5731, indicating a near-random initialization. By Epoch 2, accuracy sharply increases to 64.76%, confirming that the convolutional layers rapidly capture fundamental facial structures and discriminative patterns. Then, between Epochs 3 and 10, the model surpasses 98% training accuracy, reflecting fast convergence toward class separation boundaries. From approximately Epoch 31 onward, training accuracy reaches 100%, with loss approaching near-zero values (0.001–0.0009). This indicates that the model has effectively memorized the training distribution and fully mapped the training manifold. The validation metrics provide a more realistic view of the model’s ability to generalize to unseen subjects. The highest validation accuracy achieved is 96.08% (Epoch 145–146), with corresponding validation loss around 0.47, representing the model’s optimal generalization point. Unlike the training curve, validation accuracy exhibits controlled fluctuations between 80% and 96%, while validation loss shows noticeable oscillations (0.45–1.67). This “stochastic jitter” is expected in subjects. Despite fluctuations in validation loss, the validation accuracy remains consistently high (mostly above 85%), suggesting that decision boundaries remain stable even when prediction confidence varies. Finally, the evaluation on the completely unseen test partition provides the final measure of model performance. The test accuracy was 96.08%, which closely aligns with the peak validation performance, confirming that the model generalizes well to unseen subjects. This consistency indicates that the Stratified Group K-Fold strategy successfully reduced subject leakage and encouraged learning of subject-invariant emotional representations rather than identity-specific features.

Similarly, over JAFFE, the model demonstrates a gradual and steady learning trajectory, transitioning from low initial performance to near-perfect classification on the training set. In epoch 1, the model begins with a low accuracy of 21.33% and a high loss of 3.2208, indicating a near-random prediction state typical of an untrained network. Between Epochs 5 and 20, the model shows consistent improvement, reaching approximately 61.31% accuracy (Epoch 20). This reflects the network learning fundamental facial features such as edges, textures, and basic emotional patterns. From Epoch 20 to Epoch 50, training accuracy rises significantly from 60% to above 85%, accompanied by a steady decline in loss. This phase represents the learning of more discriminative and class-specific features. By epoch 70+, the model surpasses 94% accuracy, indicating strong internal representations of the dataset. From approximately Epoch 100 onwards, the model consistently achieves 99%–100% training accuracy, with loss reducing to 0.0287 by Epoch 150, indicating near-complete fitting of the training data distribution. The validation metrics provide insight into how well the model generalizes to unseen subjects. Validation accuracy starts very low at 9.52% (Epoch 1) and fluctuates significantly in early epochs, reflecting difficulty in generalizing beyond training samples. Between Epochs 15–30, validation accuracy improves to around 28%–38%, showing partial generalization as the model begins capturing more relevant patterns. From Epoch 30 onwards, validation accuracy exhibits high variability, oscillating between 19% and 47%, while validation loss remains relatively high (2.3–5.8 range). This “stochastic jitter” is characteristic of limited dataset size and subject-independent splits, which lead to high intra-class variation in facial expressions. The highest validation accuracy achieved is approximately 52.38% (Epoch 143), with a corresponding loss of 2.3654, representing the model’s best generalization point. The evaluation on the completely unseen test set provides the final measure of performance, which comes with the final test accuracy of 77.27%. The test performance is significantly higher than average validation accuracy, suggesting the model has learned meaningful emotional features and generalization is moderate but not perfect, as some variance exists between validation and test splits’ impact on the overall performance.

As it is observed in Table 32, the perfect scores for anger, contempt, disgust, and sadness now suggest that the model is capturing robust, identity-invariant features, not just memorizing subject-specific cues.

Still, achieving 1.0 across all metrics is rare even in this setup. Happy remains very high (F1 ≈ 0.98), indicating consistent generalization across subjects. The main difficulty lies in: Fear (F1 = 0.80): lower precision suggests confusion with visually similar expressions (often surprise or sadness). Surprise (F1 ≈ 0.94): slightly lower recall indicates missed detections, possibly due to variability in how different people express surprise. However, high ROC and G-mean across all classes reinforce that the model maintains strong class separability even under this stricter protocol. Overall, under subject-independent evaluation, this means we have a promising and likely generalizable model. The remaining errors (mainly in fear and surprise) align with known ambiguities in facial expression recognition, rather than clear methodological concerns. Overall, the model appears promising with the subject-independent evaluation. We further provide the cross-validation in Table 33 as well as in Appendix A.

Surprisingly, the manually selected folds draw completely equivalent results for the image-level split evaluation as given in Table 34. This can be attributed to the manual selection, and the results are just provided to express a comprehensive evaluation for the subject-independent evaluation. With subject-independent evaluation on JAFFE in Table 35, the interpretation shifts quite a bit because JAFFE is small, highly controlled, and identity-limited (only a few subjects). That makes generalization harder but also makes results more sensitive to class imbalance and sample size. The perfect classes (Angry, Fear, Happy) achieve 1.0 across all metrics, suggesting the model can generalize well for these expressions even across subjects. On JAFFE, this is plausible because these expressions are often very prototypical and exaggerated. However, for Disgust and Neutral, Precision/recall/F1 = 0 means the model never correctly predicted these classes. Fortunately, their ROC AUC is still high (0.90 and 0.96), which is a key clue that implies that the model can separate these classes in feature space, yet the decision threshold or class competition is failing, causing systematic misclassification. This often happens with very few samples per subject. On the other hand, Sad and Surprise have mixed performances. While Sad had high recall (1.0) but lower precision, Surprise had very low precision (0.5) but perfect recall. These patterns suggest the model is also biased toward predicting certain expressions too often. Overall, the model shows strong separability (high ROC AUC overall), meaning the learned features are meaningful.

However, the classification layer is unstable, likely due to the very small number of subjects in JAFFE and the few samples per class per subject. This makes the problem open-ended to be further handled in the future. On the other hand, the results in Table 33 using the manual option were better, and this can be attributed to the manual selection nature. Like CK+, these results are just provided to express a comprehensive evaluation for this type of evaluation. To sum up, the observed performance drop in Tables 33–37 and Figs. 28–35 when moving from image-level to subject-independent evaluation is expected and consistent with prior findings on facial expression datasets such as CK+ and JAFFE. In image-level splits, samples from the same subject may appear in both training and testing sets, allowing the model to exploit subject-specific cues and leading to optimistic accuracy estimates. On the other hand, the final findings for the five-fold cross-validation evaluation utilizing the auto option are shown in Tables 36 and 37.

In contrast, subject-independent evaluation enforces generalization to unseen identities and is therefore more challenging and realistic. Despite the decrease in accuracy (4% on CK+ and 22% on JAFFE), the model maintains competitive performance in terms of ROC and G-mean, with comparable AUC values across protocols. This indicates that the model preserves strong discriminative capability and that the drop in accuracy is primarily due to stricter generalization constraints rather than a fundamental limitation of the learned representations. In other words, the ROC scores remain largely consistent, demonstrating that the model continues to effectively separate classes even when absolute accuracy decreases. This suggests that the learned features generalize well and that performance degradation is primarily attributable to the evaluation protocol rather than model deficiency.

More importantly, the discrepancy between accuracy and AUC suggests that performance degradation is primarily due to stricter decision boundaries rather than a loss of discriminative feature quality. Notably, this behavior also highlights that accuracy alone may underestimate model performance under subject-independent constraints, whereas AUC provides a more robust measure of generalization. Lastly, a succinct comparison in Table 38 is drawn for the proposed DCNN against state-of-the-art works on all three datasets.

It is important to note that with the standard train-test split, which is widely used in the literature, the model achieved a test accuracy of 100.00%. We acknowledge that this result, while consistent with several prior works on CK+, is susceptible to the concern the reviewer raises, namely, that the model may have been exposed to subject-specific patterns during training that facilitated near-perfect classification at test time. However, with a subject-independent split, where all images belonging to held-out subjects were entirely excluded from training, the model achieved a test accuracy of 96.08%. This result is meaningfully near-perfect in this scenario and demonstrates that the model does not simply memorize subject-specific appearance features. The gap between the two protocols, 100.00% under the standard split vs. 96.08% under the subject-independent split, is itself evidence that the evaluation is sensitive to subject identity leakage, confirming that the subject-independent protocol functions as an effective safeguard against overfitting to individual subjects. Furthermore, the training logs confirm that under the subject-independent protocol, training accuracy reached 1.0000 while validation accuracy fluctuated between approximately 0.82 and 0.96 across epochs, a pattern consistent with a model that generalizes meaningfully but is not trivially overfitting to the test set. On the other hand, JAFFE is an extremely small dataset, comprising 213 grayscale images from only 10 Japanese female subjects across seven expression categories. Under the subject-independent split, where all images from held-out subjects are reserved for testing, the number of test samples per class becomes critically small, in some cases as few as one or two samples per expression category. When a model fails to correctly classify any instance of a given class in such a small test partition, the resulting precision, recall, and F1-score for that class are mathematically zero, regardless of overall accuracy. This is not solely a reflection of the model’s failure but is also a direct consequence of the extreme class sparsity that arises when applying subject-independent evaluation to a dataset of JAFFE’s scale. This limitation is well-recognized in the facial expression recognition literature, and JAFFE is generally considered unsuitable for rigorous, subject-independent evaluation due to its limited subject diversity. In contrast, the standard train-test split on JAFFE yielded substantially higher and more stable per-class performance, confirming that the model can learn meaningful expression representations from this dataset when sufficient class samples are available during training.

As shown in Table 38, recent FER studies show that model performance is strongly influenced by feature extraction capability, attention mechanisms, transfer learning, preprocessing quality, validation strategy, and dataset characteristics. In general, CK+ and JAFFE results are consistently higher than FER-2013 because they are collected under controlled environments, whereas FER-2013 contains variations in pose, illumination, occlusion, and class imbalance, making it significantly more challenging. Models incorporating attention mechanisms and hybrid deep learning frameworks generally achieved superior performance because they improved discriminative feature extraction and emphasized informative facial regions. For example, Ly et al. (2025) [7], Zhang et al. (2025) [8], and Shen & Xu (2023) [36] secured competitive results due to the integration of attention modules, residual learning, and advanced CNN architectures that enhanced spatial feature representation. Similarly, ensemble and feature fusion approaches such as Xu & Yan (2024) [14], Białek et al. (2023) [38], and Debnath et al. (2022) [42] improved robustness by combining complementary features from multiple models or descriptors. Conversely, models with low or inconsistent accuracy often suffer from insufficient preprocessing, ineffective transfer learning, shallow architectures, excessive complexity, or the absence of validation and fine-tuning procedures. For instance, Krishnasamy et al. (2025) [5], Bhagat et al. (2024) [32], Sarvakar et al. (2023) [40], and Banerjee et al. (2026) [41] achieved lower performance because their architectures either lacked sufficient discriminative capability or were prone to overfitting and poor generalization. Likewise, handcrafted feature-based methods such as Niu et al. (2021) [44] achieved only moderate performance since traditional descriptors like LBP and ORB cannot capture high-level semantic facial representations as effectively as deep learning approaches. Additionally, several studies reported high accuracy on a single dataset but demonstrated limited generalization ability because they were customized for specific datasets without robust validation. This limitation was observed in works such as Veerappan et al. (2024) [29], Xu & Yan (2024) [14], and Lopes et al. (2017) [49]. Fortunately, our developed DCNN model achieves competitive performance under the reported evaluation settings, making it a top-enriching addition to the literature, offering competitive model.

Finally, using one-holdout validation and data split (80% training, 10% validation, and 10% testing), Table 39 presents the effectiveness results, demonstrating the relative effectiveness of the proposed model compared to its baseline models. To ensure a fair comparison, the comparison study is done under an identical protocol. It is also worth stressing that these baselines were selected because they represent: (1) classical deep CNN architectures (VGG16), (2) residual learning networks (ResNet50), (3) multi-scale architectures (InceptionV3), (4) modern advanced CNN designs (ConvNeXt), and (5) the standard conventional CNN baseline to show how inferior it is to our optimized CNN-based structure. This combination ensures a balanced evaluation against both traditional and state-of-the-art approaches in facial expression recognition.

Table 39 compares the performance of a proposed Deep CNN (DCNN), several pre-trained transfer learning models (ResNet50, VGG16, InceptionV3, and ConvNeXt), and the standard CNN on three facial expression recognition datasets: CK+, JAFFE, and FER-2013. Overall, the experimental results demonstrate that the proposed DCNN is highly competitive across all datasets. This demonstrates the model’s ability to learn highly discriminative facial expression features under controlled and uncontrolled conditions. The superior performance of the proposed DCNN could be attributed to efficient feature extraction, combined with optimized hyperparameter tuning. On the other hand, transfer learning models do not always guarantee superior FER performance, especially on small or domain-specific datasets. Models pre-trained on ImageNet learn generic object features, which may not effectively capture the subtle facial muscle movements, emotion-specific patterns, and low-resolution facial textures. Moreover, the large depth and complexity of models such as ResNet50, InceptionV3, and ConvNeXt may require significantly larger datasets to enable effective feature generalization and stable learning. Compared with other pre-trained models, VGG16 demonstrated better stability and generalization, likely because its simpler architecture adapts more effectively to facial expression datasets. Lastly, the standard CNN outperformed deeper and more complex pre-trained models. However, FER-2013 remains the most challenging dataset due to its unconstrained nature, explaining the reduction in accuracy for all models.

5.3 Work’s Contribution

The key contribution of this work is not just in developing an optimized DCNN for facial emotion recognition but also in the proposed preprocessing pipeline that distinguishes itself from existing approaches. A holistic integration of data handling, model tuning, and systematic and comprehensive evaluation strategy using image-level and subject-independent splits is followed. Unlike many prior works that address FER challenges in isolation, the proposed model simultaneously tackles class imbalance, dataset bias, and generalization limitations by using three diversified datasets and employing an exclusive training-phase oversampling strategy, thereby avoiding data leakage and overfitting, an issue mostly overlooked in the literature. Additionally, unlike the bare DCNN, the effective training of the tuned DCNN, joined with developed effective pre-processing, enables it to capture subtle and high-level emotional patterns across varying facial expressions. Furthermore, the study goes beyond conventional single-dataset validation by using three datasets and incorporating subject-independent and image-level evaluation, joined with 5-fold cross-validation and cross-dataset evaluation, offering a more realistic assessment of model robustness across diverse data distributions. In contrast to most FER models that report high accuracy on specific datasets but suffer from limited generalization, the proposed approach demonstrates consistent and competitive performance across CK+, JAFFE, and FER-2013 datasets, highlighting its practical applicability. This combination of balanced data preprocessing, optimized deep feature learning, and comprehensive evaluation represents a progressive addition to the FER research. All these facts are already supported by results drawn in the results section vs. several models, including classical ML models, CNN-based structure, and state-of-the-art models.

5.4 Work’s Limitations

Despite the competitive performance of the developed DCNN, several limitations should be acknowledged. First, the model achieves high accuracy at the cost of computational efficiency, which may limit its practicality in resource-constrained or real-time applications. Second, the current approach relies on qualitative descriptions of facial regions, similar to existing methods, and does not incorporate an explicit region-selection mechanism.

6  Conclusions and Future Work

This paper presented a robust approach for facial emotion recognition based on a deep convolutional neural network, with particular attention to handling class imbalance and ensuring fair evaluation. The proposed DCNN was carefully fine-tuned and configured to efficiently extract high-level features from image data and trained to manage face expression changes while accounting for class imbalance. By applying oversampling exclusively to the training data, the proposed method effectively mitigates imbalance while avoiding data leakage and overfitting. The experimental results on CK+, JAFFE, and FER-2013 demonstrate that the model achieves strong and consistent performance across multiple evaluation metrics. According to the results secured, the proposed model has demonstrated highly competitive performance against its rivals across all datasets in a number of evaluation metrics. The use of both 5-fold cross-validation and cross-dataset evaluation provides a comprehensive assessment, confirming not only the model’s stability within individual datasets but also its ability to generalize across different data distributions. Furthermore, comparisons with classical machine learning models and state-of-the-art methods highlight the competitiveness of the proposed approach.

Finally, a comparison study between the proposed DCNN and state-of-the-art baselines is conducted. The results confirm that the proposed DCNN provides superior facial expression recognition performance compared with standard CNN and transfer learning approaches. Its high accuracy and strong generalization indicate that the model successfully captures discriminative emotional features while avoiding excessive overfitting. In contrast, pre-trained models such as ResNet50, InceptionV3, and ConvNeXt show limited adaptation to FER datasets, particularly FER-2013, due to domain mismatch and increased architectural complexity. However, unfortunately, the model’s efficacy comes at the expense of its efficiency. Therefore, the future work is dedicated to finding a lightweight model while focusing on improving generalization through domain adaptation and incorporating additional modalities to further enhance recognition performance in real-world scenarios.

Acknowledgement: Not applicable.

Funding Statement: The authors received no specific funding for this study.

Author Contributions: The authors confirm their contribution to the paper as follows: Eman Attallah H. Aljabarti: Conceptualization; methodology; validation; formal analysis; investigation; writing—original draft preparation; writing—review and editing; visualization; Ainuddin Wahid Abdul Wahab: supervision; project administration; validation; formal analysis; Mohd Yamani Idna Idris: validation; formal analysis; investigation. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: Data are openly available in a public repositories as follows: CK+: Kaggle, “CKPLUS: CK+ dataset for facial expression recognition”, 2018, https://www.kaggle.com/datasets/shawon10/ckplus. JAFFE: https://zenodo.org/records/14974867. FER-2013: https://www.kaggle.com/datasets/prilia/fer2013pluscleanedaugmballanced1.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest.

Appendix A

In this section, all detailed results for 5-cv across all three datasets are provided.

CK 5CV REPORT-Image level Split

JAFFE 5CV REPORT-Image level Split

FER-2013 5CV REPORT-Image level Split

CK+ 5CV REPORT-Subject-level Split

JAFFE 5CV REPORT-Subject-level Split

JAFFE 5CV RESULT

References

1. Rehman A, Mujahid M, Elyassih A, AlGhofaily B, Ali Omer Bahaj S. Comprehensive review and analysis on facial emotion recognition: performance insights into deep and traditional learning with current updates and challenges. Comput Mater Contin. 2025;82(1):41–72. doi:10.32604/cmc.2024.058036. [Google Scholar] [CrossRef]

2. Maes JHR, Scheper AR, Hermans D, Vissers CTWM. Improving facial emotion recognition in children with developmental language disorder: intentional or incidental training? Res Dev Disabil. 2026;169(3):105221. doi:10.1016/j.ridd.2026.105221. [Google Scholar] [PubMed] [CrossRef]

3. Kırbız S. Improving facial emotion recognition through dataset merging and balanced training strategies. J Frankl Inst. 2025;362(7):107659. doi:10.1016/j.jfranklin.2025.107659. [Google Scholar] [CrossRef]

4. Patel JB, Bhatt GT, Mangrolia JR. Facial expression recognition using machine learning techniques. Int J Adv Eng Res Dev. 2020;1(6):259–66. doi:10.1109/idea49133.2020.9170706. [Google Scholar] [CrossRef]

5. Krishnasamy N, Zade N, Khambholia D, Henry R, Gupte A. Ensemble deep learning framework for hybrid facial datasets using landmark detection: state-of-the-art tools. J Comput Cogn Eng. 2025;4(4):535–45. doi:10.47852/bonviewjcce52024451. [Google Scholar] [CrossRef]

6. Umer S, Kumar Rout R, Tiwari S, Ali AlZubi A, Mutared Alanazi J, Yurii K. Human-computer interaction using deep fusion model-based facial expression recognition system. Comput Model Eng Sci. 2023;135(2):1165–85. doi:10.32604/cmes.2022.023312. [Google Scholar] [CrossRef]

7. Ly D, Tran N, Nguyen HQ, Nguyen T, Nguyen L, Nguyen H. A graph attention network-enhanced approach to facial expression recognition using hybrid pixel-geometry features. Int J Intell Eng Syst. 2025;18(5):571–84. doi:10.22266/ijies2025.0630.40. [Google Scholar] [CrossRef]

8. Zhang D, Ma W, Shen Z, Ma Q. CAGNet: a network combining multiscale feature aggregation and attention mechanisms for intelligent facial expression recognition in human-robot interaction. Sensors. 2025;25(12):3653. doi:10.3390/s25123653. [Google Scholar] [PubMed] [CrossRef]

9. Chen D, Cao Y, Cheng X. Research on facial expression recognition method based on improved ConvNeXt. IET Image Process. 2025;19(1):e70118. doi:10.1049/ipr2.70118. [Google Scholar] [CrossRef]

10. Sowmya B, Meeradevi, Alex SA, Kanavalli A, Supreeth S, Shruthi G, et al. Machine learning model for emotion detection and recognition using an enhanced convolutional neural network. J Integr Sci Technol. 2024;12(4):786. doi:10.62110/sciencein.jist.2024.v12.786. [Google Scholar] [CrossRef]

11. Abdulazeez AM, Ageed ZS. Face emotion recognition based on machine learning: a review. Int J Inform Inf Syst Comput Eng. 2024;5(1):53–87. doi:10.34010/injiiscom.v5i1.12145. [Google Scholar] [CrossRef]

12. Yang J, Qian T, Zhang F, Khan SU. Real-time facial expression recognition based on edge computing. IEEE Access. 2021;9:76178–90. doi:10.1109/ACCESS.2021.3082641. [Google Scholar] [CrossRef]

13. Srivastav S, Khare A. Deep learning based automatic facial emotion recognition. Int J Intell Syst Appl Eng. 2024;12(11s):619–24. [Google Scholar]

14. Xu G, Yan WQ. Facial emotion recognition using ensemble learning. In: Deep learning, reinforcement learning, and the rise of intelligent systems. Hershey, PA, USA: IGI Global; 2024. p. 146–58. doi:10.4018/979-8-3693-1738-9.ch007. [Google Scholar] [CrossRef]

15. Liu C, Liu X, Chen C, Zhou K. Deep global multiple-scale and local patches attention dual-branch network for pose-invariant facial expression recognition. Comput Model Eng Sci. 2024;139(1):405–40. doi:10.32604/cmes.2023.031040. [Google Scholar] [CrossRef]

16. Canal FZ, Müller TR, Matias JC, Scotton GG, de Sa ARJ, Pozzebon E, et al. A survey on facial emotion recognition techniques: a state-of-the-art literature review. Inf Sci. 2022;582(1):593–617. doi:10.1016/j.ins.2021.10.005. [Google Scholar] [CrossRef]

17. Kumar A, Sindhwani M, Sachdeva S. Facial emotion recognition (FER) with deep learning algorithm for sustainable development. In: Sustainable engineering: concepts and practices. Berlin/Heidelberg, Germany: Springer; 2024. p. 415–34. doi:10.1007/978-3-031-47215-2_24. [Google Scholar] [CrossRef]

18. Angel JS, Andrushia AD, Neebha TM, Accouche O, Saker L, Anand N. Faster region convolutional neural network (FRCNN) based facial emotion recognition. Comput Mater Contin. 2024;79(2):2427–48. doi:10.32604/cmc.2024.047326. [Google Scholar] [CrossRef]

19. An Y, Lee J, Bak E, Pan S. Deep facial emotion recognition using local features based on facial landmarks for security system. Comput Mater Contin. 2023;76(2):1817–32. doi:10.32604/cmc.2023.039460. [Google Scholar] [CrossRef]

20. Li S, Deng W. Deep facial expression recognition: a survey. IEEE Trans Affect Comput. 2022;13(3):1195–215. doi:10.1109/taffc.2020.2981446. [Google Scholar] [CrossRef]

21. Malikovich KM, Ugli ISZ, O’Ktamovna DL. Problems in face recognition systems and their solving ways. In: Proceedings of the 2017 International Conference on Information Science and Communications Technologies (ICISCT); 2017 Nov 2–4; Tashkent, Uzbekistan. doi:10.1109/ICISCT.2017.8188594. [Google Scholar] [CrossRef]

22. Bakariya B, Singh A, Singh H, Raju P, Rajpoot R, Mohbey KK. Facial emotion recognition and music recommendation system using CNN-based deep learning techniques. Evol Syst. 2024;15(2):641–58. doi:10.1007/s12530-023-09506-z. [Google Scholar] [CrossRef]

23. Mehendale N. Facial emotion recognition using convolutional neural networks (FERC). SN Appl Sci. 2020;2(3):446. doi:10.1007/s42452-020-2234-1. [Google Scholar] [CrossRef]

24. Karnati M, Seal A, Bhattacharjee D, Yazidi A, Krejcar O. Understanding deep learning techniques for recognition of human emotions using facial expressions: a comprehensive survey. IEEE Trans Instrum Meas. 2023;72(1):5006631. doi:10.1109/TIM.2023.3243661. [Google Scholar] [CrossRef]

25. Fard AP, Mahoor MH. Ad-corre: adaptive correlation-based loss for facial expression recognition in the wild. IEEE Access. 2022;10(11):26756–68. doi:10.1109/ACCESS.2022.3156598. [Google Scholar] [CrossRef]

26. Abidi SMH, Hassan SA, Raza SM, Beliatis MJ. Advances in face recognition: a comprehensive review of feature extraction and dataset evaluation. Electronics. 2026;15(2):338. doi:10.3390/electronics15020338. [Google Scholar] [CrossRef]

27. Yalçin N, Alisawi M. Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques. Heliyon. 2024;10(20):e38913. doi:10.1016/j.heliyon.2024.e38913. [Google Scholar] [PubMed] [CrossRef]

28. Han W. Study on interactive experience design of AI in digital display of museums. Int J High Speed Electron Syst. 2026;35(5):2540658. doi:10.1142/s0129156425406588. [Google Scholar] [CrossRef]

29. Veerappan MK, Baskaran AP, Venkatachalam SB, Ramanujam RAM, Narayanan LK. Facial emotion recognition system using VGG neural network. AIP Conf Proc. 2024;3040:120024. doi:10.1063/5.0181752. [Google Scholar] [CrossRef]

30. Ezati A, Dezyani M, Rana R, Rajabi R, Ayatollahi A. A lightweight attention-based deep network via multi-scale feature fusion for multi-view facial expression recognition. arXiv:2403.14318. 2024. [Google Scholar]

31. Gupta BB, Gaurav A, Chui KT, Arya V. Deep learning-based facial emotion detection in the metaverse. In: Proceedings of the 2024 IEEE International Conference on Consumer Electronics (ICCE); 2024 Jan 6–8; Las Vegas, NV, USA. doi:10.1109/ICCE59016.2024.10444217. [Google Scholar] [CrossRef]

32. Bhagat D, Vakil A, Gupta RK, Kumar A. Facial emotion recognition (FER) using convolutional neural network (CNN). Procedia Comput Sci. 2024;235(5):2079–89. doi:10.1016/j.procs.2024.04.197. [Google Scholar] [CrossRef]

33. Tshibangu R, Tapamo JR. Improving facial emotional recognition using convolution neural network with minimal layer. In: Electronics, communications and networks. Amsterdam, The Netherlands: IOS Press; 2024. doi:10.3233/faia231252. [Google Scholar] [CrossRef]

34. Agung ES, Rifai AP, Wijayanto T. Image-based facial emotion recognition using convolutional neural network on emognition dataset. Sci Rep. 2024;14(1):14429. doi:10.1038/s41598-024-65276-x. [Google Scholar] [PubMed] [CrossRef]

35. Wang X, Wang Y, Zhang D. Complex emotion recognition via facial expressions with label noises self-cure relation networks. Comput Intell Neurosci. 2023;2023(1):7850140. doi:10.1155/2023/7850140. [Google Scholar] [PubMed] [CrossRef]

36. Shen T, Xu H. Facial expression recognition based on multi-channel attention residual network. Comput Model Eng Sci. 2023;135(1):539–60. doi:10.32604/cmes.2022.022312. [Google Scholar] [CrossRef]

37. Anwarul S, Choudhury T, Dahiya S. A novel hybrid ensemble convolutional neural network for face recognition by optimizing hyperparameters. Nonlinear Eng. 2023;12(1):20220290. doi:10.1515/nleng-2022-0290. [Google Scholar] [CrossRef]

38. Białek C, Matiolański A, Grega M. An efficient approach to face emotion recognition with convolutional neural networks. Electronics. 2023;12(12):2707. doi:10.3390/electronics12122707. [Google Scholar] [CrossRef]

39. Shahzad HM, Masood Bhatti S, Jaffar A, Rashid M. A multi-modal deep learning approach for emotion recognition. Intell Autom Soft Comput. 2023;36(2):1561–70. doi:10.32604/iasc.2023.032525. [Google Scholar] [CrossRef]

40. Sarvakar K, Senkamalavalli R, Raghavendra S, Kumar JS, Manjunath R, Jaiswal S. Facial emotion recognition using convolutional neural networks. Mater Today Proc. 2023;80(2):3560–4. doi:10.1016/j.matpr.2021.07.297. [Google Scholar] [CrossRef]

41. Banerjee D, Gothwal P, Biswas AK. ExpressNet-MoE: a hybrid deep neural network for emotion recognition. Mach Learn Appl. 2026;23(1):100830. doi:10.1016/j.mlwa.2025.100830. [Google Scholar] [CrossRef]

42. Debnath T, Reza MM, Rahman A, Beheshti A, Band SS, Alinejad-Rokny H. Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity. Sci Rep. 2022;12(1):6991. doi:10.1038/s41598-022-11173-0. [Google Scholar] [PubMed] [CrossRef]

43. Kedari P, Kapile M, Kadole D, Jaikar S. Face emotion detection using deep learning. In: Proceedings of the 2021 2nd International Conference on Advances in Computing, Communication, Embedded and Secure Systems (ACCESS); 2021 Sep 2–4; Ernakulam, India. doi:10.1109/ACCESS51619.2021.9563343. [Google Scholar] [CrossRef]

44. Niu B, Gao Z, Guo B. Facial expression recognition with LBP and ORB features. Comput Intell Neurosci. 2021;2021(1):8828245. doi:10.1155/2021/8828245. [Google Scholar] [PubMed] [CrossRef]

45. Shuvo MNR, Ashiqul M, Shamsojjaman M, Akter S, Hasan S, Khatun T. Recognizing human emotions from eyes and surrounding features: a deep learning approach. Int J Adv Comput Sci Appl. 2021;12(3):386–94. doi:10.14569/ijacsa.2021.0120346. [Google Scholar] [CrossRef]

46. Pranav E, Kamal S, Chandran CS, Supriya MH. Facial emotion recognition using deep convolutional neural network. In: Proceedings of the 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS); 2020 Mar 6–7; Coimbatore, India. doi:10.1109/icaccs48705.2020.9074302. [Google Scholar] [CrossRef]

47. Jain DK, Shamsolmoali P, Sehdev P. Extended deep neural network for facial emotion recognition. Pattern Recognit Lett. 2019;120:69–74. doi:10.1016/j.patrec.2019.01.008. [Google Scholar] [CrossRef]

48. Zhang T, Zheng W, Cui Z, Zong Y, Li Y. Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybern. 2019;49(3):839–47. doi:10.1109/tcyb.2017.2788081. [Google Scholar] [PubMed] [CrossRef]

49. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T. Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 2017;61(12):610–28. doi:10.1016/j.patcog.2016.07.026. [Google Scholar] [CrossRef]

50. Lyons M, Kamachi M, Gyoba J. The Japanese female facial expression (JAFFE) dataset. Zenodo. 1998. doi:10.5281/zenodo.3451523. [Google Scholar] [CrossRef]