Adaptive Driver State Monitoring with Temporal Reasoning and Risk Estimation for Safe Transportation

1  Introduction

Recent advances in computer vision have enabled the development of decision-support systems that utilize facial features to enhance public safety, particularly by addressing the increasing incidence of road accidents [1–3]. With the rapid development of urban infrastructure, modern cities are now interconnected through extensive road networks and advanced transportation systems, enabling millions of people to travel daily with greater convenience. However, despite these technological and infrastructural advancements, Road Traffic Accidents (RTAs) continue to be a leading cause of fatalities and injuries worldwide [4]. The high casualty rate is alarming, with many survivors left with permanent disabilities that affect their lives long-term [4]. According to Bej et al. [5], citing the Global Status Report on road safety 2018, traffic crashes result in nearly 1.3 million fatalities globally, with millions more sustaining serious injuries and facing lasting health issues [6]. RTAs have recently been identified as the eighth leading cause of death, and projections suggest they may become the seventh leading cause in the near future [4]. Driver-related factors, such as attentiveness and mental and physical condition, significantly influence road accidents [7]. Issues like fatigue, stress, drowsiness, alcohol, drugs, and distractions (such as texting or eating while driving) contribute significantly to the occurrence of accidents [8]. Drowsiness, in particular, refers to a state where drivers become unaware of their surroundings and suffer from reduced concentration, increasing the risk of crashes. According to Higgins et al. [9], citing data from the National Highway Traffic Safety Administration (NHTSA) Fatality Analysis Reporting System, there were 846 reported fatalities resulting from drowsy driving in 2014, accounting for 2.6% of all motor vehicle deaths. In 2021, RTAs involving distracted drivers resulted in 3522 fatalities and approximately 362,415 injuries [10]. Sun and Zhang [11] reported, based on a report from the Indian Ministry of Road Transport and Highways, that India witnessed 461,312 road accidents in 2022, leading to 168,491 fatalities and 443,366 injuries. Research indicates that drowsy driving may account for nearly 20% of road accidents, with around 50% of those incidents leading to severe injuries or fatalities. Drowsiness negatively impacts response time, concentration, and visual focus, all of which are critical for safe driving. While drivers often recognize their own fatigue, it is essential for them to make responsible decisions about whether to rest or continue driving.

Age-related factors are also considered the cause of road accidents, as drivers who are either too old or too young may lack the essential skills and judgment needed to operate vehicles safely [1,12]. A recent traffic accident near Seoul City Hall resulted in the deaths of around nine people and left four others injured, which started the debate over elderly drivers. The driver, a 68-year-old man, was apprehended at the scene [13]. Underage driving, defined as driving without a license before the age of sixteen, poses a serious threat to traffic safety. In Qatar, a tragic incident titled “Three Young Brothers Killed in a Road Crash” highlights the dangers associated with young drivers aged 6 to 16 who were operating a vehicle illegally, leading to a fatal accident. Similarly, Shaaban and Hassan [14] highlighted the alarming trend of underage drivers causing devastating collisions that result in the deaths of entire families, citing the report "Driver Who Struck, Killed Filipino Family Had No License". To address these issues, researchers have proposed various solutions, including wearable sensors for monitoring driver heart rates, Electroencephalography (EEG), and temperature detection, as well as vision sensors to tackle the problem [4,15,16]. Garrosa et al. [17] proposed a lightweight wearable device for continuous driver motion monitoring and physiological state. They implement the Inertial Measurement Units (IMUs) sensors at different body parts of a driver, along with Electrocardiography (ECG) and body temperature signals. Their approach provided reliable solutions for early warning signals to a driver, preventing road accidents. However, achieving effective real-time driver monitoring poses challenges in balancing conflicting demands such as high accuracy, compact model size, optimal learning parameters, and fast inference speed. Existing approaches that rely on frame-level driver state classification often fail to capture temporal dependencies, limiting their ability to assess behavior patterns over time and provide adaptive interventions. To address these limitations, realistic approaches to driver age and distraction detection-based methods are essential to safeguard both underage and elderly drivers and to assess their mental and physical condition.

This work proposes an efficient deep learning model for driver distraction monitoring, providing an innovative approach to enhance public safety and promote a secure and well-connected urban environment. The proposed framework employs an EfficientNetV2-S Convolutional Neural Network (CNN) combined with a Driver-Monitoring Coordinate Attention (DM-CA) for driver age estimation and drowsiness detection using facial feature analysis. Our primary goal is to deliver real-time warning messages to the driver regarding their condition while driving on the road. The major contributions of our work are as follows:

•   We propose an efficient deep learning framework based on EfficientNet for driver state detection and age classification. EfficientNet is employed due to its ability to extract rich and discriminative features while maintaining a lightweight architecture and low computational cost. This makes the framework suitable for real-time driver monitoring systems and practical deployment in intelligent transportation systems.

•   To further enhance feature representation, a DM-CA mechanism is integrated into the network to capture long-range spatial dependencies while preserving precise positional information. This attention mechanism enables the model to focus on critical facial regions such as the eyes and facial contours that are closely associated with driver fatigue and age-related characteristics, thereby improving feature refinement and overall detection performance.

•   We propose an adaptive driver state monitoring framework that performs temporal reasoning across sequential frames, enabling the system to analyze driver behavior over time rather than relying on a single image. This approach improves the reliability of fatigue and distraction detection while providing dynamic risk estimation for safer transportation.

•   The effectiveness of the proposed framework is evaluated on four different datasets, with extensive comparisons against state-of-the-art methods. Experimental results demonstrate that the proposed model consistently achieves better performance in terms of accuracy and efficiency, highlighting its effectiveness for driver distraction or driver fatigue detection and age classification.

•   A comprehensive ablation study is conducted to analyze the impact of different modules in the proposed model. The proposed model performance is evaluated using statistical analysis with mean standard deviation, and K-Fold Cross Validation to ensure the model generalization and robustness capabilities. Furthermore, the cross-validation performance, where the model is trained on one dataset and test on another shows the proposed model effectiveness generalization capability.

The remainder of the work is organized as follows: Section 2 presents related work in our targeted domain, offering a brief overview of driver status and age prediction using various methods. Section 3 explains our method in detail. Section 4 includes the dataset, evaluation metrics, and a discussion of the results. Finally, Section 5 concludes the paper with future challenges and possible solutions.

2  Related Work

This section primarily discusses driver attentiveness and age-related methods using computer vision technology. We begin with an overview of various existing approaches for assessing physical and mental states, such as drowsiness and fatigue levels, as well as driver attentiveness. We then delve into age prediction methods.

2.1 Driver Distraction Detection

Driver distraction detection is one of the primary research topics in the modern era of AI, with most approaches relying on face-based analysis using feature-based, machine learning-based, deep learning-based, and hybrid techniques [18–20]. Assessing driver state is considered one of the most active areas of research, as it captures driver conditions such as sleeping, yawning, talking with others, taking hands off the steering wheel, and being angry, sad, or drowsy, among others [19]. For example, the DriCare approach, presented in [21], utilizes facial landmark key points for face detection, eye blinking, eye closure, and yawning recognition. Attivissimo et al. [22] developed a system for detecting eye blinks, which is essential for identifying driver drowsiness. Zhao et al. [23] employed texture features and landmark key points to classify facial states, focusing on specific facial regions such as eyes, mouth, and nose, for fatigue detection. In another study [24], the authors analyzed bus driver drowsiness by observing their position and window, using an oblique view to estimate the percentage of eyelid closure (PERCLOS) [24]. Massoz et al. [25] introduced a novel multi-modality dataset called DROZY for drowsiness detection, significantly enhancing research in this field. Similarly, the authors of [26] presented two distinct video databases of drivers, capturing various facial characteristics, including the presence or absence of sunglasses and a range of ethnicities. This approach achieved a 60% accuracy rate for detecting yawning when the camera was positioned on the dashboard [26]. To counter noise and scale variations, a new feature extraction descriptor, called Multi-scale Histograms of Principal Oriented Gradients (MHPOG), was proposed [27]. The researchers compared results from two different databases in terms of time complexity and accuracy, yielding promising outcomes. Additionally, studies have investigated the ratios of eye closure, eye aspect, and landmark key points [28]. They analyzed blink patterns using multi-layer perceptron, random forest, and Support Vector Machine (SVM) classifiers. Anitha et al. [29] focused on enhancing face detection performance and tracking driver eye movements in input videos. Jabbar et al. [30] proposed a deep learning method for drowsiness detection. The main contribution of their work is the compression of a heavy deep learning model to a lightweight model, which can be implemented on Android applications.

Despite the advancements, these approaches have limitations, including reliance on traditional machine learning methods, the need for extensive feature engineering, and being less robust, labor-intensive, and time-consuming. Consequently, there is a shift toward investigating deep learning-based approaches. Several researchers have utilized deep learning models to address the limitations of traditional machine learning, achieving improved performance. For instance, Verma and Choudhary [31] employed a VGG16 model for a driver monitoring system that uses facial expression recognition to enhance attentiveness while driving. Another approach [32] implemented a face-cropping mechanism followed by a deep learning model to classify individuals as either sleep-deprived or rested, deploying it on Android phones. Ghoddoosian et al. [33] proposed a real-world database containing 30 h of video focused on drowsiness. Their method focused on a Hierarchical Multi-scale Long Short-Term Memory (HMLSTM) network for blink features detection. A machine learning framework was developed to predict mental fatigue in industrial control room operators using facial data and an improved Karolinska Sleepiness Scale [34]. Experimental results show that the LightGBM model achieved the best performance with 93.33% accuracy, demonstrating its potential for real-time fatigue monitoring in industrial environments. Recurrent Neural Networks (RNNs) play a crucial role in detecting driver states, including normal blinking and the onset of sleep, by learning from a sequence of features [21,35–37]. Quddus et al. [38] proposed a convolutional LSTM-based model to detect drowsiness by capturing eye movements, yielding promising results. Magan et al. [39] employed an EfficientNetB0 model to assess drivers’ drowsiness levels. This approach combines CNN with RNN and fuzzy logic support to evaluate fatigue levels, achieving an average accuracy of 55%–65% while reducing computational complexity. Similarly, a deep learning model combined with functional near-infrared spectroscopy was proposed in [40] to create a passive brain-computer interface for drowsiness detection. Furthermore, CNN-based methods were investigated to analyze functional brain maps, achieving higher performance. The research identified thirteen specific channels that exhibited heightened activity during drowsiness [40]. Zhao et al. [41] presented a data-driven approach for fatigue detection using a multi-modal fusion of ECG and video signals. Recently, Hijji et al. [1] introduced a three-layer CNN for efficient and effective driver drowsiness detection. They conducted experiments using a self-created dataset consisting of five different classes: active, angry, sad, sleepy, and yawning, obtaining optimal results with higher accuracy and reduced model complexity. However, their self-curated dataset comprises videos with similar frames, which may hinder performance in real-time monitoring. Wang et al. [42] presented a large-scale benchmark dataset consisting of 21 distracted classes and one active class, captured from an RGB and Infrared (IR) camera from four different views in five different vehicles. They employed six widely used baseline architectures ResNet50, MobileNetV3, ShuffleNetV2, SqueezeNet, GhostNetV1, and EfficientNetB0 achieving promising performance on both RGB and IR modalities. Similarly, the method presented in [43] utilized a faster RCNN for hands and face detection, surpassing the previous method and achieving an accuracy of 94.2%. Zhao et al. [44] introduced a hidden conditional random fields model for smartphone usage detection by analyzing features of the face, mouth, and hand from images captured by a mounted camera on the vehicle dashboard. In another approach, the researcher attempts to detect seven tasks performed by drivers on a daily basis, including normal driving, using a smartphone (answering or texting), checking mirrors (rear, right, and left), and setting up a video device in the vehicle. They utilized a Kinect camera that captures both depth and color information. The authors extracted 42 different features and applied random forests for classification. Wang et al. [45] developed a real-time ECG-based driver fatigue detection method using sample entropy. Horng et al. [18] presented a real-time mouth-state detection system that combines Facial Landmark Detection with an optimized BiGRU–CLPSO model for deployment on edge devices. Their approach achieved 96.57% accuracy and 98.25% precision, outperforming CNN, LSTM, MLP, and XGBoost while demonstrating strong generalization and efficiency for real-time applications. Thaher et al. [20] summarizes face detection methods for occluded faces, including traditional, deep learning, and hybrid approaches. It highlights the effectiveness of deep learning and Transformer-based models while identifying key challenges in robustness and efficiency. Furthermore, Khan et al. [46] introduced EfficientNetB0 integrated with channel attention mechanisms for real-time driver behavior monitoring. They conducted experiments on two benchmark datasets: the State Farm Distracted Driver Detection (SFD3) and AUC Distracted Driver (AUCD2), achieving promising results. However, while these models address the limitations of prior approaches, their higher computational requirements, lower inference speed, and the demand for large amounts of data for model training limit real-time deployment on resource-constrained Internet-of-Things (IoT) devices.

2.2 Age Estimation

Facial image analysis plays a crucial role in age classification, which is a rapidly growing field of study due to its wide range of applications [47], including human-computer interaction, soft biometrics, security, police investigations, and age-oriented commercial advertising. Numerous research studies have been published in this area, focusing on both traditional features [48–51] and deep learning [52–56]. The deep learning model outperforms the handcrafted method with better accuracy. In [57], researchers introduced a deep learning model for age classification utilizing the IMDB-WIKI dataset. The study presented in [58], employed a Deep Regression Forests (DRF) approach for age classification, utilizing an end-to-end method for feature extraction and classification. In [59], a Directed Acyclic Graph (DAG)-CNN was used for age prediction, incorporating multi-scale feature extractions and fusing their associated scores with multiple classifiers. In [60], the authors proposed a novel approach to age prediction that takes facial emotion into account. The method involves the concurrent learning of age and emotion by utilizing a graphical model with a hidden layer to explore their correlation. Additionally, Liu et al. [61] introduced an ordinal deep learning mechanism for learning features related to age estimation and face representation. Our literature review revealed that existing models for age and fatigue detection are resource-intensive and require further performance improvement. Consequently, we proposed a lightweight deep learning model with a DM-CA to enable efficient and effective fatigue and age detection.

3  The Proposed Framework

In this section, we provide a brief overview of the proposed method for driver state and age detection. The framework captures live video streams to assess driver age and their state in real time, raising alerts when it detects signs of fatigue, as illustrated in Fig. 1.

Figure 1: The generic overview of the proposed model.

3.1 Backbone Features Extractor

We employed EfficientNetV2-S as the backbone for feature extraction, enhanced by a DM-CA. EfficientNetV2-S was selected due to its impressive performance on the ImageNet dataset, achieving 83.9% Top-1 and 96.7% Top-5 accuracy while maintaining a compact model size and high inference speed, making it suitable for edge IoT applications. EfficientNet is a family of models optimized for parameter efficiency and floating-point operations (FLOPs), as proposed by Tan and Le [62]. The baseline model, EfficientNet-B0, is derived through Neural Architecture Search (NAS), which aims to maximize model accuracy within predefined computational constraints. After discovering EfficientNet-B0, a compound scaling strategy is applied to uniformly scale the network’s depth, width, and input resolution:

Depth=αϕ,Width=βϕ,Resolution=γϕ(1)

subject to the constraint

α⋅β2⋅γ2≈2(2)

where ϕ is a compound scaling coefficient. This strategy generates a family of models ranging from EfficientNet-B1 to EfficientNet-B7. However, EfficientNet encounters challenges when trained on large-resolution images, resulting in slow training speeds. Additionally, the extensive use of depthwise convolutions (DWCs) in the early layers results in reduced efficiency on modern hardware accelerators. In [63], a smaller and more efficient network named EfficientNetv2 was proposed, characterized by the utilization of the Fused Mobile Inverted Bottleneck Convolution (FMBC) mechanism and a progressive learning approach. In the FMBC block, the expansion and depthwise convolution operations are fused into a single convolution, which can be formulated as:

Xf=σ(BN (Convk×k(X)))(3)

where X denotes the input feature map and σ(⋅) represents the SiLU (Swish) activation function as given in Eq. (4).

SiLU(x)=x⋅σ(x)(4)

EfficientNetV2 consists of multiple stages composed of Mobile Inverted Bottleneck Convolution (MBC) and FMBC blocks, as visualized in Fig. 2a,b, respectively. The MBConv block, originally introduced in MobileNetV2, follows an inverted residual structure and integrates the Squeeze-and-Excitation (SE) attention mechanism, given an input feature map.

X∈RH×W×C(5)

Figure 2: Basic building block of attentions, (a) MBConv, (b) Fused-MBConv, and (c) DM-CA.

The MBConv block first applies channel expansion using a 1×1 convolution as given by Eq. (6).

Xe=σ(Conv1×1(X)),Ce=t⋅C(6)

where t is the expansion ratio. Subsequently, a depthwise convolution is applied:

Xdw(i,j,c)=∑m,nXe(i+m,j+n,c)⋅Kdw(m,n,c)(7)

This significantly reduces computational complexity. The resulting feature map is refined using the SE attention mechanism. First, a global average pooling operation is performed:

zc=1HW∑i=1H∑j=1WXdw(i,j,c)(8)

followed by two fully connected layers to generate channel-wise attention weights:

s=σ(W2δ(W1z))(9)

where δ(⋅) denotes the Rectified Linear Unit (ReLU) activation function. These weights are then used to recalibrate the feature maps:

Xdw^(i,j,c)=sc⋅Xdw(i,j,c)(10)

Finally, a 1×1 projection convolution is applied to produce the block output:

Y=Conv1×1(Xdw^)(11)

with a residual connection added when the input and output dimensions are identical.

In the proposed work, we utilized the EfficientNetV2-S model, which incorporates FMBC blocks in the initial layers, starting with a standard 3 × 3 CL followed by three MBC and FMBC layer, as shown in Fig. 2a,b, respectively. The last layer contains a 1 × 1 convolution and max-pooling followed by a fully connected layer, as explained in [63]. During the experiments, we found that the EfficientNetV2-S alone has limited ability to extract rich and minute details from facial regions for effective age and fatigue level detection. Therefore, in the proposed work, a DM-CA is used to enhance the feature learning capabilities.

3.2 Driver-Monitoring Coordinate Attention (DM-CA)

The MBConv blocks in EfficientNetV2-S employ SE attention to recalibrate channel-wise responses. SE attention is effective at modelling inter-channel dependencies but encodes no positional information, which is essential for distinguishing the localized facial cues that signal driver fatigue (eye closure, gaze deviation, mouth movement patterns) from the broader postural cues that signal distraction (head pose, reaching behind, operating the radio, leaving the steering wheel). Coordinate Attention (CA) [64] addresses this gap by aggregating features along the height (H) and width (W) directions separately and producing two complementary 1-D attention maps that retain positional structure. The original CA formulation was nevertheless designed for general-purpose recognition and exhibits three properties that limit its suitability for in-cabin driver monitoring under challenging illumination, head-pose, occlusion, and low-resolution conditions. (i) features are aggregated along each direction with average pooling alone, smoothing over the sharp local boundaries such as eye corners, mouth edges, eyelid contours that distinguish open from closed eyes or speaking from yawning. (ii) the H- and W-branches contribute equally by construction, although frontal driver faces are structurally vertical (eyes above nose above mouth) and the two spatial axes carry informationally asymmetric content. (iii) the joint attention map is formed as the outer product of two 1-D sigmoid maps, which is mathematically rank-1 and implicitly assumes the two axes are separable, even though the most discriminative regions (e.g., the eye region) are localized in both axes simultaneously.

To address these limitations, we introduce a DM-CA block that incorporates three targeted modifications, illustrated in Fig. 2c. Mixed directional strip pooling. For an input feature map x∈RH×W×C, the pooled response of the c-th channel along the vertical and horizontal directions is computed as the means of average and maximum strip pooling:

zch(h)=12,[1W∑0≤i<Wxc(h,i)+max0≤i<Wxc(h,i)](12)

zcw(w)=12,[1H∑0≤j<Hxc(j,w)+max0≤j<Hxc(j,w)](13)

The average term preserves smooth contextual information while the maximum term retains sharp boundary responses that average pooling otherwise suppresses.

Asymmetric directional gating. The pooled descriptors zh and zw are concatenated along the strip axis and passed through a shared 1×1 bottleneck convolution with batch normalisation and a non-linear activation δ(⋅). The resulting representation is split back into the H- and W-streams, expanded to Cchannels by per-branch 1×1 convolutions Fh and Fw, passed through a sigmoid σ, and then modulated by two learnable scalar gates αh,αw∈R:

ah=αh⋅σ,(Fh,(δ,(BN,(Conv1×1,([zh;zw])))H))(14)

aw=αw⋅σ,(Fw,(δ,(BN,(Conv1×1,([zh;zw])))W))(15)

where [⋅;⋅] denotes spatial concatenation along the strip axis and (⋅)H, (⋅)W indicate the split-back into the H- and W-streams. The scalars αh and αw are initialized at 1 and learned end-to-end with the rest of the network, allowing the model to re-weight the two axes from data rather than assuming equal contribution.

Cross-axis depthwise fusion. The two directional attention maps are combined into a joint attention map by broadcast Hadamard product and then refined by a single lightweight 3×3 depthwise convolution before the final sigmoid:

A=σ,(DWConv3×3,(ah⊙aw)),y=x⊙A(16)

where ⊙ denotes broadcast Hadamard product. The depthwise convolution adds a small non-separable receptive field that captures cues localized in both H and W simultaneously, replacing the rank-1 outer-product joint map of standard CA.

DM-CA explicitly encodes spatial position along both axes; compared with standard CA, it removes the smoothness, symmetry, and separability assumptions that hinder localization of fine-grained facial and postural cues under illumination variation, head-pose change, partial occlusion, and low-resolution input. Consequently, DM-CA strengthens the network’s ability to focus on discriminative driver-state regions while preserving the computational efficiency required for in-vehicle deployment.

3.3 The Proposed Architecture

The proposed model operates through a mechanism that includes feature extraction using the EfficientNetV2-S backbone, followed by feature refinement with a DM-CA, and fully connected layers and a SoftMax classifier. We initially fine-tuned the EfficientNetV2-S for feature extraction due to its high performance and low computational complexity. The DM-CA is then integrated with the backbone architecture for enhanced feature refinement. This DM-CA receives 7 × 7 × 1280 features from the previous layers, where global average pooling is applied to reduce feature dimensions, and a dropout layer is implemented to mitigate overfitting. Finally, the refined features are fed into a dense layer with a SoftMax activation function for classification, as detailed in Algorithm 1, which outlines a step-by-step implementation of the proposed framework. During training, the model was trained for 50 epochs with an input size of 224 × 224 × 3 and a batch size of 16, utilizing stochastic gradient descent (SGD) as the optimizer, with a learning rate of 1e−3 and a momentum of 0.9. In the testing phase, the trained model is deployed on an IoT edge device for effective driver distraction detection. When driver distraction is detected, an instant alarm is generated to alert the driver about their level of distraction, as illustrated in Fig. 1.

Beyond frame-level predictions, the framework incorporates a temporal memory module that stores sequential predictions over a time window of length T. Let the frame-level prediction probabilities for driver states at frame t be:

pt=[ptActive,ptSleeping,ptYawning,ptAngry,ptSad](17)

The temporal memory module aggregates these predictions over the past T frames to compute behavior-level metrics such as average drowsiness or yawning frequency:

p¯tSleeping=1T∑i=t−T+1tpiSleeping,p¯tYawning=1T∑i=t−T+1tpiYawning(18)

This temporal aggregation strategy helps reduce short-term prediction fluctuations and improves the stability of driver-state estimation during continuous monitoring. The temporal window length T is empirically selected to balance prediction stability and system responsiveness during continuous driver monitoring. A shorter temporal window may increase sensitivity to temporary prediction noise and transient facial variations, whereas an excessively large temporal window may delay risk detection and reduce responsiveness to sudden driver-state changes. Therefore, the adopted fixed-window aggregation strategy is designed to smooth temporary fluctuations while maintaining computational efficiency and real-time operational capability suitable for edge-based deployment scenarios.

In the proposed framework, fatigue-related temporal events are further evaluated using duration-aware persistence analysis. The system monitors whether high sleeping or yawning probabilities remain continuously active for predefined temporal intervals. These duration thresholds are computed relative to the input video frame rate to maintain consistent behavior analysis across different acquisition settings; in the current implementation, sleeping behavior persistence is evaluated over a Δsleep=2s interval, while yawning persistence is evaluated over a Δyawn=1s interval. This strategy reduces false alarms caused by transient facial variations or short-term prediction noise while improving temporal stability during continuous driver monitoring.

A risk-aware decision module then computes a per-frame driver risk score Rt by linearly combining the temporally smoothed probabilities of the dominant fatigue and attention states:

Rt=w1⋅P¯tsleep+w2⋅P¯tyawn−w3⋅P¯tactive(19)

where P¯t(.) denotes the temporally smoothed probability of the corresponding class at frame t, and w1,w2,w3 are non-negative weighting coefficients reflecting the relative contribution of each behavior to overall driver risk. In this work, we use w1=0.6, w2=0.3, and w3=0.4, assigning the largest positive weight to sustained eye-closure (the most severe fatigue indicator), a smaller positive weight to yawning, and a subtractive contribution from sustained engaged driving so that observable attentive behaviour suppresses the risk score. The weighting strategy is designed to provide an interpretable and computationally efficient risk estimation mechanism suitable for real-time deployment. Alert decisions are not driven by thresholds on Rt directly. Instead, the smoothed per-class probabilities are passed to duration-aware persistence counters:

Statust={DANGER,P¯tsleep≥θsleep for ≥ΔsleepWARNING,P¯tyawn≥θyawn for ≥ΔyawnSAFE,otherwise(20)

where θsleep=0.7 and θyawn=0.4 are class-probability thresholds, and Δsleep, Δyawn are the persistence intervals. The risk score Rt is computed and logged continuously as an interpretable summary of behavioral state, but the binary alert transition is governed by sustained-duration persistence on the dominant safety-critical classes. This persistence-based design reduces excessive alert fluctuations caused by frame-level prediction noise and ensures that warnings are issued only for genuinely sustained risk events. The feedback adaptation loop continuously updates the temporal memory and risk evaluation, enabling the system to adjust alerts in real-time and provide goal-driven interventions to the driver. The proposed framework training and testing procedure is detailed in Algorithm 1.

4  Experimental Results

This section discusses system configurations, evaluation parameters, datasets, and achievable results through both quantitative and qualitative assessments, along with comparative analysis with state-of-the-art approaches.

4.1 System Configuration

The proposed model was implemented using a dedicated Nvidia RTX 3090 GPU, with 32 GB of system RAM and an AMD Ryzen 5 5600X 6-Core Processor. The model was implemented in the Python programming language using the Keras framework with TensorFlow as the backend.

4.2 Performance Parameters

The proposed model is evaluated using several metrics, including accuracy, F1-score, recall, and precision. These metrics are derived from a confusion matrix, which helps analyze True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). The metrics can be calculated as follows:

Accuracy =(TPTP+TN+FP+FN)(21)

Precision =(TPTP+FP)(22)

Recall =(TPTP+FN)(23)

F1−measure=2 ∗ (precision∗recallprecision+recall)(24)

4.3 Datasets and Results

Data collection is a crucial aspect of any computer vision-based research. Both the quantity and quality of the collected data significantly impact on the model’s outcomes. Therefore, we conducted experiments using various datasets, including UTKFace [65] for age classification, the Fatigue [1], modified Fatigue datasets [1], and a 100-driver [42] for driver distraction detection.

4.3.1 Results Using the UTKFace Dataset

For driver age classification, there are several datasets of facial images that are publicly available on the internet, such as the All-Age-Faces dataset, the Labeled Faces in the Wild (LFW) dataset [66], the IMDB-WIKI [67] dataset, and the UTKFace dataset. In this work, we used the UTKFace [65] dataset, similar to Hijji et al. [1], for better results acquisition. UTKFace is a large-scale dataset consisting of 23,708 facial images with a long age span ranging from 0 to 116 years old. This dataset has a huge variation in facial expression, resolution, occlusion, illumination, pose, etc. Hijji et al. [1] modified this dataset by dividing it into three classes: underage (6–16), middle age (18–60), and overage (60+). We used this dataset similarly to Hijji et al. [1] and the sample images are shown in Fig. 3.

Figure 3: The sample images of the UTKFace dataset.

Initially, we evaluated the proposed model using the classification report of the UTKFace dataset, as given in Table 1. Our model achieved precision, recall, and F1-scores of 90.62%, 86.56%, and 88.54% for the middle-aged class. For the overage class, the precision, recall, and F1-scores were 94.91%, 97.67%, and 96.27%, respectively. In the underage class, these metrics were 91.97%, 92.64%, and 92.30%. The confusion matrix for the UTKFace dataset, shown in Fig. 4, illustrates the number of accurate and misclassified samples. The proposed model attained accurate predictions of 87%, 98%, and 93% for the middle-aged, overage, and underage classes, respectively.

Figure 4: Confusion matrix of the proposed model over the modified UTKFace dataset.

Furthermore, in comparison with state-of-the-art methods using UTKFace dataset, the proposed model achieved higher accuracy. Table 2 indicates that the proposed model obtained 15.76%, 11.46%, 8.76%, and 2.76% higher accuracy as compared to AlexNet [1], VGG16 [1], ResNet50 [1], and FADS [1].

4.3.2 Results Using the Fatigue Dataset

Recently, Hijji et al. [1] created a fatigue dataset for detecting various driver states. This dataset includes five different classes such as Active, Angry, Sad, Sleeping, and Yawning. In the dataset, the active, sleeping, and yawning classes contain 2000 images. The data was collected from 40 students at Islamia College Peshawar, Pakistan. Additionally, the angry and sad classes were supplemented with images from the KDEF benchmark dataset [68] to enhance the dataset’s robustness. Sample images from the Fatigue dataset are shown in Fig. 5.

Figure 5: The sample images of the fatigue dataset.

The proposed model performance in terms of a confusion matrix using the Fatigue dataset is shown in Fig. 6. It can be seen that the Yawning class achieved 100% accuracy, while the other classes yielded nearly similar results. The classification report of our model, provided in Table 3, examines the effectiveness of the proposed model.

Figure 6: Confusion matrix of the proposed model over fatigue dataset.

Furthermore, Table 2 presents a comparative investigation of the proposed model against existing methods over Fatigue dataset. Our model outperformed AlexNet [1], VGG16 [1], ResNet50 [1], MobileNet [1], and FADS [1] in terms of accuracy, achieving improvements of 7.36%, 1.06%, 11.36%, 5.86%, and 1.36%, respectively.

4.3.3 Results Using the Modified Fatigue Dataset

The dataset proposed by Hijji et al. [1] for driver state detection contains multiple highly similar consecutive frames, which may increase data redundancy and lead to model overfitting with reduced generalization capability on unseen data. To address this limitation, the Fatigue dataset was modified by removing visually repetitive frames with minimal facial variation within each class. The filtering process was performed to preserve representative facial appearance variations while reducing redundant temporal samples that could bias the learning process. Additionally, to further improve data diversity and increase the number of challenging samples, supplementary images for the Active and Yawning classes were incorporated from the drowsiness dataset [69]. The drowsiness dataset consists of four classes: closed eye, open eye, yawning, and normal. Specifically, the yawning and normal classes contain 723 and 725 training samples, respectively, while the closed eye and open eye classes each contain 726 samples. In this work, only the normal and yawning classes were utilized because they contain full facial information similar to the Fatigue dataset used in the proposed work. The remaining classes were excluded since they primarily focus on isolated eye-region features rather than complete facial representations. To maintain labeling consistency, all incorporated samples were manually verified according to the target driver-state categories before integration into the modified dataset. The final modified Fatigue dataset consists of five classes: Active, Angry, Sad, Sleeping, and Yawning. Each class was organized into separate training, validation, and testing subsets to support standardized model evaluation. The training subset contains 299 Active, 337 Angry, 394 Sad, 222 Sleeping, and 249 Yawning images. The validation subset contains 126 Active, 135 Angry, 151 Sad, 76 Sleeping, and 113 Yawning images. Finally, the testing subset contains 113 Active, 124 Angry, 130 Sad, 81 Sleeping, and 78 Yawning images. The adopted preprocessing and dataset refinement strategy were designed to improve data diversity and model robustness while maintaining compatibility with real-world driver monitoring scenarios. Some sample images from the modified fatigue dataset are shown in Fig. 7.

Figure 7: Sample images of the modified fatigue dataset.

The experimental result of the proposed model in terms of confusion matrix using a modified Fatigue dataset is presented in Fig. 8. For the active class, the proposed model achieved an accuracy of 100%, while the angry and sad classes both attained performance levels of 91% and 95%. The sleeping and yawning classes performed better, with accuracy of 98% and 96%, respectively. Table 4 provides a classification report detailing precision, recall, and F1-measure. The report indicates that the angry class had the lowest performance, while the active and yawning class exhibited the highest performance.

Figure 8: Confusion matrix of the proposed model using modified fatigue dataset.

The proposed model also investigated with different attention mechanisms in driver distraction detection research. We conducted a comprehensive ablation across ten backbone variants (NASNetMobile, MobileNetV3-Small/Large, EfficientNet-B0/B1/B2, EfficientNetV2-B0/B1/B2, and the proposed EfficientNetV2-S) combined with four attention mechanisms: channel attention (SE), spatial attention, soft attention, and the proposed DM-CA. The results in Table 5 (panels a–e) show that introducing any attention mechanism consistently improves over the no-attention baseline, with the choice of mechanism mattering more for stronger backbones than weaker ones. On the proposed EfficientNetV2-S backbone, channel attention reaches 94.29% accuracy (+0.38% over the 93.91% baseline), spatial attention 94.48% (+0.57%), soft attention 94.67% (+0.76%), and the proposed DM-CA module 95.62% achieved the highest value. The proposed model achieved 1.71% higher accuracy over the no-attention baseline and 0.95% higher accuracy over the strongest alternative attention (soft attention), with consistent improvements of +0.73, +0.62, and +0.68 in precision, recall, and F1-measure, respectively. The proposed backbone also achieves the highest accuracy within every panel (a–e), confirming that the benefit of DM-CA is not depending on a specific attention configuration. Thus, the results clearly demonstrate that our model is an effective solution for driver distraction detection.

We further investigated the proposed model performance with K-Fold-Cross Validation (KFCV) parameters, which divides the original dataset into K sub-samples. In this strategy, the proposed model is iteratively trained K a number of times, where each time a different subset of the dataset is utilized for training, validation, and testing. This procedure assists in making sure that each data point is used for validation exactly once, and the final evaluation metric is averaged over the k iterations. KFCV is particularly effective for evaluating the model generalization and robustness capability, as it provides a more precise assessment of the model performance when compared to a single train-test split mechanism. Table 6 investigated the proposed model’s performance across five different folds. It is evident that each fold yields results with slight variations. The most favorable outcome is observed in fold 1, while fold 4 exhibits the least favorable performance. The average precision, recall, F1-measure, and accuracy of all five folds are 95.30%, 95.48%, 95.38%, and 95.04%, respectively.

4.3.4 Results Using the 100-Driver Dataset

To further evaluate the robustness and generalization capability of the proposed framework, additional experiments were conducted on the publicly available 100-Driver dataset [42], a large-scale, diverse benchmark designed for distracted-driver classification under realistic driving conditions. The dataset contains 245,263 RGB images collected from 100 drivers across 5 vehicles (two sedans, two SUVs, and one van), recorded by 4 cameras placed at the front-left (D1), front (D2), front-right (D3), and side-right (D4) positions (Fig. 9a). The annotations cover 22 distracted-driving behaviors including C1 (Drive Safe), C2_Sleep, C3 (Yawning), C4 (Talk Left), C5 (Talk Right), C6 (Text Left), C7 (Text Right), C8 (Make Up), C9 (Look Left), C10 (Look Right), C11 (Look Up), C12 (Look Down), C13 (Smoke Left), C14 (Smoke Right), C15 (Smoke Mouth), C16 (Eat Left), C17 (Eat Right), C18 (Operate Radio), C19 (Operate GPS), C20 (Reach Behind), C21 (Leave Steering Wheel), and C22 (Talk to Passenger), listed in Fig. 9b. The recordings span diverse age groups (20–60 years), driving experience (1 to >10 years), weather and lighting conditions, and appearance variations (clothing, masks, sunglasses, hats), making 100-Driver substantially more representative of in-cabin deployment scenarios than smaller laboratory datasets.

Figure 9: Samples visualization of 100-Driver dataset.

To ensure a fair comparison and to eliminate the risk of identity-level data leakage, we adopted the subject-wise evaluation protocol released by the dataset authors [42]. In this protocol, drivers are partitioned into mutually exclusive train, validation, and test groups, where 47, 6, 12 drivers data were selected for train, validation, test in the split, and the number of images in each class is given in Table 7. No driver appears in more than one subset, so the test set contains only previously unseen drivers, and the reported accuracy reflects the model’s ability to generalize to new individuals rather than to memorize identity-specific features. During training the model, the dataset was used in the form released by the original authors [42], whose own down-sampling and outlier-removal procedures are part of the dataset specification rather than study-specific manipulations.

The proposed model is further evaluated using precision (Pre), recall (Rec), and F1-score (F1) on four datasets named D1, D2, D3, and D4, with the per-class results summarized in Table 8. Across the 22 classes, the model attains an average F1-score of 81.78%, 74.65%, 82.04%, and 83.92% on D1, D2, D3, and D4, respectively, with corresponding overall accuracies of 82.05%, 77.35%, 82.42%, and 84.28%. These results indicate consistent and competitive performance across all four datasets. The best results are obtained on D4, where the averages of Pre, Rec, and F1 reach 84.04%, 84.31%, and 83.92%. On D3, the proposed model obtained 83.45%, 81.88%, and 82.04%, and D1 records 82.37%, 82.01%, and 81.78% for the same three metrics. On the more challenging D2 dataset, the model still maintains balanced behavior, with Pre, Rec, and F1 of 76.91%, 76.40%, and 74.65%, respectively. At the class level, several categories are recognized with very high reliability: for example, C2 attains an F1-score of 98.87% on D1, 98.48% on D2, 97.48% on D3, and 98.43% on D4; C18 achieves F1-scores of 95.57%, 98.59%, 96.20%, and 90.42% across the four datasets; and C22 reaches 94.20%, 97.32%, 95.85%, and 98.42%, respectively. These results collectively demonstrate that the proposed model delivers robust and well-balanced Pre-Rec trade-offs across diverse datasets and a large number of classes.

To further analyze the classification behavior of the proposed model, confusion matrices for all four datasets are reported in Fig. 10, where sub-figures (a), (b), (c), and (d) correspond to D1, D2, D3, and D4, respectively. The diagonal entries indicate the proportion of correctly classified samples for each of the 22 classes (C1–C22), while the off-diagonal entries represent inter-class confusions. As shown in Fig. 10, the diagonal of every matrix is clearly dominant, confirming that the model preserves strong discriminative capability across diverse data distributions. Classes such as C2, C18, and C22 are recognized with near-perfect accuracy on all four datasets, attaining diagonal values of 1.00, 0.99, and 0.91 on D1; 0.98, 1.00, and 0.99 on D2; 0.95, 0.96, and 0.94 on D3; and 0.99, 0.88, and 0.99 on D4. Similarly, C16 achieves diagonal scores of 0.97, 0.95, 0.84, and 0.63 across the four settings, respectively, whereas C17 consistently maintains values between 0.93 and 0.99 across all settings, demonstrating the robustness of the proposed model for classes with distinctive feature patterns. A small number of consistent inter-class confusions can also be observed; for instance, samples from C14 are frequently misclassified as C1 in all four datasets (with confusion values of 0.20, 0.64, 0.20, and 0.21 on D1, D2, D3, and D4, respectively), suggesting a strong visual similarity between the two categories rather than a dataset-specific artifact. On D2, additional confusions are observed for C9 (0.50 predicted as C10), C11 (0.25 predicted as C13), and C13 (0.16 predicted as C14), which together explain its comparatively lower average performance relative to the remaining datasets. In contrast, on D3 and D4 the off-diagonal mass is more diffuse and significantly smaller, with most cells below 0.10, indicating that the model produces sharper decision boundaries on these datasets. Overall, the confusion matrices confirm the quantitative results, demonstrating that the proposed model delivers consistent, well-balanced, and dataset-agnostic classification performance across a large number of classes.

Figure 10: The proposed model performance over the 100-Driver dataset, where (a–d), are four different views.

In Table 9, we assess the performance of the proposed model against Wang et al. [42] methods. They implemented six widely used baseline architectures ResNet50, MobileNetV3, ShuffleNetV2, SqueezeNet, GhostNetV1, and EfficientNetB0 on the D1, D2, D3, and D4 datasets. Among the baselines, ShuffleNetV2 yields the lowest performance, while the proposed model achieves the highest accuracy on all four datasets. More specifically, ShuffleNetV2 attains accuracies of 70.0%, 64.6%, 67.0%, and 74.7% on D1, D2, D3, and D4, respectively, which are 12.05%, 12.75%, 15.42%, and 9.58% lower than those of the proposed model. Although EfficientNetB0 and SqueezeNet perform better than the remaining baselines with EfficientNetB0 obtaining the second-best accuracy of 79.8% and 72.3% on D1 and D2, and SqueezeNet obtaining the second-best accuracy of 79.6% and 82.5% on D3 and D4, their results are still consistently lower than those of the proposed model, which reaches 82.05%, 77.35%, 82.42%, and 84.28% accuracy on the four datasets. A similar trend can be observed for precision, recall, and F1-score metrics, where the proposed model attains average F1-scores of 82.78%, 74.65%, 82.04%, and 83.92%, outperforming the strongest baseline by 2.54%, 2.31%, 3.2%, and 2.63% on D1, D2, D3, and D4, respectively, as shown in Table 9.

4.4 Statistical Analysis

To verify the stability and reproducibility of the proposed model, we repeat each experiment five times under identical settings and report the per-run results together with the best score and the mean ± standard deviation in Table 10. As shown in Table 10, the proposed model consistently obtains high accuracy across the seven evaluation settings, with mean values of 92.53%, 99.03%, and 95.37% on UTKFace, FAT, and M-FAT, and 82.04%, 76.42%, 82.13%, and 84.13% on D1, D2, D3, and D4, respectively. The corresponding best scores, namely 93.17%, 99.40%, 95.74%, 82.29%, 77.43%, 82.42%, and 84.48%, indicate that even the worst-performing run remains close to the overall mean. More importantly, the standard deviations across the five repetitions remain very small in every setting, ranging from 0.18 on D1 to 0.66 on D2, with FAT, M-FAT, D3, and D4 yielding standard deviations of only 0.35, 0.34, 0.24, and 0.27, respectively. These low variation values, all below 1.0%, demonstrate that the proposed model is highly stable and is not sensitive to random factors such as weight initialization or training order. Among all evaluation settings, FAT yields the highest mean performance (99.03%), while D2 represents the most challenging setting and also exhibits the largest variability (σ = 0.66), which is in line with the higher inter-class confusion observed for this dataset in the confusion-matrix analysis.

4.5 Cross-Dataset Evaluations

To evaluate the generalization capability of the proposed model, we conduct a series of cross-dataset experiments in which each method is trained on one dataset and tested on another, with the average accuracies reported in Table 11. As shown in Table 11, the proposed model consistently outperforms all six baseline architectures on every one of the 14 cross-dataset configurations considered, demonstrating its ability to learn more transferable and dataset-agnostic feature representations. On the FTG ↔ M-FTG pair, the proposed model attains accuracies of 90.7% and 96.9% for FTG → M-FTG and M-FTG → FTG, respectively, surpassing the best baseline (EfficientNetB0) by 2.3% and 3.4%. The advantage becomes more noticeable on the more challenging D1–D4 transmissions; for instance, on D2 → D3 the proposed model reaches 51.5%, exceeding the second-best method, GhostNetV1 by 13.2%, while on D2 → D1 and D3 → D1 it attains 41.2% and 25.8%, outperforming the strongest baselines by 9.9% and 10.2%, respectively. Similar improvements are observed on D1 → D2 (59.6% vs. 52.1%) and D3 → D2 (40.5% vs. 31.8%), confirming that the proposed model transfers more reliably across datasets than the lightweight baselines. All methods, including ours, exhibit a significant drop in performance when D4 is involved as either the source or the target (e.g., D1 → D4: 8.3%, D2 → D4: 7.1%, D4 → D1: 6.8%, D4 → D2: 5.9%), which suggests a large distributional gap between D4 and the remaining datasets; nevertheless, the proposed model still ranks first in every such setting. In contrast, lightweight baselines such as ShuffleNetV2 frequently yield the lowest cross-dataset accuracies, dropping to 3.4% on D4 → D2 and 7.8% on D3 → D1. Overall, by achieving the highest accuracy on all fourteen cross-dataset configurations in Table 11, the proposed model demonstrates substantially stronger cross-domain generalization than the state-of-the-art architectures.

4.6 Model Complexity Analysis

Subsequently, the proposed model is analyzed in terms of computational complexity, model size, and inference time using three different hardware platforms, as given in Table 12. Computational complexity and model size are the two main factors associated with the inference time of any deep learning model. To estimate the computational complexity of our model, we examined the giga-floating-point operations (GFLOPs) and the size in MB. As reported in Table 12, the proposed EfficientNetV2-S + DM-CA model has 20.66 M parameters, a forward-pass cost of 5.73 GFLOPs, and a model size of 79.93 MB. We provided a feasibility analysis of the proposed model across three different hardware architectures, i.e., a CPU, a GPU, and an edge device, which is an NVIDIA Jetson Nano Developer Kit with four GB of unified memory; this device consists of a quad-core ARM Cortex-A57 CPU at 1.43 GHz and a 128-core Maxwell GPU. The inference times of the proposed model are tested on these three platforms. On the desktop CPU, the model attains a mean inference latency of 81.45 ms and a throughput of 12.50 FPS. On the desktop GPU, the same model executes at a mean latency of 12.09 ms, corresponding to a throughput of 82.69 FPS. On the Jetson Nano, the trained model was exported via ONNX and compiled into a TensorRT FP32 engine, achieving a mean inference latency of 64.39 ms and a throughput of 15.38 FPS. The proposed model achieved high FPS rates of 82.69, 12.50, and 15.38 for the GPU, CPU, and Jetson Nano (TensorRT FP32), respectively, with 20.66 M parameters and a model size of 79.93 MB. Hence, the proposed model achieved a balanced trade-off among the accuracy, model size, GFLOPs, and inference speed, which increases the potential for this model to be deployed on edge devices for real-time decision-making.

4.7 Qualitative Analysis

In this section, we evaluate our model through qualitative analysis, as demonstrated in Fig. 11. The visualized results include test images sourced from highly diverse scenarios that were excluded from the training phase of the proposed model. Fig. 11 illustrates our model’s strengths in facial feature analysis for driver safety systems. However, certain areas require improvement. The last column in the second-to-last row of Fig. 11 highlights examples of misclassification. The last image in the second row shows a middle-aged driver misclassified as underage. In the third-row last image, the model mistakenly indicates that a yawning driver is active. The fourth row’s last image is misclassified as sleeping due to the driver wearing glasses, which obscures the eye features from the model. Finally, in the last image in the last row featuring an older driver, the model inaccurately classifies the fatigue status as sleeping because of extreme visual similarity. These inaccuracies arise from the inherent challenges posed by such visual similarities in the dataset.

Figure 11: Qualitative results of the proposed model in consideration of real-time age and fatigue detection.

4.8 Temporal Stability and Alert Behaviour

Fig. 12 presents the temporal behavior of the proposed framework across representative test sequences. The blue curves plot the smoothed sleeping probability p¯tSleeping over time, while the embedded facial snapshots indicate the underlying driver state (Active, Yawning, or Sleeping). Three distinct regimes are visible. During active driving, the probability remains consistently low and stable, forming a quiet baseline that does not approach the alert thresholds and confirms a low false-positive rate under normal conditions. Yawning events produce moderate, short-duration spikes in the curve; because each spike falls below the persistence requirement of Δyawn=1 s, transient elevations do not trigger an alert in isolation, demonstrating that the duration-aware rule effectively suppresses short-term fluctuations. Sleeping episodes, in contrast, produce sharp and sustained high-probability segments (highlighted in the figure) that exceed both the probability threshold (θsleep=0.7) and the duration threshold (Δsleep=2 s), correctly activating the DANGER state. The clear visual separation between transient yawn-related fluctuations and prolonged sleep-related plateaus shows that the temporal smoothing and the persistence rule act jointly to distinguish behaviorally meaningful events from short-term prediction noise. Across the analyzed sequences, this produces stable transitions between SAFE, WARNING, and DANGER states without spurious alarm flicker, supporting the suitability of the proposed framework for continuous real-time driver monitoring on edge devices.

Figure 12: Temporal drowsiness probability curves of the proposed driver monitoring framework. Low values correspond to active driving, moderate increases indicate yawning, and sustained high-probability segments (highlighted in red) trigger the DANGER state based on temporal thresholds.

5  Conclusions and Future Work

This work presents an efficient and robust framework for driver distraction detection using a CNN enhanced with a DM-CA and a temporal decision strategy. The proposed model outperforms existing approaches in terms of accuracy, stability, and real-time effectiveness. First, we design a lightweight CNN architecture capable of extracting discriminative features while maintaining computational efficiency, making it suitable for deployment on edge-IoT and in-vehicle systems. Second, the integration of a DM-CA enhances feature refinement by selectively emphasizing critical driver regions, thereby improving the recognition of driver state. In addition to spatial feature enhancement, this work introduces a temporal aggregation and duration-based alert mechanism to improve reliability in real-world scenarios. Instead of relying solely on frame-level predictions, the proposed framework accumulates evidence across consecutive frames and activates warning or danger states only when predefined temporal thresholds are satisfied. This strategy effectively reduces false positives caused by transient driver activity or momentary misclassifications, while ensuring high sensitivity to sustained drowsiness events. Extensive experiments conducted on four different datasets demonstrate the effectiveness of the proposed approach over existing methods. The temporal probability analysis further validates the stability and robustness of the system under realistic driving conditions. These contributions establish a comprehensive, practical, and deployment-ready driver monitoring solution that can be seamlessly integrated into intelligent transportation systems, thereby enhancing road safety, improving driver awareness, and reducing the risk of fatigue-related accidents.

5.1 Limitations

While the proposed framework demonstrates competitive performance across four publicly available datasets and successful deployment on edge hardware, several limitations should be acknowledged. First, frame-level confusion between behaviorally similar mouth-open classes, most notably Yawning and Talking, remains a residual error mode of the proposed model. Although the duration-aware persistence rule prevents this confusion from propagating into spurious WARNING or DANGER alerts, the per-frame error itself is real and arises because the proposed DM-CA module operates on a single frame and cannot encode the temporal dynamics that distinguish the two behaviors. Second, the model operates exclusively on RGB images and therefore exhibits reduced performance under low-illumination or night-time driving conditions, where the visual cues used by the spatial-attention branch are partially lost; integration of complementary near-infrared (NIR) input is required to obtain consistent performance across the full 24-h driving cycle. Third, although the proposed model is substantially more compact than heavier vision backbones, its computational cost (5.73 GFLOPs) and model size are still relatively high for the most resource-constrained automotive-grade microcontrollers; further compression would be needed to widen the range of platforms on which the framework can be deployed without auxiliary GPU support.

5.2 Future Work

Future work is directed at the three limitations identified above. First, to address the Yawning–Talking frame-level confusion, we plan to extend the proposed duration-aware persistence module into a learned temporal model, a lightweight LSTM or a temporal convolution that captures mouth-opening velocity and frequency rather than only the instantaneous spatial appearance of the mouth region, thereby explicitly separating behaviors that differ in their temporal dynamics. Second, to address the limited robustness under night-time driving, we plan to extend the proposed framework to multi-modal RGB + NIR input, using the daytime RGB and night-time NIR sequences available in the 100-Driver benchmark for joint training, so that the system maintains consistent performance across the full day–night driving cycle. Third, to further reduce computational cost and model size for deployment on more resource-constrained automotive-grade platforms, we plan to apply post-training compression strategies, including INT8 quantization, structured channel pruning, and knowledge distillation, in which a larger version of the proposed model serves as a teacher and transfers its representations to a compact student network with the explicit objective of reducing both GFLOPs and on-disk footprint while preserving the temporal stability and edge-deployability demonstrated in this work.

Acknowledgement: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)(RS-2025-00523019), supported by the InnoCORE program of the Ministry of Science and ICT (N10260002) and was also supported by the “Regional Innovation System & Education (RISE)” through the Seoul RISE Center, funded by the Ministry of Education (MOE) and the Seoul Metropolitan Government. (2026-RISE-01-007-04).

Funding Statement: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)(RS-2025-00523019), supported by the InnoCORE program of the Ministry of Science and ICT (N10260002) and was also supported by the “Regional Innovation System & Education (RISE)” through the Seoul RISE Center, funded by the Ministry of Education (MOE) and the Seoul Metropolitan Government. (2026-RISE-01-007-04).

Author Contributions: Hikmat Yar: Conceptualization, Data curation, Methodology, Software, Writing—original draft, Writing—review & editing. Imran Ullah Khan: Formal analysis, Methodology, Validation, Writing—review & editing. Naqqash Dilshad: Data curation, Investigation, Validation, Writing—review & editing. Weiwei Jiang: Formal analysis, Validation, Visualization, Software. Heung Soo Kim: Funding acquisition, Project administration, Supervision, Resources. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: The 100-Driver, UTKFace, Drowsiness, and Fatigue datasets are available through the links provided in the manuscript. The Modified Fatigue dataset generated through preprocessing of the original Fatigue dataset is available from the authors upon reasonable request.

Ethics Approval: The current study only involved data preprocessing, modification, and analysis of existing datasets. Given the minimal-risk nature of this secondary analysis, additional ethics approval was not required.

Conflicts of Interest: The authors declare no conflicts of interest.

References

1. Hijji M, Yar H, Ullah FUM, Alwakeel MM, Harrabi R, Aradah F, et al. FADS: an intelligent fatigue and age detection system. Mathematics. 2023;11(5):1174. doi:10.3390/math11051174. [Google Scholar] [CrossRef]

2. Liu F, Chen D, Zhou J, Xu F. A review of driver fatigue detection and its advances on the use of RGB-D camera and deep learning. Eng Appl Artif Intell. 2022;116(1):105399. doi:10.1016/j.engappai.2022.105399. [Google Scholar] [CrossRef]

3. Sun Z, Miao Y, Jeon JY, Kong Y, Park G. Facial feature fusion convolutional neural network for driver fatigue detection. Eng Appl Artif Intell. 2023;126:106981. doi:10.1016/j.engappai.2023.106981. [Google Scholar] [CrossRef]

4. Michelaraki E, Katrakazas C, Kaiser S, Brijs T, Yannis G. Real-time monitoring of driver distraction: state-of-the-art and future insights. Accid Anal Prev. 2023;192(3):107241. doi:10.1016/j.aap.2023.107241. [Google Scholar] [PubMed] [CrossRef]

5. Bej D, Rakshit S, Mal AK, Mahapatra R. A cost-effective system for triggering alarm to distracted drivers/nurses. Comput Electr Eng. 2019;76:24–39. doi:10.1016/j.compeleceng.2019.03.001. [Google Scholar] [CrossRef]

6. Zhao L, Yang F, Bu L, Han S, Zhang G, Luo Y. Driver behavior detection via adaptive spatial attention mechanism. Adv Eng Inform. 2021;48(1):101280. doi:10.1016/j.aei.2021.101280. [Google Scholar] [CrossRef]

7. Huang F, Yang C, Weng W, Chen Z, Zhang Z. CM-FusionNet: a cross-modal fusion fatigue detection method based on electroencephalogram and electrooculogram. Comput Electr Eng. 2025;123(3):110204. doi:10.1016/j.compeleceng.2025.110204. [Google Scholar] [CrossRef]

8. Lv X, Zheng G, Zhai H, Zhou K, Zhang W. Driver fatigue detection method based on temporal–spatial adaptive networks and adaptive temporal fusion module. Comput Electr Eng. 2024;119(6):109540. doi:10.1016/j.compeleceng.2024.109540. [Google Scholar] [CrossRef]

9. Higgins JS, Michael J, Austin R, Åkerstedt T, Van Dongen HPA, Watson N, et al. Asleep at the wheel—the road to addressing drowsy driving. Sleep. 2017;40(2):zsx001. doi:10.1093/sleep/zsx001. [Google Scholar] [PubMed] [CrossRef]

10. Department of Transportation US. Traffic safety facts research note. National Highway Traffic Safety Administration (NHTSA). 2023 [cited 2026 Jan 1]. Available from: https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/813443. [Google Scholar]

11. Sun H, Zhang M. A lightweight model for distracted driver detection based on neural architecture search and coordinate attention. Comput Electr Eng. 2025;123(6):110235. doi:10.1016/j.compeleceng.2025.110235. [Google Scholar] [CrossRef]

12. Lam LT. A neglected risky behavior among children and adolescents: underage driving and injury in New South Wales, Australia. J Saf Res. 2003;34(3):315–20. doi:10.1016/S0022-4375(03)00026-4. [Google Scholar] [PubMed] [CrossRef]

13. The Korea Times. 9 dead, 4 injured as car plows into pedestrians in central Seoul. [cited 2026 Jan 1]. Available from: https://www.koreatimes.co.kr/www/nation/2024/07/113_377825.html. [Google Scholar]

14. Shaaban K, Hassan HM. Underage driving and seat-belt use of high school teenagers in Qatar. J Transp Saf Secur. 2017;9(Suppl 1):115–29. doi:10.1080/19439962.2016.1212445. [Google Scholar] [CrossRef]

15. Liu Y, Lan Z, Cui J, Sourina O, Müller-Wittig W. Inter-subject transfer learning for EEG-based mental fatigue recognition. Adv Eng Inform. 2020;46(4):101157. doi:10.1016/j.aei.2020.101157. [Google Scholar] [CrossRef]

16. Lee C, An J. LSTM-CNN model of drowsiness detection from multiple consciousness states acquired by EEG. Expert Syst Appl. 2023;213(3):119032. doi:10.1016/j.eswa.2022.119032. [Google Scholar] [CrossRef]

17. Garrosa M, Ceccarelli M, Sánchez SS, Russo M. Design and validation of a low-cost vehicle driver monitoring device. Mech Syst Signal Process. 2025;235(9):112939. doi:10.1016/j.ymssp.2025.112939. [Google Scholar] [CrossRef]

18. Horng MF, Nguyen TL, Nguyen TT, Shieh CS, Guo LY, Hung CF, et al. Real-time mouth state detection based on a BiGRU-CLPSO hybrid model with facial landmark detection for healthcare monitoring applications. Comput Model Eng Sci. 2026;146(1):1–10. doi:10.32604/cmes.2025.075064. [Google Scholar] [CrossRef]

19. Wang MS, Jeong NT, Kim KS, Choi SB, Yang SM, You SH, et al. Drowsy behavior detection based on driving information. IntJ Automot Technol. 2016;17(1):165–73. doi:10.1007/s12239-016-0016-y. [Google Scholar] [CrossRef]

20. Thaher T, Mafarja M, Saffarini M, Mohamed AHHM, El-Saleh AA. A comprehensive review of face detection techniques for occluded faces: methods, datasets, and open challenges. Comput Model Eng Sci. 2025;143(3):2615–73. doi:10.32604/cmes.2025.064857. [Google Scholar] [CrossRef]

21. Deng W, Wu R. Real-time driver-drowsiness detection system using facial features. IEEE Access. 2019;7:118727–38. doi:10.1109/ACCESS.2019.2936663. [Google Scholar] [CrossRef]

22. Attivissimo F, D’Alessandro VI, Di Nisio A, Scarcelli G, Schumacher J, Lanzolla AML. Performance evaluation of image processing algorithms for eye blinking detection. Measurement. 2023;223:113767. doi:10.1016/j.measurement.2023.113767. [Google Scholar] [CrossRef]

23. Zhao L, Wang Z, Wang X, Liu Q. Driver drowsiness detection using facial dynamic fusion information and a DBN. IET Intell Transp Syst. 2018;12(2):127–33. doi:10.1049/iet-its.2017.0183. [Google Scholar] [CrossRef]

24. Mandal B, Li L, Wang GS, Lin J. Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans Intell Transport Syst. 2017;18(3):545–57. doi:10.1109/tits.2016.2582900. [Google Scholar] [CrossRef]

25. Massoz Q, Langohr T, François C, Verly JG. The ULg multimodality drowsiness database (called DROZY) and examples of use. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV); 2016 Mar 7–10; Lake Placid, NY, USA. doi:10.1109/WACV.2016.7477715. [Google Scholar] [CrossRef]

26. Abtahi S, Omidyeganeh M, Shirmohammadi S, Hariri B. YawDD: a yawning detection dataset. In: Proceedings of the 5th ACM Multimedia Systems Conference. Singapore: ACM; 2014. p. 24–8. doi:10.1145/2557642.2563678. [Google Scholar] [CrossRef]

27. Song F, Tan X, Liu X, Chen S. Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recognit. 2014;47(9):2825–38. doi:10.1016/j.patcog.2014.03.024. [Google Scholar] [CrossRef]

28. Maior CBS, das Chagas Moura MJ, Santana JMM, Lins ID. Real-time classification for autonomous drowsiness detection using eye aspect ratio. Expert Syst Appl. 2020;158(3):113505. doi:10.1016/j.eswa.2020.113505. [Google Scholar] [CrossRef]

29. Anitha J, Mani G, Venkata Rao K. Driver drowsiness detection using Viola Jones algorithm. In: Smart intelligent computing and applications. Singapore: Springer; 2020. p. 583–92. doi:10.1007/978-981-13-9282-5_55. [Google Scholar] [CrossRef]

30. Jabbar R, Al-Khalifa K, Kharbeche M, Alhajyaseen W, Jafari M, Jiang S. Real-time driver drowsiness detection for Android application using deep neural networks techniques. Procedia Comput Sci. 2018;130:400–7. doi:10.1016/j.procs.2018.04.060. [Google Scholar] [CrossRef]

31. Verma B, Choudhary A. Deep learning based real-time driver emotion monitoring. In: 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES); 2018 Sep 12–14; Madrid, Spain. doi:10.1109/ICVES.2018.8519595. [Google Scholar] [CrossRef]

32. García-García M, Caplier A, Rombaut M. Sleep deprivation detection for real-time driver monitoring using deep learning. In: Image analysis and recognition. Cham, Switzerland: Springer International Publishing; 2018. p. 435–42. doi:10.1007/978-3-319-93000-8_49. [Google Scholar] [CrossRef]

33. Ghoddoosian R, Galib M, Athitsos V. A realistic dataset and baseline temporal model for early drowsiness detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); 2019 Jun 16–17; Long Beach, CA, USA. p. 178–87. doi:10.1109/cvprw.2019.00027. [Google Scholar] [CrossRef]

34. Chen Y, Chen J, Xie X, Yi W, Ji Z. Prediction of mental fatigue for control room operators: innovative data processing and multi-model evaluation. Mathematics. 2025;13(17):2794. doi:10.3390/math13172794. [Google Scholar] [CrossRef]

35. Liu W, Qian J, Yao Z, Jiao X, Pan J. Convolutional two-stream network using multi-facial feature fusion for driver fatigue detection. Future Internet. 2019;11(5):115. doi:10.3390/fi11050115. [Google Scholar] [CrossRef]

36. Ji Y, Wang S, Zhao Y, Wei J, Lu Y. Fatigue state detection based on multi-index fusion and state recognition network. IEEE Access. 2019;7:64136–47. doi:10.1109/ACCESS.2019.2917382. [Google Scholar] [CrossRef]

37. Ed-doughmi Y, Idrissi N. Driver fatigue detection using recurrent neural networks. In: Proceedings of the 2nd International Conference on Networking, Information Systems & Security; 2019 Mar 27–29; Rabat, Morocco. p. 1–6. doi:10.1145/3320326.3320376. [Google Scholar] [CrossRef]

38. Quddus A, Shahidi Zandi A, Prest L, Comeau FJE. Using long short term memory and convolutional neural networks for driver drowsiness detection. Accid Anal Prev. 2021;156(1):106107. doi:10.1016/j.aap.2021.106107. [Google Scholar] [PubMed] [CrossRef]

39. Magán E, Sesmero MP, Alonso-Weber JM, Sanchis A. Driver drowsiness detection by applying deep learning techniques to sequences of images. Appl Sci. 2022;12(3):1145. doi:10.3390/app12031145. [Google Scholar] [CrossRef]

40. Tanveer MA, Khan MJ, Qureshi MJ, Naseer N, Hong KS. Enhanced drowsiness detection using deep learning: an fNIRS study. IEEE Access. 2019;7:137920–9. doi:10.1109/access.2019.2942838. [Google Scholar] [CrossRef]

41. Zhao L, Li M, He Z, Ye S, Qin H, Zhu X, et al. Data-driven learning fatigue detection system: a multimodal fusion approach of ECG (electrocardiogram) and video signals. Measurement. 2022;201(8):111648. doi:10.1016/j.measurement.2022.111648. [Google Scholar] [CrossRef]

42. Wang J, Li W, Li F, Zhang J, Wu Z, Zhong Z, et al. 100-driver: a large-scale, diverse dataset for distracted driver classification. IEEE Trans Intell Transport Syst. 2023;24(7):7061–72. doi:10.1109/tits.2023.3255923. [Google Scholar] [CrossRef]

43. Le THN, Zheng Y, Zhu C, Luu K, Savvides M. Multiple scale faster-RCNN approach to driver’s cell-phone usage and hands on steering wheel detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); 2016 Jun 26–Jul 1; Las Vegas, NV, USA. p. 46–53. [Google Scholar]

44. Zhao CH, Zhang BL, He J, Lian J. Recognition of driving postures by contourlet transform and random forests. IET Intell Transp Syst. 2012;6(2):161–8. doi:10.1049/iet-its.2011.0116. [Google Scholar] [CrossRef]

45. Wang F, Wang H, Fu R. Real-time ECG-based detection of fatigue driving using sample entropy. Entropy. 2018;20(3):196. doi:10.3390/e20030196. [Google Scholar] [PubMed] [CrossRef]

46. Khan T, Choi G, Lee S. EFFNet-CA: an efficient driver distraction detection based on multiscale features extractions and channel attention mechanism. Sensors. 2023;23(8):3835. doi:10.3390/s23083835. [Google Scholar] [PubMed] [CrossRef]

47. Angulu R, Tapamo JR, Adewumi AO. Age estimation via face images: a survey. EURASIP J Image Video Process. 2018;2018(1):42. doi:10.1186/s13640-018-0278-6. [Google Scholar] [CrossRef]

48. Wang JG, Yau WY, Wang HL. Age categorization via ECOC with fused Gabor and LBP features. In: 2009 Workshop on Applications of Computer Vision (WACV); 2009 Dec 7–8; Snowbird, UT, USA. doi:10.1109/wacv.2009.5403085. [Google Scholar] [CrossRef]

49. Lu J, Liong VE, Zhou J. Cost-sensitive local binary feature learning for facial age estimation. IEEE Trans Image Process. 2015;24(12):5356–68. doi:10.1109/tip.2015.2481327. [Google Scholar] [PubMed] [CrossRef]

50. Nguyen DT, Cho SR, Park KR. Human age estimation based on multi-level local binary pattern and regression method. In: Future information technology. Berlin/Heidelberg, Germany: Springer; 2014. p. 433–8. doi:10.1007/978-3-642-55038-6_67. [Google Scholar] [CrossRef]

51. Sai PK, Wang JG, Teoh EK. Facial age range estimation with extreme learning machines. Neurocomputing. 2015;149(4):364–72. doi:10.1016/j.neucom.2014.03.074. [Google Scholar] [CrossRef]

52. Dehghan A, Ortiz EG, Shu G, Masood SZ. DAGER: deep age, gender and emotion recognition using convolutional neural network. arXiv:1702.04280. 2017. [Google Scholar]

53. Dornaika F, Arganda-Carreras I, Belver C. Age estimation in facial images through transfer learning. Mach Vis Appl. 2019;30(1):177–87. doi:10.1007/s00138-018-0976-1. [Google Scholar] [CrossRef]

54. Han H, Jain AK, Wang F, Shan S, Chen X. Heterogeneous face attribute estimation: a deep multi-task learning approach. IEEE Trans Pattern Anal Mach Intell. 2018;40(11):2597–609. doi:10.1109/tpami.2017.2738004. [Google Scholar] [PubMed] [CrossRef]

55. Huerta I, Fernández C, Segura C, Hernando J, Prati A. A deep analysis on age estimation. Pattern Recognit Lett. 2015;68(6):239–49. doi:10.1016/j.patrec.2015.06.006. [Google Scholar] [CrossRef]

56. Ranjan R, Zhou S, Chen JC, Kumar A, Alavi A, Patel VM, et al. Unconstrained age estimation with deep convolutional neural networks. In: 2015 IEEE International Conference on Computer Vision Workshop (ICCVW); 2015 Dec 7–13; Santiago, Chile. p. 351–9. doi:10.1109/iccvw.2015.54. [Google Scholar] [CrossRef]

57. Rothe R, Timofte R, Van Gool L. Deep expectation of real and apparent age from a single image without facial landmarks. Int J Comput Vis. 2018;126(2):144–57. doi:10.1007/s11263-016-0940-3. [Google Scholar] [CrossRef]

58. Shen W, Guo Y, Wang Y, Zhao K, Wang B, Yuille A. Deep regression forests for age estimation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2018 Jun 18–23; Salt Lake City, UT, USA. p. 2304–13. doi:10.1109/cvpr.2018.00245. [Google Scholar] [CrossRef]

59. Taheri S, Toygar Ö. On the use of DAG-CNN architecture for age estimation with multi-stage features fusion. Neurocomputing. 2019;329(1):300–10. doi:10.1016/j.neucom.2018.10.071. [Google Scholar] [CrossRef]

60. Lou Z, Alnajar F, Alvarez JM, Hu N, Gevers T. Expression-invariant age estimation using structured learning. IEEE Trans Pattern Anal Mach Intell. 2018;40(2):365–75. doi:10.1109/tpami.2017.2679739. [Google Scholar] [PubMed] [CrossRef]

61. Liu H, Lu J, Feng J, Zhou J. Group-aware deep feature learning for facial age estimation. Pattern Recognit. 2017;66(12):82–94. doi:10.1016/j.patcog.2016.10.026. [Google Scholar] [CrossRef]

62. Tan M, Le QV. EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning (ICML); 2019 Jun 9–15; Long Beach, CA, USA. p. 6105–14. [Google Scholar]

63. Tan M, Le QV. EfficientNetV2: smaller models and faster training. In: International Conference on Machine Learning. Proceedings of the 38th International Conference on Machine Learning; 2021 Jul 18–24; Virtual. p. 10096–106. [Google Scholar]

64. Hou Q, Zhou D, Feng J. Coordinate attention for efficient mobile network design. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2021 Jun 20–25; Nashville, TN, USA. p. 13708–17. doi:10.1109/cvpr46437.2021.01350. [Google Scholar] [CrossRef]

65. Zhang Z, Song Y, Qi H. Age progression/regression by conditional adversarial autoencoder. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21–26; Honolulu, HI, USA. p. 4352–60. doi:10.1109/cvpr.2017.463. [Google Scholar] [CrossRef]

66. Huang GB, Learned-Miller E. Labeled faces in the wild: updates and new reporting procedures. Amherst, MA, USA: University of Massachusetts Amherst; 2014. [Google Scholar]

67. Rothe R, Timofte R, Van Gool L. DEX: deep EXpectation of apparent age from a single image. In: Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW); 2015 Dec 7–13; Santiago, Chile. p. 252–7. doi:10.1109/iccvw.2015.41. [Google Scholar] [CrossRef]

68. Sajjad M, Zahir S, Ullah A, Akhtar Z, Muhammad K. Human behavior understanding in big multimedia data using CNN based facial expression recognition. Mob Netw Appl. 2020;25(4):1611–21. doi:10.1007/s11036-019-01366-9. [Google Scholar] [CrossRef]

69. Perumandla D. Drowsiness dataset. [cited 2025 Sep 20]. Available from: https://www.kaggle.com/datasets/dheerajperumandla/drowsiness-dataset. [Google Scholar]