A Learning-based Multi-Frame Visual Feature Framework for Real-Time Driver Fatigue Detection (2025)

Abstract: Driver fatigue is a significant factor contributing to road accidents, highlighting the need for reliable and accurate detection methods. In this study, we introduce a novel learning-based multi-frame visual feature framework (LMVFF) designed for precise fatigue detection. Our methodology comprises several clear and interpretable steps. Initially, facial landmarks are detected, enabling the calculation of distances between eyes, lips, and the assessment of head rotation angles based on 68 identified landmarks. Subsequently, visual features from the eye region are extracted, and an effective visual model is developed to accurately classify eye openness. Additionally, features characterizing lip movements are analyzed to detect yawning, thereby enriching fatigue detection through continuous monitoring of eye blink frequency, yawning occurrences, and head movements. Compared to conventional single-feature detection approaches, LMVFF significantly reduces instances of fatigue misidentification. Moreover, we employ various quantization and compression techniques for multiple computation stages, substantially reducing the latency of our system and achieving a real-time frame rate of 25-30 FPS for practical applications.

## introduction with the rapid expansion of urban road networks, as well as the increasing mileage of train and highspeed rail lines, the frequency of traffic accidents has risen significantly. among the various factors contributing to these accidents, fatigued driving has become one of the most concerning. the fastpaced demands of modern life have led to an alarming increase in the number of drivers experiencing fatigue behind the wheel. as shown in fig. 1 , fatigue severely @xcite and heightens the risk of poor decision-making, all of which can result in catastrophic outcomes, including serious injuries, fatalities, and significant property damage. although intelligent road perception @xcite and autonomous driving @xcite technologies emerge, fatigue warning remains the most effective method to prevent fatigue-related accidents @xcite @xcite . fatigued driving poses significant risks, largely because drivers often fail to recognize its symptoms until it is too late. a particularly hazardous symptom is micro-sleep, brief, involuntary episodes of attention loss lasting only a few seconds. during these lapses, vehicles traveling at high speeds may drift off the road or collide with other vehicles. relevant research @xcite has indicated that fatigued driving can be as perilous as driving under the influence of alcohol, as both conditions severely impair the ability of a driver to respond promptly and effectively to unexpected changes in traffic conditions. given the severe risks associated with fatigued driving, researchers globally have emphasized the urgent need for effective preventive measures. to address this critical issue, various advanced detection algorithms @xcite @xcite have been developed to monitor driver fatigue in real-time. these algorithms leverage physiological signals, facial analysis, and behavioral patterns to detect early signs of drowsiness and issue timely warnings, thereby reducing the likelihood of accidents. the ongoing enhancement and deployment of these technological solutions are vital in mitigating the dangers posed by fatigued driving and improving road safety for all drivers. existing fatigued driving detection methods can be classified into two categories, i.e., traditional and learning-based methods. traditional methods rely on rule-based approaches, statistical analysis, and signal processing techniques. since handcrafted features from traditional methods @xcite have limited representation capability, they often fail to capture complex fatigue patterns accurately and may exhibit poor adaptability under varying real-world driving conditions. due to their powerful modeling capability, learning-based approaches @xcite @xcite have become increasingly preferred for developing advanced driver fatigue detection systems, presenting promising opportunities. however, existing learning-based methods often suffer from suffer from narrow feature coverage, limited interpretability, and excessively bulky models. to address these issues, we propose a learningbased multi-frame visual feature framework named lmvff, which integrates multi-dimensional feature modeling within a lightweight design. first, the framework performs facial landmark detection and calculates distances between eyes, distances within the lip region, and head rotation angles derived from 68 facial landmarks to preliminarily assess fatigue. it then extracts visual features from the eye region to construct a visual model that determines eye openness. subsequently, it integrates open and closed lip-region features to identify yawning behaviors, evaluating driver fatigue based on blinking and yawning frequencies to minimize misjudgments caused by single-feature reliance. comprehensive experiments on various datasets demonstrate that the proposed method achieves high accuracy in fatigue detection. ## architecture the proposed learning-based multi-frame visual feature framework (lmvff) for fatigued driving detection is illustrated in fig. 2 . since our multiframe version (see fig. 2 (a) ) is built on the singleframe one (see fig. 2 (b) ), this section will detail the single-frame method and then transition the multi-frame version. face detection. face images collected from various scenarios in train and vehicle cabins are initially pre-processed. a face detection model is trained using libfacedetection @xcite , achieving real-time performance at approximately 150 fps. to minimize false positives, we also integrate the onet network from mtcnn @xcite , effectively balancing speed and accuracy. face landmark detection. leveraging publicly available and self-collected datasets, we annotate images with 68 face landmarks. a landmark detection model is developed using mo-bilenetv2 @xcite combined with depthwise separable convolutions @xcite and a weighted loss function to enhance precision in landmark localization. mobilenetv2 is a highly efficient convolutional neural network architecture designed for resource-constrained devices. its lightweight design makes it particularly wellsuited for real-time tasks such as face landmark detection, where speed and accuracy are critical. face alignment and feature extraction. after landmark detection, the face image is aligned by computing a similarity transformation matrix @xcite based on these facial landmarks: ## experiment training and testing data: we utilize multiple publicly available facial landmark datasets, such as 300w @xcite and aflw @xcite , for pretraining. additionally, we collect a substantial amount of frontal and profile face data as the test set. the test data is divided into five groups (as shown in evaluation metrics: for the landmark detection task, we employ nme (normalized mean error @xcite ) as the evaluation metric, which measures the average distance between the detected and ground-truth landmark to assess the facial recognition accuracy of the model. we adopt mse (mean squared error @xcite ) as the loss function in training to optimize the landmark prediction performance. meanwhile, for face detection task, we employ mtcnn @xcite network, while landmark detection is performed using different versions of mobilenetv2 and vgg models, with tests conducted on various network depths (as shown in ## conclusion in conclusion, we present a learning-based approach for detecting driver fatigue through multiframe facial analysis, incorporating multiple optimization strategies to enhance detection accuracy and system latency. we employ mobilenetv2 as the backbone for landmark detection and apply enhanced facial feature weights to improve eye landmark perception accuracy. based on these perceived landmarks, we crop the eye region and introduce an effective eye state classification model to accurately identify closed eyes. similarly, the mouth region is cropped using facial landmarks, enabling a new mouth state classification model that significantly boosts yawning detection accuracy. lastly, we leverage euler angles to determine head orientation. collectively, the features from these techniques enable robust identification of potential driver fatigue.