Driver body condition monitoring system based on human-computer interaction

. Tired driving is still a significant cause of traffic accidents, often causing huge economic losses to society.Therefore, how to avoid driver fatigue has become a meaningful problem. The purpose of this paper is to review and summarize various research methods of driver fatigue monitoring system at home and abroad, so as to reduce the probability of traffic accidents caused by fatigue driving. The main content of this paper is to review the existing fatigue driving monitoring technologies such as electrocardiogram (ECG), electroencephalogram (EEG), HRV, eye tracking system, CNN, facial state analysis, emotion recognition, 5G and brain wave recognition, and infer the fatigue level of drivers by using these technologies to extract various clues that usually represent the level of human vigilance. Assess cognitive load and identify emotional states to monitor tired driving. By discussing the advantages and disadvantages of various fatigue monitoring systems at home and abroad, this paper proposes an innovative method, which transforms the previous single-mode monitoring system into a multi-mode monitoring system, integrates multiple biosensors, extracts relevant features from the biosensor data and processes the data reasonably in order to understand the driver ’ s physiological state more comprehensively. This method is expected to improve the accuracy of fatigue monitoring system.


Introduction
Cars have become one of the most commonly used means of transportation.While people enjoy the convenience of transportation brought by cars, they also bring a series of problems:Frequent occurrences of traffic accidents have led to a significant magnitude of human casualties and substantial economic ramifications.As documented in the 2019 report issued by the China National Bureau of Statistics, the scope of traffic accident victims in China during that year amounted to 275,125, with the resultant fatality count from these incidents standing at 52,388 [1].Noteworthy efforts have been undertaken by both domestic and international scientific communities to meticulously investigate systems capable of real-time detection of drivers' physiological conditions and facial expressions.These systems serve the dual purpose of providing timely driver alerts and intervening in vehicular operations, thereby yielding profound advancements in mitigating the incidence of traffic-related mishaps.The objective of this work is to review the previous research results on driver monitoring system, propose innovative new ideas, make critical evaluation, and finally predict the future development direction.
The real-time monitoring system for the assessment of driver physiological and affective states assumes a pivotal role in upholding road safety standards.This system is designed with the primary objective of preempting traffic accidents arising from unanticipated exigent circumstances, including but not limited to fatigue and illness, which can collectively give rise to compromised driver states.Previous research in this area has explored multiple approaches to driver monitoring, including a range of physiological and behavioral indicators.These studies have utilized various techniques such as electrocardiograms (ECG), electroencephalography (EEG) [2], electromyography (EMG), electrooculogram (small town) [3], facial state analysis, emotion recognition, and implementation of neural networks and deep learning algorithms [4].By utilizing these techniques, researchers have made significant progress in detecting driver fatigue, assessing cognitive load, and identifying emotional states.
However, developing an integrated monitoring system that combines these different methods is essential to obtain accurate and reliable results to prevent dangerous driving due to unexpected situations.This paper proposes an integrated biosensor to detect driver fatigue.Through the meticulous surveillance of vital indicators and physiological parameters, encompassing notable metrics such as heart rate [5], electroencephalogram (EEG) activity [6], and ocular motility patterns [7], the system aptly discerns the manifestation of fatigue and offers prognostic insights into the driver's capacity for sustained vigilance.When the system detects signs of fatigue reaching a critical threshold, it can prompt the driver to stop and rest within a specified time frame, ensuring improved safety during long journeys.To sum up, the development of a comprehensive driver condition monitoring system is essential to ensure road safety.By integrating biosensors to detect a driver's fatigue threshold, the system can help prevent traffic accidents caused by poor driving and improve overall road safety.
This paper is divided into three chapters.Chapter 1 will undertake an extensive review of methodologies employed in both domestic and international research domains.The subsequent chapter, designated as the deliberative segment, will commence by conducting a comprehensive analysis of the merits and demerits inherent in prevailing methodologies.Subsequently, it will engage in a nuanced examination of points of innovation within the realm of study.The third chapter is the conclusion, which will first review the results of our work, and then put forward our prospects for future research.Figure 1 shows the outline of this thesis.

An EEG-based and ECG-based driver fatigue detection.
There have been research findings, the physiological signal, such as eye features, ECG, EEG is a good indicator of driver's fitness and fatigue.
The uses EEG measurements, the foundation for which most effective offline tiredness monitoring device is built.Additionally, An online iteration denoted as "Mind Switch" has been developed, which employs a specialized headband outfitted with electrodes positioned upon the driver's cranium to ascertain cerebral wave patterns [8].
EEG signals have emerged as one of the most reliable and predictive indicators for describing an individual's level of alertness, among numerous physiological indicators available.The software, created by Saroj K.L. Lal, can assign EEG data from these four stages into four channels represented by color panels (see Figure 2), and it can identify four different states of brain function during the transitional phase and arousal phase.This software may one day be used to warn drivers of their fatigue [9].2.1.2.An HRV-based driver fatigue detection.M. Patel et al. investigated using HRV (heart rate variability) as a physiological parameter to assess the level of fatigue after monitoring the level of fatigue using several physiological indicators such as EEG and EOG (See Figure 3).Their study used HRV, which provides quantitative physiological markers of driving fatigue, combined with artificial intelligence systems such as neural networks that provide in-vehicle alerts.It suggests that a warning system could be created [10].Given the advancement in non-intrusive methodologies, heart rate variability (HRV) has emerged as a valuable signal for the real-time assessment of driver fatigue.Rahul Bhardwaj and colleagues, for instance, extracted temporal domain statistics, nonlinear attributes, and frequency domain components from a cohort of 10 participants (see Figure 4).Experimental findings demonstrate that the suggested model can be successfully utilized to track driving condition [11].

An EM-CNN-based driver fatigue detection.
Unlike the above monitoring methods, CNN does not require expensive physiological sensors like traditional physiological parameter monitoring and must be attached to the human skin, causing discomfort to the driver.
In their study, Zuopeng Zhao et al. propose a novel approach by incorporating both eye and mouth features.They adopt a hierarchical conventional MTCNN (Multi-Task Cascaded Convolutional Neural Network) network architecture (see Figure 5).This architecture is utilized for feature extraction and spatial recognition, replacing conventional image processing techniques (see Figure 6).They proposed a detection method based on EM-CNN to evaluate the eyes and oral conditions (see Figure 7) [12].This method has good robustness and is suitable for the actual driving environment.

An eye tracking based Driver Fatigue detection.
Also as a non-invasive system, Qiang et al. developed a prototype computer vision system to monitor driver alertness in real time.First, development and extract usually represents the hardware and image algorithm required for various visual clues that are fatigue (see Figure 8).Then, fatigue is simulated.An elaborate probabilistic framework has been formulated to methodically amalgamate diverse visual cues alongside pertinent contextual information, culminating in the derivation of a robust and uniform fatigue index.These visual cues encompass aspects encompassing eyelid motion, gaze patterns, cranial gestures, and facial demeanor.The fatigue detection system employs eye movement detection to identify driver fatigue and issues a warning within a mere half-second.Additionally, it records the driver's performance in a graphical format, as depicted in Figure 9.The experimental results conclusively demonstrate the system's reliability and effectiveness in monitoring fatigue [13].[14].Zotye Auto M11, M12, and Shaanxi Automobile heavy truck have implemented the driver fatigue monitoring system developed by Tsinghua University.On the other hand, the Hongqi H7 DSM system, developed by FAW Group, relies on the driver's eye and head movement data in conjunction with the vehicle's driving trajectory to assess the driver's current physical condition.It is worth noting that the monitoring systems used by many other automobile companies predominantly rely on foreign products [14].
Undoubtedly, in tandem with the rapid evolution of 5G technology, networking advancements, and novel energy-propelled vehicles, a plethora of novel constituents within the automotive manufacturing domain have arisen.Concurrently, China's scientific and technological community has proffered a multitude of innovative approaches poised to seamlessly accommodate developmental trajectories while engendering enhanced efficacy:

A driver physiological state monitoring system leveraging the synergistic capabilities of 5G
technology and the Internet of Vehicles paradigm.The driver's physical condition monitoring system leverages the advancements in 5G and Internet of Vehicles (IoV) technologies to enhance its capabilities.The design concept of the system is outlined in Figure 10, serving as a reference for the overall architecture and implementation approach.The system employs high-definition infrared cameras and CCD image sensors to capture detailed facial information of the driver, along with the surrounding vehicle environment data.By utilizing advanced algorithms, the system analyzes this data to determine the driver's current driving state.This not only compensates for the inaccuracy of traditional monitoring methods, but also avoids driver discomfort.In addition, when the system detects that the current vehicle is in a state of fatigue driving, it will evaluate the current driver's fatigue level and adopt different early warning methods according to the degree of fatigue driving to reduce discomfort.Misjudgment due to accurate judgment.If the system detects that the two cars are in a state of severe fatigue, it will alert the surrounding vehicles and actively send information to the local traffic management [15].
Figure 10.The design concept of this system [16].

Facial feature detection fatigue driving detection system based on deep learning.
The system applies Open-CV for grayscale preprocessing of real-time captured images (see Figure 10).Following this, directional gradient histogram techniques are utilized to detect and extract the facial area.The Dlib model is then employed to calibrate 68 facial feature points accurately.Subsequently, enhanced algorithms for detecting blinking, yawning, and nodding are utilized to calculate the aspect ratio of the eyes and mouth, as well as the Euler angle of the head posture.These calculated values are compared with corresponding thresholds to determine the driver's fatigue state, initiating appropriate warning measures.Experimental results demonstrate an impressive system accuracy of up to 97% (see Figure 11) [17].

The advantages and limitations of foreign existing methods.
For foreign driver fatigue monitoring methods, the following summarizes the advantages and disadvantages of each technology.In the realm of fatigue assessment utilizing EEG and ECG methodologies, scholars have devised sophisticated fatigue monitoring software.This software demonstrates the capacity to accurately discern the three distinct stages of fatigue through vigilant observation of variations within the EEG spectrum, contingent upon the monitoring of pertinent physiological parameters of the driver.These techniques have proven to be highly accurate and suitable for real-time monitoring, but require physical contact with the driver (e.g., attaching electrodes).It is invasive and can cause impact and distress to the driver [8].
Researchers have devised a fatigue monitoring approach that utilizes neural networks.This method, based on EM-CNN and MTCNN models, accurately tracks facial features and locates key points to monitor driver fatigue.The system's high accuracy and ability to simulate real driving conditions make it an effective solution [12].Studies show that neural networks are typically up to 90% accurate, but validating their accuracy requires large data sets for training, validation and testing.Otherwise, neural network accuracy cannot be fully validated with limited data sets [9].The analysis of heart rate fluctuations provides the basis for the tracking accuracy of neural networks.To enhance the accuracy of neural networks in analyzing heart rate variability (HRV), it is crucial to gather diverse data from subjects with different physiological parameters.This allows the neural network to effectively learn and adapt to the various changes in HRV.Furthermore, studies have demonstrated the potential of combining HRV with neural networks and other artificial intelligence systems to develop internal alarm and early warning devices [10].However, in addition to the fact that reduced HR has been validated to be associated with fatigued driving states, the direction of change is inconsistent for other HRV measures.More specific knowledge is required to understand the changes of HRV at different levels, the causes of fatigue, and its relationship with drivers.The fatigue detection system based on heart rate variability has a wide range of accuracy.However, because of the different in the experimental setup, it is difficult to the result of the comparison research, experiment research and detection performance of large doses of road and modeling independent agent is very poor [14].The driver's fatigue state is monitored through the detection of their eye movements.This approach allows for the recording of the driver's eye movement patterns in graphical form.The system also captures and stores the velocity changes over time and correlates them with changes in clamping pressure.These data are archived on the hard disk for future reference and analysis.Therefore, this method is conducive to monitoring the symptoms of driver fatigue in advance and giving warnings with high accuracy.It's also non-invasive [13].However, due to the immaturity of eye tracking technology, different eye tracking technologies have their own advantages and disadvantages, such as sampling rate, accuracy and interference, etc., so there are still certain limitations in their application in human-computer interaction, mainly including accuracy and freedom, "Midas contact" problem and algorithm problem.

The advantages and limitations of internal existing methods.
For the current driver condition monitoring system in China, there are some undeniable advantages, but there are still some shortcomings.Driver condition monitoring system used in vehicles, the advantages of the acoustic system, such as byd's body red flag of the DSM system, Volvo DAC system, and so on, is that it can record the driver's current real-time physiological signals and vehicle driving.Information.State.Camera and a variety of sensors in the car.Thus, the driver's current state can be monitored and warned more accurately.However, these systems can only warn the driver of fatigue driving and dangerous driving behavior, but they cannot effectively intervene in the current driving state of the vehicle and the driver's driving behavior.Even if intervention can be made, the current vehicle must be equipped with other L2 level driver assistance systems such as ACC and turned on, but these systems are currently only equipped with high-configuration versions of some models and are expensive, and many people do not choose to buy high-configuration models when buying vehicles.Therefore, it is difficult to cooperate with the driver condition monitoring system to effectively intervene the driver under the fatigue driving state.Even some systems such as DAC will only enter the working state when the speed is above 65km/h, and it is difficult to comprehensively monitor the driving state of the driver.
The driver status monitoring system based on 5G and vehicle networking technology offers several advantages.It leverages the low latency and expansive network coverage of 5G technology, alongside high-definition infrared cameras and multiple sensors, to enable real-time and accurate monitoring and analysis of both the vehicle's state and the driver's physiological signals.Additionally, the system is capable of accurately monitoring and analyzing the driver's state of fatigue in real time when detected.The bee fire alarm is used to remind the driver currently driving fatigue.When the driver could not control the current vehicle driving conditions, the system will remind the surrounding vehicles and automatically inform the local traffic management departments.However, the algorithm of such a system is relatively complex, and the physiological signals of human beings and the driving state of the vehicle are greatly affected by other factors, and the safety valve value is unstable, so the system is prone to misjudgment and unnecessary impact on the surrounding vehicles, traffic management departments and drivers.For the driver status recognition system based on EEG signals, the system can judge the current status of the driver more accurately, and has strong real-time, and has a relatively abundant warning time.However, this system requires monitoring devices to be in contact with the driver's relevant body parts, and this kind of mandatory interference is easy to make it difficult for the driver to drive easily and produce resistance, so it is difficult to promote and apply.

Introduce innovative approaches
Based on existing research, the following are innovative ideas for driver monitoring using biosensor technology in combination with machine learning algorithms: The utilization of multiple biosensors in a multi-mode monitoring system enables a holistic perspective on the driver's physiological state [18].The integration of heart rate monitor, skin conducting sensor, eye tracked device and EEG sensor can fully evaluate the driver's pressure level, cognitive load, fatigue and emotional state.This method makes the driver more accurate, more detailed, and improves safety and personal help.
Implementing an approach to integrated monitoring of drivers using biosensors involves several steps and considerations.Here is a rough overview of how to implement the process (see Figure 13): Through the integration of biosensor technology and advanced machine learning algorithms, a powerful and efficient driver monitoring system can be developed, considering the driver's physiological state for enhanced safety and attentiveness.Yet, prior to implementation in real-world environments, rigorous testing and validation are imperative to ensure the system's reliability and security.
The proposed driver monitoring approach in this chapter offers innovative advantages through the utilization of biosensor technology and machine learning algorithms.However, it is essential to acknowledge that certain limitations exist alongside these advancements.First, a major weakness is the lack of extensive experimental data to support the system's performance in a variety of driving scenarios and conditions.In addition, individual differences between drivers, such as age, health status, and physiological changes, can lead to the complexity of using biosensor data to accurately interpret and predict their state.The technical feasibility of seamlessly integrating multiple biosensors in a driving environment and ensuring their long-term reliability and durability is a major challenge.Addressing these limitations is essential to improve the effectiveness of the system and ensure its usefulness in practical applications.Robust testing, continuous data collection, and advances in sensor technology are important steps to overcome these challenges and develop comprehensive and reliable driver monitoring solutions.

Conclusion
This paper has reviewed various research methods on driver monitoring system and facial expression recognition system at home and abroad.Foreign research methods show high accuracy in monitoring driver fatigue and alerting driver fatigue, but some methods are intrusive or affected by external factors.In China, the current on-board driver condition monitoring system has advantages in accurately capturing physiological signals and driving states, but lacks effective intervention capabilities.The main findings of this paper show that a machine learning model integrating multiple biosensors can provide a comprehensive understanding of a driver's physiological state, thereby improving safety and personalized assistance.The implications of the conclusions of this paper for future research are to fill the gaps in the current driver integrated biosensor monitoring system and improve road safety.Innovative approaches such as real-time AI analytics and personalized monitoring are expected to develop in the future.Future research should focus on AI implementation, personalized monitoring, and addressing ethical issues to pave the way for safer driving practices and positive societal impacts.

Authors Contribution
All the authors contributed equally and their names were listed in alphabetical order.

Figure 1 .
Figure 1.The outline of this thesis.

Figure 2 .
Figure 2. The data distribution into an alert (green) on the panel: Phase 1 represents the transition from alertness to exhaustion, Phase 2 the transitional-post transitional phase, and Phase 3 the post-transitional phase [9].

Figure 4 .
Figure 4.The average diagrams for 10 subjects were extracted during a two-hour simulated driving activity, considering (a) time domain (HR) and (b,c) frequency domains (HR) and(B, C) frequency domain (high-frequency power, low-frequency power).The calculations were performed every 5 minutes[11].

Figure 7 .
Figure 7. Flow diagram of a proposed method for driver fatigue detection [12].

Figure 11 .Figure 12 .
Figure 11.The caculate of faciai feature detection system [17].2.2.4.Brainwave-based driver condition monitoring device.The fatigue driving early warning device is composed of EEG acquisition module, fatigue detection integration module, and alarm device (see Figure 12(a)) for specific composition), and the brainwave acquisition uses the TGAM module of Shennian Technology Company (as shown in Figure 12(b)), which is more efficient and accurate for the collection of brainwave signals.By conducting software debugging, a waveform diagram is obtained.This waveform is transmitted using Bluetooth wireless technology to the frequency analysis and processing module.The module is responsible for analyzing and processing EEG signals, enabling the identification of frequency patterns associated with driver fatigue.Once identified, the system triggers the early warning device to emit a timely warning signal[18].
Proceedings of the 2023 International Conference on Machine Learning and Automation DOI: 10.54254/2755-2721/40/20230628 2.2.Summary of domestic research methods 2.2.1.The current status of the driver's condition monitoring system in China.Currently, in China, where science and technology continue to advance, many research results have been successfully applied to automobiles, for example, BYD BAWS system through infrared sensors, cameras and other accessories to collect the driver's eyes, head information to monitor driving fatigue.Geely Auto introduced Volvo's DAC monitoring technology, which determines the driver's current state by monitoring vehicle operation information, driver head information, and the driver's eye state