Driver Drowsiness Behavior Detection and Analysis Using Vision-Based Multimodal Features for Driving Safety

Rui Li, Howard Brand, Aditya Gopinath, Srivatsav Kamarajugadda, Liang Yang, Weitian Wang, Bing Li

Research output: Contribution to journalConference articlepeer-review

5 Scopus citations


Driving inattention caused by drowsiness has been a significant reason for vehicle crash accidents, and there is a critical need to augment driving safety by monitoring driver drowsiness behaviors. For real-time drowsy driving awareness, we propose a vision-based driver drowsiness monitoring system (DDMS) for driver drowsiness behavior recognition and analysis. First, an infrared camera is deployed in-vehicle to capture the driver's facial and head information in naturalistic driving scenarios, in which the driver may or may not wear glasses or sunglasses. Second, we propose and design a multi-modal features representation approach based on facial landmarks, and head pose which is retrieved in a convolutional neural network (CNN) regression model. Finally, an extreme learning machine (ELM) model is proposed to fuse the facial landmark, recognition model and pose orientation for drowsiness detection. The DDMS gives promptly warning to the driver once a drowsiness event is detected. The proposed CNN and ELM models are trained in a drowsy driving dataset and are validated on public datasets and field tests. Comparing to the end-to-end CNN recognition model, the proposed multi-modal fusion with the ELM detection model allows faster and more accurate detection with minimal intervention. The experimental result demonstrates that DDMS is able to provide real-time and effective drowsy driving alerts under various light conditions to augment driving safety.

Original languageEnglish
JournalSAE Technical Papers
Issue numberApril
StatePublished - 14 Apr 2020
EventSAE 2020 World Congress Experience, WCX 2020 - Detroit, United States
Duration: 21 Apr 202023 Apr 2020


Dive into the research topics of 'Driver Drowsiness Behavior Detection and Analysis Using Vision-Based Multimodal Features for Driving Safety'. Together they form a unique fingerprint.

Cite this