  A smart vehicle should be able to monitor the actions and behaviors of the
human driver to provide critical warnings or intervene when necessary. Recent
advancements in deep learning and computer vision have shown great promise in
monitoring human behaviors and activities. While these algorithms work well in
a controlled environment, naturalistic driving conditions add new challenges
such as illumination variations, occlusions and extreme head poses. A vast
amount of in-domain data is required to train models that provide high
performance in predicting driving related tasks to effectively monitor driver
actions and behaviors. Toward building the required infrastructure, this paper
presents the multimodal driver monitoring (MDM) dataset, which was collected
with 59 subjects that were recorded performing various tasks. We use the Fi-
Cap device that continuously tracks the head movement of the driver using
fiducial markers, providing frame-based annotations to train head pose
algorithms in naturalistic driving conditions. We ask the driver to look at
predetermined gaze locations to obtain accurate correlation between the
driver's facial image and visual attention. We also collect data when the
driver performs common secondary activities such as navigation using a smart
phone and operating the in-car infotainment system. All of the driver's
activities are recorded with high definition RGB cameras and time-of-flight
depth camera. We also record the controller area network-bus (CAN-Bus),
extracting important information. These high quality recordings serve as the
ideal resource to train various efficient algorithms for monitoring the driver,
providing further advancements in the field of in-vehicle safety systems.
