6DDirect H+B: Body-Aware Head Pose Estimation
In a well-functioning democratic constitutional state, it is crucial for professionals such as journalists and politicians to work unimpeded, yet threats against them have increased, necessitating surveillance. Manual surveillance requires extensive manpower and is limited in effectiveness, leading to the adoption of computer vision systems. Our research aims to enhance surveillance by accurately predicting head and body rotation, as well as gaze direction, in surveillance footage.
To achieve this, we developed a model named 6DDirect H+B that accurately determines the 6D poses of head and body for multiple individuals in surveillance images. This model addresses challenges such as occlusions, varying subject distances from the camera, and diverse lighting conditions. By integrating localization, classification, and rotation learning within a unified framework using a fine-tuned YOLOv5 backbone, our approach enhances the accuracy of rotation estimation. Then, we apply 6DDirect H+B to the task of gaze direction estimation, using an LSTM to leverage changes in head and body rotations over time to predict where a person is looking, to demonstrate the effectiveness of our approach.
In this repository, we provide the weights for our trained models and small part of the data for GazeNet demo. For code, please see our repository.
Head Pose and Body Orientation
These models predict head and/or body 3D rotation using the 6D representation.
Models trained on AGORA
- 6DDirectMHP: agora_m_1280_e300_r40_lw010_bs32_H.pt
- 6DDirect H+B: agora_m_1280_e300_t40_bs32_b04_h06.pt
Models trained on CMU
- 6DDirectMHP: cmu_m_1280_e200_t40_bs32_H.pt
- 6DDirect H+B: cmu_m_1280_e200_t40_bs32_b00_h02.pt
Models trained on GAFA
- 6DDirect H+B: gafa_ptAGORA_720_e50_t40_b128_b04_h06.pt
Gaze Direction Estimation
This LSTM predicts 3D gaze direction by using the head and body rotations over 7 frames.
- GazeNet: gazenet.ckpt
Data for GazeNet Demo
To test GazeNet on a small subset of GAFA data, we have this zip.
- kitchen: kitchen.zip