Classical visual attention models neither consider social cues, such as faces, nor auditory cues, such as speech. However, faces are known to capture visual attention more than any other visual features, and recent studies showed that speech turn-taking affects the gaze of non-involved viewers. In this paper, we propose an audiovisual saliency model able to predict the eye movements of observers viewing other people having a conversation. Thanks to a speaker diarization algorithm, our audiovisual saliency model increases the saliency of the speakers compared to the addressees. We evaluated our model with eye-tracking data, and found that it significantly outperforms visual attention models using an equal and constant saliency value for all faces.
By using speaker diarization algorithm REF when the weight for active speakers was increased, their audio-visual attention model significantly outperformed the visual saliency model with equal weights for all faces.
An audiovisual attention model for natural conversation scenes
2014 IEEE International Conference on Image Processing (ICIP)
2014 IEEE International Conference on Image Processing (ICIP)
Computer Science