Speaker identification refers to the task of localizing the face of a person who has the same identity as the ongoing voice in a video. This task not only requires collective perception over both visual and auditory signals, the robustness to handle severe quality degradations and unconstrained content variations are also indispensable. In this paper, we describe a novel multimodal Long Short-Term Memory (LSTM) architecture which seamlessly unifies both visual and auditory modalities from the beginning of each sequence input. The key idea is to extend the conventional LSTM by not only sharing weights across time steps, but also sharing weights across modalities. We show that modeling the temporal dependency across face and voice can significantly improve the robustness to content quality degradations and variations. We also found that our multimodal LSTM is robustness to distractors, namely the non-speaking identities. We applied our multimodal LSTM to The Big Bang Theory dataset and showed that our system outperforms the state-of-the-art systems in speaker identification with lower false alarm rate and higher recognition accuracy. Speaker identification is one of the most important building blocks in many intelligent video processing systems such as video conferencing, video summarization and video surveillance, etc. It aims to localize the face of the speaker associated with the ongoing voices. To achieve this task, collective perception over both visual and auditory signals is indispensable. In the past few years, we observe the rapid advances in face recognition and speech recognition respectively by using Convolutional Neural Networks are not uncommon. These effects significantly degrade the performance of many existing CNN based methods. Second, the state-of-the-art convolutional network based face model is trained with still images. Its application in sequential data as well as its connection to recurrent networks is less explored and it is not straightforward to extend CNN based methods to the multimodal learning setting. Third, a practical system should be robust enough to reject distractors, the faces of non-speaking persons indicated by the red bounding boxes in figure 1b, which adds additional challenges to the task. Despite the rich potential of both CNN and RNN for facial and auditory data, it is still unclear if these two
This work is further extended in the temporal domain with multimodal LSTM by REF .
13 F eb 2 01 6 Look , Listen and Learn-A Multimodal LSTM for Speaker Identi fication