Visual voice activity detection as a help for speech source separation from convolutive mixtures
Audiovisual speech source separation consists in mixing visual speech processing techniques (e.g., lip parameters tracking) with source separation methods to improve the extraction of a speech source of interest from a mixture of acoustic signals. In this paper, we present a new approach that combines visual information with separation methods based on the sparseness of speech: visual information is used as a voice activity detector (VAD) which is combined with a new geometric method of separation. The proposed audiovisual method is shown to be efficient to extract a real spontaneous speech utterance in the difficult case of convolutive mixtures even if the competing sources are highly non-stationary. Typical gains of 1820dB in signal to interference ratios are obtained for a wide range of (22) and (33) mixtures. Moreover, the overall process is computationally quite simpler than previously proposed audiovisual separation schemes.
speech source separation convolutive mixtures visual speech processing voice activity detector speech enhancement highly non-stationary environments