Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition (2023)

Abstract: Audio-visual speech recognition (AVSR) provides a promising solution to ameliorate the noise-robustness of audio-only speech recognition with visual information. However, most existing efforts still focus on audio modality to improve robustness considering its dominance in AVSR task, with noise adaptation techniques such as front-end denoise processing. Though effective, these methods are usually faced with two practical challenges: 1) lack of sufficient labeled noisy audio-visual training data in some real-world scenarios and 2) less optimal model generality to unseen testing noises. In this work, we investigate the noise-invariant visual modality to strengthen robustness of AVSR, which can adapt to any testing noises while without dependence on noisy training data, a.k.a., unsupervised noise adaptation. Inspired by human perception mechanism, we propose a universal viseme-phoneme mapping (UniVPM) approach to implement modality transfer, which can restore clean audio from visual signals to enable speech recognition under any noisy conditions. Extensive experiments on public benchmarks LRS3 and LRS2 show that our approach achieves the state-of-the-art under various noisy as well as clean conditions. In addition, we also outperform previous state-of-the-arts on visual speech recognition task.

## introduction the world surrounding us involves multiple modalities, including vision, audio, text, etc., which complement each other and jointly comprise human perception @xcite . audio-visual speech recognition (avsr) leverages both audio and visual modalities to understand human speech, which provides a promising solution to ameliorate the noise-robustness of audio-only speech recognition with noise-invariant lip movement information @xcite ). 1 code is available at https://github.com/yuche n005/univpm . (b) video stream audio stream restored clean audio recognition mapping & transfer fusion fusion recognition viseme bank phoneme bank figure 1: illustration of noisy audio-visual speech recognition. (a) mainstream avsr approaches with noise adaptation. (b) our framework constructs visemephoneme mapping for modality transfer, which restores clean audio from visual signals to enable speech recognition under any noisy conditions. however, most existing efforts still focus on audio modality to improve noise-robustness considering its dominance in avsr, where audio modality contains much richer information to represent speech content than visual modality @xcite . current mainstream approaches introduce noise adaptation techniques to improve robustness foot_0 , inspired by robust speech recognition @xcite . most of them leverage noise-corrupted training data to strengthen robustness @xcite @xcite , and recent works extend it to selfsupervised learning scheme @xcite . based on that, latest works introduce speech enhancement as front-end to denoise before recognition @xcite . despite the effectiveness, these methods are usually faced with two practical challenges. first, they require abundant labeled noisy audio-visual data for network training, which is not always available in some real-world scenarios @xcite . second, the well-trained model may not adapt to new-coming noise scenes in practical applications 2 , resulting in less optimal model generality @xcite . therefore, our research idea in this paper is leveraging visual modality to develop a general noise-robust avsr system while without dependence on noisy training data. we may gain some inspirations from human perception mechanism of noisy audio-visual speech. neuroscience studies @xcite find that human brain will unconsciously rely more on the lip movement to understand speech under noisy conditions @xcite . during this process, instead of directly recognizing lip movement, human brain will first transfer it to speech signal in auditory cortex for further understanding @xcite . with prior knowledge of lip-audio mapping, human brain can restore informative clean audio from lip movement under any noisy conditions to aid in speech understanding @xcite . motivated by above observations, we propose a universal viseme-phoneme foot_1 mapping approach (univpm) to implement modality transfer, which can restore clean audio from lip movement to enable speech recognition under any noisy conditions. we first build two universal memory banks to model all the visemes and phonemes via online balanced clustering. based on that, an adversarial mutual information estimator is proposed to construct strong viseme-phoneme mapping, which enables final lip-to-audio modality transfer via retrieval. as a result, our system can adapt well to any testing noises while without noisy training data. empirical results show the effectiveness of our approach. our contributions are summarized as: viseme-phoneme mapping. viseme-phoneme mapping is important to many visual-audio learning tasks, including speech recognition (chan et al., fusion restored clean audio recognizer "i am happy" (b) inference on any noisy data visual front-end audio front-end transfer fusion restored clean audio "i am happy" audio stream (clean) video stream audio stream (noisy) viseme bank phoneme bank clustering recognizer figure 2: illustration of our proposed univpm. (a) training on clean audio-visual data to construct universal viseme-phoneme mapping. (b) inference on any noisy data with restored clean audio from modality transfer.