O pen SR : Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment (2023)

Abstract: Speech Recognition builds a bridge between the multimedia streaming (audio-only, visual-only or audio-visual) and the corresponding text transcription. However, when training the specific model of new domain, it often gets stuck in the lack of new-domain utterances, especially the labeled visual utterances. To break through this restriction, we attempt to achieve zero-shot modality transfer by maintaining the multi-modality alignment in phoneme space learned with unlabeled multimedia utterances in the high resource domain during the pre-training, and propose a training system Open-modality Speech Recognition ( OpenSR ) that enables the models trained on a single modality (e.g., audio-only) applicable to more modalities (e.g., visual-only and audio-visual). Furthermore, we employ a cluster-based prompt tuning strategy to handle the domain shift for the scenarios with only common words in the new domain utterances. We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods. To the best of our knowledge, OpenSR achieves the state-of-the-art performance of word error rate in LRS2 on audio-visual speech recognition and lip-reading with 2.7% and 25.0%, respectively.

## introduction speech recognition @xcite @xcite ) (e.g., audio-visual speech recognition known as avsr) transcribs visual and audio data into text form, building a bridge between multi-media speech @xcite @xcite and natural language @xcite ., * equal contribution. † corresponding author. 2022, 2021; @xcite . among them, asr (automatic speech recognition) and lip-reading (vsr, visual speech recognition) are twin tasks transcribed using only audio and only vision, respectively. audio utterances with clear pronunciation are sufficient for asr training, most of which can be easily collected from recordings of telephone conversations and audiobooks @xcite . while current lip-reading training systems require mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions around the lip area are clearly captured @xcite . the significant difficulty of collecting labeled visual utterances hinders the training of lip-reading models suitable for new domains or low-resource domains, resulting in the relatively low-speed development of domain-specific lip-reading models, compared with asr models. since audio speech is easier to obtain, can we use audio utterance alone to train lip-reading models for target domains? humans, once they have mastered a language, can immediately predict the lip movements of a phoneme @xcite . similarly, if we align the phoneme represen-tations of acoustic and image fragments with each other, we can then apply the term distribution and syntax specification of the target domain learned from the audio utterance to lip-reading. building on this novel idea, we employ the audio-visual aligned encoder, such as av-hubert, co-trained on a large number of multi-modality high resource domain utterances, to align the multi-modality utterances in the same phoneme space, and train a domain-specific decoder from phoneme space to text using the labeled audio-only utterances. since the audio-visual encoder is trained to embed different modalities of the same phoneme near one another (for instance, the visual phoneme feature of /tu:/ is aligned with its audio phoneme feature), we can flexibly transfer the knowledge of the target domain (e.g., the mapping from phoneme /tu:/ to homophones two and too, and the syntax used to distinguish them) from a single modality (i.e., audio) to more modalities (i.e., visual-only and audio-visual). as shown in figure 1 , the zero-shot lip-reading performance of our method is even better than most full-shot methods. furthermore, we propose to enhance the discrimination of visually similar words (e.g., pet and bet, which are audio distinguishable but visually similar @xcite ) with more readily available utterances containing only common words. however, the uneven word distribution causes the sequence modeling to be easily corrupted by domain shift during fine-tuning @xcite . we propose a cluster-based prompt tuning strategy, cluster prompt, only tuning the limited parameters of the fine-grained prompt-embedding of each visual phoneme cluster, to prevent excessive impact on sequence modeling. the main contributions are as follows: ## ethical discussion lip-reading has many applications, including instruction dictation in public areas or a noisy environment and information accessibility for the hearing impaired. opensr makes it possible to quickly build domain-specific lip-reading models for low-resource domains (lack of labeled visual utterances), which greatly enhances the fairness of lip-reading technology across the domains and languages. for speech recognition, there may be concerns about the risk of information leakage. but in fact, as mentioned above, the lip-reading model has a relatively high requirement on the visual utterances used for lip-reading, requiring mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions around the lip area are clearly captured. in general, only close-range cameras or online meetings have similar video conditions, which ensures that the lip-reading model will not be abused in potentially privacy revealing scenarios such as surveillance videos.