VHASR : A Multimodal Speech Recognition System With Vision Hotwords (2024)

Abstract: The image-based multimodal automatic speech recognition (ASR) model enhances speech recognition performance by incorporating audio-related image. However, some works suggest that introducing image information to model does not help improving ASR performance. In this paper, we propose a novel approach effectively utilizing audio-related image information and set up VHASR, a multimodal speech recognition system that uses vision as hotwords to strengthen the model’s speech recognition capability. Our system utilizes a dual-stream architecture, which firstly transcribes the text on the two streams separately, and then combines the outputs. We evaluate the proposed model on four datasets: Flickr8k, ADE20k, COCO, and OpenImages. The experimental results show that VHASR can effectively utilize key information in images to enhance the model’s speech recognition ability. Its performance not only surpasses unimodal ASR, but also achieves SOTA among existing image-based multimodal ASR.

## introduction asr model @xcite takes audio as input and produces corresponding transcription. one effective method to improve the model's asr performance is to increase both the volume of training data and the number of model parameters. we are now in the era of large language models (llms) @xcite , which have been developed across various domains @xcite . in the speech domain, there are also many llms that demonstrate impressive asr capabilities @xcite . however, this approach can be expensive. a more cost-effective alternative is to introduce additional information related to speech into the model. this information can be presented in either textual or visual forms. the asr system that utilizes audio-related information from various modalities is referred to as multimodal asr. hotwords, which are terms in certain professional fields or words that are easily confused with other homonyms, are common textual cues. there have been many studies on how to freely customize hotwords and improve the recall of hotwords @xcite . it is also possible to use captions as textual information @xcite . visual cues can be in the form of video or image. audio-visual speech recognition (avsr) enhances the accuracy of speech recognition by capturing lip movement information of characters in video @xcite . image-based multimodal asr extracts visual feature from image associated with speech to correct transcription errors. we abbreviate image-based multimodal asr as ibsr. because the lip movement information of video's role is closely linked to his speech, it influences nearly every word in the transcribed text. in contrast, ibsr only impacts a subset of the words as the image is only associated with specific audio clips @xcite . ibsr currently lacks a universal and effective method for utilizing image information, leading to various experimental results in different studies. some works @xcite have a positive effect by incorporating image information, while others @xcite @xcite , have the opposite effect. in this paper, we propose a novel approach effectively utilizing audio-related image information and set up vhasr, a multimodal speech recognition system that utilizes vision hotwords to enhance the model's speech recognition capability. it calculates the similarity between different modalities to improve the effectiveness of cross-modal fusion. drawing inspiration from text hotwords, we utilize vision transformer (vit) to partition images into multiple visual tokens and consider each visual token as a vision hotword. our system adopts a dual-stream architecture. one stream is the asr stream, which receives audio information and produces transcribed text. the other stream is the vision hotwords (vh) stream, which receives vision hotwords and audio hidden features, and generates corresponding text. in the vh stream, we calculate the similarity between audio and vision hotwords to reduce the weight of vision hotwords with low similarity. this process helps to extract fine-grained image information. when inferring, vhasr first transcribes the text separately from the asr stream and the vh stream, and then merges the outputs. we ensure the high accuracy of the merged output by comparing the similarity of different modalities. specifically, we first calculate the audio-image similarity to discard the vh stream if the similarity is low. then, we calculate the image-text token similarity to compare the asr stream and vh stream outputs by tokens. finally, tokens with higher similarity are selected for the merged output. we evaluate the proposed model on four datasets: flickr8k, ade20k, coco, and openimages. the experimental results show that vhasr can effectively utilize critical information in images to improve the model's asr performance. its perfor-mance is not only better than ordinary unimodal asr models but also surpasses existing ibsr models. the contributions of this paper are as follows: ## related work image-based multimodal asr. @xcite introduce a multimodal speech recognition scenario which utilizes images to assist the language model in decoding the most probable words and rescoring the top hypotheses. @xcite propose an end-to-end multimodal asr system implemented by lstm (graves and graves, 2012). they apply visual adaptive training @xcite to finetune a pretrained asr model with visual data, and leverage visual information to initialize model's encoder and decoder. srinivasan et al. (2020b) present a model for multimodal asr that utilizes visual feature from object proposals. they integrate the features of object proposals into a visual representation by utilizing their attention distribution as weights, and incorporate this visual representation into the model via a hierarchical attention mechanism. oneat , ȃ and cucu (2022) combine speech and visual embeddings using two fusion approaches. one approach fuses along the embedding dimension, and another fuses along the sequence dimension. they find that the first method performs better. han et al. (2023) propose a novel multimodal asr model called vilas, which is based on the continuous integrate-and-fire (cif) mechanism (dong and xu, 2020). it can integrate image and caption information simultaneously or separately to facilitate speech recognition. chang et al. (2023) propose a multimodal asr system for embodied agents. their model is based on transformer (vaswani et al., 2017), where the visual feature vector is concatenated to the decoder's input word embedding at every timestep of generation. function of image information. srinivasan et al. (2020a) conduct the experiment called audio cor-speech encoder predictor speech decoder vision encoder vh encoder speech adapter asr stream 𝑋 1:𝑇 𝐻 1:𝑇 𝐸 sampler vision adapter vh decoder 𝐻 1:𝑁 ′ 𝐷 𝑁 ′ 𝑌 𝐴 ′′ 𝑌 𝐴 ′ ℒ 𝐴𝑆𝑅 ℒ 𝑀𝐴𝐸 𝑌 𝑉 ′ 𝑌 𝑀 ′ ℒ 𝑉𝐴 𝐸 1:𝑁 ′ 𝑎 𝐸 1:𝑁 ′ 𝑎 𝐸 1:𝑁 ′ 𝑐 𝐸 1:𝑁 ′ 𝑠 ℒ 𝑉𝐻 𝑍 𝐻 1:𝐾 𝑉 𝐻 1:𝐾 𝐸 𝑆 𝑉 ′ 𝐴 𝑆 𝑉𝐴 𝑆 𝑉𝑇 ′ ruption, in which they mask the words related to nouns and places with silence and white noise, respectively. the study demonstrates that visual representations help in recovering words that are masked in the input acoustic signal. srinivasan et al. (2020c) think the previous work has only masked a fixed set of words in the audio, which is an unrealistic setting. so, they propose a method called randwordmask, where masking can occur for any word segment to improve the audio corruption experiment. kumar et al. (2023) propose two effective asr error correction methods: one employs a gated fusion method to concatenate visual and textual features, while the other utilizes image's caption as correction model's prompt. both methods demonstrate that visual information helps restoring incorrect words in transcription. in short, image information helps to recover incorrect words in transcription that are caused by masked acoustic signals or asr model's error. ## conclusion we propose vhasr, a multimodal speech recognition system that utilizes vision hotwords to strengthen the model's speech recognition ability. our system features a dual-stream architecture, consisting of an asr stream and a vh stream that firstly transcribe separately and then combine their outputs. by leveraging vision hotwords, the vh stream concentrates on key visual information, allowing for precise transcription of words associated with images. in the merging phase, the vh stream assists the asr stream in correcting any mis-transcribed words related to images, thereby ensuring high accuracy in the final transcription. we conduct comprehensive experiments on flickr8k, ade20k, coco, and openimages, which showcase the effectiveness of vision hotwords and the robust asr performance of vhasr.