Multi-Modal Retrieval For Large Language Model Based Speech Recognition (2024)

Abstract: Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multi-modal large language models, it is important to extend the pure text based methods to incorporate other modalities in retrieval as well for applications across the wide spectrum of machine learning tasks and data types. In this work, we propose multi-modal retrieval with two approaches: kNN-LM and cross-attention techniques. We demonstrate the effectiveness of our retrieval approaches empirically by applying them to automatic speech recognition tasks with access to external information. Under this setting, we show that speech-based multi-modal retrieval outperforms text based retrieval, and yields up to improvement in word error rate over the multi-modal language model baseline. Furthermore, we achieve state-of-the-art recognition results on the Spoken-Squad question answering dataset.

## introduction the wide adoption of large language models (llms) has driven new application areas levering this technology. one such direction is jointly modeling multi-modal inputs and outputs with a single generative llm model. in the speech domain, models such as those proposed by @xcite , jointly model text and audio by tokenizing speech signals into discrete units. with an expanded vocabulary encompassing tokens of multiple modalities, this modeling approach has been used in both single task @xcite and multi-task settings, with @xcite arguing that the multi-task training of speech-llms improves overall generalization of the model through synergies across tasks and modalities. with an 1 these two authors contributed equally to this work. with generalization capabilities, multi-modal llms such as audiopalm @xcite and seamless @xcite , have targeted many tasks, including automatic speech recognition (asr). these models rely on the decoder language model (lm) to generate the output text transcription when consuming tokenized speech as a prompt. because the multi-modal llm approach can leverage the use of a large text-only corpus for training and multi-tasking, the approach has an advantage compared to traditional asr models such as recurrent neural network transducers (rnn-t) @xcite or whisper-like architectures @xcite , which rely primarily on paired audio-transcription data. however, enterprise-grade asr systems often include further advances, such as functionality to incorporate auxiliary information to assist decoding accuracy, which has yet to be fully addressed with these new multi-modal llm-based approaches. perhaps the two most common approaches for incorporating auxiliary information for asr have been shallow fusion with an external lm @xcite @xcite and contextual biasing @xcite @xcite . shallow fusion with an external lm is a modular way to bias the asr model at inference time by interpolating the probability distribution of the asr model with that of the external lm. shallow fusion though, can suffer from a loss of generality since the external lm does not have direct access to acoustic information. neural biasing, meanwhile, resolves this issue by ingesting the auxiliary information directly into the acoustic model training @xcite . yet, neural biasing is ultimately limited in the number of contextual documents it can ingest since attention over an ever larger number of documents renders the method less effective through dilution. we aim to address both of these limitations in this work through retrieval augmentation. retrieval augmentation is a well known approach to improve existing lms for ingesting additional information @xcite @xcite @xcite @xcite @xcite . the overarching idea of all retrieval augmented models is to use an external knowledge base, i.e. retrieval corpus, to improve lm performance. during inference, the retrieval corpus is queried for relevant context and information. the query usually consist of a key computed using an encoder model followed by a search step to find the closest neighbors to the key -typically in a cosine similarity or euclidean distance sense. the retrieved neighbors are provided to the retrieval augmented lm as additional inputs, which can be used as prompts or cross-attended over. ## proposed approach in the following sections, we describe the multimodal speech-llm and the modelling approaches to incorporate the retrieved context. ## experiments in the following sub-sections, we describe datasets used for experiments, model adaptation specifics, and retrieval data construction. ## 11008 speech was encoded into continuous vectors with a pre-trained hubert model @xcite with ∼ 1 billion parameters and further discretized using 2000 k-means centroids. text was tokenized using the corresponding sentence piece model for all models. we used greedy search for the small model and beam search with beam width of two for the large model. ## conclusion we investigate use of retrieval augmented multimodal lms for ingesting dynamic context and domain adaptation in speech recognition. we showed that a multi-modal lm can be effectively used for contextualizing retrieval database with audio, leading to an improvement of 10-50% (absolute) in retrieved token accuracy and recall compared to the text-only counterpart. furthermore, we compared masked-lm (bert) with a similar sized multimodal lm as key encoder for constructing a text based retrieval database. overall, the multi-modal lm fared better for next token retrieval by 6-10% (absolute), whereas the bert model had better recall for subsequent tokens by 3-4% (absolute). we considered two different approaches for domain adaptation: a knn-lm and a cross-attention based neural approach. both approaches are studied using two different model sizes: a small model with ∼ 300m parameters and a larger model with ∼ 7b parameters. in all cases we used a multimodal lm to encode the keys used for retrieval. domain adaptation using cross-attention outperformed the knn-lm for both small and large models by 30-100% relative. the large multi-modal lm with cross-attention outperformed whisper model by 40% relative for the entity heavy spoken-squad qa dataset, achieving state-of-the-art results. additionally, the multi-modal lm's parameters can be shared between the key encoder and decoder. activation from the early layers can be computed once, used for retrieval and re-used with the retrieved context in the modified top layers for retrieval augmented generation. this reduces memory footprint of the model, which can be an important consideration for embedded applications. future directions of this work could be integration of more modalities (e.g video, sound etc.) into the retrieval process for improving model performance in various applications.