Transforming LLM s into Cross-modal and Cross-lingual Retrieval Systems (2024)

Abstract: Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn’t require speech data during LLM pre-training and can exploit LLM’s multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data.

## introduction llms have demonstrated their effectiveness in modelling textual sequences to tackle various downstream tasks @xcite @xcite . this effectiveness has led to the development of powerful llms capable of modelling text in a wide range of languages. the abundance of textual data in different languages across the internet has fueled the progress of multi-lingual models @xcite @xcite . on the other hand, speech technologies are prevalent in smartphones and personal assistants, but their we expand the embedding layer of our backbone llm to support the additional discretized speech tokens, that are extracted from a pre-trained speech encoder. at the same time, we tokenize the corresponding transcripts with the llm tokenizer. we encode the speech tokens and transcripts separately and train the model with a contrastive loss over the dot product between speech and transcript embeddings. various efforts have explored solutions to the speech-text data scarcity problem @xcite @xcite . works such as speechmatrix @xcite use separate speech and text encoders to mine semantically similar utterances that are neighbors in an embedding space. however, these approaches are limiting because they require speech and text encoders that have aligned representation spaces. we posit that we can retrieve speech and text utterances by aligning both modalities within the embedding space built from a single pre-trained llm. we take inspiration from previous works that use pre-trained llms to perform automatic speech recognition (asr) and automatic speech translation (ast) @xcite @xcite @xcite . our intuition is that we can perform the speech and text alignment leveraging the capabilities of text-only llms without requiring two separate models. in this paper, we propose converting llms into speech and text de retrieval systems without requiring speech pre-training and outperform previous methods with significantly less data. by discretizing speech into acoustic units @xcite , we extend our llms embedding layer and treat the acoustic units as ordinary text tokens. consequently, we transform our llm into a retrieval system via a contrastive loss allowing us to match speech and text utterances in various languages. our contributions are the following: ## method we train a transformer-based de model that encodes speech and text given a dataset @xmath0, y i )}, where x i is a speech utterance and y i is its transcription. we denote the speech and text embeddings as x @xmath1, respectively, where e is a transformer-based de that encodes speech and text. ## data and tasks appendix a.3 details our training and evaluation datasets along with the number of languages in each dataset, the split we used, and the size of each dataset. we focus on the following retrieval tasks: speech-to-text retrieval (s2t) involves retrieving the corresponding transcription from a database given a speech sample. in s2t, we train on covost-2 @xcite speech utterances and their transcriptions. covost-2 is a large multilingual speech corpus derived from wikipedia expanding over 21 languages and provides translation to and from english. we use fleurs @xcite to evaluate s2t performance on 102 languages. fleurs is an n-way parallel dataset containing speech utterances from flores-101 @xcite human translations. to evaluate s2t, we report recall at 1 (r@1) rates for retrieving the correct transcription for every speech sample and word error rate (wer). ## model figure 1 shows an illustration of our model. we initialize our dual encoder from palm 2 xxs @xcite and append a linear projection layer after pooling the outputs along the sequence length dimension. the embedding and linear projection layers are initialized randomly. after initializing our model from palm 2, we use a contrastive loss @xcite . appendix a.1 includes more details on our training setup. we will refer to our proposed model as palm 2 de. ## experiments we train our de model to perform s2t, where the task is to retrieve the corresponding transcription given a speech sample. we train on the 21 languages from covost-2 and evaluate our model using the s2t portion of fleurs in 102 languages. ## related work the success of pre-trained llms have motivated the application of these models in different modalities. @xcite transformed speech into pseudo-text units to introduce the task of generative spoken language modeling. @xcite introduced a framework to generate audio with long-term consistency. consequently, @xcite showed that speechlms benefit from being initialized from pre-train llms while @xcite demonstrated that pre-trained llms can be adapted to various tasks that required text and speech understanding. on the other hand, several works aim to build joint speech and text representations @xcite . @xcite introduced w2v-bert which combines masked language modeling and contrastive learning to create speech representations. @xcite jointly pre-trains on speech and text from unsupervised speech and text data. recently, @xcite employed separate speech and text encoders to generate embeddings in over 200 languages. nevertheless, there is still a lack of understanding of whether joint speech and text representations can be built from a single encoder. we fill this gap by using pre-trained llms to jointly train on speech samples and their transcriptions to show that our approach is capable of speech-text matching in 102 languages. ## conclusion we present an effective approach to developing a speech-to-text de from a text-only llm. our findings suggest that by using a text-only llm as a backbone model, we can drastically outperform previous approaches using considerably less speech-to-text training data. additionally, we find that we can improve zero-shot speech translation by simply combining readily available translation and s2t data. we showcase our findings in 102 languages for s2t and 4 languages in s2tt; opening up the possibility of using speech-to-text de's in different cross-model and cross-lingual settings. using equation 1, our multi-modal de will learn from paired speech and text embeddings (x i , y i ), where y i is considered as a positive example to x i while all other examples where @xmath2. the model should learn to bring the positive transcriptions closer to the corresponding speech sample, while pushing away all the other negative transcriptions. in our training, the positive and negative distinction is done within the training batch. hence, we apply an in-batch softmax as part of our loss computation. lastly, sim() is a similarity function formulated as the dot product between the speech sample and the transcription embeddings. to train our model, we use the sum of a contrastive loss with a spreadout loss @xcite of both the speech and text embeddings. we calculate the contrastive loss @xcite in a bidirectional way, by adding the loss in the speech-to-text and the text-to-speech direction. we use the adam (kingma and ba, 2014) optimizer with a learning rate of 1.0 × 10 -3 with linear ramp cosine decay scheduler with 2.5k warm up steps. we use a dropout probability of 0.1 and train for 100k steps with a batch size of 1024.