L a RA : Large Rank Adaptation for Speech and Text Cross-Modal Learning in Large Language Models (2024)

Abstract: Integrating speech and text capabilities into large language models (LLMs) is a challenging task and we present Large Rank Adaptation (LaRA) for effective cross-modal integration of speech and text in the LLM framework. Unlike conventional LoRA, our method requires significantly larger ranks comparable to the pretrained weights to accommodate the complexities of speech-text cross-modality learning. The approach utilizes HuBERT to convert speech into discrete tokens and fine-tunes the pretrained LLM to adapt to cross-modal inputs and outputs. The work employs a Hi-Fi GAN vocoder to synthesize speech waveforms from the generated speech units. The initial studies use the Librispeech corpus to teach the model the relationships between speech and text, and Daily Talk, which involves dialog conversations, to adapt for interaction. The proposed work demonstrates adaptation for spoken and text conversations. However, the proposed framework can be easily extended to other cross-modal applications.

## introduction in humans, speech is the most preferred mode of communication. it encompasses rich information such as tone, emotion, gender, and speaker information. on the other hand, text is a less complex mode, making it better suited for storage and computational tasks. with the recent advancement of larger deep learning models, the text domain has seen revolutionary changes @xcite @xcite . applications such as summarizing, conversation, and translation have shown tremendous ability to reason with the input data. speech, unlike text, is a high bit rate and complex signal. understanding its semantics is crucial for language comprehension. training a large language model (llm) with speech from scratch is computationally intensive and requires an enormous amount of training data, which is not commonly available @xcite . considering the capabilities of llms, the scope of expanding them to speech domain which already possesses the knowledge in the text domain, can be explored @xcite . this approach to enhancing the capabilities of llms is a significant area of research for further exploration. in this paper, we propose large rank adaptation (lara) foot_0 , an approach that utilizes the existing knowledge of pretrained large language models for cross-modal applications. the main contributions of the paper are as follows: the remaining paper is divided in the following manner. section 2 extends the introduction and gives an insight into the current literature. section 3 details the architecture of the model. section 4 discusses the data corpus and formatting, finetuning, and experiments carried out. section 5 provides a comprehensive analysis of the work. finally, section 6 concludes the paper. ## related work recent advancements in unified models for speech and text tasks have demonstrated the potential for a single architecture to handle diverse input and output modalities effectively. since we worked on fine-tuning the decoder-only model, we looked into similar work done in the literature. one significant development is the decoder-only model introduced by lauragpt @xcite , which can perform tasks such as speech recognition, speech-totext translation, machine translation, text-to-speech synthesis, and speech emotion recognition. this model processes continuous audio inputs and produces discrete audio outputs, exemplifying the capability of a unified model to seamlessly integrate multiple functions. the necessity of fine-tuning the entire parameters of large models is often computationally expensive and resource-intensive. to address this, a more efficient method involves training low-rank decomposed matrices and attaching them to the dense layers, thereby reducing fine-tuning costs significantly @xcite . this approach has paved the way for adapting large language models (llms) for various downstream tasks. for instance, the audiopalm model described in @xcite combines the linguistic strengths of the text-based palm-2 with the paralinguistic capabilities of audiolm, using separate embedding matrices for audio and text. this dual approach enhances the model's ability to process complex audio and text tasks. similarly, speechgpt, as detailed in @xcite , employs a threestep training process to fine-tune the llama-13b pretrained model, incorporating a low-rank adapter to improve performance across different modalities. other notable works including the listen, think, and understand (ltu) model @xcite , which extends the llama llm for general audio reasoning, and the ltu audio-speech (ltu as) model @xcite , which combines whisper and llama 2 to understand both environmental sounds and speech. these advancements underscore the ongoing efforts to develop com-prehensive models capable of addressing a wide array of speech and text-related tasks, pushing the boundaries of what unified multitask learning can achieve. the work by @xcite focuses on speech-to-text applications, capturing nuances in audio for tasks like speech translation by adapting a low-rank adaptation (lora). another similar approach is discussed in @xcite , emphasizing the importance of detailed audio representation in enhancing speech recognition accuracy. from the literature we observe that on text-based llms interact with the speech modality @xcite @xcite @xcite . some studies have also explored the multi-modality in llm using low-rank adapters. furthermore, models designed to combine speech and text have typically been pretrained with both types of data. so, our work aims to develop a model that can be easily adapted for speech by leveraging the knowledge from the pretrained text large language model using adapters. ## method as shown in figure 1 , our methodology begins by converting speech input into discrete tokens using a speech tokenizer based on the hubert architecture @xcite . in this work, we use llama-2 7b foot_1 as our base llm @xcite . the speech tokens are added to the llm's vocabulary, and the embedding layer is resized accordingly. the embedding for speech tokens are initially set using a gaussian random distribution and then replaced with hubert's quantized hidden representations projected to match the llm's dimensions using a projection layer. cross-modal learning involves creating input sequences with speech tokens followed by text tokens and vice versa. the llm is trained to predict the next token, whether speech or text, based on the previous tokens, allowing it to associate speech units with corresponding text. for efficient training, large rank adaptation (lara) technique is proposed, usage of high-rank adapters to enable the llm to adapt to the new speech modality while preserving its text generation capabilities. during inference, the input sequence with speech and text tokens is processed, and the generated speech tokens are converted into speech waveform using a unit hifi-gan vocoder @xcite , which synthesizes the final speech output. ## experiments this section describes the experimental setup and datasets used for evaluating the proposed approach. the experiments were conducted using the librispeech-360 and dailytalk datasets, which contain speech recordings and corresponding transcriptions. it also provides details on the dataset statistics, experimental configurations, and hyperparameter settings used for training and evaluation. ## analysis we employed the cross-entropy loss for training the network, which is defined as: this objective reflects the model's cross-modal understanding between speech and text, as it learns to associate speech units with corresponding text by optimizing the likelihood of generating the correct token sequence from the cross-modal input consisting of both speech and text tokens. ## conclusion and future work in this paper, we presented a large-rank adapter for llama-2 7b llm. our approach shows the adaptation of the speech modality for the pretrained llm with a significantly larger rank of 2048. the findings underscore the importance of adapting existing text-based llms to incorporate speech modality without the need for training from scratch. future work will focus on extending this framework to other cross-modal applications, further enhancing the versatility and applicability of large language models in multimodal contexts.