Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training (2024)

Abstract: Recent advancements in language modeling have led to the emergenceof Large Language Models (LLMs) capable ofvarious natural language processing tasks.Despite their success in text-based tasks, applying LLMs to the speech domainremains limited and challenging. This paper presents BLOOMZMMS, a novel modelthat integrates a multilingual LLM with a multilingual speech encoder,aiming to harness the capabilities of LLMs for speech recognition and beyond.Utilizing a multi-instructional training approach, we demonstrate the transferabilityof linguistic knowledge from the text to the speech modality.Our experiments, conducted on 1900 hours of transcribed data from 139 languages,establish that a multilingual speech representation can be effectivelylearned and aligned with a multilingual LLM. While this learned representationinitially shows limitations in task generalization, we address this issue bygenerating synthetic targets in a multi-instructional style.Our zero-shot evaluation results confirm the robustness of our approach acrossmultiple tasks, including speech translation and multilingual spoken languageunderstanding, thereby opening new avenues for applying LLMs in the speech domain.

## introduction language modeling task involves predicting subsequent text tokens based on a context of preceding ones @xcite . training a language model (lm) requires only raw text samples, as portions of these samples function as their labels, facilitating a self-supervised learning (ssl) approach. the widespread availability of machinereadable text online, coupled with advancements in computational power, has led to the rise of large lms (llms) in recent years. these llms not only generate highly fluent natural text but also encode higher-level knowledge within their parameters. this enables them to tackle natural language processing tasks like reading comprehension and machine translation based only on task specific instructions, without needing labeled data @xcite . ssl has recently made significant strides in the speech domain @xcite . most applications of ssl in speech employ an encoder that transforms raw speech signals into high-level representations, serving either as a fixed feature extractor @xcite or a tunable pretrained model for various downstream tasks @xcite . incorporating of ssl pretrained encoders into encoder-decoder speech recognition models has dramatically reduced the amount of labeled data required for effective training @xcite . however, using ssl pretrained decoders in such models is relatively rare. in certain instances, ssl is part of a joint training process that seeks to learn a shared speech and text representation @xcite . however, this approach often demands a large dataset and considerable computational resources. recent work has begun to harness the powerful text generation capabilities of decoder-only llms by incorporating them as the decoder component of encoder-decoder speech processing models. @xcite adopt the llama-7b llm for speech translation to english by training a speech encoder from scratch using filter bank acoustic features, 14,000 hours of internal speech data in 14 languages, and outputs of internal translation system as synthetic targets. outputs of speech encoder are aligned with the text token embedding space using ctc pretraining and downsampled by averaging of consequative frames with the same ctc output label. @xcite adopt the gpt2 xl llm for fully-formatted english speech recognition by training a speech encoder from scratch using filter bank acoustic features, and 75,000 hours of internal transcribed english speech data. ctc loss is applied to speech encoder outputs as a part of the main training process and speech representations are downsampled by removal of frames classified as ctc blank labels with a predefined threshold. @xcite adopt the llama-7b llm for longform english speech recognition by incorporating the hubert-large ssl pretrained speech encoder and finetuining it on the librispeech dataset containing 960 hours of transcribed english speech. outputs of the speech encoder are downsampled by a convolutional module trained as a part of the main training process. @xcite adopt the llama-7b llm for speech recognition in 8 languages by training a speech encoder from scratch using filter bank acoustic features and the multilingual librispeech dataset containing 50,000 hours of transcribed speech in the same 8 languages. speech encoder is pretrained with ctc loss and its outputs are downsampled by simple discarding of every n frames. @xcite combine an internal pretrained llm with an internal pretrained speech encoder and finetune it on the automatically transcribed librilight dataset containing 60,000 hours of english speech. the training is performed with a combination of the speech transcription and speech continuation tasks. the resulting model is utilized for the spoken language answering task. most of these studies rely on conventional filter bank features for speech encoding and do not incorporate an ssl pretrained speech encoder, necessitating a large amount of training data. moreover, scant attention has been given to leveraging the linguistic knowledge stored in llms for tasks beyond mere transcription and for languages other than english. to address these challenges, we propose bloomzmms, a model that fuses a multilingual llm (bloomz @xcite ) with a multilingual speech encoder @xcite ). we argue that multi-instructional training is crucial for transferring linguistic knowledge from the text to speech modality. our experiments demonstrate that training on 1900 hours of transcribed data from 139 languages yields a multilingual speech representation compatible with a multilingual llm in the context of automatic speech recognition (asr) task. although this representation does not generalize well to other tasks, we show that the issue can be mitigated by generating additional synthetic targets. our zero-shot evaluations confirm this approach's effectiveness across various tasks, including spoken language translation (slt) and multilingual spoken natural lan-guage inference (nli). our training recipes and models are released under the apache-2.0 license foot_0 . ## method the proposed method is outlined in figure 1 . our model comprises the pretrained speech encoder, llm and an intermediate adaptor module that maps the output of the speech encoder to the latent space of the text token embeddings of the llm. we train the adaptor module using pairs of speech recordings and their corresponding text transcriptions, denoted as x and y transcription respectively, and keep the parameters of the speech encoder and the llm frozen. the objective of the adaptor training is to make its output h adaptor obtained from the input speech x as close as possible to the text embedding sequence of the ground truth transcription h @xmath0, where lmembedding is the token embedding layer of the llm. similarly to previous works on the llm adaptation to the speech modality @xcite , our training process comprises of the two stages: an alignment of the speech encoder output with the llm token embedding space, and an integrated optimization of the complete model with the llm. an attempt to omit either of the two stages in our process leads to the lack of training convergence. we hypothesize that the different training stages help the adaptor to learn different subtasks like segmentation, ordering and the actual token embedding prediction. at the first stage of the training, h adaptor is projected to the llm tokens' logits using the frozen output linear layer of the llm (which is often a transposed token embedding layer), and the connectionist temporal classification (ctc) loss @xcite is minimized between the llm token probabilities obtained from the token logits and the transcription: at the second stage, h adaptor is concatenated with the token embeddings of the prefix and postfix parts of a text prompt. this joint sequence is then passed through the self-attention layers of the llm and projected with the transposed token embedding weight matrix w (also serving as the output layer of the llm) to obtain the llm prediction. the cross-entropy (ce) loss is minimized between the prediction of the llm for this sequence and the expected llm output. in case of the speech recognition task, we set the prompt prefix and postfix to "repeat the sentence: " and ". " respectively: ## conclusion in this paper we present bloomzmms, a multilingual multitask speech processing model that combines a multilingual llm and a pretrained multilingual speech encoder. our investigation into two training strategies revealed their combined efficacy in a broad spectrum of spoken language processing tasks, a conclusion bolstered by zero-shot evaluations on multiple benchmarks.