Toward Joint Language Modeling for Speech Units and Text (2023)

Abstract: Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model’s learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability.

## introduction speech and language processing research has largely focused on spoken and written language separately. however, the integration of speech and text in a single model holds potential benefits. speech data contains prosodic information that does not exist in the text, which can help in modeling dialogues. on the other hand, text data from sources like wikipedia can provide structural knowledge that is not available in most speech datasets. moreover, the amount of written text on the internet exceeds the size of any available speech dataset. the impressive performance of text large language models (llms) has caused a revolution in natural language processing @xcite figure 1 : an illustration of our workflow. we tokenize speech signals into discrete units and mix them with text to create speech-text data. our sutlm is then trained on a combination of speech-only, text-only, and speech-text data. more details on the data formats can be found in in this work, we aim to fill the gap between text-only and speech-only lms by developing and studying design choices for a joint speech unit and text language model (sutlm). for speech, we use a self-supervised learning (ssl) speech model, i.e. hubert @xcite , to convert continuous speech signals into speech units. we then combine the units with text data to train an lm that models speech units and text jointly. we convert speech-only, mixed speech-text, and text-only data into token sequences (as shown in figure 1 and to evaluate the sutlm, automatic metrics are developed to quantify the cross-modal ability of the lms. we also fine-tune our models on downstream tasks for spoken language understanding. we fine-tune the sutlms on either the speech or text data and test them on either speech or text to understand how well the models learn to align the two modalities. our main contributions are: ## method we start with a dataset of sentences @xmath0, s foot_1 , . . . , s n }, where a sentence s i is composed of a sequence of t i tokens (z i 1 , z i 2 , . . . , z i t i ), where z i j can be either text or speech units. the sutlm is trained to predict the next token z i j given its prior context z @xmath1. we maximize the log-probability of the data n @xmath2 in the following sections, we describe how we construct token sequences from speech and text. an example of our data formats can be found in ## evaluation metrics we introduce automatic metrics that require no fine-tuning to evaluate the sutlm. fine-tuning is a common approach to assess the quality of pretrained models @xcite @xcite . however, it is a timeconsuming process and the reliability of the experiments highly depends on the hyper-parameter selection process. furthermore, there is no reliable metric to measure the cross-modal ability of lms. in light of this, we propose context retrieval accuracy (cra), a new metric that does not require fine-tuning, to evaluate the cross-modal ability of an sutlm. ## conclusion our study on joint language modeling for speech units and text involved developing evaluation metrics and fine-tuning the model on speech and text data. we found that using mixed speech-text data improves the model's cross-modal ability and performance on both automatic metrics and downstream tasks. our study sheds light on the benefits of considering both speech and text in building language models. we hope that this research will motivate the research community to further explore the integration of speech and text data for more comprehensive language modeling. future work in this area could involve investigating the optimal balance between speech and text data in model training and exploring ways to handle multi-modal data beyond the speech-text domain.