SPLAT : Speech-Language Joint Pre-Training for Spoken Language Understanding (2021)

Abstract: Spoken language understanding (SLU) requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. To boost the models’ performance, various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. However, the inherent disparities between the two modalities necessitate a mutual analysis. In this paper, we propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules. Besides conducting a self-supervised masked language modeling task on the two individual modules using unpaired speech and text, SPLAT aligns representations from the two modules in a shared latent space using a small amount of paired speech and text. Thus, during fine-tuning, the speech module alone can produce representations carrying both acoustic information and contextual semantic knowledge of an input acoustic signal. Experimental results verify the effectiveness of our approach on various SLU tasks. For example, SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%.

## introduction spoken language understanding (slu) tackles the problem of comprehending audio signals and making predictions related to the content. slu has been widely employed in various areas such as intent understanding @xcite @xcite , question answering @xcite , and sentiment analysis @xcite . early approaches leverage a two-step pipeline: use automatic speech recognition (asr) to transcribe input audio into text, and then employ language understanding models to produce ⇤ equal contribution. the work was done when yu-an chung was interning at microsoft. results. however, such cascaded system has several drawbacks. first, the transcription produced by the asr module often contains errors, which adversely affects the language understanding module's prediction accuracy. second, even if the transcription is perfect, the rich prosodic information of speech (e.g., tempo, pitch, and intonation) is inevitably lost after asr. in comparison, humans often leverage these information to better understand and disambiguate the content. therefore, there has been a rising trend of end-to-end approaches to retain information from audio signals to carry out the understanding task @xcite @xcite . while end-to-end slu methods are effective, they often suffer from a shortage of labeled training data, especially when the target task is in a novel domain. one solution is to leverage self-supervised training as is done in pre-trained language models. examples like bert @xcite , gpt @xcite , and roberta @xcite are first pre-trained on large-scale unannotated text in a self-supervised fashion to learn rich textual representations before being finetuned on downstream tasks with a modest amount of labeled data. borrowing this idea, several pretraining methods have been proposed for speech, e.g., wav2vec @xcite , contrastive predictive coding @xcite , autoregressive predictive coding @xcite @xcite , and decoar @xcite , to capture contextual representations from unlabeled speech data. nevertheless, these methods leverage only acoustic data and mainly focus on modeling the acoustic information during pre-training. as a result, the produced representations may not be optimal for language understanding tasks. to solve these problems, we propose a novel speech-language joint pre-training framework, figure 1 : overview of splat. first, the speech and language modules are separately pre-trained using speech and text data via masked language modeling (mlm). in practice, we directly employ the bert base model released by @xcite to be the language module. then, by leveraging a small amount of paired speech and text data, either a sequence-level alignment loss l seq or a token-level alignment loss l tok is applied to align the representations from both modules in a shared latent space (only l seq is shown here). during alignment, the language module is kept frozen and only the speech module is updated. before aligning the two modules, there is an optional step to update the bert base -initialized language module via mlm using the text portion from the paired data. this optional step aims to adapt the language module to the speech domain to facilitate later alignment. after pre-training, the language module is discarded and only the speech module is used in downstream tasks. splat. splat contains a speech module and a language module for multi-modal understanding. the speech module is a transformer encoder trained from scratch and the language module is initialized from bert. both modules leverage large-scale unannotated data for pre-training via masked language modeling. in the speech module, each frame is seen as a token and is replaced with zero vector with a certain probability. for each masked frame, we minimize the l1-distance between the predicted frame and the original frame. then, to make the speech module aware of the contextual information extracted from the language module, we design an alignment loss to align the representations from both modules in a shared latent semantic space. in detail, we propose two alignment methods, a sequence-level one and a token-level one, that leverage a small amount of paired speech and text to minimize the disparity between the acoustic representations from the speech module and the textual representations from the language module. in this way, the speech representations will carry not only the acoustic information but also the contextual knowledge from the text. after this alignment, when text input is absent during fine-tuning, the speech module alone can produce representations that bridge the speech input and the language understanding output. we conduct extensive evaluations on several downstream slu tasks, including fluent speech commands for intent detection, switchboard for dialog act classification, cmu-mosei for spoken sentiment analysis, and spoken squad for spoken question answering. splat achieves superior results in all datasets. for example, splat improves the previous state-of-the-art performance on the spoken squad dataset by more than 10%. furthermore, we show that splat can perform well even given just a tiny portion of the labeled training data in downstream tasks. ## related work spoken language understanding in recent years, due to its flexibility and effectiveness, endto-end spoken language understanding (slu) has been proposed and applied to various tasks @xcite @xcite . for instance, @xcite use an autoencoder to initialize the slu model. @xcite pre-train the model to recognize words and phonemes, and then fine-tune it on downstream tasks. @xcite pre-train the model to categorize graphemes, and the logits are fed into the classifier. in most of these approaches, the model pre-training requires annotated speech, e.g., word or phonemes corresponding to audio signals. as a result, the massive unlabeled speech data cannot be utilized by these models. self-supervised pre-training for language pretrained models have achieved great success in both language and speech domains. in language, bert @xcite , roberta @xcite , unilm @xcite , and bart @xcite have been successfully applied to natural language inference @xcite , question answering @xcite , and summarization @xcite . these pretrained models leverage self-supervised tasks such as masked language modeling (mlm), next sentence prediction, and de-noising autoencoder. self-supervised pre-training for speech in speech, wav2vec @xcite leverages contrastive learning to produce contextual representations for audio input; vq-wav2vec @xcite and wav2vec 2.0 @xcite further propose to discretize the original continuous audio signals in order to enable more efficient mlm training with transformer @xcite . pre-trained speech models have been applied to asr @xcite @xcite , phoneme recognition @xcite , speech translation @xcite , and speech synthesis @xcite , to name a few. nevertheless, an slu model must incorporate both acoustic and language understanding capabilities to project speech signals to semantic outputs. thus, a pre-trained model for slu needs to address tasks beyond a single modality. speech and language joint pre-training recently, slu applications have prompted joint pretraining on both speech and text data. speech-bert @xcite applies mlm to pairs of audio and transcripts. however, there are several crucial differences to compared to our work. first, speechbert contains a phonetic-semantic embedding module that requires forced alignment to first segment speech into word segments to obtain. second, both the pre-training and fine-tuning phases of speechbert require both speech and text input, since it is designed for a specific spoken question answering task. however, many slu tasks only take speech as input, which does not align with the design of speechbert. in contrast, our model can learn to align acoustic and textual representations using just (a small amount of) paired data during pre-training, and only needs speech input for downstream tasks. @xcite propose to align speech and language embeddings in a method similar to ours. however, there are several key differences. first, denisov and vu (2020) employ the encoder of a pre-trained asr model, which already requires plentiful of annotated speech to obtain. our model, on the other hand, conducts self-supervised learning to pre-train the speech module using unannotated speech. secondly, besides sequence-level alignment, we propose a token-level alignment method, which is suitable for token-level downstream tasks. last but not least, our model uses a much smaller paired speech and text for alignment (10 hours) than denisov and vu (2020) (1,453 hours), yet still largely outperforms their method in intent detection and dialog act classification. ## method in this section we present splat, a framework for learning joint contextual representations of speech and language. the model consists of a speech module and a language module that share a similar architecture and learning algorithm. the pretraining of splat is divided into two steps. first, we individually pre-train the speech and language modules using unannotated speech and text, respectively. then, we leverage a simple yet effective alignment task that uses only a small amount of paired speech and text data to align the representations from both modules in a shared latent semantic space such that the information learned by the language module is transferred to the speech module. after pre-training, the language module is discarded and only the speech module is used in downstream tasks. below we formally describe the procedures for pre-training the speech ( §3.1) and language modules ( §3.2), and the alignment loss ( §3.3) for aligning the representations from the two modules. figure 1 provides an overview of the pre-training procedures of splat. ## conclusions spoken language understanding (slu) tasks require an understanding of the input audio signal and its underlying semantics. in this paper, we present a novel speech-language joint pre-training framework, splat, to carry out both speech and language understanding tasks during pre-training. besides a self-supervised training on the speech and language modules, we propose two methods to align the semantic representations from both modules using a modest amount of labeled speech data. the speech module can quickly adapt to downstream tasks and achieve superior results on various slu datasets including intent detection, dialog act classification, spoken sentiment analysis, and spoken question answering. this joint pre-training also makes the model less sensitive to the amount of labeled training data in downstream domains. for future work, we plan to integrate automatic speech recognition and natural language generation into our framework to achieve good results on spoken language generation tasks.