Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment (2023)

Abstract: Recently, speech-text pre-training methods have shown remarkable success in many speech and natural language processing tasks. However, most previous pre-trained models are usually tailored for one or two specific tasks, but fail to conquer a wide range of speech-text tasks. In addition, existing speech-text pre-training methods fail to explore the contextual information within a dialogue to enrich utterance representations. In this paper, we propose Speech-text Pre-training for spoken dialog understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog pre-training model. Concretely, to consider the temporality of speech modality, we design a novel temporal position prediction task to capture the speech-text alignment. This pre-training task aims to predict the start and end time of each textual word in the corresponding speech waveform. In addition, to learn the characteristics of spoken dialogs, we generalize a response selection task from textual dialog pre-training to speech-text dialog pre-training scenarios. Experimental results on four different downstream speech-text tasks demonstrate the superiority of SPECTRA in learning speech-text alignment and multi-turn dialog context.

## introduction in recent years, speech-text pre-training, which learns universal feature representations from a large training corpus @xcite @xcite , has achieved significant success in both uni-modal @xcite and multi-modal @xcite downstream tasks. existing speech-text pre-training works mainly employed multi-modal self-supervised pre-training objectives, such as cross-modal masked data modeling @xcite and crossmodal contrastive learning @xcite , which align the speech utterance representation to the corresponding text sentence representation. despite the remarkable progress of previous speech-text pre-training models, there are still several technical challenges to constructing an effective and unified speech-text pre-training model for spoken dialog understanding, which are not addressed well in prior works. first, previous models are mainly tailored for specific speech-text tasks, such as speech-to-text translation @xcite and speech-language understanding @xcite , failing to conquer a wide range of speechtext tasks. although @xcite proposed a unified speech-text pre-training for speech translation and recognition, it fails to exploit the temporality of an input speech sequence and cannot learn the fine-grained speech-text alignment. second, limited exploration has been attempted to bridge the gap between plain speeches/texts and human conversations. in particular, existing speech-text pre-training methods fail to explore the context information within a dialog. nevertheless, spoken dialog understanding needs to effectively process context information so as to help the system better understand the current utterance, since humans may omit previously mentioned entities/constraints and introduce substitutions to what has already been mentioned. in this paper, we propose speech-text dialog pre-training for spoken dialog understanding with explicit cross-modal alignment (spectra), which is the first-ever speech-text dialog pretraining model. we illustrate the framework of our method in figure 1 and details in figure 2 . the backbone of spectra is composed of a text encoder, a speech encoder, and a fusion module, learning semantic/acoustic information and the interaction between them, and pre-trained on a largescale real-world multi-modal (speech-text) dialog corpus. we propose two pre-training objectives to learn better context-aware speech/text representations for spoken dialog understanding @xcite . specifically, to consider the temporality of speech modality, we design a novel temporal position prediction task to capture the speech-text alignment by predicting the start and end time of each textual word in the corresponding speech waveform. in addition, to learn the characteristics of spoken dialogs @xcite , we devise a cross-modal response selection objective to consider the context information within each dialog. our contributions are summarized as follows: ## related work uni-modal pre-training in recent years, pretrained language models (plms), such as bert (kenton and toutanova, 2019), roberta @xcite , and gpt @xcite have been proposed and applied to many nlp tasks, yielding impressive performances. plms benefit from the rich linguistic knowledge in large-scale corpora @xcite . inspired by the success of plms in nlp tasks, several speech pretraining models, such as wav2vec @xcite , hubert @xcite , and wavlm @xcite , were proposed to learn highquality universal speech representations from massive speech data. multimodal pre-training compared to multimodal pre-training for vision-and-language tasks, speech-text pre-training is relatively less explored. speechbert @xcite jointly trained multimodal representations based on a single bert for spoken question-answering. ctal @xcite extended the original transformer to crossmodal by modifying the attention mechanism of the transformer decoder. st-bert @xcite combined a pre-trained acoustic model with bert and took phoneme posterior and subword-level tokenized text as input. @xcite explored multimodal pre-training model in extremely lowresource data scenarios. clam @xcite employed contrastive and multirate information inherent in audio and lexical inputs to align acoustic and lexical information. stpt @xcite proposed a multi-task learning framework to integrate different modalities in speech-text pre-training. ## method in this section, we introduce the model architecture and pre-training objectives of spectra. ## analysis 5.1 ablation study to better understand the effectiveness of our spec-tra pre-training method, we investigate the influence of pre-training components and dialog history on the overall performance of spectra. we report the ablation test results in impact of pre-training to demonstrate the efficiency of multi-modal pre-training, we directly use uni-modal encoders and randomly initialize the modality fusion module. we observe a significant performance drop by comparing (a) "w/o multimodal pre-training" to other pre-training settings on all five datasets. in particular, setting (a) directly collapses on the erc task, which is a complicated and conversational scenario. this verifies the necessity of cross-modal pre-training and aligning speech-text modalities. in addition, by comparing spectra and setting (b) "using less pre-training data", we can find that using more pre-training data can further improve the performance of our model. impact of tpp and crs by comparing the setting (c) "w/o tpp" to spectra, the performances on all five datasets drop to different extents, which case ground-truth by spectra by "w/o tpp" given text #1 leave ✓ leave ✗ complain i am going to sit in traffic for 45 minutes and return this. #2 inform ✓ inform ✗ praise that's tommy. he's lead organizer, total badass. ## conclusion in this paper, we proposed our model spectra, the first speech-text dialog pre-training model. considering the temporality of speech and text modalities, we introduced a novel temporal position prediction pre-training task to learn word-level speechtext alignment. to capture multi-modal dialog context in our model, we generalized the response selection task into multi-modal scenarios. extensive experiments show that our pre-training method can learn better cross-modal interactions as well as multi-modal contextual information and significantly outperformed other strong baselines. in the future, we would like to extend speech-text dialog pre-training to more modalities or generative tasks.