The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation (2023)

Abstract: End-to-end spoken language understanding (SLU) remains elusive even with current large pretrained language models on text and speech, especially in multilingual cases. Machine translation has been established as a powerful pretraining objective on text as it enables the model to capture high-level semantics of the input utterance and associations between different languages, which is desired for speech models that work on lower-level acoustic frames. Motivated particularly by the task of cross-lingual SLU, we demonstrate that the task of speech translation (ST) is a good means of pretraining speech models for end-to-end SLU on both intra- and cross-lingual scenarios. By introducing ST, our models reach higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks. To verify the effectiveness of our methods, we also create new benchmark datasets from both synthetic and real sources, for speech summarization and low-resource/zero-shot transfer from English to French or Spanish. We further show the value of preserving knowledge for the ST pretraining task for better downstream performance, possibly using Bayesian transfer regularizers.

## introduction modern artificial intelligence is characterized by large pretrained language models (ptlms) with strong language capabilities to be adapted to various downstream tasks. the success of ptlms rests on carefully-designed pretraining tasks to bestow the capability we expect on the model. current ptlms are mostly trained on self-supervised tasks, which started from masked language modelling (mlm) and next sentence prediction (nsp) in bert @xcite , but recently evolved into more difficult ones such as whole word @xcite or span masking @xcite ), text infilling, and token deletion @xcite . while the rather simple nsp has been replaced by sentence permutation, document rotation @xcite , and sentence order prediction @xcite . all those efforts introduced more challenges in the pretraining phase to mine stronger semantic supervision signals out of unlabelled data. such semantic-rich supervision is particularly relevant for pretrained spoken language models like wav2vec2 @xcite and hubert @xcite based on mlm on (sub-)phonetic units from lower-level audio signals, which are less informative and require models to carry out additional labor on acoustics. therefore, their high-level capacities are more restricted. this may explain why automatic speech recognition (asr) models finetuned upon them with paired data still have a role in fully end-to-end (e2e) slu, often as a pretrained feature extractor @xcite . unlike the cascaded slu in which asr produces transcripts for text processing, in such e2e systems asr as an auxiliary or additional pretraining task provides strong supervision to explicitly link audio to representations that correspond to the denser and semantic-richer textual space, which is valuable for downstream understanding tasks. on texts, self-supervised objectives are rather effective thanks to enormous text data with high information density, but supervised tasks are still used in many cases, machine translation (mt) being an often-seen one. a pioneer of the current ptlm paradigm, cove @xcite , is a seq2seq model pretrained on mt that achieved the then state-of-the-art on various downstream tasks. @xcite further validate language capabilities of mt on morphological, syntactic, and semantic levels, and t5 @xcite uses an ensemble of supervised tasks including mt. furthermore, when trained with inputs of multiple languages, the model encoder may align and push representations for inputs in different lan-guages with similar meaning together to have the same output in the target language, thanks to the guidance from paired data @xcite . with this semanticcentric language agnosticity, such an encoder can achieve few/zero-shot transfer to another language in downstream tasks @xcite . inspired by those works, we hypothesize that the counterpart of multilingual mt on speech, i.e., e2e multilingual speech translation (st) that directly maps speech of various languages to texts in other languages, will also be effective as a pretraining task on e2e slu, for three critical advantages: admittedly, st data is only available in a limited number of language pairs, but for each covered language, there are infinite number of diverse downstream slu tasks with only rich data in english. it is a practical need to enroll various such languages to an english-only model trained on each specific slu task. therefore, as shown in figure 1 , we may pretrain the model on speech translation between english and the target language french in both directions (i.e. en↔fr), and then fine-tune on downstream tasks with an additional classifier, reusing the encoder. we show the benefit of our method on a variety of tasks for semantic understanding of speech, including mono-& multilingual intent classification (ic), spoken question answering (sqa), as well as speech summarization, for which we create a synthetic dataset following @xcite . then we show the strong advantage on cross-lingual transfer to french. all the experiments are focused on comparing st with other tasks like asr as the pretraining or auxiliary task to verify our core hypothesis above. in addition, to show that our method applies to other languages as well, we also conducted experiments using spanish as the target language. this is evaluated by creating the french and spanish version of the english ic benchmark slurp @xcite , using both real and synthetic sources. on all the tasks, our approach outperforms previous baselines and asr pretraining, often by a large margin. furthermore, unlike knowledge for selfsupervised objectives loosely connected to target slu tasks, knowledge to handle tasks with closer link to semantics such as st will be more valuable, following our core hypothesis. hence it should be helpful to preserve such knowledge instead of direct fine-tuning with the risk of catastrophic forgetting. therefore, we introduce multi-task learning as well as bayesian regularizers for knowledge preservation, namely l2-sp @xcite and ewc @xcite , which show benefits especially in low-resource cases. to summarize, our contributions are three-fold: 1. we demonstrate the effectiveness of speech translation pretraining on multiple slu tasks, especially in cross-lingual transfer cases. our code, models, and datasets will be released at https://github.com/idiap/ translation-aided-slu . ## model pretraining as in figure 1 , we first build a speech translator using an architecture established by @xcite , that connects pretrained models on speech and text with a cnn adaptor: audio signals are fed into the lower half of the multilingual wav2vec2, xlsr-53 @xcite , to extract the (sub-)phonetic representations into a 320x-downsampled sequence. the upper half (12 layers) of xlsr is discarded for computational efficiency, as those parameters are found focused on mlm pretraining and less useful for downstream tasks @xcite . given the phonetic level embeddings produced by the half xlsr, the task is similar to machine translation to map them to the output text, for which we leverage the mt model based on mbart @xcite . while the length of this sequence is still much longer than the corresponding text. to better align it with typical textual embeddings as in mbart inputs, a 3-layer 8x-downsampling cnn adaptor is inserted. a target embedding is then prepended to specify the target language or task, similar to the target token used in mbart. to promote language agnosticity, we do not indicate the source language. furthermore, it has been found that explicitly promoting language agnosticity may help zero-shot transfer @xcite , hence we've also attempted to add language adversarial training on the encoder outputs during pretraining and fine-tuning, using a language classifier of a 2-layer mlp to predict the language of the input speech, with gradient reversal layer to explicitly align the representations between different languages. based on the architecture, we fine-tune the model using a combination of the @xmath0, and the @xmath1, both derived from ted talks, plus the @xmath2, with texts further cleaned and sentences that are too long or contain foreign characters removed. unlike @xcite , the whole model is fine-tuned for best pretraining results. to compare, we also experiment with pretraining on the task of asr instead. as the data are paired with both translations and ## cross-lingual transfer for cross-lingual transfer, ic models trained on slurp are then applied on/fine-tuned to french/spanish data below: datasets a french version of slurp, slurp-fr, is created to evaluate the cross-lingual transfer capabilities of the model, which is based on massive @xcite , a translation of slurp texts into multiple languages. with the same input domain and output categories, zero-shot transfer becomes possible. we first produce the audio for the 16.5k french samples in massive with a 7:2:1 train-dev-test split using google tts from four different wavenet-based speakers. then we invite two native french speakers to read out a total of 477 randomly-selected category-balanced held-out utterances, forming the real test set. to mimic slurp, we record the audio indoors with two microphones under both near-field and far-field conditions. we also define a 100-shot per category subset with 4.5k samples in total to simulate a condition with even lower resource. slurp-es is created in a way similar to slurp-fr with 16.5k spanish samples in massive, though we are unable to create a real set. experiments the advantage of our method on cross-lingual transfer is evaluated under the fulldata, 100-shot, and zero-shot cases, using different pretraining strategies compared to the none model trained on slurp without further supervision but directly upon the multilingual self-supervised pretrained models. hence it is noteworthy that all the compared models have been pretrained in a multilingual way. as given in in addition, we build a cascaded system which first translates the speech into english text using our st model, with bleu scores of 26.08/26.34/21.56 on dev/test/real. then a bart-based textual slurp model with 85.7% test accuracy is used. this essentially zero-shot system gives a competitive 62.2% real accuracy, which poses challenges to the future development of e2e cross-lingual models. the model on spanish, based on en+fr+es st/asr pretraining along with training on slurp in an identical protocol, shows similar results as in ## pretraining knowledge preservation methods as mentioned above, the knowledge to perform asr/st that connects speech and semantic-rich texts could be valuable for downstream tasks, which motivates us to use joint training above that maintains the performance on the pretraining tasks. this is verified by the considerable performance improvement or gap between the joint and single models. however, it is computationally intensive and requires access to the pretraining data. experiments we experiment with these regularizers, targeted on st-pretraining on slurp and minds-14, plus the st+asr pretraining on minds-14 which has a considerable 1.32% accuracy gap. we try to use various weights α for l2-sp regularization ranging from 1e-5 to 1e-2. then we inspect the distribution of the approximated f i , which ranges from 1e-20 to 1e-5 as in appendix f. for optimization stability we clamp the weight αf i above 1e-2, and use ewc weights of 2e2, 2e4, 2e6, and 2e7 to roughly match the magnitude of those for the l2-sp regularizer. results are shown in figure 2 , and for minds-14 the average accuracies are reported. in the case of slurp, it is possible that the amount of data is already sufficient that the preservation of the pretraining knowledge could be helpful only if it is carried out in a fully adaptive way, namely joint training. therefore, the regularizers lead to limited help or even harm to the accuracy when the weight is large. however, under the low-resource condition in minds-14, both regularizers are effective. as in @xcite , although being more flexible and adaptive, ewc doesn't necessarily lead to better transfer learning. this is consistent with our observations: both regularizers can successfully overcome the accuracy gap or even go beyond the joint training model under an appropriate weight, while the best regularizer varies in different cases, though the more adaptive ewc has a chance to reach better results as in the minds-14 st+asr case. in this way, we demonstrate the effectiveness of bayesian parameter-preserving regularizers for transfer learning on such large pretrained models. ## related work translation as an auxiliary task it has been found that representations from mt models capture various aspects of the input utterance such as syntax @xcite , morphology @xcite , and also semantic inferences @xcite . hence mt has been established as a pretraining task as in cove @xcite for various downstream tasks. but unlike this paper, recent works on the direction has been focused on multilingual and cross-lingual cases, starting from attempts to reuse mt representations as sentence embeddings for text classification @xcite , and, particularly often, for semantic similarity and bi-text mining @xcite @xcite . as for pretraining ptlms to be fine-tuned, mt proves effective for downstream cross-lingual tasks on few-shot and zero-shot transfer @xcite , while often accompanied with similar tasks like translation language modelling @xcite , cross-lingual mlm @xcite , and dictionary denoising @xcite . particularly, mt has been used as an auxiliary task for cross-lingual intent classification on texts @xcite @xcite , and is widely used on cross-lingual generation, including summarization @xcite @xcite , simplification @xcite , question generation @xcite , and data-to-text generation @xcite . end-to-end slu cascaded slu methods work on asr transcripts, for which error propagation is a major challenge @xcite . hence recently end-to-end methods have gained popularity @xcite , especially with the performance gap compared with cascaded systems mitigated in many cases thanks to the ptlm paradigm. besides directly fine-tuning existing ptlms on speech @xcite , there are also explorations for end-to-end interface to connect pretrained models on speech and text @xcite @xcite , as well as joint speech-text modelling, pretraining, or distillation @xcite @xcite @xcite , prompt tuning for ptlms @xcite , combining ptlm features @xcite , and multitask learning with asr @xcite . bayesian transfer learning viewing the pretrained model not as a point estimation but a distribution is critical for continual learning as in ewc @xcite , and the idea has been also applied to transfer learning to regularize fine-tuning as in l2-sp for image classification @xcite , though similar regularizers have been used on @xcite and asr @xcite . more recently, @xcite propose to approximate the prior using sgd trajectory as in swag @xcite for transfer learning. ## conclusion we confirm our hypothesis that speech translation can be a powerful pretraining and joint-training means for various end-to-end models on tasks involving semantic understanding of speech. particularly, it benefits multilingual scenarios and crosslingual transfer, including the zero-shot case. we also create two new datasets for the above tasks. furthermore, we demonstrate the effectiveness of bayesian regularizers to preserve the knowledge from pretraining for downstream tasks.