THE IWSLT 2021 BUT SPEECH TRANSLATION SYSTEMS (2021)

Abstract: The paper describes BUT’s English to German offline speech translation (ST) systems developed for IWSLT2021. They are based on jointly trained Automatic Speech Recognition-Machine Translation models. Their performances is evaluated on MustC-Common test set. In this work, we study their efficiency from the perspective of having a large amount of separate ASR training data and MT training data, and a smaller amount of speech-translation training data. Large amounts of ASR and MT training data are utilized for pre-training the ASR and MT models. Speech-translation data is used to jointly optimize ASR-MT models by defining an end-to-end differentiable path from speech to translations. For this purpose, we use the internal continuous representations from the ASR-decoder as the input to MT module. We show that speech translation can be further improved by training the ASR-decoder jointly with the MT-module using large amount of text-only MT training data. We also show significant improvements by training an ASR module capable of generating punctuated text, rather than leaving the punctuation task to the MT module.

## introduction speech translation (st) systems are intended to generate text in target language from the audio in source language. the conventional st systems are cascade ones, including (in the most popular form) three blocks i.e., an asr, punctuation/segmentation module and an mt model @xcite @xcite . both automatic speech recognition system (asr) and machine translation (mt) models are independently trained, and the mt model processes the asr output text (asr hypotheses) to generate translations. in a cascade system, the advance-ments in asr and mt can be directly extended to st. these models can also leverage on the availability of large asr and mt data-sets, and some of the state-of-the art st systems are still cascade ones. recently, end-to-end st systems have become widely popular. an end-to-end st can directly generate text in target language from the audio in source language. these models are simpler in structure and they are more suitable for operating in streaming fashion. most end-to-end speech translation systems are variants of encoderdecoder architecture with attention models @xcite @xcite . this category includes the popular transformer models, which have been adapted for training @xcite . in @xcite , a better performance of st was achieved by initializing the encoder and decoder modules from pre-trainied asr and mt systems, respectively. very-deep transformer models have been trained with stochastic depth for training end-to-end st models in @xcite . the use of relative positional embeddings has also improved the performance of transformer @xcite . one major drawback or end-to-end st is the data availability, i.e., paired speech-to-translation data is scarce compared to asr or mt data. data augmentations and use of synthetic data have been explored in @xcite to mitigate the issue. unlike end-to-end st systems, the data for training cascade systems is easily available and less costly. a brief survey of existing approaches and their principal limitations are discussed in @xcite . despite multiple advantages, the cascade systems suffer from a major drawback: propagating erroneous early decisions into mt models, which then cause degradation in the trans-lation performance. to mitigate this degradation, rather than passing a single asr output sequence to mt model, other forms such as lattices, n-best hypotheses and continuous representations have been explored in @xcite @xcite @xcite . in this work, we use our jointly trained automatic speech recognition-machine translation (joint-asr-mt) model previously described in @xcite . joint-asr-mt model is a cascade system, but it has a differentiable path between asr and mt modules. to create such differentible path, the continuous hidden representations (corresponding to each output token) from the asr decoder are passed to the mt-model. the hidden continuous tokens corresponding to each output token are the attention-weighted value vectors in the last layer of the transformer decoder. we refer to these continuous representations as"context vectors" as proposed in @xcite . existing large separate asr training data and mt training data can be used to pre-train these modules; then, the pre-trained modules are jointly optimized using a small amount of speech translation data. the joint optimization mitigates the degradation in performance due to erroneous early decisions. in this paper, we generate german translation from english speech, and we focus on two main contributions: (1) we train different mt models that can translate normalized text or punctuated text. it is known that mt-models translating punctuated text provide superior performance, therefore, we propose to train an asr system that can generate the punctuated text. we confirm that such asr system provides superior performance in asr-mt pipeline. (2) we use the internal continuous representations from the asr-decoder as the input to mt module. in section 6, we show that speech translation can be further improved by adapting asr-decoder to the mt module. this is achieved by training the asr-decoder jointly with the mt-module using a large amount of text-only mt training data. ## datasets and pre-processing the datasets used for training various models are described in table . 1. asr-train-set and mt-train-set are used for pre-training asr and mt models respectively. the pre-trained models are fine-tuned using asr-mt-train-set. all models are evaluated using mustc-common test set. ## automatic speech recognition (asr) asr systems trained in this work are built on transformer asr models @xcite @xcite . the asr models have 12 encoder and 6 decoder layers with 4096 feed-forward units and 1024 attention dimension with 16 heads. models are initially trained with asr-train-set and are later fine-tuned with asr-mt-train-set. a thresholding mechanism is used for pruning away the noisy end-of-sequence (eos) tokens from beam search @xcite . models are trained with 30k warm-up updates and a checkpoint is saved after every 8k updates. the training is stopped with an early stopping criterion. 8best check-points are averaged and the averaged weights are used for decoding the hypothesis. vectorized beam search @xcite was used for decoding the asr hypotheses with a beam size of 10. further in this paper, asr models described in this section are referred to as ext.asr models (externally trained asr models). two different asr systems were trained for generating normalized text (norm-asr) and punctuated text (punc-asr), and their performances are reported in asr-lm: a transformer language model was trained on english text @xcite . the model has 6 layers, with 4096 feed-forward units and 1024 attention dimension with 8 heads. the model is initially pre-trained on librispeech lm corpus and it is later fine-tuned on english text ## machine translation systems(mt) transformer models @xcite are also at the core of mt-systems. they have 6-encoder and 6-decoder layers with 4096 feed-forward units and 1024 attention dimensions and have 16 heads. the models are optimized with 30k warm-up updates and a check-point is saved every 8k updates. training is stopped using an early stopping criterion. 8-best check-points are averaged and the averaged weights are used for decoding the hypotheses. the noisy eos tokens are pruned out using @xcite . vectorized beam @xcite search has been used for decoding the hypotheses with a beam size of 8. a large variance in the performance is observed w.r.t the decoding hyper-parameters such as maximum target sequence length and length-bonus. the maximum length of the target sequence is computed by multiplying the input sequence length with lengthratio: 1.2 was found as optimal on the development set. to control the length of the output sequence, the log-likelihood scores of the hypotheses are penalized by additive token insertion penalties. the optimal value for this penalty is tuned as a hyper-parameter on the development set. the hypothesis text is de-tokenized and bleu score is evaluated using moses toolkit. all the bleu scores reported in this paper are computed using the de-tokenized, punctuated german text using multi-bleu-detok.perl. the performances of the mt systems are reported in table . 3. all bleu scores reported in this paper are computed using punctuated text as reference. in the architecture of the model is same as asr-lm mentioned in section 3. a shallow fusion between the mt-model and the mt-lm language model is performed. as shown in ## jointly trained asr-mt systems the model has two modules: asr and mt; their architecture is same as described in sections 3 and 4 respectively -see block diagram in figure 1 and full description of the model in @xcite . the context vectors from the final layer of the asr-decoder are used as inputs to the mt module. passing context vectors from asr to mt models while training has also been explored in @xcite . both the models are jointly optimized using a multi-task cross-entropy (asr cross-entropy and mt cross-entropy) -both losses are also shown in figure 1 . during the inference, beam search has been used to obtain the asr hypotheses, and the corresponding context vectors obtained from the asr model are used by the mt model for generating translations. the mt model also uses a beam search, and the final st hypotheses is obtained by a coupled search (vydana et al., 2021) using the joint-likelihood from asr and mt: y * = arg max y @xmath0, ## adapting asr decoder to the mt module joint-asr-mt models are jointly optimized by having an end-to-end differentiable path from speech to translations. the internal continuous representations from the asr-decoder are used as the input to mt module. speech translation can be further improved by adapting asrdecoder to the mt module. this is achieved by training the asr-decoder jointly with the mtmodule using large amount of text-only mt training data. the weights for the model are initialized from trained joint-asr-mt model. speech translation data (asr-mt-train-set) is used to finetune joint-asr-mt model using a multi-task loss. apart from that, the data from the mt-train-set is used to jointly train the asr-decoder and the mtmodule of joint-asr-mt model. we alternately update the model using multi-task loss described in section 4 and the adaptation loss as described in this section. a block diagram describing this training is presented in figure 2 . the input text sequence is given to the asr-decoder and a sequence of zeros is considered as the encoder output sequence of the asr model (i.e.,h asr in figure 2 ). the context vectors computed from these two sequences are used for training the mt-module. note that similar method has been adopted in @xcite for improving the performance of asr system using only text data. this training further improves the performance as will be shown in section 7. ## speech translation results results for the various configurations of speech translation systems are given in row 1 corresponds to the conventional cascade system, where the ext.asr systems generates the n-best hypotheses of discrete token sequences and an ext.mt uses these token sequences for generating the translations as described in eq. 1. we consider this system achieving bleu 23.20 as a baseline. usually, transformer-asr decoder uses the partial output hypothesis and extends it by a new token with every autoregressive decoding step. for the system in row 2, ext.asr generates the complete hypothesis and asr module from joint-asr-mt is "asked" to extend it by one more token. as a byproduct "context vectors" (the continuous representations) are generated for the whole sequence -these are then passed to the mtmodule in joint-asr-mt model to generate translation. compared to row 1 of column a, we see a degradation in performance (bleu-20.19). this can be attributed to having only small amount of speech translation training data, which is not sufficient for robustly training the joint-asr-mt systems. for the systems in row 3, ext.asr generates the asr hypotheses which are used by ext.mt similar to the system described in row 1; the hypotheses from ext.asr are used by joint-mt similarly to the system described in row 2. to generate the translation, the hypotheses form both models are ensembled as follows: for each output token, a weighted average of log-softmax outputs from the two mt models is computed. this weighted average is used in the beam-search to compute the n-best partial hypotheses. these partial hypotheses are further extended by both the models to generate the log-softmax outputs for next tokens. we can see that this ensembling system achieves a bleu score of 24.02 and outperforms the cascaded baseline. the systems in rows 4-6 are essentially the same as the ones in rows 1-3, respectively, except that now, the asr module from joint-asr-mt system is directly used to produce the n-best asr hypotheses and the corresponding context vectors. rows 4-6 show the same trend as rows 1-3 with slightly improved performance; these improvements are mainly due to better performing asr system: as described in section 2.2, training asr systems only on asr-mt-train-set (data from mustc, iwslt and europarl with erroneous transcriptions) did not lead to convergence. however, when the same data is used to train joint-asr-mt model for speech translation task, we observe that the asr module in this model trained well. the reason for that is that the asr-module is not directly trained on erroneous transcriptions, instead, it is trained to produce transcriptions that lead to good translations. this training can be seen as a form of light supervision which can mitigate the problem with the erroneous transcriptions. at the end, this system trained only on asr-mt-train-set achieves better asr performance (wer 16.14%) compared to ext.asr (wer 18.20%), which is pre-trained on asr-train-set (approx 2000hrs) and fine-tuned on erroneous asr-mt-train-set. similar trend will be observed with the systems in columns b, c and d. the systems described in rows 7-9 are similar to those from rows 1-3, except that the asr hy-potheses are obtained by ensembling the ext.asr and asr-module in joint-asr-mt model. the ensembling is performed in a similar way as described for the mt-system (row 2). all the ensemble systems in rows 3, 6, and 7-9 are ensembled giving equal weight to both the systems, except for row 10, where the ensemble weights are tuned on the development set. for all these systems, we can see that the ensembling consistently improves the performances. the systems in column b are similar to the ones in column a, but for the joint-asr-mt model, the weights of asr and mt module are initialized from the ext.asr and ext.mt. only then, the joint-asr-mt model is fine-tuned using asr-mt-train-set. comparing column-a and column-b, we can see that such pre-training has significantly improved the performance. we also see that the mt system using continuous representations (joint-mt) (row 5; bleu 23.97) outperforms the system with the ext.mt (row 4; bleu-23.86) and similar trend can be seen in columns c and d. this is in contrast to the system in column a where we did not use enough data for training the joint-asr-mt model; now, with the pre-training, the joint-asr-mt model is effectively trained on the same amount of data as the ext.mt systems. the systems in column c are similar to the ones in column b, but the asr and mt modules used here are punc-asr (asr systems which can generate punctuated text) and punc-mt (mt systems which can process punctuated text as input), respectively. we can see that the systems from column-c perform significantly and consistently better than the corresponding ones in column-b. this shows that it is more effective to train an asr module to generate punctuated text rather than leaving the punctuation task to the mt module. note that the asr performances reported in columns c and d is computed including the punctuation symbols, which results in higher wers. finally, the systems in column d are the same as the ones in column c except that we additionally use the asr decoder adaptation scheme described in section 6. the consistent improvements observed in column d as compared to column c show the effectiveness of this adaptation scheme. they are able to make use of the large amount of text-only mt training data to train also the asr decoder in order to tighten the coupling between asr-decoder and mt-module. apart from improving mt-module, this adaptation has also improved the performance of asr-decoder on its own. this can be observed by comparing wer's of row 4 in columns c and d. the results of passing the n-best hypotheses from asr to mt models are presented in ## conclusion in this work, we have explored joint-training of asr-mt models for speech translation. initializing these models from pre-trained asr and mt models has helped in better optimization. the joint training has improved the performance of the asr module significantly as the additional mt module has provided better (light) supervision in the context of erroneous asr transcripts. adding the punctuation information into the input text improves the performance of the mt-model greatly. in line with this observation, use of asr system generating punctuated text also improves the mt performance significantly in a cascade pipeline. use of the mt text only data to adapt the asr decoder to the mt module in the joint-asr-mt model further improves the performances of these systems. the systems trained in this work are offline models and their performances needs to be studied from the perspective of online or streaming models.