Multilingual Speech Translation KIT @ IWSLT 2021 (2021)

Abstract: This paper contains the description for the submission of Karlsruhe Institute of Technology (KIT) for the multilingual TEDx translation task in the IWSLT 2021 evaluation campaign. Our main approach is to develop both cascade and end-to-end systems and eventually combine them together to achieve the best possible results for this extremely low-resource setting. The report also confirms certain consistent architectural improvement added to the Transformer architecture, for all tasks: translation, transcription and speech translation.

## introduction the neural sequence-to-sequence models have revolutionalised both automatic speech recognition (asr) and machine translation in many different aspects, from performance @xcite to various forms such as multimodal @xcite and multilingual @xcite @xcite . after multilingual text translation has been established, the recent focus is naturally shifted to multilingual speech translation especially with a series of public speech corpora with multiple translation being released @xcite @xcite . recent evaluation campaigns in speech translation have seen a fierce competition between traditional cascade systems and end-to-end counterparts @xcite @xcite . the competition without a doubt would continue in multilingual speech translation especially in a low-resource condition. however, the competition between two modeling schemes suggests that each of them possesses its own strengths and advantages. notably the cascade models can easily benefit from the separated optimized architectures of each subtask and enjoy the larger available datasets, while the end-to-end models can theoretically avoid error propagation. this manuscript describes the translation system for the multilingual tedx task with the aim of combining the strong points of both approaches. we showed that optimizing the cascade models is necessary to bootstrap a powerful end-to-end model, while in the end combining their powers based on ensembling gives promising results. ## dataset overview the multilingual tedx corpus @xcite provided us with the 5 languages spanish (es), french (fr), italian (it), portuguese (pt) and english (en). while speech audio is available for the first 4 languages, text translation is available for all 20 language pairs, and the speech translation parallel data is largely more scarce than the other two. the data statistics is shown in it is noticeable that the training data is severely lacking for speech translation when the number of sentences is only a fraction of the asr or mt resources. as a result, our initial plan was to generate synthetic translations from the available transcripts, that can effectively increase the data size for training end-to-end slt models. ## general enhancement for transformer models in this section, we describe the overall model descriptions that were applied in all three tasks. transformers @xcite are constructed with blocks of transformation functions including self-attention and feed-forward neural networks. self-attention transforms a sequence of states using themselves as queries, keys and values, building up hierarchical representational powers since the output states are the weighted-sum of the input states that can be flexibly learned during training. relative attention @xcite further improves the interaction between states by assigning learnable weights for each relative position. @xcite incorporated this mechanism into speech models by extending the partially learnable relative positions in @xcite to attend to all positions in the sequence bidirectionally. furthermore, the transformer models are strengthened by using dual feed-forward (ffn) layers per block instead of one @xcite . as such, one feed-forward network block precedes the initial self-attention in either encoder and decoder. the outputs of both ffn layers are scaled by 0.5. besides, it is possible to help training deep transformer better by using relu-inspired activation functions that do not suffer from dead neurons. gelu @xcite and silu @xcite are combined with gated linear units @xcite , as used in our activation functions. in most of our experiments and in the eventual submission, all of the above enhancements were incorporated. ablation studies are unfortunately not fully possible because of the time constraint, but will be provided to depict the improvement of each addition. ## speech recognition our speech recognition models are built based on both the lstm and the speech deep transformer @xcite enhanced with bidirectional relative attention @xcite . while lstm models have been intensively experimented for the best results @xcite , transformers have been recently adopted to this task with strong results @xcite . for the four languages in the multilingual tedx, we trained both multilingual transformers and lstm models on the combination of the datasets, using the factorization scheme. the lstm has 6 encoder layers and 2 decoder layers with 1024 hidden units in each layer. the sole attention layer between encoder and decoder is an 8-head dotproduct attention. on the other hand, we experimented the transformers with the "large" models having 16 encoder layers and 6 decoder layers with 1024 units in the hidden layers. the models are trained with adam and an inverse square-root learning rate schedules with 4096 warm-up steps following the same setting as @xcite for up-to 120k steps or early-stopping on the development set. in order to facilitate training, layers are randomly dropped with the highest rate of 0.5 and linearly reducing from top to bottom @xcite . due to the relatively small size of the dataset, regularization is added with dropout probability 0.35 in all layers, and spec augmentation with dropped frequency range is @xmath0. removing the punctuations and using the evaluation scripts in the same repo with @xcite gave us 11.0, 13.88, 13.38 and 14.14 error rates for spanish, italian, french and portuguese respectively, which are significantly lower than the hybrid lf-mmi provided. ## machine translation our multilingual machine translation is built based on the universal multilingual framework @xcite @xcite , in which the vocabulary is shared between languages using a bpe size of 16000 merging units. thanks to the relatively small data size, the translation task was used to measure the incremental improvement of various features, including the relative attention and the macaron feed-forward layers. therefore, experiments were carried out using the base-setting of transformer as the starting point. dropout was increased to 0.35 together with word dropout @xcite at both encoder and decoder to help the models counter overfitting. the output language is controlled by the language embedding vectors added directly to the word embedding at every timestep @xcite . the language pairs are randomly sampled based on the training size of each pair (no temperature was used). training is done using the adaptive learning rate for adam, with maximum learning rate at 0.7 achieved after 4096 warming-up steps, and is often early-stopped after 60000 training steps, each is approximately 48000 words. regularization is further improved via data diversification @xcite . carrying a similar idea of back-translation @xcite ) that generates synthetic labels for untranslated monolingual data, the main idea of data diversification is to popularize the available training data with synthetic translation of both source sentences and target sentences. according to the algorithm presented in @xcite , the training process is divided into rounds in which the training data is incrementally added with synthetic data coming from the refining models themselves. starting from the original training data in round 0, we use the best settings in round n to translate the source and target sentences in the training to the counterpart language and add the synthetic translation pairs to the current training data, proceeding to round n + 1. each synthetic pair consists of one original sentence and one synthetic sentence. the idea is the combination of backtranslation, model distillation @xcite and data augmentation @xcite without any additional data. interestingly, thanks to the multilingual property, it is also possible to translate one sentence to a range of languages after each round, leading to different options and a massive amount of sentences to be added. however, it was empirically found out that the method did not scale after 1 round, and massively translating to all languages did not improve the training data. therefore, after round 0, the best configuration which is an ensemble is used to generate synthetic parallel data for round 1 by just translating each sentence to the same language in the original dataset. the translation result is seen in in terms of improvement, it can be seen that even though in this extreme low-resource scenario, using more complicated architecture obtained better translations. a combination of relative attention, macaron ffn and 16 layers of depth allowed us to improve the baseline by 0.95 bleu points, in which the relative attention seems to be the most useful. ensembling multiple models is, as expected but costly to improve the results further. data diversification was very effective after the first round, by improving the average score by nearly 1 bleu point. italian-related language pairs enjoyed up to 2 bleu points, due to the lowest amount of original sentences. this result somewhat went against the initial expectation, because by not changing the sampling method, the data ratio for those languages was even lower than in round 0. we obtained the best configuration for text translation with ensembles on round 1. proceeding to round 2 unfortunately did not produce any further improvement, which might be reasoned by the dominance of synthetic sentences in terms of quantity. ## end-to-end speech translation naturally, end-to-end speech translation is developed at the last stage to benefit from the previous stages. the asr models serve as providing the slt with the pretrained encoder, while we used the mt model to fill the gaps, i.e translate all available asr data. this allows us to increase the amount of training data for slt significantly, especially for languages such as italian and french. architecture wise, we only used transformers for slt, that followed the same training procedure with asr due to the fact that the encoders are transferred from the transformer asr models. the results are shown in compared with pre-distillation, a lot of languagepairs enjoyed a significant improvement of up to 26 bleu points, such as the sample italian audio inputs, thanks to the distillation models changing zero-shot to supervised settings. the supervised language pair that was mostly improved is spanish-french (12 bleu points). finally, in this particular slt setup, we found that it is useful to ensemble cascade and slt models in a multi-modal manner. in the literature, it has been observed that each approach has its own strength. while the components of the cascade can be easily tuned individually because asr and mt have lower mapping complexity than slt, the end-to-end models can avoid error-propagation that plagues cascade systems. an ensemble suggests that we can combine the strengths of two approach, yet only available in certain experimental settings that leaves audio segmentation out of the scope. here the ensemble is done by simply using the same bpe vocabulary for the mt and slt models, and average the output probabilities of the mt and slt models for every timestep. the result showed that this intuition can help improve the results further. ## final submission our final submissions include an ensemble of e2e and cascade as primary, with the e2e model served as the contrastive. the official results are shown in in the final results, we can see that the ensemble quality depends on the asr performance, which can be seen in test sets with spanish audio and french audio. at the relatively low error rate, combining two approaches provides a significant boost to the translation quality. however, for french samples the deterioration of the cascade makes the combination worse than the sole end-to-end solu- the evaluation also suggests us to investigate into zero-shot translation for multilingual slt, which is extremely difficult because of the modality difference between the source and target sequences.