A mazon A lexa AI ’s System for IWSLT 2022 Offline Speech Translation Shared Task (2022)

Abstract: This paper describes Amazon Alexa AI’s submission to the IWSLT 2022 Offline Speech Translation Task. Our system is an end-to-end speech translation model that leverages pretrained models and cross modality transfer learning. We detail two improvements to the knowledge transfer schema. First, we implemented a new loss function that reduces knowledge gap between audio and text modalities in translation task effectively. Second, we investigate multiple finetuning strategies including sampling loss, language grouping and domain adaption. These strategies aims to bridge the gaps between speech and text translation tasks. We also implement a multi-stage segmentation and merging strategy that yields improvements on the unsegmented development datasets. Results show that the proposed loss function consistently improves BLEU scores on the development datasets for both English-German and multilingual models. Additionally, certain language pairs see BLEU score improvements with specific finetuning strategies.

## introduction multilingual spoken language translation (slt) enables translation of audio into text in multiple languages. traditionally, slt is solved by cascading automatic speech recognition (asr) models, which convert audio to transcribed text, with text-to-text translation models. end-to-end (e2e) models, such as fair speech translation system @xcite , allow a single model to translate from speech to text. recent advances in e2e models show comparable results with cascaded architectures @xcite . our baseline end-to-end speech translation system leverages large-scale pretrained models on dif-ferent data modalities following the approach proposed by @xcite . we adopt dynamic dual skew divergence (ddsd) loss function @xcite to replace cross entropy (ce) for effective knowledge transfer from pretrained textto-text (t2t) translation model to speech-to-text (s2t) translation model through joint task training. we observe that ddsd consistently outperforms ce across all language directions. our multilingual model supports translation of english (en) audio to german (de), japanese (ja) and chinese (zh). we find that finetuning this model based on language groups can improve the performance of the model. additionally, we find that finetuning models by considering alternate translations can lead to subtle improvements in the overall performance of the models. while working with unsegmented data, we show that using a custom audio segmentation strategy can improve the translation performance by around +2.0 bleu points. on iwslt 2022 blind test sets, our system achieves 22.6, 15.3, and 30.4 bleu score for @xmath0, @xmath1, and @xmath2. on the progression test set, our e2e speech translation system performs on par with iwslt 2021 winning cascaded system @xcite . ## base model we adopt the end-to-end speech translation system proposed by @xcite , which takes both text and speech as input for translation task. the model's encoder consists of a text encoder and a speech encoder for each input data modality, respectively. the text encoder is a 12 layer transformer architecture initialized from the pretrained mbart encoder @xcite . the speech encoder is a 24 layer transformer architecture in which we initialize the speech feature extractor and first 12 layers from pretrained wav2vec 2.0 model @xcite . the remaining 12 layers of the speech encoder share weights with the text encoder. between the speech encoder and text encoder, an adaptor @xcite of 3 1-d convolution layers with a stride of two are inserted to compress the speech encoder output by a factor of eight. the model's decoder is initialized from mbart decoder and is shared by two data modalities. we alter the original model architecture to decoupled the mbart output layer and embedding layer instead of using a shared projection layer. ## finetuning approaches to avoid overfitting and moderate generalization, we finetune the base model with a proposed sampling loss algorithm. in addition, we experiment with the effect of finetuning on languages with similar linguistic typology or vocabulary to see if there is negative transferring with the multilingual setting. finally, we test the consequence of using in-domain data. the motivation for sampling loss comes from a hypothesis that the ground truth translations may lack diversity. we can make the translation model more robust and increase end-phrase diversity by training with alternate translations to supplement the ground truth translations. to achieve this, we clone the t2t components from the trained base model and use beam search as a mechanism to generate the alternate translations to guide the s2t components. during the beam search, the target probabilities of all the nodes visited are considered during loss computation as illustrated in figure 2 . we reuse the dynamic dual skew divergence loss to train the student model, and this is the only loss applied during our sampling loss finetuning. we recognize that other sampling strategies could also generate alternative translations. a similar approach is explored in mixed cross entropy loss @xcite . while mixed cross entropy loss achieves the same effect as sampling loss, sampling loss considers the complete target distribution as ground truth while training the student model. ## experimental setup in this section, we first describe the datasets and hyperparameters settings used in our model training experiments, followed by a brief introduction of our audio segmentation approach that improves our model performance on unsegmented datasets. 1 there is a small part of german is sov. ## results and analyses in this section, we present our experimental results and analyses. all the reported results are obtained from a single run using one of the following model settings: the corpora and target languages used in a model training are denoted in superscript and subscript, respectively. if no superscript or subscript appears, all the available corresponding corpora or target languages have been used. for example, ddsd de means a bilingual @xmath3.1. as for the evaluation datasets, if our model can directly handle the size of a given audio clip, such as the audio in the must-c dataset, we directly use the provided data. otherwise, we use the algorithm described in section 5.1 to split audio clips into smaller chunks. ## conclusion in this paper, we adapt and improve the existing dual skew divergence loss by dynamically balancing the model's quality and diversity via the ddsd text-guided loss. the ddsd text-guided loss outperforms the baseline cross-entropy loss on all the experimented language arcs. we observe that for ce and ddsd loss, one-to-one models always outperform one-to-many multilingual models, however ddsd reduces the performance gap between them. we also consider three different finetuning approaches: sampling loss, language grouping, and domain adaption. overall, mixed results are observed and none of the finetuning strategies stand out from the others. in addition, the results of the segmentation experiments indicate that the translation quality can be boosted by presenting audios that are longer than the majority of the training data since more context can be taken into consideration. our submitted end-to-end speech translation system achieves on par performance with the best cascaded system from iwslt 2021.