D i D i Labs’ End-to-end System for the IWSLT 2020 Offline Speech T ranslation T ask (2020)

Abstract: This paper describes the system that was submitted by DiDi Labs to the offline speech translation task for IWSLT 2020. We trained an end-to-end system that translates audio from English TED talks to German text, without producing intermediate English text. We use the S-Transformer architecture and train using the MuSTC dataset. We also describe several additional experiments that were attempted, but did not yield improved results.

## introduction the performance of end-to-end speech translation systems at iwslt has been approaching that of cascaded systems, with the gap shrinking to 1.5 bleu points in 2019 @xcite . with additional effort, end-to-end systems could finally surpass cascaded systems. the 2020 task required participants to translate audio from english ted talks to german text. we trained several different end-to-end speech translation systems. we used the must-c dataset to train models for speech translation and speech recognition, the europarl-st dataset for speech recognition @xcite , and the wmt-19 news commentary dataset for text translation @xcite . our best performing model used an encoder that was first pre-trained for english speech recognition, and then finetuned for speech translation. this system scored 17.1 bleu on the must-c test set. ## experimental framework our models used the s-transformer architecture of di gangi et al.. this is an adaptation of the transformer architecture @xcite for speech inputs. the encoder performs a 2-d convolution on the audio input before applying self-attention as in the transformer. another distinction is that the decoder operates at the character level, instead on the byte-pair encoding (bpe) tokenization that is typically used with transformer models for text. the system uses a 512-dimensional embedding in the self-attention layers. each of the encoder and decoder have 8headed attention and 6 self-attention layers. the models have 32,132,040 parameters. each of our models were run on a single nvidia tesla p-100 gpu. we used a batch size of 8, and the adam optimizer with a learning rate of 0.005 and an inverse square root warm-up schedule starting from 0.0003 for the first 4000 training steps. each model was trained for up to 50 epochs, stopping early when validation loss had not decreased for 10 consecutive epochs. we trained 6 models using different methods. we used the german transcripts and german audio from europarl-st for decoder pre-training. we used the wmt news commentary parallel corpus for text translation. all other experiments used the must-c dataset. ## using wav2vec inputs the must-c corpus represents the input audio using 40-dimensional mel-filterbank features. @xcite presented wav2vec: unsupervised pre-training to learn speech representations, with improved speech recognition results. we attempted to apply this same approach to speech translation, replacing the mel-filterbank features with wav2vec features as input to the system. we use the pre-trained model released in the fairseq library 2 to compute features for the must-c dataset. wav2vec features are 512dimensional vectors, but the mel-filterbank features are 40-element vectors. we applied principal component analysis (pca) to reduce the wav2vec vectors to 40 dimensions to match the existing architecture. to reduce the computational load, we simply computed the pca transformation on the first segment of the training set, and then applied the same transformation matrix to each subsequent sample. we then attempted to pre-train the encoder for english asr using the same s-transformer architecture as before, in section 3.2. however, this model does not successfully learn to transcribe english audio during pre-training. after fine-tuning, it cannot translate english audio and also gets a score of 0 bleu. we suspected our dimensionality reduction from 512 to 40 was too crude, losing too much information. to see if this was the case, we also attempted to use the full 512-dimensional wav2vec features as input, and increased the system layer widths accordingly. however, computational constraints limited us to only training on 20,000 segments of the must-c training set. however, this model also does not successfully learn to transcribe english audio during pre-training. after fine-tuning, it still cannot translate english audio and also gets a score of 0 bleu. ## text translation multi-task training strong text translation systems are often trained on many millions of sentences, if they are available. transcribing audio and translating is more expensive than finding parallel sentences, so the must-c corpus is considerably smaller than text translation corpora. we hypothesized that additional training on translation data would improve performance. we pre-trained an english to german mt system that shared the decoder with our s-transformer system in section 3.2, in order to improve the decoder's translation ability. this model used a standard transformer encoder, not the s-transformer. unfortunately after training, this model was not able to successfully learn to translate text, though this same corpus has been successfully used in previous work @xcite . we did not conduct further experiments trying to use the shared decoder in this model for speech translation. model bleu 1. baseline s-transformer model 0.00 2. #1 + encoder pre-trained on english asr 17.1 3. #1 + decoder pre-trained on german asr 0.00 4. #1 + #2 + #3 16.8 5. #2 + wav2vec preprocessing 0.00 6. #1 + text translation multi-task training 0.00 we have presented several different experiments in training end-to-end speech translation system based on the s-transformer architecture. unfortunately, none of the experiments we presented were able to improve performance on the must-c test set relative to the models of di @xcite . with more work, the ideas we attempted could produce stronger systems in the future.