The U niversity of H elsinki Submission to the IWSLT 2020 Offline S peech T ranslation Task (2020)

Abstract: This paper describes the University of Helsinki Language Technology group’s participation in the IWSLT 2020 offline speech translation task, addressing the translation of English audio into German text. In line with this year’s task objective, we train both cascade and end-to-end systems for spoken language translation. We opt for an end-to-end multitasking architecture with shared internal representations and a cascade approach that follows a standard procedure consisting of ASR, correction, and MT stages. We also describe the experiments that served as a basis for the submitted systems. Our experiments reveal that multitasking training with shared internal representations is not only possible but allows for knowledge-transfer across modalities.

## introduction an effective solution for performing spoken language translation (slt) must deal with the evident challenge of transferring the implicit semantics between audio and text modalities. an end-to-end slt system must hence appropriately address this problem while simultaneously performing accurate machine translation (mt) @xcite . in last year's iwslt challenge, both end-toend and cascade systems yielded similar results @xcite . it follows that this year's iwslt offline speech translation challenge focuses on whether "the cascaded solution is still the dominant technology in spoken language translation" @xcite . for our participation on this task, we train both cascade and end-to-end systems for slt. for the end-to-end system, we use a multimodal approach trained in a multitask fashion, which maps the internal representations of different encoders into a shared space before decoding. for the cascade approach, we use a pipeline of three stages: (i) automatic speech recognition (asr), (ii) punctuation and letter-case restoration, and (iii) mt. we focus on exploiting the knowledge-transfer capabilities of a multitasking architecture based on language-specific encoders-decoders @xcite @xcite . this idea has been proposed and studied in the multilingual scenario @xcite @xcite , however, we adapt it to be used in a multimodal scenario. regarding different modalities (in this case, audio and text) as different languages when training the model, allows us to employ a cross-modal intermediate shared layer for performing slt in an end-to-end fashion. by jointly training this layer, we aim for the the model to combine the semantic information provided in the text-to-text mt tasks with the ability to generate text from audio in the asr tasks. ## data preprocessing the mt, asr and end-to-end slt systems have been trained on different subsets of the allowed training corpora. data for the end-to-end slt system. we use europarl-st (iranzo-sánchez et al.), iwslt2018 @xcite and @xcite , a total of 433k utterances after cleaning some corrupt files or with other problems in the sampling. we extracted 80-dimensional mel filterbank features for each sentence-like segment using our own implementation. text data for the end-to-end slt system. for the text data of the multimodal end-to-end slt system, we use a total of ∼51m sentence pairs from corpora specified in first, we extract six feature values for each of the sentence pairs. in particular, we apply the following features: characterscore, crossentropy, languageid, nonzeronumeral, terminalpunctuation and wordalign, each of which is defined in @xcite . secondly, we train a logistic regression classifier based on those features. the classifier is trained only on wit 3 , must-c, europarl-st and iwslt18, which are multimodal datasets with speech-to-text and text-to-text data. this allows the system to adapt to text translations that are associated with speech translations. finally, we use the classifier to assign a cleanness score ranging from 0 to 1 for all sentence pairs in all corpora. the data is then ranked based on the cleanness score, after which a portion of noisy pairs is removed from the tail. our preliminary translation experiments showed that removing up to 40% of the data improves the translation quality, leaving us ∼30.5m sentence pairs of training data, which are then used in all our end-to-end experiments. audio for the cascade system. we have extracted 40-dimensional filterbank features with speaker normalization for each sentence-like segment of the must-c, how2 @xcite and mozilla common voice @xcite corpora using xnmt @xcite . after getting rid of audio files that were too short (less than 0.4 seconds), corrupted, or no longer available for download from youtube, some 1.2m clean utterances remained for training the asr system, and 30k for validation. on the target side, we use two contrastive preprocessing pipelines: text data for the cascade system. in our slt pipeline, the data we applied for our restoration and translation models have some overlap and some differences. for training, both models use the text data from the iwslt 2018 speech translation corpus, the must-c training set, news commentary v14, europarl v9, and rapid 2019. the translation model also uses data from the opensubti-tles2018 dataset, which the restoration model does not since this dataset is particularly noisy in terms of punctuation and letter cases. conversely, the restoration model also uses data from the how2 and mozilla common voice datasets, which the translation model does not use as they do not contain german text. the translation model uses the iwslt development set from 2010 and test sets from 2011-2015 as validation data, while the restoration model uses them as supplementary training data in order to reinforce domain bias, using only the must-c development set for validation. initially, we "clean" the output of our asr model to remove segments containing musical note characters ( ), and repeating phrases that were consistently hallucinated during silence, applause, laughter or noise in the audio (e.g. in our case, "shake. fold."), as well as parts of segments that designate the speaker (e.g. "audience: ..."). subsequently, we use the same preprocessing pipeline for the cleaned asr output as we do for all of our text data. for this, we start by removing nonprinting characters, normalizing punctuation, and retokenizing the text using the corresponding utilities from the moses toolkit @xcite . afterwards, we apply subword segmentation via sentencepiece @xcite , using a joint english-german bpe model with a vocabulary size of 32 000 for all of our translation models, and an english unigram model with a vocabulary size of 24 000 for the restoration stage of our cascade slt, both trained on all of the data used for the translation and restoration models combined. before the training of the restoration model, the training data was run through a moses truecaser model (trained on the same selection of training data as the restoration model) as an additional step before segmentation. this step removes sentenceinitial capitalization for words that would not be capitalized otherwise, ensuring that differences in distributions of words appearing in sentence-initial positions does not influence case restoration for the model. once truecased and segmented, we assign the processed data as the target for the restoration model, and continue to strip punctuation and lowercase the target to generate the source. this configuration comes with the useful side effect of the model learning to generate truecased output, which may be beneficial for mt. ## experiments in this section we report on the experiments that lead up to our final submissions. the experiments on this section have been trained, validated and tested on the respective splits of the must-c dataset. as a first stage, we focused on selecting the multitask training strategy that performed better. having the three modalities enaudio, entext and detext as possible inputs, and both text modalities as possible outputs, there can be up to 64 combinations where audio is an input foot_1 without taking into account the cases where the text encoder is shared between german and english. we considered the 5 scenarios depicted in figure 1 and present its results in all the models were trained using the same set of hyperparameters. at the time we ran these experiments, the final version of the audio encoder was not ready for deployment, so we used a 4-layered pyramidal cnn+rnn encoder adaptation from @xcite with 512 hidden units and pooling factors of (1,1,2,2) after each layer, respectively. for the text encoders, we applied embedding layers of 512 dimensions, four stacked bidirectional lstm layers with 512 hidden units (256 per direction). we use attentive text decoders composed of two unidirectional lstm layers with 512 units. regarding the shared attention bridge layer, we used 100 attention heads with 1024 hidden units each. training is performed using the adam optimizer (kingma and ba, 2015) with a learning rate of 0.0002 and batch size 32 for all source-target pairs, for at most 100,000 steps per language pair foot_2 . at this stage, we apply a uniform language-rotating scheduler. isolating the effect of multitasking from the effect of weighting the scheduling distribution helped us understand the importance of weighting it with respect to the batch size. our preliminary bleu scores foot_3 for these models are low. we, however, justify our choice to include them given the low performance of other experiments in similar scenarios reported in the literature. namely, @xcite reported 9.55 bleu training on the same set with a transformer based architecture, the only paper that trains and tests on the same set, and thus the only truly comthe well-known sensitivity to hyperperparameter choice of the transformer architecture is also visible in our transformer-based audio encoders. we performed hyperparameter tuning on opt3 multitask training configuration (figure 1 (d) ). this resulted in a performance of a 9.53 bleu score on german translations and 47.63 on the english, a clear increase from the untuned models that got at most 1 bleu point in any of them. the final hyperparameter setup consists of: training was done with 8,000 warmup steps, using an adam optimizer with learning rate 2 and noam decay method, accumulation count of 8 to have an approximate effective batch size of 256 for the audio utterances, dropping utterances above the length of 5500, and a language rotating scheduler that uses the inverse of the batch size as weights 5 . 5 in case of training opt3, the weights assigned to enau-we also tried other strategies such as (i) using 3, 4 and 6 stacked filterbanks as different channel inputs for the cnns to reduce the input size instead of dropping utterances, (ii) using specaugment @xcite layers (2 frequency masks of width 20 and 2 time masks of width 50) to produce a data augmentation effect while training, (iii) including layer normalization after the attention bridge, (iv) using the positional embeddings of our transformer-based audio encoder in other places of the encoder or not using them at all. unfortunately, none of them produced as effective improvements as what we describe above. we note that it is probable that using milder hyperparameters for specaugment could be beneficial. ## results from the insights gained out of our experiments on the must-c dataset, for our submission, we train a system using the data as described in section 3 with the training configuration opt3 (see figure 1 (d) ) and the hyperparameters that yielded the best results. further, we decided to try out an additional training configuration we had not previously tried out: enaudio as input and detext and entext as output, which we refer to as opt6. configurations from figure 1 use both modalities as input, whereas opt6 separates them by using onlyaudio input and only-text output. this might be the reason why opt6 outperformed them when tested on the must-c test set. further experimentation would be required to make this statement conclusive. one of our main aims in participating in this task is to test our multitask architecture; for this reason we submit our best slt system as primary system and the cascade approach with subword segmentation as contrastive baseline. we would like to note that, unfortunately, at the time of submission, our end-to-end systems had not converged yet. for the sake of consistency, these have been benchmarked with the must-c test set as well. the results are reported in ## conclusion in this paper we present our work for the iwslt2020 offline speech translation task, along with the set of experiments that led to our final systems. our submission includes both a cascaded baseline and a multimodal system trainable in a multitask fashion. our work shows that it is possible to train a system that shares internal representations for transferring the implicit semantics between audio and text modalities. the nature of the architecture enables end-to-end slt, while at the same time providing a system capable of performing asr and mt. although this represents an important step in multimodal mt, there is still a lot of room for improvement in the proposed systems. in future work, we would like to implement more sophisticated audio encoders, such as the s-transformer. this, along with using the same amount of data during training, will allow us to draw a truly fair comparison between both end-toend and cascade approaches.