The U niversity of E dinburgh’s B engali- H indi Submissions to the WMT 21 News Translation Task (2021)

Abstract: We describe the University of Edinburgh’s Bengali \leftrightarrow Hindi constrained systems submitted to the WMT21 News Translation task. We submitted ensembles of Transformer models built with large-scale back-translation and fine-tuned on subsets of training data retrieved based on similarity to the target domain.

## introduction we present the university of edinburgh's participation in the wmt21 news translation shared task on the @xmath0. we followed the constrained condition, i.e. only using the data provided by the organizers. the training data for these language pairs consisted of noisy crawled data, and was mostly out-of-domain with respect to the validation and test domain. therefore, most of our efforts concentrated on fine-tuning models to adapt to the target domain. we also explore multiple back-translation methods, and ensembles of models trained and fine-tuned with different methods. building our systems consisted of the following steps, each of which is described in more detail in the remaining sections of this paper: we also report some methods that we tried to use but did not work in section 8. ## model configuration our models follow the transformer-big architecture @xcite : 6 layers of encoders and decoders, 16 heads, an embedding size of 1024, a unit size of 4096, etc. we found that smaller transformer architectures performed worse. all models are trained with the same vocabulary of 32k sentencepiece subwords @xcite to allow ensembling. we use a shared vocabulary between source and target, as well as tied embeddings @xcite . we tried other vocabulary sizes too: 5k, 10k, and 20k, though all of them had similar performance. we also included several special tokens in the vocabulary, of which we finally used only one for tagged back-translation @xcite . we train models with 32gb dynamic batch size and an optimizer delay @xcite of 3 with the adam optimizer (kingma and ba, 2015) under a learning rate of 0.0003, until we see no improvement within 10 consecutive validation steps. all models were trained with the marian nmt toolkit (junczys-dowmunt et al., 2018) foot_0 3 datasets and cleaning ## training with synthetic data in each language direction, we trained 4 models with different seeds. we then ensembled these 4 models to back-translate @xcite all the provided monolingual data. we used this translated data in many different ways as described in the remainder of this section. tagged back-translation following caswell et al. (2019) , we prefixed a @xmath1, combined the data with the clean parallel data, and trained new models. ## results ## unsuccessful attempts in this section, we document some methods that we tried to use, but which did not work at all or did not result in better systems. dual conditional cross-entropy filtering our initial cleaning effort was to use dual conditional cross-entropy (junczys-dowmunt, 2018) to selffilter the parallel data, which yielded no useful results. we also randomly split the data into two halves, trained translation models on each half, to score and filter the other half of the data -this method did not work either. we conclude that these methods are not suitable in this scenario where we do not have any clean data, however small, to train the initial cleaning model. copied monolingual data we attempted to synthesize training data by copying @xcite and transliterating foot_7 monolingual data in the target language to source. in this way, we obtained pseudo parallel data that could potentially improve the decoder side of a translation model without harming the encoder much. transfer learning we also explored utilizing dataset from another language in the form of model pre-training. following aji et al. (2020), we initialize our bengali↔hindi model weights, excluding the embeddings, from our english↔german submission to wmt21 @xcite . these methods above did not increase bleu, except that transliterated monolingual data brought a tiny improvement. model pre-training achieved the convergence faster, but did not achieve better final bleu. consequently, we did not carry out any further experiments with these methods.