CUNI E nglish- C zech and E nglish- P olish Systems in WMT 20: Robust Document-Level Training (2020)

Abstract: We describe our two NMT systems submitted to the WMT 2020 shared task in English<->Czech and English<->Polish news translation. One system is sentence level, translating each sentence independently. The second system is document level, translating multiple sentences, trained on multi-sentence sequences up to 3000 characters long.

## introduction in this paper, we describe our two nmt systems submitted to the wmt 2020 news translation shared task: "cuni-transformer" (charles university transformer, sentence-level) and "cuni-doctransformer" (document-level). we trained them for english↔czech and the former one also for english↔polish (no parallel document-level data was provided for english-polish, thus we could not train the latter one). ## common settings both our systems are implemented in the ten-sor2tensor framework @xcite and have the same transformer @xcite architecture -transformer_big with 12 encoder layers instead of the default 6 (while keeping 6 layers in the decoder). the 32k joint english-czech subword vocabulary is exactly the same as used by @xcite and @xcite , which are the systems we submitted to wmt in the last two years. also most of the hyperparameters (except for the encoder depth) and the training regime are the same. the main improvement of our sentence-level system relative to our last-year submission stems from using slightly larger and better-filtered training data -czeng 2.0 @xcite sentences (see our document-level system was further improved as described in sections 3 and 4. ## document-level training our last-year document-level submission @xcite introduced a method of training-data context augmentation, where multiple consecutive sentences (within original documents) are merged together into multi-sentence sequences (of parallel source-target data). the sentences within each sequence are separated with a special token, so that we can easily extract the sentence alignment after decoding. the length of the sequences was limited by 1000 characters and 200 subwords (i.e. any sequences longer than any of the limits in either source or target were discarded from training). an important aspect of this method is that it extracts all possible sequences from the documentlevel training data. for example, given a document with 5 sentences s1-s5, we extract sequences s1, s1-2, s1-s3, s2, s2-s3, s2-s4, s3, s3-s4, s3-s5, s4, s4-s5 and s5, while ignoring sequences s1-s4, s1-s5 and s2-s5 because these are longer than the limits. note that this way of context augmentation implicitly upsamples sentences from longer documents relative to sentences from shorter documents. a sentence appearing in n windows of at most 1000 characters is present n times in the augmented training data. thus this year, we simply sample nonoverlapping sequences of sentences: s1-s3, s4-s5. there are many documents shorter than the limits in czeng 2.0, including many single-sentence documents (from sources without document-level annotation). thus, there are naturally occurring trainng sequences which are shorter than the limits and we checked the model is capable of translating also single sentences. in addition to this change in data sampling, we increased the sequence length limit to 3000 characters and 750 subwords. ## document-level decoding there are many possible ways how to use document-level models (trained as described in the previous section) at decode time. because of the increased limits of training sequences, we increased also the decoding limits two times: pre-context of up to 400 characters, main content of up to 1000 characters and post-context of up to 1800 characters minus the length of the pre-context and main content. ## robust training with noising to make the model more robust to real-world usergenerated data, we added a noise to the training data. we followed an approach of @xcite and made the source side of the training data more noisy by introducing both grammatical and spelling errors. the basic set of noising operations introducing grammatical errors consisted of the following operations: token replacement with one of its spelling dictionary proposals, token deletion and insertion and swapping of two nearby words. moreover, we also allowed to replace phrases with one of their most frequent variants, add or delete punctuation and allowed to strip diacritics. we applied this technique only to our @xmath0. in preliminary experiments, we observed substantial improvements on artificially noised dev sets, but slight worsenings on wmt dev sets, which contain just a very small amount of typos and other errors (on the source side). we thus decided to mix the noised training data with the original unnoised data 1:1. this resulted in approximately the same bleu on the original dev sets as without noising, while keeping the improved results on artificially noised dev sets. for time constraints, we decided to not use any noising in the document-level @xmath1, as well as in our @xmath2. ## results in tables 2 3 4 5 , we report bleu scores on the new-stest2020 for all the systems submitted to wmt. 1 for @xmath3, we report also gender coreference accuracy scores based on winomt testset results @xcite . for @xmath4, we report also manual document-level quality evaluation by @xcite of 269 wmt2020 test-suites sentences (i.e. not sentences from newstest2020). the official manual evaluation on newstest2020 is not available yet. we can see that while our doctransformer was not the best system according to bleu, it scored well according to the other two reported metrics, being the best system in @xmath5. this could be caused by the low reliability of bleu (and other metrics based on similarity with reference) for highquality mt, or by the domain mismatch -the testsuites contain also other domains than news. unfortunatelly, we cannot answer this question before further analysis using the official wmt manual evaluation, once it is done and published. finally, we present several translation examples in the appendix. the source english documents were taken from the wmt2019 newstest and the same examples were selected already by @xcite . ## conclusion we suceeded to improve our baseline system cuni-t2t-2018 @xcite by using better training data, doubling the encoder depth (to 12 layers) and by robust training with source-side noising. while all these three techniques are well-known, we show improvements in improving the last-year wmt state of the art in english-czech translation. we improved also our document-level system (cuni-doctransformer) by more careful data sam