CUNI systems for WMT 21: Multilingual Low-Resource Translation for I ndo- E uropean Languages Shared Task (2021)

Abstract: This paper describes Charles University sub-mission for Terminology translation shared task at WMT21. The objective of this task is to design a system which translates certain terms based on a provided terminology database, while preserving high overall translation quality. We competed in English-French language pair. Our approach is based on providing the desired translations alongside the input sentence and training the model to use these provided terms. We lemmatize the terms both during the training and inference, to allow the model to learn how to produce correct surface forms of the words, when they differ from the forms provided in the terminology database.

## introduction the goal of the task was to translate text from catalan into occitan, italian and romanian. additionally, use of parallel corpora which combine the evaluated languages with english, french, portuguese and spanish was permitted. the choice of the languages from the same family invites to explore how to take advantage of similarities between the languages. one way to exploit similarities between the languages translated by an nmt model is to train a single joint model for multiple languages. this way, parameters representing rules and features which are common for multiple languages can be shared and better estimated due to a larger amount of training examples related to them. another approach which can be effective when source and target languages are very similar is character-level processing of the text. since most of the differences between catalan and occitan are straightforward orthographic variations, we hypothesize that the translation model would benefit from being able to manipulate the text at character level instead of larger subwords. we also explore making use of language similarity in spoken form, aside from written form. languages from the same language group may be more mutually intelligible in their spoken form rather than in the written form. for instance, based on our anecdotal observations, native speakers of czech report better understanding of spoken rather than written polish. this is mainly due to polish orthography, which is regular but uses various digraphs, making polish texts less comprehensible for common czech speakers. phonemic representations may be even more helpful for languages with irregular spelling. instead of using automatically acquired phonemic representation as one of the inputs, we rather focus on strengthening robustness of our translation models by teaching them to produce this representation as an additional task. some of our models are thus trained to provide machine translation as well as grapheme-to-phoneme conversion (g2p) of the source. ## main features of our approach the core of our approach lies in leveraging multilingual training data, various subword granularity and phonemic representation of texts by multi-task learning. all our models are instances of the transformer architecture @xcite as implemented in the mariannmt @xcite . for the final submissions, we trained several models in multiple stages and tuned the decoding hyperparameters. moreover, we applied character-level rescoring for the catalan-occitan submissions. ## datasets apart from the catalan, occitan, romanian and italian data, we take advantage of the data in other languages allowed by the shared task organizers: spanish, french and english (we did not use portuguese corpora). we used datasets specified by the task organizers, namely paracrawl, globalvoices, europarl, jw300, wikimatrix, multiccaligned, opus100, books and bible. ## results in this section, we report bleu @xcite and chrf2 citepopovic-2015-chrf scores on development and test sets provided by the organizers. we did not rerun test set evaluations for all the models, so for a small number of configurations we only show scores on the development sets. ## conclusion we described our submission to the shared task, which ranked first according to the majority of the 7 except for occitan bilingual model, which uses a vocabulary of 8k tokens.