The U niversity of E dinburgh- U ppsala U niversity’s Submission to the WMT 2020 Chat Translation Task (2020)

Abstract: This paper describes the joint submission of the University of Edinburgh and Uppsala University to the WMT’20 chat translation task for both language directions (English-German). We use existing state-of-the-art machine translation models trained on news data and fine-tune them on in-domain and pseudo-in-domain web crawled data. Our baseline systems are transformer-big models that are pre-trained on the WMT’19 News Translation task and fine-tuned on pseudo-in-domain web crawled data and in-domain task data. We also experiment with (i) adaptation using speaker and domain tags and (ii) using different types and amounts of preceding context. We observe that contrarily to expectations, exploiting context degrades the results (and on analysis the data is not highly contextual). However using domain tags does improve scores according to the automatic evaluation. Our final primary systems use domain tags and are ensembles of 4 models, with noisy channel reranking of outputs. Our en-de system was ranked second in the shared task while our de-en system outperformed all the other systems.

## introduction and challenges the task's aim is to create machine translation (mt) systems to enable task-oriented communication between a service agent and a customer speaking different languages (english and german respectively). like most dialogues, the texts can show strong context sensitivities, as the customer and the agent engage in a common activity and continually react to each other's utterances @xcite . however, the dialogues, which relate to ordering or reserving products and services from a limited set of providers, also follow fairly strong scripts and are anchored in a small discourse universe defined by the products on offer. their context sensitivity is therefore counterbalanced by domain-specific conventions and expectations. our design choices are informed by an initial manual inspection of the training data and a baseline translation, which revealed that the main challenges relate to idiomaticity: incorrect or poor translation of english idioms, named entities and politeness markers (e.g. formal vs. informal forms of address, or poor translation of english sir) and an incorrect use of domain-specific terminology. almost always, the problems were the result of an excessively literal translation of the source text, and this literalness also frequently affected the reference translations themselves too. surprisingly, we found few instances of phenomena explicitly requiring context to be correctly translated (e.g. we did not find pronominal anaphora to be a major problem in the dialogues examined). 2 the contextdependent instances we did find were more taskspecific (e.g. english enjoy! should be translated differently depending on whether it is about a pizza (guten appetit!) or a film (viel spaß!)). we therefore focus on domain adaptation and general context modelling strategies. our submissions are based on existing state-of-the-art mt systems for news translation, which we fine-tune on in-domain and pseudo-in-domain data. we also experiment with (i) adapting the models to the different speaker roles and to the different tasks during fine-tuning and (ii) exploiting preceding context through a simple but effective method of concatenating previous sentences to the current one. our code and models are publicly available. 3 2 data the task data consists of parallel task-oriented dialogues between an agent (english) and a customer (german) across six domains: (i) ordering pizza, (ii) making auto repair appointments, (iii) ordering a taxi, (iv) ordering movie tickets, (v) ordering coffee and (vi) making restaurant reservations. the dialogues were initially in english, retrieved from a subset of the taskmaster-1 dataset @xcite and then manually translated into german at unbabel. 4 although the speaker tags are provided for each utterance, the conversations are not explicitly marked with their task domain. the task being to translate the agent's utterances from english into german and the customer's utterances from german to english, we evaluate each translation direction separately, using only the agent's utterances for en-de translation and the customer's utterances for de-en. for training however, we use the full set of 13,845 utterances for both directions. ## approaches we explore four approaches, each of which is detailed below: (i) pretraining using additional data sources, (ii) speaker adaptation, (iii) domain adaptation and (iv) incorporating previous context. pretraining to account for the limited indomain data, we use pre-existing mt models trained for the wmt'19 news task @xcite and then continue training on pseudo-indomain web crawled data from the paracrawl project foot_4 @xcite , before fine-tuning on the in-domain chat training data. we compare two different base systems for each language direction: uedin models foot_5 ( @xcite and fair models @xcite . the pseudo-in-domain data on which training is continued is created by filtering paracrawl data using dual conditional noisy cross-entropy filtering (junczys-dowmunt, 2018). this consists in training a neural language model for each language on the task training data, and jointly scoring each parallel sentence in paracrawl using the two models. we take the top scoring 2.5 million subset of the original 34 million en-de sentences (those that most resemble the task data). speaker adaption distinguishing between the two speaker roles is important as they have different contributions to the dialogue; the customer's utterances are short, interrogative and informal, while the agent's utterances are often long, informative and more formal. we adapt our models to each speaker by using the speaker identity (provided with the task data) as a pseudo-token @xcite : we prepend a speaker tag to each utterance on both the source and the target side. ## experimental setup we compare two neural mt base system types, both wmt'19 news translation task submissions: uedin (university of edinburgh; @xcite . all models are transformer-big models @xcite : 6 encoder and 6 decoder layers, model dimension of 1024, 16 heads except that uedin has a feedforward dimension of 4096 for both the encoder and decoder, and fair models increase this dimension to 8192 in the encoder. uedin models are implemented in marian @xcite and fair models in fairseq @xcite . both model types are trained on parallel and backtranslated monolingual data from the wmt'19 news translation shared task @xcite . for our final submission (using the base fair model), we also use noisy channel reranking @xcite , which requires mt models in both directions and a (target) language model. we describe the data processing techniques in appendix a and list the hyper-parameters in appendix b. ## experimental results and analysis we report automatic evaluation results in section 5.1 and provide a qualitative manual comparison in section 5.2. ## discussion and future work we observe that fine-tuning the wmt'19 newsadapted models on in-domain chat data is a strong baseline. the addition of tags, though helpful, has limited gains on bleu, and the addition of context (intuitively an important component for any dialogue related task) actually degrades results. we speculate that this is due to the nature of the original dataset, which has limited linguistic diversity and utterances that are mostly context-independent @xcite . the overall translation of this dataset was of excellent quality, allowing easy understanding of the dialogues. however, the translated chats exhibit translationese and in some cases lacked naturalness, also the case of the references themselves. an interesting avenue for data collection would be a spontaneous generation of chats in two different languages which can roughly follow the same discourse as in @xcite .