Unbabel- IST at the WMT Chat Translation Shared Task (2022)

Abstract: We present the joint contribution of IST and Unbabel to the WMT 2022 Chat Translation Shared Task. We participated in all six language directions (English ↔ German, English ↔ French, English ↔ Brazilian Portuguese). Due to the lack of domain-specific data, we use mBART50, a large pretrained language model trained on millions of sentence-pairs, as our base model. We fine-tune it using a two step fine-tuning process. In the first step, we fine-tune the model on publicly available data. In the second step, we use the validation set. After having a domain specific model, we explore the use of kNN-MT as a way of incorporating domain-specific data at decoding time.

## introduction in recent years, neural machine translation (nmt) has seen remarkable advances due to the increasingly powerful models @xcite @xcite . the translation of conversational text is an important and challenging application for machine translation, specially in the customer support domain, since international companies have an increasing need to offer customer support in various languages. however, this domain has not been substantially explored in machine translation research. in the chat translation shared task, the goal is to understand the context's impact in conversational text translation, and to study the feasibility of multilingual systems for customer support translation. this year, the focus was on the case in which we have a centralizing costumer support with english speaking agents and a translation layer between agent and costumer, allowing the communication with customers which speak different languages. in this paper we discuss our submission to this task. our submitted system covers all 3 language pairs: english-german, english-brazilian portuguese, and english-french, in both directions: we translate the agent utterance from english to the other language and the customer utterances from the other language to english. as no training data is provided for this task, we recur to the use of the pre-trained multilingual machine translation model mbart50 ( §2.1; @xcite ) and perform domain adaptation through fine-tuning ( §2.2) with domain-specific data and by retrieving similar examples from domain-specific datastores ( §2.3). to increase the size of training examples that can be used to fine-tune the model and to create the domain-specific datastore we search for similar examples on publicly available datasets ( §3.1) and perform back-translation of the provided monolingual data ( §3.2). ## models in this section, we describe the model that we used to tackle this shared task. we start by describing the base model. then, we describe the techniques used to adapt the base model to customer support chat translation. ## data the data provided by the shared task organization is part of a corpus called maia corpus. it consists of parallel data of chats between an agent (english) and a customer (brazilian portuguese, german or french) across one domain: customer support conversation. thus, there are a total of 6 translation directions. one of the main obstacles of this domain is the lack of parallel data publicly available. to make the task closer to a real case scenario, the shared task organization has only provided bilingual validation sets and monolingual data, for all languages. as already mentioned, finding parallel data for this specific domain is challenging. the only exception is the dataset from wmt 2020 shared task on chat translation @xcite . unfortunately, it only contains two translation directions: english to german and german to english. therefore, to circumvent the lack of domain-specific data available to fine-tune the model and to add to the datastores, we perform data augmentation ( §3.1) and back-translate the monolingual data provided ( §3.2). ## experiments in this section, we describe the experiments we made, to allow us to choose the best model to submit to the shared task. ## conclusions we presented the joint contribution of ist and unbabel to the wmt 2022 chat translation shared task. first, we perfomed fine-tuning of a large pretrained model, mbart50. then we perfomed knn-mt using multiple datastores to incorporate domain-specific data at decoding time. through experiments we show that the combination of the proposed methods is a good way of performing domain adaptation when we have few domain-specific data available. as we are dealing with conversational content it would be interesting to incorporate context information. unfortunately the few experiments that we have performed using context did not improve the performance of our models. as future work, one interesting line of research is how to incorporate the context information together with augmented retrieval approaches. these can be complementary to each other leading to translation quality improvements.