Findings of the WMT 2020 Shared Task on Chat Translation (2020)

Abstract: We report the results of the first edition of the WMT shared task on chat translation. The task consisted of translating bilingual conversational text, in particular customer support chats for the English-German language pair (English agent, German customer). This task varies from the other translation shared tasks, i.e. news and biomedical, mainly due to the fact that the conversations are bilingual, less planned, more informal, and often ungrammatical. Furthermore, such conversations are usually characterized by shorter and simpler sentences and contain more pronouns. We received 14 submissions from 6 participating teams, all of them covering both directions, i.e. En->De for agent utterances and De->En for customer messages. We used automatic metrics (BLEU and TER) for evaluating the translations of both agent and customer messages and human document-level direct assessments (DDA) to evaluate the agent translations.

## introduction despite the significant progress in neural machine translation (nmt) in the last years @xcite , most systems still operate at sentence-level, disregarding the context of previous sentences. it has been pointed out that ignoring the context may degrade the quality of translations, leading to incorrect choice of pronouns, lexical inconsistency, and incoherence @xcite . this is particularly relevant in the context of bilingual chat translation, which normally consists of short messages, referencing each other, and where the correct lexical choice to translate a speaker might have been uttered in a previous turn by the other speaker. numerous systems have been proposed recently to address document-level translation (tiedemann * these authors contributed equally. @xcite @xcite @xcite @xcite @xcite @xcite , focusing on extending both long short-term memory (lstm) @xcite and transformer @xcite with additional encoders or decoders to incorporate previous sentences context. however, often, the approaches are developed for single speaker and document-like tasks. by contrast, in this shared task, we focus on the online multispeaker and multi-lingual setting, where each participant in the conversation speaks in their native language. this task has been first considered by @xcite . in the first round of the chat translation shared task, we propose translating dialogues with two speakers, where the first speaker is speaking in the @xmath0. moreover, we tailor this task for a specific use case: translating conversational text of the customer support chats. in this setting the utterances of the german speaking customer are translated using a machine translation system into english. then, the replies of the english speaking agent are translated into german and sent to the customer. translating conversational text, in particular customer support chats, is an important and challenging application task for machine translation technology. this type of content has so far not been extensively explored in prior mt research, largely due to the lack of publicly available data sets. prior related work has mostly focused on movie subtitles and european parliament speeches. to alleviate this problem, we created a corpus for this shared task, bcontrast( §2), which is translated from english into german and is based on the monolingual taskmaster-1 corpus @xcite . the main motivation of this shared task is to analyze the challenges posed by conversational data as a content type, which has a broad application in industry-level services. in this content type, the text is usually not carefully well formatted, frequently contains typos, abbreviations, and inconsistent casing, usually with shorter sentences, often informal and ungrammatical. since chat sessions are interactive, the task of translating conversations can be seen as a two-in-one task, modelling both dialogue and document-level translation at the same time. in order to evaluate the translation quality of the participating systems we use both automatic metrics (bleu @xcite and ter @xcite ), and human evaluation, consisting of direct assessment (da). for da, we define the evaluation process similarly to last year's wmt news translation task @xcite with document-level context and following the set of recommendations of @xcite . however, differently than the news task, here we rely on professional translators instead of a crowd. this is mainly based on the observations of @xcite , which provides evidence of the professional translators having better judgment and ability to detect fine-grained phenomena. six teams participated in this first campaign of the chat translation shared task, with 14 runs in total. all teams submitted both @xmath1. in §4, we describe each system in more details. ## bilingual conversational data one of the main challenges of bilingual conversation translation is the lack of publicly available data sets targeted for the task. the most commonly used datasets are movie subtitles @xcite , european parliament speeches @xcite , and conversations extracted from the public forums such as ubuntu dialogue corpus @xcite . these corpora, however, usually involve more than two speakers, contain a significant amount of noise (e.g. speakers information missing in the case of movie subtitles), and usually cover very broad domains. for the chat translation task, we aim to develop a common ground for mt researchers to train and test their solutions by providing common training, validation, and test sets, as well as a common shared task definition. unfortunately, due to the general data protection regulation (gdpr), most commercial enterprises cannot distribute publicly their proprietary data. therefore, we opted for using the taskmaster-1 corpus @xcite , which includes monolingual (english) taskbased dialogues in six domains: (i) ordering pizza, (ii) creating auto repair appointments, (iii) setting up ride service, (iv) ordering movie tickets, (v) ordering coffee drinks, and (vi) making restaurant reservations. we used this corpus for creating the data of our shared task. since the main goal of this task is to enable multilingual speakers communicate with each other in their native language, we used the unbabel translation service foot_0 to translate the utterances of both speakers into the target language (german). in this process, the conversations (originally in english) were first automatically translated into german and then manually post-edited by unbabel editors, who are native german speakers. having the conversations in both languages allows us to simulate bilingual conversations in which one speaker, the customer, speaks in german and the other speaker, the agent, answers in english. ## task description a critical challenge faced by international companies today is delivering customer support in several different languages. one solution to this challenge is centralizing support with english speaking agents and having a translation layer in the middle to translate from the customer's language into the agent's (english) and vice versa. the ideal solution for this environment needs to consider the context of both sides which are in different languages, and also needs to be robust to the noisy input since the text here represents a higher degree of noise com ## participants six participants submitted their systems to the chat translation shared task. although the @xmath2.e. customer side) was optional, all participants submitted their systems for both directions. in total, 14 runs were submitted (although only primary submissions were considered for human evaluation). team institution naverlabs naver labs europe uedinuppsala univ. of edinburgh, uppsala univ. indtaowang individual participant (tao wang) tencent tencent umaryland university of maryland ujordan jordan u. of science and technology ## evaluation procedures for the first round of the chat translation shared task we follow the standard procedure of wmt shared tasks and evaluate both on automatic metrics and human evaluation with context. even though automatic metrics provide a cheap mechanism to evaluate machine translation (mt) systems outputs, they do not tell the whole story for highperforming systems @xcite . for example, recent "sentence-level human parity" claims do not seem to hold when the context of the document is considered @xcite , and metrics such as bleu @xcite fail to correlate properly with human assessment @xcite . in this edition of the shared task, we aim for both automatic and manual evaluations. ## discussion the results of the automatic scores of both agent and customer side of all the submitted systems are reported in on the customer side we observe different behaviours and more diverse scores. in fact, the differences of the bleu scores of the baseline and the submissions vary from -7.2 up to +12.7. this means that in a few cases our submitted systems fall behind the baseline by -7.2 bleu scores. the ter scores show a similar behaviour and the differences of the scores of the submitted systems with the baseline varies from +8.2 (in the worst case) to -9.2 (in the case of best performing system). given the fact that our references for this direction (i.e @xmath4. typos, wrong casings, etc) it is difficult to make a final and strong conclusion for this direction. we plan to investigate this aspect further. in addition to the overall da scores of the submissions one might ask how they perform on the more detailed aspects such as sentences with different lengths or sentences containing pronouns. in order to address the first question, we analyzed the human scores for each system with respect to different intervals of lengths (i.e., different bins), namely 1-5 words, 6-10 words, 11-15 words, and, finally, 16+ words. to this end we can condition either (i) on the source sentence, or (ii) on the reference sentence, or (iii) on the generated translations of each system. among these, we focused on (i) which provides more insights and is fairer comparison for all the systems. the english sentences containing pronouns is another aspect that we analyzed further and compared the performances of the submitted systems when there is a pronoun in the sentence. specifically, we compute the scores for sentences which contain at least one instance of pronoun it. finally, we note that three of the submitted primary systems do not leverage the document-level context and use only the sentence-level information. due to the data size and content proposed for the first edition of the chat translation shared task, this is to be expected as there is some level of repetition and similarity among different conversations. however, by looking at the results, we notice that approaches with document-level context seem to benefit from human evaluation when compared to the automatic metrics. ## conclusions we presented the results of the first edition of the wmt20 chat translation shared task. for the purpose of this task, we created a bilingual english-german dialogue corpus, bcontrast, which is publicly available on the website of the task. it is based on the monolingual taskmaster-1 corpus @xcite which was originally created in english. we translated around 18k of conversations of this corpus into german using the professional translators and used it as the in-domain corpus of the shared task. this year we received 14 submissions from 6 different teams, all of them covering both directions (i.e. customer and agent). in addition to the automatic metrics (i.e. bleu and ter) we performed an extensive direct assessment with document-level context using professional translators and used the results of these manual evalua-tions to rank the participating systems. the previous sentences of each conversion provide the annotators with more context to have a more reliable assessment of the translations. due to the constraints posed by our data, this year we were able to perform the manual evaluation only on the agent side (i.e. @xmath5. however, we aim at assessing both sides in the futures tasks.