Findings of the WMT 2022 Shared Task on Chat Translation (2022)

Abstract: This paper reports the findings of the second edition of the Chat Translation Shared Task. Similarly to the previous WMT 2020 edition, the task consisted of translating bilingual customer support conversational text. However, unlike the previous edition, in which the bilingual data was created from a synthetic monolingual English corpus, this year we used a portion of the newly released Unbabel’s MAIA corpus, which contains genuine bilingual conversations between agents and customers. We also expanded the language pairs to English↔German (en↔de), English↔French (en↔fr), and English↔Brazilian Portuguese (en↔pt-br).Given that the main goal of the shared task is to translate bilingual conversations, participants were encouraged to train and test their models specifically for this environment. In total, we received 18 submissions from 4 different teams. All teams participated in both directions of en↔de. One of the teams also participated in en↔fr and en↔pt-br. We evaluated the submissions with automatic metrics as well as human judgments via Multidimensional Quality Metrics (MQM) on both directions. The official ranking of the systems is based on the overall MQM scores of the participating systems on both directions, i.e. agent and customer.

## introduction with the significant translation quality improvements brought by newer machine translation (mt) approaches in the last years, we can start using mt to translate non-conventional content types such as bilingual and multilingual conversations. these new applications pose new challenges to mt systems and require new solutions to deal with them. ## the maia corpus one of the biggest challenges of bilingual conversation translation, especially for customer support conversations, is the lack of appropriate publicly available datasets. to alleviate this issue, in the first edition of the chat translation shared task, farajian et al. introduced the bcontrast corpus that was based on a monolingual english corpus, taskmaster-1 @xcite . they translated the selected conversations into german mimicking a scenario in which an agent and a customer are communicating in their native languages. however, even this dataset was just an approximation of a real customer support conversation due to the fact that: 1) the original conversations in the taskmaster-1 corpus were created by using crowdsourced workers interacting with each other to complete a specific task; and 2) the conversations were not truly genuine bilingual conversations since german segments were all just translations of the original english sentences. this year, we made advancements by releasing the unbabel's maia dataset: a corpus that is truly composed of entire, genuine and original bilingual conversations from four different clients of the unbabel database. the conversations are from clients that gave written consent on using this data for research purposes as long as in accordance with the general data protection regulation (gdpr). in this corpus, the original segments of customers and agents are translated into their corresponding target languages via the mtpe process foot_0 , done by the experienced translators of the unbabel community that have demonstrated consistently high quality within the respective language pair. mt segments were produced with a mixed of online mt services and internal ones. the corpus is released under the creative commons public license attribution-noderivatives 4.0 international (cc by-nd 4.0) and can be freely used for research purposes only. please note that, as the license states, no commercial uses are permitted for this corpus. this data was collected within the maia project @xcite . the unbabel's maia dataset foot_1 contains more than 40k segments from more than 900 conversations in three language pairs (and a total of 6 language directions): english @xmath0de), english @xmath1fr) and english @xmath2pt-br). the breakdown of the corpus by language pair and direction is presented in anonymization process. to make the conversations publicly available and in accordance with the general data protection regulation (gdpr), we anonymized them first automatically by using the unbabel proprietary anonymization tool and then by manually verifying the data. this resulted in 12 different anonymization categories, each presented by a specific token that are presented in ## task description similarly to the first edition of the chat translation shared task, in this edition we focused on a critical challenge faced by international companies that are providing customer support in several different languages. one common approach to deal with this challenging requirement is centralizing the customer support with english speaking agents and having a translation layer in the middle to translate from the customer's language into the agent's language (e.g. english) and vice versa. the ideal solution for this environment needs to be able to properly handle all its aforementioned issues including the context-related challenges, the noisy inputs and multilingualism, among others. in the second edition of the chat translation shared task we provide real genuine bilingual data for three different language pairs and encouraged the participants to make use of the bilingual context present in the conversations and to submit translations for both directions of the three language pairs. to emphasize on the importance of this aspect, we decided to rank the participating teams based on the overall quality of their primary submissions on both directions using a manual quality evaluation methodology through mqm annotations. and, finally, we asked the participants to submit a maximum of three mt outputs per language pair direction, one primary and a maximum of two contrastive outputs. due to time and budget constraints we performed the manual evaluation only for the primary submissions, while all the systems are evaluated using the automatic evaluation metrics (comet, chrf, and sacrebleu). for more details on the evaluation process please see §5. ## participants the participants were asked to submit at most three systems per language pair direction, one primary and a maximum of two contrastive ones. moreover, the submitting team was required to explicitly indicate their primary and contrastive submissions. we received eighteen submissions from four different teams: bjtu-wechat (two primaries and four contrastives), iitp-flipkart (two primaries and four contrastives), hw-tsc (one primary and two contrastives), and unbabel-ist (one primary and two contrastives). the first three teams participated only for en@xmath3de, en@xmath4pt-br). all the participating systems follow a two step training in which a generic model is trained first on a large amount of publicly available data and then fine-tuned on the task data. the systems are different in the following aspects: i) the pretraining step, in which some use the publicly available models like mbart and facebook-fair's wmt 2019, and the others train their own generic models, ii) the model architecture, in which some use deep encoder-decoder transformers, and others use multi-encoder transformers, iii) the fine-tuning stage and the data used in that step, and iv) the translation directions, in which some use bilingual models for each direction and some use a single multilingual model to cover all the language pairs and directions. ## evaluation procedures similarly to the previous edition, we evaluated the systems' performance both automatically and manually. this year we used comet, chrf and sacre-bleu as the automatic metrics and mqm @xcite for the human evaluation. due to time and budget constraints, the manual mqm evaluation was performed on the primary submissions only while all the submissions were evaluated using the automatic metrics. as mentioned earlier, the official rankings of the participating teams were based on the overall mqm score of their translations for the whole conversation, i.e. both customer and agent sides. ## discussion in this section we analyze the results of the automatic and human evaluation of the systems from different aspects. ## conclusions we presented the results of the wmt 2022 chat translation shared task. this year, we provided the participants with anonymized genuine bilingual customer support conversations for development and test sets. the conversations are part of the maia corpus, a corpus that we introduced here for the first time that aim to provide the best possible research ground for this very particular domain. four different teams participated in en@xmath5fr, and en$pt-br. all participants covered both directions (i.e. customer and agent). we evaluated submissions with automatic metrics (i.e. comet, chrf, and sacrebleu) and primary submissions with mqm human evaluation. the mqm evaluations were conducted under an adaptation of the mqm framework @xcite , that is tailored to assess customer support translated content @xcite , providing a rich analysis of the type of errors that, we hope, will foster future mt research in this domain.