The U niversity of M aryland’s Submissions to the WMT 20 Chat Translation Task: Searching for More Data to Adapt Discourse-Aware Neural Machine Translation (2020)

Abstract: This paper describes the University of Maryland’s submissions to the WMT20 Shared Task on Chat Translation. We focus on translating agent-side utterances from English to German. We started from an off-the-shelf BPE-based standard transformer model trained with WMT17 news and fine-tuned it with the provided in-domain training data. In addition, we augment the training set with its best matches in the WMT19 news dataset. Our primary submission uses a standard Transformer, while our contrastive submissions use multi-encoder Transformers to attend to previous utterances. Our primary submission achieves 56.7 BLEU on the agent side (en→de), outperforming a baseline system provided by the task organizers by more than 13 BLEU points. Moreover, according to an evaluation on a set of carefully-designed examples, the multi-encoder architecture is able to generate more coherent translations.

## introduction recent advances have made mt a widespread tool for asynchronous consumption of text. the dream of dissolving language barriers, however, will not be fulfilled until mt enables two or more people carry on a synchronous conversation, each speaking their native languages. building translation systems that enable seamless conversations between an english-speaking customer support agent and a german-speaking customer is the goal of wmt20's shared task of chat translation @xcite . in participation of this shared task, we focused on the agent side, translating english utterances into german. our methods are inspired by @xcite and @xcite , explicitly leveraging broader context to address coreference and cohesion to improve translation quality. ## related work one of the main challenges for translating discourse arises from ambiguities of sentences when they are taken out of context, as mt models often do @xcite . especially in dialogue, sentences tend to reference entities in previous sentences, which necessitates using cross-sentential information to translate a given sentence. individual words can be translated in different ways, significantly varying the meaning of the resulting sentence in a larger context @xcite . in addition, dialogue in the customer support domain is a distinctive and spontaneous category of text, with colloquialisms, errors and minimal revisions. all of these deviations can accumulate error throughout the course of a conversation. dialogue translation: specific interests in translating dialogues can be found as early as lee and kim (1997)'s work on korean-english dialogue translation based on syntactic patterns and n-grams. though their model parses sentences into speech acts instead of generating full-sentence translations, they have pointed out the importance of context (previous sentence) in interpreting the current sentence properly. the most relevant recent work is @xcite , in which contexts for both source-side and target-side are utilized as additional generation conditions for the decoder in their nmt model. several variants of the model architecture and the attention mechanism are explored. however, their experiments are conducted on europarl and opensubtitles. the former is formal language and the latter scripted conversations of movies and tv. here, in contrast, chat data is informal unscripted real-world language. ## experiments we conducted varied experiments in the english-german direction. we included the english reference of the customer utterances as training data for the scope of these experiments, even though this would not be available in a production setting. this was a strategy to provide more training pairs to our models, knowing that the english references for customers is natural language, according to task organizers. ## discussion our results largely agree with those of @xcite , chiefly that combining knowledge from a previous "context" sentence can improve the model's ability to improve translation quality when measured against sentences whose translations require anaphora considerations. to accommodate this, we produced one set of sentences which require coreference and cohesion resolution, and one set of sentences that have invalid resolution. we found that each submitted system scored worse on the invalid set compared to the valid set, but the difference was more staggering ( our work and submission to the shared task can be viewed with several caveats in mind, which may explain the sub-optimal performance of the contrastive systems compared to the primary system. first, we used hyperparameters consistent with a context-agnostic pretrained model in order to have a fair comparison for evaluation and because these presumably have been well-tuned for the original model. it may be the case that different hyperparameters would work better for this particular data and the slightly larger architectures used for the contrastive submissions. it would be worth strategizing with better hyperparameter optimization. secondly, we use the provided target sides of the @xmath0. since both the agent and customer sides of this datasets were actually produced in english (the latter being translated with human-corrected machine translation), these additional utterances are likely higher quality than we would get from back-translating in a real test setting. ## conclusions in this paper, we discussed our methods for training and submitting the outputs of three models for the wmt20 shared task of chat translation. each system was based off a transformer model pretrained on wmt17 en-de news to provide better fluency. our best system achieves a test bleu of 56.7, improving over the provided baseline by more than 13 bleu points, and less than 4 points behind the best shared task submission. though we were unable to show that a context-aware model produced better translation quality than the context-agnostic model on the given dataset, our coherence evaluations indicated that it can produce better translations when measured against references needing context for coreference and cohesion resolution. this was validated both in terms of bleu and by model scoring of references.