C ontra CAT : Contrastive Coreference Analytical Templates for Machine Translation (2020)

Abstract: Recent high scores on pronoun translation using context-aware neural machine translation have suggested that current approaches work well. ContraPro is a notable example of a contrastive challenge set for English→German pronoun translation. The high scores achieved by transformer models may suggest that they are able to effectively model the complicated set of inferences required to carry out pronoun translation. This entails the ability to determine which entities could be referred to, identify which entity a source-language pronoun refers to (if any), and access the target-language grammatical gender for that entity. We first show through a series of targeted adversarial attacks that in fact current approaches are not able to model all of this information well. Inserting small amounts of distracting information is enough to strongly reduce scores, which should not be the case. We then create a new template test set ContraCAT, designed to individually assess the ability to handle the specific steps necessary for successful pronoun translation. Our analyses show that current approaches to context-aware NMT rely on a set of surface heuristics, which break down when translations require real reasoning. We also propose an approach for augmenting the training data, with some improvements.

## introduction machine translation is a complex task which requires diverse linguistic knowledge. the seemingly straightforward translation of the english pronoun it into german requires knowledge at the syntactic, discourse and world knowledge levels for proper pronoun coreference resolution (cr). the german third person pronoun can have three genders, determined by its antecedent: masculine (er), feminine (sie) and neuter (es). previous work @xcite @xcite proposed evaluation methods for pronoun translation. this has been of special interest in context-aware nmt models that are capable of using discourse-level information. despite promising results @xcite @xcite , the question remains: are transformers @xcite truly learning this task, or are they exploiting simple heuristics to make a coreference prediction? to empirically answer this question, we extend contrapro @xcite )-a contrastive challenge set for automatic @xmath0. our adversarial attacks on contrapro show that context-aware transformer nmt models can easily be misled by simple and unimportant changes to the input. however, interpreting the results obtained from adversarial attacks can be difficult. the results indicate that nmt uses brittle heuristics to solve cr, but it is not clear what those heuristics are. in general, it is challenging to design attacks based on modifying contrapro that can test specific phenomena that may be of interest. ## coreference resolution in machine translation addressing discourse phenomena is important for high-quality mt. apart from document-level coherence and cohesion, anaphoric pronoun translation has proven to be an important testing ground for the ability of context-aware nmt to model discourse. anaphoric pronoun translation is the focus of several works in context-aware nmt @xcite @xcite @xcite . however, the choice of an evaluation metric for cr is nontrivial. bleu-based evaluation is insufficient for measuring improvement in cr @xcite without carefully selecting or modifying test sentences for pronoun translation @xcite . alternatives to bleu include f 1 , partial credit, and oracle-guided approaches @xcite @xcite . however, @xcite show that these metrics can miss important cases and propose semi-automatic evaluation. in contrast, our evaluation is completely automatic. we focus on scoring-based evaluation @xcite , which works by creating contrasting pairs and comparing model scores. accuracy is calculated as how often the model chooses the correct translation from a pool of alternative incorrect translations. @xcite manually create such a contrastive challenge set for @xmath1. contrapro @xcite follows this work, but creates the challenge set in an automatic way. we show that making small variations in contrapro substantially changes the scores. our work is related to adversarial datasets for testing robustness used in natural language processing tasks such as studying gender bias @xcite @xcite , natural language inference @xcite and classification @xcite . @xcite propose a model for pronoun translation evaluation trained on pairs of sentences consisting of the reference and a system output with differing pronouns. however, as @xcite point out, this fails to take into account that often there is not a 1:1 correspondence between pronouns in different languages. as a result, a system translation may be correct despite not containing the exact pronoun in the reference, and incorrect even if containing the pronoun in the reference, because of differences in the translation of the referent. moreover, introducing a separate model which needs to be trained before evaluation adds an extra layer of complexity in the evaluation setup and makes interpretability more difficult. in contrast, templates can easily be used to pinpoint specific issues of an nmt model. our templates follow previous work @xcite @xcite where similar tests are proposed for diagnosing nlp models. ## do androids dream of coreference translation pipelines? imagine a hypothetical coreference pipeline that generates a pronoun in a target language, as illustrated in this idealized pipeline would produce the correct pronoun in the target language. the coreference steps resemble the rule-based approach implemented in stanford corenlp's coref-annotator @xcite . however, nmt models are currently unable to decouple the individual steps of this pipeline. we propose to isolate each of these steps through targeted examples. ## model we use a transformer model for all experiments and train a sentence-level model as a baseline. the context-aware model in our experimental setup is a concatenation model @xcite (concat) which is trained on a concatenation of consecutive sentences. concat is a standard transformer model and it differs from the sentence-level model only in the way that the training data is supplied to it. the training examples for this model are modified by prepending the previous source and target sentence to the main source and target sentence, respectively. the previous sentence is separated from the main sentence with a special @xmath2, on both the source and target side. this also applies to how we prepare the contrapro and contracat data. we train the concatenation model on opensubtitles2018 data prepared in this way. we remove documents overlapping with contrapro. preprocessing details and model hyper-parameters are presented in the appendix. ## templates automatic adversarial attacks offer less freedom than templates as many systematic modifications cannot be applied to the average sentence. thus, our templates are based on the hypothetical coreference pipeline in section 3 that target each of the three steps: i) markable detection, ii) coreference resolution and iii) language translation. our minimalistic templates draw entities from sets of 25 animals, 20 human professions @xcite , 15 foods, and 5 drinks, along with associated verbs and attributes. we use these sets to fill slots in our templates. animals and foods are natural choices for subject and object slots referenced by it. restricting our sets to interrelated concepts with generically applicable verbs-all animals eat and drink-ensures semantic plausibility. other object sets, such as buildings, had more semantic implausibility issues and were not included in the final corpus. ## augmentation we present an approach for augmenting the training data. while challenging for nlp, we focus on a narrow problem which lends itself to easier data manipulation. our previous analyses show that our model is capable of modeling the gender of nouns. however, they also show a strong prior to translate it to es and very little cr capability. our goal with the augmentation is to break off the strong prior and test if this can give rise to better cr in the model. we attempt to do this by augmenting our training data and call it antecedent-free augmentation (afa). we identify candidates for augmentation as sentences where a coreferential it refers to an antecedent not present in the current or previous sentence (e.g., i told you before. <@xmath3. → ich habe dir schonmal gesagt. <@xmath4.). we create augmentations by adding two new training examples where the gender of the german translation of "it" is modified (e.g., the two new targets are "ich habe dir schonmal gesagt. <@xmath5." and "ich habe dir schonmal gesagt. <@xmath6."). the source side remains the same. an additional example is shown in ## conclusion in this work, we study how and to what extent cr is handled in context-aware nmt. we show that standard challenge sets can easily be manipulated with adversarial attacks that cause dramatic drops in performance, suggesting that nmt uses a set of heuristics to solve the complex task of cr. attempting to diagnose the underlying reasons for these results, we propose targeted templates which systematically test the different aspects necessary for cr. this analysis shows that while some type of cr such as pleonastic and event cr are handled well, nmt does not solve the task in an abstract sense. we also propose a data augmentation approach which substantially improves performance on some metrics, but it does not change the general conclusions we infer from the templates. future work should be evaluated on our adversarial attacks and contracat, which we publicly release, to realistically estimate the ability of nmt to robustly do cr.