Evaluating and Improving the Coreference Capabilities of Machine Translation Models (2023)

Abstract: Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: How well MT models learn coreference resolution via implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language.Following, we evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.

## introduction machine translation (mt) may require coreference resolution to translate cases where the source and target language differ in their grammatical properties. for example, consider translating "the trophy didn't fit in the suitcase because it was too small" from english to french: "le trophée ne rentrait pas dans la valise car elle était trop petite" @xcite . in french, suitcase ("valise") is grammatically feminine, and trophy ("trophée") is masculine, while the source-side english does not encode grammatical noun gender. this requires an mt model to infer that "it" refers to the suitcase (and not the trophy) to correctly produce the feminine inflection for the phrase "it was too small" ("elle était trop petite"), whereas an incorrect coreference resolution may produce the masculine inflection ("il était trop petit") corresponding to the trophy. figure 1 : mt models can be compared to source-side coreference resolvers. an example translation from english (turquoise) to french (yellow). our method first identifies the grammatical gender of the mentions in the target language marked in purple (female) and blue (male), followed by inferring the source side clusters (orange), through gender agreement. such texts evade lexical one-to-one translation, and instead demand source-side coreference resolution as a prerequisite for a correct translation. the prominent end-to-end approach to mt assumes that translation models implicitly learn source-side coreference resolution by observing aligned sourcetarget pairs, without intermediate coreference supervision. while the importance of addressing such semantic phenomenon has been stated in various works @xcite , it was also observed that the ubiquitous bleu metric @xcite does not adequately quantify it @xcite . this work addresses the following research question: how well does mt learn coreference when compared against explicit coreference supervision? answering this question can improve our understanding of the way mt models operate and also has practical implications: if implicit supervision lags behind monolingual training, it would motivate integration between end-to-end mt approaches and explicitly-supervised monolingual components. in section 3 we devise an evaluation paradigm that reduces mt output to source-side coreference resolution predictions by inferring coreference clusters from source inputs and predicted target translations. e.g., in the previous example, a feminine inflection for the pronoun "it" in french can infer linking "it" with "suitcase", while a masculine french inflection links "it" with "trophy", as shown in figure 1 . this approach allows us to distill the coreference resolution abilities of mt models and compare them against state-of-the-art coreference resolution models, trained explicitly on the task. we use this approach to evaluate the coreference capabilities of several commercial and open source mt systems, translating from english to six target languages. we conduct our experiments in both synthetic (winomt and wino-x; @xcite and naturalistic settings (bug; @xcite . our results show that state-of-the-art coreference resolvers vastly outperform mt models on several benchmarks, indicating that explicit supervision may lead to better coreference performance. following this finding, in section 4, we develop methods for improving coreference in mt, both implicitly and explicitly. our implicit approach consists of fine-tuning mt models on texts that specifically require many coreference decisions, thus exposing the model to more implicit coreference signal. our explicit approach further enriches source sentences with predicted coreference markers. we show that these approaches improve coreference over the end-to-end mt approach, achieving comparable or better results than much larger mt models, both commercial systems and open-source. more broadly, our approach can be applied to improve the translation of other semantic phenomena that diverge in realization between source and target languages, such as plurality in second-person pronouns @xcite or tense marking @xcite . ## background: gender bias in mt we start our work by extending the methodology developed in @xcite , which relies on target-side morphology to infer the translated gender of certain professions. in particular, assuming a dataset of english sentences d, where each instance includes gold coreference annotation between a human entity and its pronoun (e.g., "the doctor asked the nurse to help her with the procedure."), they evaluate gender bias from english to language t with morphological gender in the following manner: the gender bias of m is then defined as the difference in performance between stereotypical and anti-stereotypical gender role assignments. we use a similar setup to address a different question: rather than evaluating the gender bias of the model, we evaluate its coreference abilities, which may be hindered by bias, but also by the inherent difficulty to infer coreference in the absence of an explicit training signal. ## mt models fare poorly against coreference resolvers the approach taken in winomt is limited as it restricts the evaluation to sentences with a known gender in english, indicated by a gendered pronoun of a human entity (e.g., her). consider the sentence in figure 1 : "the trophy didn't fit in the suitcase because it was too small" with the coreference cluster {suitcase, it}. step 3 in stanovsky et al.'s method will fail to assign a coreference label to the translation, because "it" does not have a gender in english. in this section, we extend the winomt approach in order to estimate more general coreference abilities of mt models. to achieve this, we note that many languages have gender agreement between pronouns and the noun that they refer to. therefore, correct targetside gender agreement requires (implicitly) resolving the source-side coreference of the relevant entities. as exemplified in figure 1 , a reader of the french translation would infer from the gender inflection of "it" whether it refers to "suitcase" or the "trophy". i.e., a feminine pronoun ("elle") would agree with the feminine noun suitcase ("valise"), while a masculine pronoun ("il") would agree with the masculine noun trophy ("trophée'). we therefore formulate a new metric quantifying the ability of the mt model to implicitly resolve source-side coreference (henceforth, target-side consistency), defined as the proportion of instances in which the morphological gender of an entity (e.g., "suitcase") matches that of its referring pronoun (e.g., "it") in the target language t . this metric examines whether an mt model is consistent in its coreference decisions, regardless of whether it correctly inferred the coreference relations in the input text. indeed, some texts may keep the english ambiguity in the translation, and hence absolve the mt model from resolving coreference. for example, in the sentence "the battery didn't fit in the suitcase because it was too small", both battery and suitcase are feminine in french. our proposed metric will correctly indicate that the mt model was successful in such cases (albeit trivially). the metric thus serves as an upper bound on the mt model's coreference abilities. we note that while this framing uses the morphological gender inflection of common nouns, it is different in motivation from measures of gender bias. in our example above, gender inflection allows us to determine whether an mt model correctly employs common sense rather than examining whether it tends to prefer stereotypical gender norms. while a model's gender bias may explain some loss in coreference abilities, the model's ability to resolve coreference need not be aligned with the degree of its bias (e.g., a random gender assignment would result in unbiased performance, but very poor coreference ability). most importantly, by considering the gender of the entity and the pronoun, we obtain mention clusters which can be compared against those produced by coreference resolution models. in our example figure, both the first mt model and the coreference model produce the correct clustering: {{trophy}, {suitcase, it}}, while the second mt model errs by producing: {{trophy, it}, {suitcase}}. another aspect of our evaluation methodology is its generality. our method does not require a reference translation or make any particular assumptions about the generated output. as there are generally many correct translations, this flexibility allows us to accurately assess the model's coreference abilities. for instance, our methodology does not assume the gender of the entity's translation as can be seen in the first example in ## improving mt coreference consistency in the previous section we showed that the coreference performance of mt systems, obtained through an implicit signal, seems inferior to that of coreference resolution learned from an explicit signal. this result raises the question of whether we can leverage dedicated conference resolvers to improve the consistency of mt coreference. to address this question, we propose two data augmentation techniques that leverage a sourceside english coreference model, and show that finetuning on them indeed improves coreference resolution in mt. augmented fine-tuning with instances which require coreference resolution. first, we run a coreference resolution model on the source-side sentences. we then consider two approaches for constructing the augmented fine-tuning data: (1) coref data with all sentences that have nonsingleton clusters and (2) gender data, a subset of coref data where there is at least one non-singleton cluster with a gendered pronoun (he, she, her, him, adding explicit source-side coreference markers. second, we use the non-singleton clusters from the coreference model to add inline coreference markers in the source sentences. for our example sentence, this process produces the following source-side sequence: "the trophy didn't fit in @xmath0, indicating that "suitcase" and "it" are coreferring. ## related work the study of coreference has a long tradition in machine translation. a long line of work uses pronoun translation as a way of measuring coreference, since bleu-based evaluation was shown to be insufficient for measuring improvement in coreference @xcite . an alternative evaluation methodology is using automatic reference-based methods that produce a score based on word alignment between the source, reference translation, and translation output, and identification of pronouns in them, such as auto-prf @xcite and @xcite . nevertheless, a later human meta-evaluation showed ## conclusion our work is the first to present an automatic methodology for assessing the coreference capabilities of mt models, that can be applied in any target language and does not require any target side annotations. furthermore, to the best of our knowledge, we are the first to conduct a large-scale multilingual coreference evaluation study on prominent opensource and commercial mt models, and compare them against state-of-the-art coreference resolvers on three challenging benchmarks. finally, based on the superior results of coreference resolvers, we propose a novel approach to improve the coreference capabilities of mt models, that outperforms or achieves comparable results to strong and larger mt models. despite this substantial gain, there is still a performance gap between our model and state-of-the-art coreference resolvers. we hope that our work, and specifically our automatic evaluation methodology, will encourage future research to improve the coreference capabilities of mt models. future work can expand our approach to account for number and person agreement phenomena, investigate how to extend our approach to more coreference clusters and more mentions per cluster in intra-sentential as well as inter-sentential settings. moreover, we intend to investigate how different morphological attributes affect mt models' coreference abilities.