It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020)

Abstract: As machine translation (MT) systems progress at a rapid pace, questions of their adequacy linger. In this study we focus on negation, a universal, core property of human language that significantly affects the semantics of an utterance. We investigate whether translating negation is an issue for modern MT systems using 17 translation directions as test bed. Through thorough analysis, we find that indeed the presence of negation can significantly impact downstream quality, in some cases resulting in quality reductions of more than 60%. We also provide a linguistically motivated analysis that directly explains the majority of our findings. We release our annotations and code to replicate our analysis here: https://github.com/mosharafhossain/negation-mt .

## introduction machine translation (mt) has come a long way, steadily improving the quality of automatic systems, relying mostly on the advent of neural techniques @xcite and availability of large data collections. as neural mt (nmt) systems start to become ubiquitous, however, one should take note of whether the improvements, as measured by evaluation campaigns such as wmt @xcite @xcite or iwslt shared tasks @xcite or other established benchmarks, do in fact mean better quality, especially with regards to the semantic adequacy of translations. we focus specifically on negation, a single yet quite complex phenomenon, which can severely impact the semantic content of an utterance. as nmt systems increasingly improve, to the point of claiming human parity in some high-resource language pairs and in-domain settings @xcite ), 1 fine-grained semantic differences become 1 we direct the reader to @xcite and (toral increasingly important. negation in particular, with its property of logical reversal, has the potential to cause loss of (or mis-)information if mistranslated. other linguistic phenomena and analysis axes have gathered significant attention in nmt evaluation studies, including anaphora resolution @xcite and pronoun translation @xcite , modality @xcite , ellipsis and deixis @xcite , word sense disambiguation @xcite , and morphological competence @xcite . nevertheless, the last comprehensive study of the effect of negation in mt pertains to older, phrase-based models @xcite . in this work, we set out to study the effect of negation in modern nmt systems. specifically, we explore: ## negation 101 negation at its most straightforward-simple negation of declarative sentences-involves reversing @xcite for further examination of such claims. the truth value of a sentence. skies are blue, for example, becomes skies are not blue. clausal negation of an existing utterance, defined roughly as negation of the entire clause rather than a subpart, produces a second utterance whose meaning contradicts that of the first. 2 without stipulations about special contexts (e.g. referring to skies in two different places), it is not possible for both utterances to be true. constituent negation involves negation of a non-clausal constituent, such as happy in the universe is not happy with us. finally, lexical negation refers to two different phenomena. the first are words formed by adding negative affixes, such as non-starter or unhappy. the second are words with no negative affixes whose meanings nonetheless convey negation, such as the verb lack in this engine lacks power. negation is a core property of human language; every language provides mechanisms for expressing negation, and, cross-linguistically, there are a small number of strategies used for its expression @xcite @xcite . following @xcite , languages may use three different mechanisms for clausal negation: a) negative particles (e.g. english not or german nicht); b) negative morphemes or affixes (e.g. lithuanian ne-); and/or c) negative auxiliary verbs, (e.g. finnish ei), a strategy in which (generally) inflection moves from the main verb to the auxiliary verb. depending on the language, constituent and lexical negation may be coded using the same mechanisms as clausal negation, or with entirely different mechanisms. negative pronouns (e.g. english nothing or nowhere), negative adverbials (never), and negative polarity items (any) add additional complexity to the expression of negation. even within negation, cross-linguistic variability is immense, ranging from different linguistic strategies, to different structural relationships between linguistic elements, even to differences in interpretation. @xcite show that the interpretation of conjunction, disjunction, and negation can differ cross-linguistically. in english "not x and y" can be interpreted as either ¬x ∧¬y ("neither x nor y") or ¬x ∨¬y ("not x or not y") while in hungarian (and russian, italian, etc.) only the first interpretation is allowed. such differences are potential error sources for translating negation. ## experimental setup we follow the setup of the conference on machine translation (wmt), in particular, the 2018 and 2019 shared tasks @xcite . we compare reference and system translations using normalized direct assessments. direct assessments are scores from 0 to 100 provided by researchers or crowd workers. in order to account for the hundreds of human annotators with potentially different criteria, raw direct assessments are normalized using the mean and standard deviation of each annotator. the normalized direct assessment is the average of the sentence-level direct assessments. we refer to normalized direct assessment with z-score or simply z. z-scores are the official ranking criterion in wmt competitions, and are preferred to automated metrics to assess the quality of mt. nevertheless, most of the mt community still relies on automated metrics for development and system comparisons. thus, we also work with three automated metrics, in particular bleu @xcite , chrf++ @xcite , and meteor @xcite . in the remainder of the paper we present two complementary analyses. first, we investigate the role of negation in machine translation with an emphasis on numeric evaluation (section 4). second, we investigate from a linguistic perspective what makes translating negation difficult (section 5). ## quantitative analysis we conduct a thorough quantitative analysis of the wmt18 and wmt19 submissions around six questions investigating the role of negation in mt. ## conclusions we show, through both quantitative and qualitative analysis, that negation remains problematic for modern nmt systems. though negation is ubiquitous and universal in its semantic effect, its realization varies tremendously from language to language. typological similarity with respect to negation seems to correspond to better translation of negation, at least for the language pairs we investigate. looking forward, we propose to harness linguistic insights about particular languages to better translate negation, and to devise fine-grained evaluation metrics to capture the adequacy of negationinvolving translations. elena voita, rico sennrich, and ivan titov. 2019. when a good translation is wrong in context: context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 1198-1212, florence, italy. association for computational linguistics. elena voita, pavel serdyukov, rico sennrich, and ivan titov. 2018. context-aware neural machine translation learns anaphora resolution. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1264-1274, melbourne, australia. association for computational linguistics. mike zhang and antonio toral. 2019. the effect of translationese in machine translation test sets. in proceedings of the fourth conference on machine translation (volume 1: research papers), pages 73-81, florence, italy. association for computational linguistics.