Revisiting Negation in Neural Machine Translation (2021)

Abstract: In this paper, we evaluate the translation of negation both automatically and manually, in English–German (EN–DE) and English– Chinese (EN–ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced networks, although the performance varies between language pairs and translation directions. The accuracy of manual evaluation in EN→DE, DE→EN, EN→ZH, and ZH→EN is 95.7%, 94.8%, 93.4%, and 91.7%, respectively. In addition, we show that under-translation is the most significant error type in NMT, which contrasts with the more diverse error profile previously observed for statistical machine translation. To better understand the root of the under-translation of negation, we study the model’s information flow and training data. While our information flow analysis does not reveal any deficiencies that could be used to detect or fix the under-translation of negation, we find that negation is often rephrased during training, which could make it more difficult for the model to learn a reliable link between source and target negation. We finally conduct intrinsic analysis and extrinsic probing tasks on negation, showing that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation in hidden states but nevertheless leave room for improvement.

## introduction negation is an important linguistic phenomenon in machine translation, as errors in translating negation may change the meaning of source sentences completely. there are many studies on negation in statistical machine translation (smt) @xcite @xcite @xcite , but studies on negation in neural machine translation (nmt) are quite limited and results are partly conflicting. for example, @xcite find that negation is still challenging, whereas @xcite show that nmt models almost make no mistakes on negation using 130 sentences with negation from three language pairs as the evaluation set. hence, it is still not clear how well nmt models perform on the translation of negation. in this paper, we present both automatic and manual evaluation of negation in nmt, in english-german (en-de) and english-chinese (en-zh). the automatic evaluation is based on contrastive translation pairs and studies translation from english into german/chinese (@xmath0. the manual evaluation targets translation in all four translation directions. we find that the modeling of negation in nmt has improved with deeper and more advanced networks. the contrastive evaluation shows that deleting negation from references is more confusing to nmt models compared to inserting negation into references. for the manual evaluation, nmt models make fewer mistakes on negation in en-de, than in en-zh, and there are more errors on negation in de/@xmath1. moreover, under-translation is the most prominent error type in three out of four directions. the black-box nature of neural networks makes it hard to interpret how nmt models handle the translation of negation. in @xcite , neither attention weights nor layer-wise relevance propagation can explain why negation is under-translated. we are interested in whether the information about negation is not well passed to the decoder. thus, we investigate the negation information flow in nmt models by raw attention weights and attention flow @xcite . we demonstrate that the under-translation of cues is not caused simply by a lack of negation information transferred to the decoder. we further explore the mismatch between source and target sentences-negation cues appearing only on the source side or only on the target side. we find that there are roughly 17.4% mismatches in the training data in zh-en. these mismatches could confuse nmt models and make the learning harder. we suggest to distill or filter training data by removing the sentence pairs with mismatches to make the learning easier. in addition, we conduct intrinsic analysis and extrinsic probing tasks, to explore how much information about negation has been learned by nmt models. the intrinsic analysis based on cosine similarity shows that nmt models can distinguish negation and non-negation tokens very well. the probing results on negation detection reveal that nmt can encode a lot of information about negation in hidden states but still leaves much room for improvement. moreover, encoder hidden states capture more information about negation than decoder hidden states. ## evaluation in this section, we present the results of both automatic and manual evaluation on negation in en-de and en-zh, to get a more comprehensive picture of the performance on translating negation. ## interpretation there are few studies on interpreting nmt models with respect to negation. since ## conclusion in this paper, we have explored the ability of nmt models to translate negation through evaluation and interpretation. the accuracy of manual evaluation in @xmath2, @xmath3, @xmath4, and @xmath5.7%, 94.8%, 93.4%, and 91.7%, respectively. the contrastive evaluation shows that deleting a negation cue from references is more confusing to nmt models than inserting a negation cue into references, which indicates that nmt models have a bias against sentences with negation. we show that nmt models make fewer mistakes in en-de than in en-zh. moreover, there are more errors in de/@xmath6. we also have investigated the information flow of negation by computing the attention weights and attention flow. we demonstrate that the negation information has been well passed to the decoder, and that there is no correlation between the amount of negation information transferred and whether the cues are under-translated or not. thus, we consider attempts to detect or even fix under-translation of cues via an analysis or manipulation of the attention flow to have little promise. however, our analysis of the training data shows that negation is often rephrased, leading to cue mismatches which could confuse nmt models. this suggests that distilling or filtering training data to make grammatical negation more consistent between source and target could reduce this under-translation problem. in addition, we show that nmt models can distinguish negation and non-negation tokens very well, and nmt models can encode substantial information about negation in hidden states but nevertheless leave room for improvement. moreover, encoder hidden states capture more information about negation than decoder hidden states; negation cues are better modeled in lower encoder layers while negation events and tokens belonging to negation scope are better modeled in higher encoder layers. overall, we show that the modeling of negation in nmt has improved with the evolution of nmt -with deeper and more advanced networks; the performance on translating negation varies between language pairs and directions. we also find that the main error types on negation have shifted from smt to nmt-under-translation is the most frequent error type in nmt while other error types such as reordering were equally or more prominent in smt. we only conduct evaluation in en-de and en-zh, and german/chinese and english are very similar in expressing negation. it will be interesting to explore languages have different characteristics on negation in the future, such as italian, spanish, and portuguese, where double negation is very common.