Predicting the Focus of Negation: Model and Error Analysis (2020)

Abstract: The focus of a negation is the set of tokens intended to be negated, and a key component for revealing affirmative alternatives to negated utterances. In this paper, we experiment with neural networks to predict the focus of negation. Our main novelty is leveraging a scope detector to introduce the scope of negation as an additional input to the network. Experimental results show that doing so obtains the best results to date. Additionally, we perform a detailed error analysis providing insights into the main error categories, and analyze errors depending on whether the model takes into account scope and context information.

## introduction negation is a complex phenomenon present in all human languages. @xcite put it beautifully when he wrote "negation is what makes us human, imbuing us with the capacity to deny, to contradict, to misrepresent, to lie, and to convey irony." broadly speaking, negation "relates an expression e to another expression with a meaning that is in some way opposed to the meaning of e" @xcite . the key challenge to understanding negation is thus to figure out the meaning that is in some way opposed to e-a semantic and highly ambiguous undertaking that comes naturally to humans in everyday communication. negation is generally understood to carry positive meaning, or in other words, to suggest an affirmative alternative. for example, john didn't leave the house implicates that john stayed inside the house. @xcite show that comprehending negation involves considering the representation of affirmative alternatives. while not fully understood, there is evidence that negation involves reduced access to the affirmative mental representation @xcite . @xcite provide evidence that humans switch to the affirmative alternative in binary scenarios (e.g., from not red to green when processing the figure could be red or green. the figure is not red). in such multary scenarios, however, humans keep the negated representation unless the affirmative interpretation is obvious from context (e.g., humans keep not red when processing the figure is red, green, yellow or blue. the figure is not red.). from a linguistic perspective, negation is understood in terms of scope and focus (section 2). the scope is the part of the meaning that is negated, and the focus is the part of the scope that is most prominently or explicitly negated @xcite . identifying the focus is a semantic task, and it is critical for revealing implicit affirmative alternatives. indeed, the focus of negation usually contains only a few tokens, and it is rarely grammatically modified by a negation cue such as never or not. only the focus of a negation is actually intended to be negated, and the resulting affirmative alternatives range from implicatures to entailments as exemplified below (focus is underlined, and affirmative alternatives are in italics): he reported the incident to his superiors, but not until confronted with the evidence. the board learnt about the millions of dollars wasted in duplicate work, but not the details. in this paper, we experiment with neural networks for predicting the focus of negation. we work with the largest corpus annotating the focus of negation (pb-foc, 3,544 negations), and obtain the best results to date. the main contributions of this paper are: (a) neural network architecture taking into account the scope of negation and context, (b) experimental results showing that scope information as predicted by an automated scope detector is more beneficial than context, (c) quantitative analysis profiling which foci are easier and harder to predict, and (d) detailed qualitative analysis providing insights into the errors made by the models. crucially, the scope detector we leverage to predict focus is trained with cd-sco, a corpus created independently of pb-foc (section 2). our results suggest that negation scopes may transfer across (a) genres (short stories vs. news) and (b) negation types (all negations vs. only verbal negations, i.e., when the negation cue modifies a verb). ## background it is generally understood that negation has scope and focus. scope is "the part of the meaning that is negated" and includes all elements whose individual falsity would make the negated statement strictly true @xcite . consider the following statement (1) john doesn't know exactly how they met. this statement is true if one or more of the following propositions are false: (1a) somebody knows something, (1b) john is the one who knows, (1c) exactly is the manner of knowing, and (1d) how they met is what is known. thus, the scope of the negation in statement ( 1 ) is (1a-d). the focus of a negation is "the part of the scope that is most prominently or explicitly negated", or in other words, the element of the scope that is intended to be interpreted as false to make the overall negative true @xcite . determining the focus consists in pinpointing which parts of the scope are intended to be interpreted as true and false given the original statement. without further context, one can conclude that the intended meaning of statement (1) is john knows how they met, but not exactly, or alternatively, that (1a-b, 1d) are intended to be interpreted as true, and (1c) as false. this interpretation results from selecting as focus (1c), i.e., the manner of knowing. we summarize below corpora annotating scope and focus of negation, emphasizing the ones we work with. the survey by @xcite provides a more comprehensive analysis including corpora in languages other than english. ## previous work in addition to identifying negation cues and resolving the scope and focus of negation, there is work showing that processing negation is important for natural language understanding in general. in particular, sentiment analysis benefits from processing negation @xcite . for example, like generally carries positive sentiment, but not when modified by a negation cue (e.g., don't like). @xcite introduce the idea of contextual polarity, and note that negation may intensify rather than change polarity (e.g., not good vs. not only good but amazing). @xcite present a set of heuristic rules to determine sentiment when negation is present, and @xcite show that information about the scope of negation is beneficial to predict sentiment. outside sentiment analysis, @xcite point out that neural machine translation struggles translating negation, and point to focus detection as a possible solution. neural networks are hard to interpret, but there is evidence that they learn to process negation-to a certain degree-when trained to predict sentiment analysis. @xcite visually show that neural networks are capable of meaning composition in the presence of, among others, negation and intensification. @xcite show that an lstm architecture is capable of determining sentiment of sequences containing negation such as not good and not bad. these previous works train a model for a particular task (i.e., sentiment analysis) and then investigate whether the model learnt anything related to negation that is useful for that task. unlike them, we target focus of negation detection-and the resulting affirmative alternatives-and work with task-independent negations. ## predicting the focus of negation we approach the task of predicting focus of negation as a sequence labeling task with a neural network. we first describe the network architecture, and then present quantitative results. section 5 presents a detailed error and qualitative analysis. ## error and qualitative analysis to better understand the strengths and weaknesses of our models, we perform a detailed qualitative analysis of the errors made in predicting focus. negation is a complex semantic phenomenon which interacts with other aspects of the meaning and structure of sentences, and this complexity is reflected in the diversity of errors. we perform the analysis over all 712 negations in the test set, investigating how linguistic properties of the negated sentences influence performance across the four models (baseline, scope, context, and combined); we consider nearly 3,000 predictions in total. the counts in this section reflect instance-model pairings; it could happen, for example, that three of the four models predict the wrong focus for a sentence with a particular linguistic property. for some sentences, multiple error types are relevant. we identify three broad categories of errors: syntactic (5.1), semantic (5.2), and other (5.3). there are multiple error types within each category, and each error type is associated with a particular linguistic property of the negated sentence. here we focus on the most frequently-occurring error types per category, as these offer the greatest insight into specific strengths and weaknesses of the models. the distribution of error categories across the four models is shown in representative examples from pb-foc for each error type appear in tables 5 , 6 , and 7. for each example, we show the full sentence, with predicted scope (as output by the scope detector trained with cd-sco) between double angle brackets and semantic roles in square brackets. for each negated sentence, the table shows the gold focus (gf) foot_0 and the predicted focus (pf), along with the model(s) responsible for the incorrect prediction. ## conclusions negation is generally understood to carry positive meaning, or in other words, to suggest affirmative alternatives. predicting the focus of negation (i.e., pinpointing the usually few tokens that are actually negated) is key to revealing affirmative alternatives. in this paper, we have presented a neural architecture to predict the focus of negation. we work with pb-foc, a corpus of verbal negations (i.e., when a negation cue grammatically modifies a verb) in which one semantic role is annotated as focus. experimental results show that incorporating scope of negation information yields better results, despite the fact that we train the scope detector with data in a different domain (short stories vs. news). these results suggest that scope of negation transfers across domains. our best model (nn + scope) obtains the best focus prediction results to date. a quantitative analysis shows that this model is robust across most role labels ( in addition to state-of-the-art results, we have presented a detailed qualitative analysis. we discover three main error categories (syntactic, semantic, and other) and 8 error types after manual analysis of the predictions made by the four models with all test instances. we draw two main insights from the qualitative analysis. first, including scope information solves many syntactic errors but introduces semantic errors (recall that scope information is beneficial from a quantitative point of view). second, the lower results after including context, at least with the current architecture, are largely due to additional semantic errors via distractors in the previous and next sentences. e the e carrier e has e not e yet e turned e a e profit e n e n e n e n e n e y e n e n e a0 e a0 e am-mod e am-neg e am-tmp e v e a1 e a1 token emb. (1024) negated verb emb. (50) srl emb. (200) e i_s e i_s e o_s e i_s e i_s e i_s e i_s scope emb. (200) e i_s figure 2: input representation of our neural model. the symbol ⊕ denotes concatenation, not addition. the left and right dotted rectangles in figure 1 . the previous and next sentences in the example shown in figure 1 are "stateswest operates four twinengine turboprop aircraft, connecting 10 cities in california, arizona and nevada" and "the former president of firstsouth f.a., a defunct arkansas thrift, pleaded guilty to conspiring to inflate the institution's earnings by concealing worthless loan guarantees" respectively. like in the baseline model, we map each word of the adjacent sentences to its 1,024-dimensional elmo embedding vector before feeding them into the recurrent network. each network component consists of a 2-layer bidirectional lstm with 50 hidden units. a dropout rate of 30% is applied to the recurrent layers. we add an attention layer on top of the final bilstm layer. more specifically, we adopt the word-attention technique proposed by @xcite . the attention weights from both networks are concatenated with the final hidden representation of the base 3-layer bilstm network (figure 1 ). subsequently, the additional network components are trained with the original bilstm network.