Learning Negation Scope from Syntactic Structure (2020)

Abstract: We present a semi-supervised model which learns the semantics of negation purely through analysis of syntactic structure. Linguistic theory posits that the semantics of negation can be understood purely syntactically, though recent research relies on combining a variety of features including part-of-speech tags, word embeddings, and semantic representations to achieve high task performance. Our simplified model returns to syntactic theory and achieves state-of-the-art performance on the task of Negation Scope Detection while demonstrating the tight relationship between the syntax and semantics of negation.

## introduction negation is a semantic phenomenon in natural language which varies significantly. for example, "sherlock did not solve the case" contains a simple negation cued by "not." however, there are many cue words such as "without" and "nothing," and word affixes like "un-" which instantiate negation, and their effect on meaning can be different and dependent on context. we approach the meaning of negation using a logical semantics, in which natural language negation is expressed with the negation operator on logical expressions @xcite , such as in ¬solve(sherlock, case). further, in truth-theoretic logic the meaning of negation is simply the inverted truth value of this expression. capturing the meaning of a negation cue in language can thus be understood as simply identifying the negated expression. this is the task of negation scope detection (nsd): given a negation cue in a sentence, identify the sentence tokens which make up the negated expression. this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . ## previous work there have been a variety of approaches to nsd. the *sem shared task dataset is the only resource available with semantic annotations (other nsd datasets are annotated for different goals), but there are many models which use *sem to compare with @xcite . @xcite show the most recent, best-performing model on *sem. it is a bil-stm sequence tagging model which produces inscope/out-of-scope classifications for each word, and jointly learns word embeddings with additional part-of-speech features for each token. however, further analysis @xcite shows the model is over-reliant on punctuation like commas (",") which often mark scope boundaries in english and especially in conan doyle's older style of writing (the content of the *sem dataset). the model does not learn a very robust semantics without these markers, yet also does not make use of the full syntactic structure of sentences. @xcite address this with a dependency-lstm model which processes dependency trees using encodings both for words and dependency relations. on a modified *sem dataset this model slightly improves over the bilstm when they are ensembled together, showing again that the bilstm method lacks some structural understanding. the original winner of *sem, u io 1 @xcite , uses an svm classifier on constituents that contain the cue. constituents are shown to be useful, but this is not enough to capture discontinuous scopes (see example ( 2 )) which do not align to a single constituent. read adds extra heuristics to help with this, and @xcite continues from this with a comprehensive set of syntactic heuristics deployed on dependency graphs. this model performs well and demonstrates the strength of syntax by itself for capturing negation semantics. @xcite and @xcite approach nsd "in the semantic domain." packard first induces a minimal recursion semantics parse of a sentence @xcite , then crawls it to identify negated predicates and arguments. though intuitive, inducing a full parse may overshoot the problem -by itself the crawler doesn't compete because it loses much information during parsing and backtracking to sentence words. ensembling with u io 1 provides a boost over u io 1 itself, which indicates that mrs parsing provides complementary information, possibly due to induction of additional structure. ## a structural approach we argue for a solution returning to syntax which makes per-word scope judgements conditioned on the full syntactic structure of the sentence. section 1.1 discusses the theoretical basis in syntax for negation semantics and section 2 details successful models on the task. while syntax is used by several models to a degree, full trees are rarely used and recent models rely on additional features like word embeddings to achieve high performance. combinatory categorial grammar (ccg) is a nearly context-free constituency grammar capable of describing complex phenomena like coordination @xcite . ccg also has transparency between syntax and semantics: a ccg syntactic parse may be transformed into a logical form in terms of events, similar to the sherlock example in section 1. @xcite separately formalizes a theory for computing the polarity of lexical items using a ccg-based calculus, which encourages an automated approach to learning negation scope by example. figure 1(a) shows the ccg parse tree for example (1). commonly, simple independent clauses in english form one continuous scope span, which is distinguishable in syntax. the scope of negation in (1) is simply the complement clause. however, it is not always the case that negation scope aligns cleanly to subtrees, like in example figure 1(b) , the parse tree for sentence (2). ccg provides structural insight which alleviates this problem. in this example ccg's explicit modeling of coordination makes it straightforward to identify the subject to the left of the coordinated dependent clauses. s[dcl] s[dcl] s[dcl] s[dcl]\np s[dcl] s[dcl]\np s[b]\np pp np it it it it it it it it it it it it it it it it it pp/np of of of of of of of of of of of of of of of of of (s[b]\np)/pp come come come come come come come come come come come come come come come come come (s[dcl]\np)/(s[b]\np) could could could could could could could could could could could could could could could could could np[nb] n good good good good good good good good good good good good good good good good good np[nb]/n no* (s[dcl]\np)/s[dcl] thought np i s/s , , s/s sir s/s , , s/s well s[dcl] s[dcl][conj] s[dcl] s[dcl]\np s[dcl]\np[conj] s[dcl]\np (s\np)\(s\np) away away away away away away away away away away away away away away away away away s[dcl]\np np nothing* (s[dcl]\np)/np gives gives gives gives gives gives gives gives gives gives gives gives gives gives gives gives gives conj and s[dcl]\np is a deep one np he he he he he he he he he he he he he he he he he conj but s[dcl] , , s[dcl] understanding how semantics is built from phrase structures is key to our method. analyses from additional sentences support this (see appendix for parse trees). ## methods the global belief tree recursive neural network of @xcite is adapted to solve this task. the gb-trnn takes as input a binarized syntax tree (satisfied by ccg) and starting state vectors for each word in the sentence, referred to now as tree leaves. it first recursively combines constituent states from leaves to root in an upward pass, building up to a single, global state vector. conditioned on this state, the downward pass recursively unfolds the global vector following the parse tree back down from the root to the leaves. the output for each constituent in the downward pass can be used to produce classifications conditioned on the entire tree. we add additional syntactic target inputs in both passes and refer to this new architecture as a structural tree recursive neural network (strnn). these additional inputs represent the current kind of ccg combination. in the upward step the parent tag (composed category) is additionally passed in with the two child states, while in the downward step the two child tags (decomposed categories) are passed in addition to the parent state. the strnn learns one embedding matrix for ccg syntactic categories e @xmath0, where v ccg is the ccg category vocabulary (∼400 tags) and the embedding size @xmath1. figure 2 shows a diagram of the strnn. upward pass. the initial state u i for a leaf constituent i is the result of the transformation matrix @xmath2. this consumes a category embedding and the cue feature c to produce an initial upward state u i of size @xmath3. c is a binary indicator expressing if the word is a cue. note that only a word's ccg category and cue status are shown to the model. the strnn first makes an upward pass through the syntax tree. this pass recursively combines two child constituent states u lef t and u right as well as the target parent category embedding s @xmath4. a weight matrix w ↑ ∈ r (2h+s)×h is learned for this operation. the top level hidden state of size h, global ↑ , is then transformed with the matrix @xmath5. this is recursively unfolded down the tree to the leaves using the learned matrix the recursive cell consumes two recurrent inputs following paulus: the downward parent state d parent and upward state u parent . we supply additional inputs which instruct the cell how to decompose the state vector, the target child category embeddings s lef t , s @xmath6. the cell produces a double-wide state vector which is split into left-and right-child state vectors d lef t and d right . completing the downward pass results in a globally-informed information state d j for each constituent j in the syntax tree. classifications are produced using the matrix @xmath7. c takes as input, for any tree constituent j, the concatenation of u j and d j vectors. this can be applied in the same way to tree leaves (words) and to any higher tree constituent to produce a scope judgement. optimization. the model is semi-supervised. the *sem dataset provides supervision signal only for which words are in scope, not general constituents. however, like the gb-trnn, the strnn produces classifications for all tree constituents, learning the general pattern from the supervision of words. during training the adam optimizer was used with cross-entropy loss and 0.001 initial learning rate. regularization is important with a small training set, and we used dropout following @xcite with recurrent connections set to 0.2 and others 0.5. 5 results ## conclusions we show the strnn model effectively predicts negation scope using syntactic parse trees. state-ofthe-art performance is achieved on the *sem2012 shared task without identifying individual words or extracting features beyond syntax. this result reverts to earlier theories about the relationship between syntax and negation semantics. both word embeddings and semantic reasoning require large resource overhead in model parameters and processing, but efficient use of syntax is all that is needed for high performance on this task.