Negation-Instance Based Evaluation of End-to-End Negation Resolution (2021)

Abstract: In this paper, we revisit the task of negation resolution, which includes the subtasks of cue detection (e.g. “not”, “never”) and scope resolution. In the context of previous shared tasks, a variety of evaluation metrics have been proposed. Subsequent works usually use different subsets of these, including variations and custom implementations, rendering meaningful comparisons between systems difficult. Examining the problem both from a linguistic perspective and from a downstream viewpoint, we here argue for a negation-instance based approach to evaluating negation resolution. Our proposed metrics correspond to expectations over per-instance scores and hence are intuitively interpretable. To render research comparable and to foster future work, we provide results for a set of current state-of-the-art systems for negation resolution on three English corpora, and make our implementation of the evaluation scripts publicly available.

## introduction negation is a complex semantic phenomenon in natural language that "transforms an expression into another expression whose meaning is in some way opposed to the original" @xcite . it occurs frequently, with the proportion of sentences with negation in english corpora ranging between 9 and 32% @xcite . natural language processing (nlp) applications that may benefit from negation resolution include sentiment analysis @xcite and information extraction. negation is also still a challenge in machine translation @xcite @xcite and natural language inference @xcite . negation resolution @xcite refers to the task of automatically retrieving the elements of a sentence that are affected by test sentences (gold scope) . . . . . . . system . . . a system b tp / fp / fn tp / fp / fn (1) . . if not, . i . . 'll . . . . have to do with you. 0 / 4 / 0 0 / 0 / 0 (2) he made no . . . . . . . remark, but the 1 / 0 / 2 3 / 0 / 0 matter remained in his thoughts. (3) well, mrs. warren, i can see that 16 / 0 / 0 10 / 2 / 6 you have any particular cause () for concern, nor . . . do . i . . . . . . . . . . . understand . . . . why . i, . . . . . . . whose . . . . . time . . is . . of . . . . . . . some . . . . . . value, . . . . . . . should . . . . . . . . . interfere . . in . . . . the . . . . . . . matter. instance-based p/r: 66.7/77.8 94.4/87.5 f1: 71.8 90.8 the negation introduced by a cue. the cue's scope is "the part of the meaning that is negated" @xcite . the task is difficult due to the multitude of ways in which negation may be expressed. despite having been a continuously active research area especially since two shared tasks @xcite , building robust computational models is far from being a solved task, in part due to a lack of annotation standards @xcite . negation resolution has traditionally been addressed by heavily relying on syntactic parses (e.g. sanchez @xcite @xcite . recently, end-to-end neural approaches to modeling negation resolution @xcite @xcite have claimed superior performance. a significant problem in the field of negation resolution is that due to the large variety of possible evaluation setups and metrics, and the use of standard scoring scripts vs. custom implementations, it is not obvious how to meaningfully compare existing approaches and benchmark new models. differences in evaluation setup include, for example, whether gold cues are provided for scope resolution, and which subset and variations of evaluation metrics are used. in the conll 2010 shared task on detecting hedges and resolving their scope @xcite , only exact cue and scope matches were counted. the *sem 2012 shared task on negation resolution @xcite proposed additional less strict metrics based on token-level matching. the organizers of both tasks explicitly state that their evaluation is intended as a starting point and that further work on defining a scope evaluation measure that better captures the impact of partial matches is necessary. in this paper, we take a step back and revisit the evaluation of negation resolution in order to provide a more unified picture of the relative performance of different systems. as a result, we propose a negation-instance based evaluation framework that defines intuitively interpretable and linguistically motivated metrics, facilitating a graded scoring of cue and scope matching. as illustrated by figure 1 , a gradual judgment of a system's scope resolution capability may be misleading if simply computing f1 over tokens in gold or predicted scopes. in (1), system a marked a non-existent scope, which may be detrimental to a user's trust in the system. in (2), a did not mark the main event, i.e., extracting a correct logical representation would be impossible. in (3), despite getting a bad recall score, b's output captures all relevant arguments and complements of the negated proposition headed by "understand." the core of our proposed method is to normalize precision and recall scores per instance and compute an expectation for these instance-wise match scores treating all instances equally during score aggregation, following the insight that failing on short or long negation scopes may be equally detrimental. we apply our metrics to a set of recent neural models for negation resolution, including negbert, tagging-based, and dependency-parsing based approaches, providing a competitive range of baselines for future work to compare with. our contributions are as follows. (1) we provide a concise and formal overview of existing evaluation metrics with the aim of facilitating a principled comparison of approaches to negation resolution. (2) we propose a linguistically motivated and intuitively interpretable negation-instance based scope resolution scoring framework. (3) using our (and previously proposed) metrics, we conduct a reproducibility study on negation resolution, reporting performance scores of a variety of relevant baseline systems in a uniform experimental setup. foot_0 (4) as a side result, we show that a modern transformer-based reimplementation of the taggingbased system by @xcite achieves the best negation resolution performance under most circumstances. ## related work we here introduce the linguistic terminology and concepts used in this paper, and briefly survey related work in computational linguistics. ## evaluation metrics and settings in this section, we first give an overview of the various evaluation metrics used in shared tasks and research publications. we then propose a framework for negation resolution focusing on instance-level metrics. negation resolution as a standalone nlp task is usually split into two sub-tasks, cue detection and scope resolution, where the latter depends on the former. 2 hence, scope resolution may be evaluated in two settings, with either gold cue information being given or in an end-to-end manner where systems also have to predict cues. we first introduce some notations with the aim of a unified presentation of metrics. a negation instance is a tuple (c, s) consisting of a set of cue tokens @xmath0, c 2 , ..., c c ) and a set of scope tokens @xmath1, s 2 , ..., s s ). punctuation may be excluded from these sets by definition. in the case of affix negation, the affix is treated as a separate token if annotated or predicted as a cue. i g is the set of gold standard, i p the set of predicted negation instances. ## modeling in this section, we explain the negation resolution models that we compare in our experiments in sec. 5. due to the large number of negation resolution systems, often with no published code, it is infeasible to provide unified evaluation scores for all prior work. instead, to provide competitive baselines for future work to compare with, we chose a wide range of neural architectures inspired by previous work and re-implement them. for comparability reasons, we base them all on the same robust transformer-based language model. token representation. our token embedding backbone is the transformer-based xlm-r-large language model @xcite , as well as the corresponding word-piece tokenizer. to obtain contextualized word embeddings, we take a weighted sum of the internal states corresponding to the first word piece for each token. the coefficients of this weighted sum are learned during training, employing layer dropout (see @xcite . transformer weights are finetuned during training. to determine the effect of injecting implicit syntactic knowledge into the system, in addition to using the default pre-trained xlm-r model, we also run experiments on an xlm-r-synt model that was previously fine-tuned on the task of universal dependencies parsing on the ewt treebank @xcite . ## discussion a general problem with most existing text corpora annotated for negation is that annotations are only created on the surface level. however, in the words of @xcite , "negation does not stand on its own, to be useful, it should be added as part of another existing knowledge representation." our negation-instance based framework was detailed above such that it can directly be applied to existing widely used negation corpora. however, the precision and recall scoring functions f p and f r can easily be designed in other ways. for instance, if leveraging a dependency parse of the sentence, argument structure could be approximated by taking as the set of elements in a scope not the tokens, but only the dependents of the main predicate to which the negation cue attaches. in this way, relative clauses as in (3) in figure 1 would have zero impact. thinking ahead, a truly linguistically motivated scoring function could even weight components by their importance of being detected as belonging to a scope, e.g., missing a restrictive relative clause could be penalized more than missing a non-restrictive one. if we had a dataset marking cues, scopes and foci, our scoring framework could assign a high weight for detecting the focus correctly. similarly, detecting the main predicates correctly could be incorporated into the score. we here decided against including event annotations as they are only marked on factual events and hence, in our opinion, should be evaluated as a separate task (as was done in most metrics in previous shared tasks). ## conclusion and outlook the aim of this paper is to provide a concise reference of evaluation metrics and setups for negation resolution, making it easier for nlp researchers and developers to enter the research area. our core contribution is to detail the linguistic motivation for employing a new instance-based approach to evaluating the performance of end-to-end negation resolution systems, giving credit to partial scope matches but relying on exact cue matches. we argue that this metric is well-motivated and intuitively interpretable and should hence be adopted by future studies or shared tasks. in addition, our experimental study, comparing a set of recent neural architectures on a similar basis, will serve as a reference for future work. besides implementing a variety of linguistically motivated extensions with the aim of deeper system analyses using our framework as suggested above, an important next step is to evaluate the suite of models used in this paper on further datasets in languages other than english (e.g., @xcite @xcite .