Improving negation detection with negation-focused pre-training (2022)

Abstract: Negation is a common linguistic feature that is crucial in many language understanding tasks, yet it remains a hard problem due to diversity in its expression in different types of text. Recent works show that state-of-the-art NLP models underperform on samples containing negation in various tasks, and that negation detection models do not transfer well across domains. We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking, to better incorporate negation information into language models. Extensive experiments on common benchmarks show that our proposed approach improves negation detection performance and generalizability over the strong baseline NegBERT (Khandelwal and Sawant, 2020).

## introduction negation is an important linguistic phenomenon that appears commonly in natural language but is underrepresented in common nlp benchmarks @xcite . furthermore, the checklist benchmark @xcite shows that most sentiment analyzers and machine comprehension models struggle with samples containing negation. negation is even more important in biomedical domain text, where patients are carefully defined as having/not having specific characteristics. even within the biomedical domain, there are many types of text such as clinical notes, lab reports, or research publications, each with particular characteristics in relation to the use of negation. a recent study on english texts found that negation detection models do not transfer well across domains, due to variations in expression of negation @xcite . it remains a challenge to solve negation in general, even with state-of-the-art nlp models. negation detection is typically defined as consisting of the two sub-tasks of: (1) cue detection, detecting the cue phrase that triggers the negation; and (2) scope resolution, determining the affected spans that are negated. there are three primary datasets that have been used to evaluate negation: in this work, we aim to extend the transfer learning capability of negbert @xcite through additional pre-training with task-related augmented training data, and a new masking objective. our contributions are: ## related work to date, negation detection has been heavily reliant on rule-based systems. @xcite proposed a simple system, negex, based on regular expressions to detect negation cues in a sentence given a concept of interest (the scope). negex re-mains the most popular approach to negation detection, especially in the clinical domain to determine the polarity of clinical concepts (e.g., as sourced from metamap @xcite ). further research has extended negex with syntactic information @xcite , and shown that rule-based systems can achieve relatively good performance for detecting negation, especially in the biomedical domain, but do not generalize well to other domains or datasets. to approach negation cue and negation scope detection with supervised machine learning, two classification tasks are defined: (1) finding negation tokens, and (2) classifying tokens as the first or last (or neither) token within the scope of negation. most work follows a common scheme in extracting various features from the sentence, and using a classifier to classify each token as the beginning, inside, or outside of a negation cue or scope span @xcite @xcite . recently, research has shifted to applying deep learning methods to the task. most approaches make use of rnn-based architectures to encode the input sentences, combined with a softmax layer for classification @xcite . despite the high performance on common benchmarks, results are biased by the fact that negation scope is often delimited by punctuation and other dataset artefacts @xcite . as such, they are potentially only learning domain-specific surface features rather than capturing the true semantics of negation. negbert applies a large pre-trained language model to the problem of negation detection, outperforming previous deep learning methods on negation detection, with especially high gains on scope resolution. ## method our proposed pre-training strategy consists of two main components: (1) negation-focused data collection in which we first collect relevant data that contains negation; and (2) negation-focused pretraining that makes use of the negation-focused data to emphasize negation instances, and adopts a novel negation-specific masking strategy. ## conclusion in this work, we propose a new negation-focused pre-training strategy to explicitly incorporate negation information into pre-trained language models. empirical results on common benchmarks show that the proposed strategy helps improve the performance of pre-trained language models on the negation detection task when evaluating on the same source dataset, as well as their transferability to target data in different domains. despite the gains over previous methods, the sub-optimal results on some benchmarks show that negation remains a big challenge in nlp.