Multitask Learning of Negation and Speculation using Transformers (2020)

Abstract: Detecting negation and speculation in language has been a task of considerable interest to the biomedical community, as it is a key component of Information Extraction systems from Biomedical documents. Prior work has individually addressed Negation Detection and Speculation Detection, and both have been addressed in the same way, using 2 stage pipelined approach: Cue Detection followed by Scope Resolution. In this paper, we propose Multitask learning approaches over 2 sets of tasks: Negation Cue Detection & Speculation Cue Detection, and Negation Scope Resolution & Speculation Scope Resolution. We utilise transformer-based architectures like BERT, XLNet and RoBERTa as our core model architecture, and finetune these using the Multitask learning approaches. We show that this Multitask Learning approach outperforms the single task learning approach, and report new state-of-the-art results on Negation and Speculation Scope Resolution on the BioScope Corpus and the SFU Review Corpus.

## introduction detection of linguistic phenomena like negation and speculation are key components of biomedical information retrieval systems, as they significantly alter the meaning of a sentence. while detecting these are also useful in sentiment analysis systems, and systems used to determine the veracity of information, their primary use is in biomedical systems. thus, these tasks have attracted significant interest from researchers over the years, and due to the similarity between these tasks, similar approaches have been used to address them, and parallel corpora containing annotations for both negation and speculation have also been created, including: in the example given above, not is the negation cue, as it expresses the negation in the sentence. similarly, perhaps is the speculation cue. scope resolution involves finding the word(s) that were affected by the cue word of the linguistic phenomena being considered. in the example above, the underlined words outline the scope for each cue. specifically, for the negation cue not, the word present was negatively affected by it. similarly, for the speculation cue perhaps, the words indicating an immuno deficiency were affected by it, indicating that these words have an associated uncertainty. the approaches addressing these tasks have varied significantly over the years, with recent work focusing on using transformer-based architectures to perform transfer learning, and have given the best results to date on scope resolution. on cue detection, they yield the best performance among neural models, but due to the small dataset sizes, the best performance is still given by rule-based heuristic approaches. despite the similarity among the subtasks, prior systems have looked at these tasks independently. we believe that a system can improve performance on both tasks by learning from both simultaneously, due to the similarity, which is what multitask learning is about. multitask learning involves jointly training the same architecture to perform multiple tasks. it relies on the concept of using shared knowledge between both tasks, which is what the model is forced to learn to perform well at all tasks, eventually leading to better performance on all tasks. for neural models, this is especially useful, as the lower layers can share the same input representation, thus getting more data to learn better lower level features from the input, and a task specific layer final layer can learn the task specific features. in this paper, inspired by the success of transformers, we propose a method to perform multitask learning of negation and speculation using transformer-based architectures. we explore a few design choices, and analyse the impact of these design choices. we show that our approach provides significant benefits over the normal independently trained version. we also make all our code publicly available 1 . this paper is structured as follows: section 2 contains a brief literature review, section 3 describes the methodology in detail, section 4 talks about the experimentation details, section 5 contains the results and their analysis, and section 6 contains the conclusion and future scope. ## literature review over the years, methods addressing these subtasks have ranged from simple whitelists based on frequency (rule-based), to traditional machine learning algorithms like svms, neural models like bil-stms and transformer based models. @xcite provide an extensive literature review of the methods for negation cue detection and scope resolution. for speculation cue detection, most methods used were similar to the methods used for negation cue detection. below, we summarise a few papers that addressed speculation cue detection and scope resolution. @xcite used a support vector machine (svm) with a linear kernel to detect speculation cues. once the speculation cues were identified, the parts-of-speech tags and the syntactic structure were used for scope resolution. @xcite used the igtree classifier with the help of gain ratio (timbl implementation) to classify the cues. for scope resolution, three classifiers were used to classify if a token was the first token in the scope se-1 adityak6798.github.io quence (f-scope), the last (l-scope), or neither. these three classifiers were memory-based learning (as implemented in timbl), svm and conditional random field (crf). a fourth classifier, the metalearner, used the output of the three classifiers to predict the scope classes. @xcite used a maximum entropy classification approach to find the speculation cue. for scope resolution, they used a rule-based approach. the rules operated on the dependency structure of the parser (maltparser and xle). @xcite used linguistic knowledge to detect speculation cues. this was achieved by using a semi-automatic lexical acquisition strategy as well as by using a dictionary of weighted speculation cues. in a follow-up paper @xcite , they improved on their previous work with the help of vagueness quantifiers and syntactic dependency relations. cues were detected with rules that operate on lexical information and syntactic information obtained from the stanford lexical parser. the scopes of the cues were detected with the help of the stanford lexical parser and dependency-based heuristics. ## methodology similar to @xcite ) and (britto and khandelwal, 2020), we use the transformer model (bert/xlnet/roberta) with a classification head as our base model. to jointly train the model, we propose the following additions to the model. ## experimentation details we perform experimentation on the following datasets: we believe that by training on multiple datasets, the overfitting of the models can reduce, as the datasets are fairly small in size @xcite , despite the different domains of the datasets (bio-scope corpora is from the biomedical domain, and sfu review corpus contains general online review text). hence, we also experiment with training the models on multiple datasets, and testing on the individual datasets. we use a 70-15-15 train-dev-test split. the results are reported as an average of 5 runs for training on a single dataset and an average of 3 runs for training on a combination of multiple datasets. we report the macro f1 average (token-level) score for both cue detection and scope resolution. we perform an early stopping (with a patience of 6) on the validation f1 score. since we jointly train 2 tasks, we experiment with these 2 ways to perform early stopping: we train the models using gpus available via google colaboratory. the code is publicly available. ## results and analysis to perform a better comparison of independently trained models on multiple datasets, we train bert, xlnet and roberta on bf+ba, bf+sfu, ba+sfu and bf+ba+sfu for negation cue detection and negation scope resolution. we train the model as per the paper by britto and khandelwal (2020), and average the results of 3 runs. the results are shown in tables 1 and 2 . an analysis of the results shown below is done in section 5.4. ## conclusion in this paper, we explored the realm of multitask training to jointly train the same model to perform both negation and speculation detection. we experimented with transformer-based architectures (bert, xlnet and roberta), and proposed schemes to jointly train the cue detection model for both negation and speculation, and the scope resolution model for both negation and speculation. our approach yielded improvements over the independently trained versions of the same architectures, and we reported new state-of-the-art results for both negation and speculation scope resolution on the bioscope corpus and the sfu review corpus. we also evaluated the different design choices that were involved, and observed that the combined early stopping variant gave the best overall performance. the future scope of this work would be to look at using this scheme to jointly train a model for more such tasks, like ner and sentiment analysis, along with negation and speculation detection.