Leveraging Active Learning to Minimise SRL Annotation Across Corpora (2023)

Abstract: In this paper we investigate the application of active learning to semantic role labeling (SRL) using Bayesian Active Learning by Disagreement (BALD). Our new predicate-focused selection method quickly improves efficiency on three different specialised domain corpora. This is encouraging news for researchers wanting to port SRL to domain specific applications. Interestingly, with the large and diverse \textit{OntoNotes} corpus, the sentence selection approach, that collects a larger number of predicates, taking more time to annotate, fares better than the predicate approach. In this paper, we analyze both the selections made by our two selections methods for the various domains and the differences between these corpora in detail.

## introduction the majority of natural language processing (nlp) systems are reliant on manual annotations to train supervised models. although semi-supervised and unsupervised methods are frequently employed to help adapt models to new domains, human annotation remains the gold standard for quality input. due to the high cost of human annotation, especially if the task requires expert knowledge, and the time-intensive process, this can be daunting for many applications. active learning (al) has been shown to reduce annotation requirements for a variety of nlp tasks @xcite by selecting more informative instances that are most likely to fill gaps in the model's knowledge. in this paper, we focus specifically on the nlp task of semantic role labeling (srl). the goal of srl is to identify and label the who, what, and when of predicates in a sentence. this information can be used as features in downstream applications such as information extraction @xcite , machine translation @xcite , and features prominently in abstract mean-ing representation (amr) applications @xcite . in this paper, we propose a new selection strategy tuned for srl that is based off of previous methods of using model dropout to approximate a gaussian process @xcite . we compare this to prior work on al selection for srl @xcite on four corpora in a variety of domains: ecology, earthquakes, clinical notes, and the large multi-genre ontonotes corpus. since sentences in most domains typically contain multiple predicates, there are often redundancies in choosing predicates to annotate on the sentence level. although a sentence may contain a particularly informative predicate, annotating highfrequency verbs such as "be" that co-occur in the sentence may not be beneficial. we instead use a method to select specific predicate-argument structures and compare the impact on performance as compared to selecting whole sentences instead. this method is a natural extension that allows us to even better leverage the focused annotation that active learning offers by using a more granular approach. while we find consistent early benefit in the more domain-specific corpora, this finergrained approach proves to be slower for the more diverse ontonotes. we also explore the statistical differences between these corpora, the selections our algorithm makes, and test a variety of selection batch sizes in order to shed light on expectations for use in future domains. ## background proposition bank (propbank) @xcite is verb-oriented semantic representation consisting of a predicate and its arguments. predicates are given a roleset id, which distinguishes the sense of the word, such as play.01 (to play a game) or play.02 (to play a role). each roleset has its own list of permissible semantic roles, or arguments, for play.01 play a game arg0 player arg1 game arg2 equipment arg3 opponent active learning is an iterative process by which data is selected for annotation using the model's own confidence. after initially training the model on a small amount of annotated data (referred to as the seed set), each unlabeled instance is predicted by the model and those that the model is least certain about (conventionally, by the model's outputs) are presumed to be more informative to learn from than those that the model has high certainty about. the uncertain instances can then be manually annotated and added into the training pool for the next training iteration. this process can repeast until either the performance is no longer significantly increasing or time/budget has been exhausted. previous work has shown that neural networks tend to be overconfident in their predictions, owing to their nonlinearity and tendency to overfit @xcite . therefore, more recent work @xcite ) @xcite has explored using bayesian active learning by disagreement @xcite (bald) rather than model outputs as a way of selecting informative instances for active learning for srl and other nlp tasks. by using dropout during prediction, multiple forward passes can be treated as monte carlo draws from a stochastic model. the instances that have more disagreement amongst the predictions are considered to be more informative for the model to learn from. @xcite applied bald to srl by calculating disagreement among five forward passes of the trained model using dropout, break-ing down agreement scores by individual argument labels. we describe this in more detail in section 4.1. the active learner used two alternative methods to select sentences: 1) using the average disagreement score amongst all predicates in the sentence (bald-ap) or 2) by choosing the sentences that contain the single lowest scoring predicate (bald-lsp). since bald-lsp performed best, we compare our predicate-focused bald strategy against this method on both corpora used previously (ontonotes and thyme colon) as well as two new geoscience corpora from the clearearth project @xcite . ## data we aim to provide a demonstration of active learning for srl across a variety of domains and sublanguages @xcite . some knowledge domains exhibit narrow lexical, syntactic, and semantic structures that distinguish them from more general-purpose domains. this can lower performance dramatically when testing with an off-theshelf general purpose model. special techniques that take these domain specific-structures into account are needed for adapting nlp tools to these domains, as illustrated below. thyme colon is comprised of unstructured clinical notes relating to treatment of colon cancer @xcite . this corpus contains specialised medical vocabulary for a narrow domain and a large number of formulaic sentences, such as the following example: pathology demonstrated a tubular adenoma with moderate dysplasia. this contains medical terminology (tubular adenoma, dysplasia) as well as a non-standard use of demonstrate, which includes the shortening of the pathology report to simply pathology. this particular framing re-occurs frequently in thyme colon, sometimes with show or reveal instead, and occasionally including the word report as in pathology report. we also used two distinct geoscience domains from the clearearth project @xcite : the ways that plates interact depend on their relative motion and whether oceanic or continental crust is at the edge of the lithospheric plate. ontonotes 5.0 @xcite spans multiple genres, largely consisting of news sources, but also including telephone conversations, text from the new testament, weblogs, and usenet. this popular corpus serves as a broad purpose corpus for us, as opposed to the other more specialised domains. we use a version of ontonotes that does not include files that had no manual propbank annotation performed. there still exist sentences within this version of the data that had only partial annotation, but we consider this to have a relatively small impact on performance. evaluation was performed on the standard test subset for each respective corpus. ## methods we simulated active learning using allennlp's @xcite implementation of a stateof-the-art bert-based srl model @xcite . in order to simulate active learning on each of these corpora, we partitioned the training subset of each corpus into 200 random sentences for seeding the learner, with the remainder used as the initial "unlabeled" pool for selection. the initial 200 seed sentences were the same across the three selection methods tested for each respective corpus. after initially training on the seed set, we then select a batch of either 100 predicates or a number of sentences containing approximately 100 predicates to add to the training pool using the bald predicates or bald sentences strategy described below in section 4.1 or by choosing 100 random predicates to simulate a passive learning approach. results are reported on the test subset of the respective corpora and the model was retrained with the extended training pool. we continue iterations of selection and re-training until either all the data has been selected and moved into the training pool, or the experiment performances have sufficiently plateaued. our training procedure for this model used 25 epochs or stopped early with a patience of 5 based on the validation data for the relevant corpus. ## results we present the learning curves of the different selection methods for the four corpora are presented 401 in figures 1 , 2 , 3, and 4. natural variability in training the model produces some amount of noise, most prominently during the early iterations. in order to improve readability of these learning curves, we applied a savitzky-golay filter using a window of 15 data points and using a cubic polynomial. we see consistent benefits of the bald predi-cates method at different points depending on the corpus. for colon, ecology, and earthquakes we begin to see consistent improvement for the bald predicates method over the other methods by approximately 1,500-2,000 predicates. on the other hand, for ontonotes, it only catches up to random selection around 4,500 predicates and begins to improve over it around 7,000 predicates. for this corpus, bald sentences performs better. ## analysis of selections in order to better understand the differences between the selection processes used and their variance across datasets, we examine the selections within each batch. ## corpus analysis although the new predicate selection method offers immediate benefit over bald sentences for the three sublanguage corpora, this is inconsistent with the result on ontonotes, where selecting bald sen-tences is more advantageous until about 7k predicates. in order to better understand the possible reasons for this, we compare the make-up and distribution of the corpora. these statistics are presented in we use propbank roleset id's as our measure of polysemy, since we have gold standard annotation for them in all 4 corpora. note that propbank sense distinctions are fairly coarse-grained and were generally only created when there were differences between senses with respect to the semantic roles. verbnet @xcite , framenet @xcite and wordnet @xcite would all give much higher polysemy counts. the largest and most diverse corpus in our experiments is ontonotes, although we find that in terms of ratio of total tokens to predicates, unique rolesets, and unique tokens, ontonotes is statistically more similar to the thyme colon cancer corpus than to either of the clearearth corpora. ontonotes and colon contain approximately one unique roleset per 376-403 tokens, whereas earthquakes and ecology contain one per 39 and 60 tokens, respectively. since ontonotes covers a wider diversity of text types, it's unsurprising that it contains a much more diverse set of senses compared to the other corpora. while a lemma like "take" shows up with 25 different senses in ontonotes, it only shows up in 8 senses in colon. for ontonotes, only 30% of predicate occurrences are monosemous within the context of the corpus, whereas this figure is between 54%-61% for the other three corpora. 6% of the unique predicate lemmas within ontonotes are seen in 3 or more rolesets, while this is true of only 2% of the set of lemmas in each of the other corpora. we believe this polysemy factor may contribute to the predicate selection method being disproportionately slower to improve the learning curve on ontonotes compared to the more focused domain corpora. bald predicates may be disadvantaged by more frequently choosing these rare senses even though they make up proportionally less of the training data and provide less value in terms of performance, but further investigation is needed. ## batch sizes each iteration of active learning includes selecting an arbitrary number of instances to query. the number may be static, or dynamic with larger batches being selected in the early training process and smaller batches later on. to maximally benefit from the model's feedback, in an ideal setup, each iteration would query for only one new instance, thereby minimizing the likelihood of selecting a batch of sentences with redundant information @xcite . unfortunately, this leads to the process of active learning being significantly slower due to needing to re-train a model more often. additionally, annotating a sentence at a time with long breaks in between may cost additional time on the part of the annotator due to mental context-switching and needing to load up appropriate software and resources. it would be more efficient for them to be able to annotate numerous examples in a row. our previous experiments testing the bald pred-icates method show positive results when selecting 100 predicates in a batch. this small batch size requires about 60 iterations before the learning curve plateaus for the colon corpus. we examine the effect of larger batches on the learning curves for the thyme colon and the two clearearth corpora. ## conclusion and future work in this paper, we've demonstrated that active learning can reduce annotation requirements for semantic role labeling across multiple domains by employing bayesian active learning by disagreement and using dropout to provide variability in predictions from the model. these predictions can be used to estimate the model's confidence in its predictions and select informative training instances to annotate.