Tuning Deep Active Learning for Semantic Role Labeling (2021)

Abstract: Active learning has been shown to reduce annotation requirements for numerous natural language processing tasks, including semantic role labeling (SRL). SRL involves labeling argument spans for potentially multiple predicates in a sentence, which makes it challenging to aggregate the numerous decisions into a single score for determining new instances to annotate. In this paper, we apply two ways of aggregating scores across multiple predicates in order to choose query sentences with two methods of estimating model certainty: using the neural network’s outputs and using dropout-based Bayesian Active Learning by Disagreement. We compare these methods with three passive baselines — random sentence selection, random whole-document selection, and selecting sentences with the most predicates — and analyse the effect these strategies have on the learning curve with respect to reducing the number of annotated sentences and predicates to achieve high performance.

## introduction the ability to identify the semantic elements of a sentence (who did what to whom, where and when) is crucial for machine understanding of natural language and downstream tasks such as information extraction @xcite and questionanswering systems @xcite . the process of automatically identifying and classifying the predicates in a sentence and the arguments that relate to them is called semantic role labeling @xcite . the current state-of-the-art semantic role labeling systems are based on supervised machine learning and rely on large corpora in order to achieve good performance. large corpora have been created for languages such as english @xcite , but such resources are lacking in most other languages. additionally, those corpora created may still not translate well to other in-language domains, due to sentence structure or domain-specific vocabulary. creation of additional annotated corpora requires a significant amount of time and often the hiring of domain experts, causing a bottleneck for developing advanced nlp tools for other languages and domains. active learning (al) focuses on choosing only the most informative and least repetitive instances to have annotated, thereby reducing the total needed annotation to train a supervised model, without sacrificing performance. this is done by iteratively re-training the model and assessing its confidence in its predictions in order to choose additional data for annotation that would have maximal impact on the learning rate. traditionally, practitioners use the model's probability distributions for the annotation candidates to quantify how informative a new training instance would be for the model. however, state-of-theart srl systems rely on deep learning, whose predictive probabilities are not a reliable metric of uncertainty. in lieu of this, @xcite found that we can estimate model confidence by calculating the rate of disagreement of multiple monte carlo draws from a stochastic model, accomplished by utilising dropout during forward passes. previous work @xcite has combined this finding with bayesian active learning by disagreement @xcite as a way of selecting informative instances for active learning for srl and other nlp tasks; hereafter referred to as do-bald. semantic role labeling for a single sentence is a complicated structural prediction, involving multiple predicates and varying spans. this complexity makes identifying the training examples with maximal impact more challenging. in this work, we compare two ways of aggregating confidence scores for individual predicates into a unified score to assess the usefulness of selecting a sentence for active learning. we test these strategies with two active learning approaches to calculating certainty for a predicate instance: the model's output probabilities and a granular do-bald selection method. additionally, we compare the benefits of these al approaches with three baselines: random sentence selection, random document selection, and selecting sentences with the most predicates. we will discuss the practical workflow of srl annotation and the way this must be considered in utilising active learning effectively to create new datasets. although the current standard data selection methodology for srl corpora, which typically involves selecting entire documents, leaves much room for improvement by even passive strategies, we will show that active learning can provide significant reductions in annotation of both number of sentences and number of predicates. we aim to provide this comparison within the broader context and understanding of srl annotation in practice. ## background active learning begins with the selection of a classifier, a small pool of labeled training data (also referred to as a seed set) for the classifier to initially be trained on, and a large amount of unlabeled data. al is an iterative process where the classifier is trained on the labeled data and then through some query selection strategy, an instance or instances are chosen from the unlabeled data for a human annotator to provide a label for. typically, they're chosen after the classifier attempts to predict labels for the unlabeled data and provides feedback about what instances may be the most informative. the newly annotated data is then added to the pool of labeled data that will be used to train the classifier on the next iteration. this iteration continues until some stopping criteria are met, such as the classifier's confidences about the remaining unlabeled data exceeding a certain threshold, or simply until funds or time are exhausted. proposition bank (propbank) @xcite is verb-oriented semantic representation. predicates in text are assigned a roleset id based on the sense of the word, such as play.01 (to play a game) or play.02 (to play a role). the roleset determines the permissible semantic roles, or arguments, for that predicate. the core arguments are given generalised numbered labels, arg0 roleset id: give.01 transfer arg0 giver arg1 thing given arg2 entity given to ## related work active learning has been utilised with success in numerous nlp tasks, such as named entity recognition @xcite , word sense disambiguation @xcite , and sentiment classification @xcite . in recent years, active learning has been applied to srl. since probabilities from offthe-shelf nn models may sometimes be inaccessible, @xcite proposed working around this by designing an additional neural model to learn a strategy of selecting queries. given an srl model's predictions, this query model classifies instances as requiring human annotation or not. their approach was a hybrid of active learning and selftraining. the self-training is enacted by accepting the srl model's predicted labels into the training pool for future iterations when the sentence was determined not to require human annotation. this approach requires 31.5% less annotated data to achieve comparable performance as training on the entirety of the conll-2009 dataset. @xcite compared data selection policies while simulating active learning for question-answer driven srl (qa-srl). qa-srl is a form of representing the meaning of a sentence using question-answer pairs. rather than annotating spans of text with argument names, such as propbank's arg0, annotators enumerate a list of questions relating to the actions in a sentence, such as who is performing an action and when is it happening, along with the corresponding answers from the original text. this representation provides similar coverage to propbank, but can also represent implicit arguments that aren't directly represented by the syntax. the process of identifying spans that are arguments of a predicate and the generation of questions based on the arguments were treated as independent tasks. to provide an approximate upper bound on the learning curve, they simulated active learning on the dataset, splitting the unlabeled candidates into k subsets, and selecting the subset that improved the model the most on the evaluation data. against this oracle policy, they compared the following selection strategies, sampling k random subsets to choose from: selecting a random subset, selecting the subset with the highest average token count among sentences, and selecting the subset that has the maximal average entropy over the model's predictions. the uncertainty strategy performed worse than random selection for argument span detection, and was not tested for question generation. selecting the sentences with high token counts tended to improve the f-score for argument span detection by 1-3% given an equal number of training instances (and attaining 60% on the full dataset), while being largely comparable to random selection for question generation. active learning for srl has also been applied in combination with multi-task learning @xcite , using a subset of propbank roles along with a new "greet" role. the authors compared single-and multi-task srl, both with and without active learning. under multi-task learning the model jointly learns to identify semantic roles as well as to classify tokens as entities such as "person" or "location". they introduced a set of semantic roles that accommodate conversational language and annotated a small corpus of indonesian chatbot data to provide training and testing data. by selecting sentences using model uncertainty in the single-task context, f-score was improved by less than 1% compared to randomly selecting the data. modern srl systems utilise deep learning, which poses a challenge to assessing the model's certainty in its predictions. the predictive probabilities in the output layer cannot be reliably interpreted as a measure of model certainty. @xcite proposed using dropout as a bayesian approximation for model certainty, estimating it using the variation in multiple forward passes. this dropout principle was tested on numerous nlp tasks by @xcite , including srl. for their srl experiments, they used a neural srl model based on the @xcite model, with modifications to the decoding method (instead using a crf decoder) and increasing the dropout rate from 0.2 to 0.25. in comparison to the baseline of random selection, they tested the classic uncertainty measure of using the output probabilities of the model, normalised for sentence length, with two bayesian active learning by disagreement methods for selecting additional instances: monte carlo dropout disagreement (do-bald) and bayes-by-backprop (bb-bald). the do-bald method applies dropout during multiple predictions of instances in the unlabeled pool and selects instances based on how many of those predictions disagree on the most common label of the entire sequence. this selection strategy is similar to the selection method we propose in this paper, but with several differences. the most significant difference is that the authors treat agreement between predictions as all-or-nothing, rather than allowing partial agreement based on arguments. they also are using a higher number of predictions (100 per sentence as opposed to 5 per predicate) to calculate disagreement between, which may be necessary in this allor-nothing approach. in contrast, we consider each predicate-argument label sequence independently. they tested their methods on both the conll-2005 and conll-2012 datasets, which use prop-bank annotation. while the bayesian methods were similar to the standard uncertainty selection method in the case of srl, these methods resulted in approximately 2-3% increase for f-score compared to random selection when training on the same number of tokens. these results were much more modest than results for other tasks such as ner. ## data we used two independent datasets for our experiments: the english section of ontonotes (version 5.0) @xcite with the latest frame updates @xcite and the colon cancer portion of thyme @xcite . ontonotes 5.0 consists of 1.5 million words across multiple genres. the majority of this data is sourced from news, but it also includes telephone conversations, text from the bible, and web data. thyme is comprised of clinical notes and pathology reports of colon and brain cancer patients. for our experiments, we used only the colon cancer portion. the data is split into training, validation, and test subsets. we simulated active learning on the training subset of each corpus, dividing it into an initial seed set and a set of sentences to select from. the initial seed sets for sentence-based experiments were 200 randomly chosen sentences. for the wholedocument baseline, the seed set is comprised either of documents from multiple genres, totalling 200 sentences, in the case of ontonotes; or a single patient (consisting of two clinical notes and one pathology report, totalling 195 sentences) in the case of the thyme corpus. in both cases, we utilised validation data to determine early stopping. due to the excessive computational time required to predict the standard validation sets for these corpora for every epoch for every iteration, as well as the fact that a realworld scenario would be unlikely to have such a disproportionally large validation set to perform active learning, we selected a subset of the validation data for use. in the experiments involving selecting individual sentences, we used the same randomly chosen 250 sentences. in the case of the baselines of choosing random documents, we used validation datasets approximating 250 sentences, comprised of whole documents. evaluation was performed on the standard test subset for each respective corpus. ## model we used allennlp's @xcite implementation of a state-of-the-art bert-based model @xcite . our training procedure for this model used 25 epochs or stopped early with a patience of 5. trained under the same experimental configuration on the full training subsets, this model achieves an f-score of 83.82 and 83.48 on the ontonotes and thyme datasets respectively. after training on the initial seed dataset, each iteration of active learning selected batches of 100 sentences re-trained from scratch. in the case of the whole-document baseline, for the creation of each batch, we selected random documents until the number of sentences selected met or exceeded 100. ## results out results are reported as a learning curve across number of sentences (figures 1 , 3 ) and predicates (figures 2 , 4 ) present in the training pool after each iteration. selected f-scores for the methods are reported according to number of sentences ( ## conclusions between the two proposed methods of aggregating predicate-argument structure scores into a single value to represent a sentence, averaging across them (ap) or only considering the weakest predicate (lsp), our results show the latter to be substantially better. both selecting sentences for the most predicates and selecting sentences with the predicate with the lowest do-bald agreement offer a significant 53%-60% decrease in the number of sentences required to train the model to a viable performance level. these findings are consistent for both the broad, general ontonotes corpus and the niche colon cancer clinical note domain of the thyme corpus. we assessed the performance of these selection strategies in terms of reducing both number of sentences and number of predicates annotated. typically, the srl annotation process of a large annotation project benefits most from a reduction of predicates, due to presenting annotators with batches of a specific predicate to annotate, thereby reducing the cognitive load of switching between different predicate frames. but in the case of projects attempting to develop new corpora with significant budget constraints that would most benefit from an active learning approach, the piecemeal nature of each annotation iteration makes this approach less viable and likely necessitates presenting annotators with the data sentence-by-sentence. in this case, reducing the number of sentences will have a more substantial impact than reducing the number of predicates. while both do-bald lsp and the simpler strategy of selecting sentences with high predicate density provide significant reduction in sentence annotation, only do-bald lsp simultaneously reduced predicate annotation as well.