Active Learning for Coreference Resolution using Discrete Annotation (2020)

Abstract: We improve upon pairwise annotation for active learning in coreference resolution, by asking annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. This simple modification, when combined with a novel mention clustering algorithm for selecting which examples to label, is much more efficient in terms of the performance obtained per annotation budget. In experiments with existing benchmark coreference datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour. Future work can use our annotation protocol to effectively develop coreference models for new domains. Our code is publicly available.

## introduction coreference resolution is the task of resolving anaphoric expressions to their antecedents (see figure 1 ). it is often required in downstream applications such as question answering @xcite or machine translation @xcite . exhaustively annotating coreference is an expensive process as it requires tracking coreference chains across long passages of text. in news stories, for example, important entities may be referenced many paragraphs after their introduction. active learning is a technique which aims to reduce costs by annotating samples which will be most beneficial for the learning process, rather than fully labeling a large fixed training set. active learning consists of two components: (1) a taskspecific learning algorithm, and (2) an iterative sample selection algorithm, which examines the performance of the model trained at the previous iteration and selects samples to add to the annotated figure 1 : discrete annotation. the annotator is shown the document, a span (yellow), and the span's predicted antecedent (blue). in case the answer to the coreference question is negative (i.e., the spans are not coreferring), we present a follow-up question ("what is the first appearance of the entity?"), providing additional cost-effective signal. our annotation interface can be seen in figure 5 in the appendix. in this paper, we propose two improvements to active learning for coreference resolution. first, we introduce the notion of discrete annotation (section 3), which augments pairwise annotation by introducing a simple additional question: if the user deems the two mentions non-coreferring, they are asked to mark the first occurrence of one of the mentions (see second question in figure 1 ). we show that this simple addition has several positive implications. the feedback is relatively easy for annotators to give, and provides meaningful signal which dramatically reduces the number of annotations needed to fully label a document. second, we introduce mention clustering (section 4). when selecting the next mention to label, we take into account aggregate model predictions for all antecedents which belong to the same cluster. this avoids repeated labeling that would come with separately verifying every mention pair within the same cluster, as done in previous methods. we conduct experiments across several sample selection algorithms using existing gold data for user labels and show that both of our contributions significantly improve performance on the conll-2012 dataset @xcite . overall, our active learning method presents a superior alternative to pairwise annotation for coreference resolution, achieving better performing models for a given annotation budget. ## background our work relies on two main components: a coreference resolution model and a sample selection algorithm. ## discrete annotation in discrete annotation, as exemplified in figure 1 , we present the annotator with a document where the least certain span i ("po-po", in the example) and i's model-predicted antecedent, a(i) ("locals"), are highlighted. similarly to pairwise annotation, annotators are first asked whether i and a(i) are coreferent. if they answer positively, we move on to the next sample. otherwise, we deviate from pairwise sampling and ask the annotator to mark the antecedent for i ("a volcano in mexico") as the follow-up question. 2 the annotator can abstain from answering the follow-up question in case i is not a valid mention or if it does not have an antecedent in the document. see figure 5 in the appendix for more example annotations. in section 5, we show that discrete annotation is superior to the classic pairwise annotation in several aspects. first, it makes better use of human annotation time, as often an annotator needs to resolve the antecedent of the presented mention to answer the first question. for example, identifying that "po-po" refers to the volcano, and not the locals. second, we find that discrete annotation is a better fit for mention ranking models @xcite , which assign the most-likely antecedent to each mention, just as an annotator does in discrete annotation. ## mention clustering we experiment with three selection techniques by applying popular active learning selectors like entropy or query-by-committee @xcite to clusters of spans. because our model outputs antecedent probabilities and predictions, we would like to aggregate these outputs, such that we have only one probability per mention cluster rather than one per antecedent. we motivate this with an example: suppose span i's top two most likely antecedents are y 1 and y 2 . in scenario 1, y 1 and y 2 are predicted to be clustered together, and in scenario 2, they are predicted to be clustered apart. span i should have a "higher certainty" in scenario 1 (and thus be less likely to be picked by active learning), because its two most likely antecedents both imply the same clustering, whereas in scenario 2, picking y 1 vs. y 2 results in a different downstream clustering. thus, rather than simply using the raw probability i refers to a particular antecedents, we use the probability i belongs to a certain cluster. this implies modelling y 1 and y 2 "jointly" in scenario 1, and separately in scenario 2. formally, we compute the probability that a span i belongs in a cluster c by summing p (ant(i) = y) for all y that belong in some cluster c, since i having an antecedent in a cluster necessarily also implies i is also in that cluster. this allows us to convert the predicted antecedent probabilities to in-cluster probabilities: similarly, for query-by-committee, we aggregate predictions such that we have one vote per cluster rather than one vote per antecedent: the cluster information (@xmath0. antecedents which were not predicted to be in clusters are treated as singleton clusters. additionally, to respect user annotations during the selection process, we must keep track of all prior annotations. to do this, we use the concept of must-link (ml; if two mentions are judged coreferent) and cannot-link (cl; if two mentions are judged non-coreferent) relations between mentions introduced by @xcite , and adapt it for our purposes. specifically, in our discrete setting, we build the links as follows: if the user deems the pair coreferent, it is added to ml. otherwise, it is added to cl, while the user-corrected pair (from the second question) is always added to ml. in addition, we use these links to guide how we select for the next mention to query. for example, if a cl relation exists between spans m 1 and m 2 , we will be less likely to query for m 1 , since we are slightly more certain on what m 1 's antecedent should be (not m 2 ). formally, we revise probabilities and votes p (@xmath1, which affects the selector uncertainty scores. 3 finally, following @xcite , we impose transitivity constraints, which allow us to model links beyond what has been explicitly 3 see section a.2 in the appendix for more details. however, recomputing these closures after each active learning iteration can be extremely inefficient. instead, we build up the closure incrementally by adding only the minimum number of necessary links to maintain the closure every time a new link is added. we experiment with the following clustered selection techniques: clustered entropy we compute entropy over cluster probabilities and select the mention with the highest clustered entropy: where p (@xmath2. ## analysis a major reason discrete annotation outperforms the pairwise baseline is that the number of pairwise annotations needed to fully label a document is much larger than the number of discrete annotations. in an average development document with 201 candidates per mention, the number of pairwise queries needed to fully label a document is 15, 050, while the maximum number of discrete queries is only 201 (i.e., asking for the antecedent of every mention). thus, the average document can be fully annotated via discrete annotation in only 2.6% of the time it takes to fully label it with pairwise annotation, suggesting that our framework is also a viable exhaustive annotation scheme. further analysis shows that the improvement in discrete selection stems in part from better use of annotation time for mention detection accuracy (figure 3 ) and pronoun resolution (figure 4 ), in which we measure performance only on clusters with pronouns, as identified automatically by the spacy tagger @xcite . finally, ## discussion and conclusion we presented discrete annotation, an attractive alternative to pairwise annotation in active learning of coreference resolution in low-resource domains. by adding a simple question to the annotation interface, we obtained significantly better models per human-annotation hour. in addition, we introduced a clustering technique which further optimizes sample selection during the annotation process. more broadly, our work suggests that improvements in annotation interfaces can elicit responses which are more efficient in terms of the obtained performance versus the invested annotation time.