Enhanced Labelling in Active Learning for Coreference Resolution (2020)

Abstract: In this paper we describe our attempt to increase the amount of information that can be retrieved through active learning sessions compared to previous approaches. We optimise the annotator’s labelling process using active learning in the context of coreference resolution. Using simulated active learning experiments, we suggest three adjustments to ensure the labelling time is spent as efficiently as possible. All three adjustments provide more information to the machine learner than the baseline, though a large impact on the F1 score over time is not observed. Compared to previous models, we report a marginal F1 improvement on the final coreference models trained using for two out of the three approaches tested when applied to the English OntoNotes 2012 Coreference Resolution data. Our best-performing model achieves 58.01 F1, an increase of 0.93 F1 over the baseline model.

## introduction coreference resolution (cr) is the task of resolving which noun phrases (np) in a text are referring to the same entity. it is related to entity linking, but does not involve an external knowledge base. it is an important task in information extraction, as a step in structuring the unstructured information in natural language. cr has traditionally been a difficult problem, as it is hard to accurately predict coreference links without extensive real-world knowledge. figure 1 : different types of coreference resolution. an anaphoric pair of noun phrases is marked in green, and a cataphoric pair is marked in yellow. from "t2: trainspotting" @xcite an example of different levels of cr is shown in figure 1 . the mentions "us" and "i" are both singletons, and are not coreferring with anything in this text. the noun phrase "she" is anaphoric (where the pronoun points backwards to its antecedent) with "the queen". the pronoun "you" in "you've" is coreferring with "mr begbie", but the pronoun is pointing forward to its coreferent, this type of coreference is cataphoric coreference. ## methodology the experiments reported in this paper investigate a set of different methods for conducting manual annotation during an active learning scenario. ## evaluation we compare the baseline discrete labelling system versus enhanced labelling using the standard english conll-2012 coreference resolution dataset @xcite . following both @xcite and @xcite , user labelling is simulated from the gold standard labels in the conll dataset. ## analysis and discussion the timed annotation exercise show that the cost of annotating all the labels of an entity in a text is low when the annotator has already read the text to make a judgement on the initial coreference pair. the results also show that there might be a cut tail distribution of annotation times. the majority of the multi-response annotations were faster than the initial and the single-response follow-up question responses. on average it took our annotators longer time for the initial question in our implementation of the same timed annotation exercise as in @xcite , but shorter time for the follow-up question. people working in nlp are likely to be more experienced with seeing text containing bracketed annotation. it is possible that our set of annotators were slower at responding for the initial question because of the lack of experience in nlp. one reason the average time for answering the follow-up question was lower in our setup might be that the annotators were allowed to label any instance of the occurrence, not just the first. particularly for longer texts it might be faster to label an occurrence closer to the proform entity than the first occurrence. from figure 3 we can see that the labelling approach in experiment 1 returns more labels per query than the baseline approach, through the al sessions. the same is true for experiment 4 and 2 respectively. this indicates that cataphoric occurrences contain unused information, which should be used for training. the sudden jump in successful queries in al sessions 6 and 7 for experiment 2 and 4 can partly be ascribed to an increase in document length in those sessions, even though the graph is normalised to document length. this might mean that models trained on datasets with longer documents are able to benefit more from the improved label retrieval rate. even with 10% of the labels chosen at random the combined approach retrieved more successful label pairs than the baseline system, but the final f1 score was somewhat lower. this lower score f1 was expected, as the erroneous labelling would add confusion to the model. care should therefore be taken when designing a labelling system to ensure that errors are minimised. the small improvement in the validation f1 score shown in figure 2 indicates that the added labels under the current system do not translate into having an impact on how fast high accuracy is achieved. despite this, the final f1 score on the separate test data is marginally higher for experiment 4 than the baseline experiment. this lack of impact could have several causes. as the machine learning algorithm is the same as in the baseline system, it might not be best suited to make use of the extra available information. in addition, the ontonotes dataset does not inherently support cataphoric linking of entities, so a dataset which does contain inherent cataphoric links might also contribute towards making use of the extracted data more efficiently. the negative results for experiment 3 can have multiple causes. one of these is that the algorithm for selecting replacement proform spans was purposefully conservative in choosing the closest span. this was to retain ecological validity in the annotation simulation, as an annotator would look close to the span to determine whether the error was a boundary error. ## conclusion and future research the contribution of the research in this article is the improved techniques for extracting more information from user labelling. we have seen that allowing annotators to leverage cataphoric information, especially in combination with annotating several occurrences per query, can contribute to optimising the time spent by annotators hand labelling a dataset. even though the machine learning models did not perform markedly better earlier in the training process, the amount of disregarded queries dropped by a noticeable amount just by adding cataphoric labels. we have also seen that the amount of successful label pairs per query is over 1 for the approaches allowing multiple responses. this means that it is possible to extract much more information than with previous approaches. our timed annotation exercise indicate that labelling several occurrences of an entity in the same query is faster than answering multiple queries with only one set of labels. it would be interesting to investigate whether choosing labels closer or further from the proform label would have an impact on the learning. these findings are interesting for the real world application of coreference resolution systems, particularly for long form documents, such as in the legal sector, where there is a lot more information to leverage than in short form documents. a future project would look into making changes to the machine learning model for more effective use of the new data. future research would also look into testing which interface design would best aid the human annotator in the labelling process, especially for long form documents.