A Neural Model for Aggregating Coreference Annotation in Crowdsourcing (2020)

Abstract: Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. Collecting sufficient labelled data from expert annotators to train a high-performance coreference resolution system is time-consuming and expensive. Crowdsourcing makes it possible to obtain the required amounts of data rapidly and cost-effectively. However, crowd-sourced labels can be noisy. To ensure high-quality data, it is crucial to infer the correct labels by aggregating the noisy labels. In this paper, we split the aggregation into two subtasks, i.e, mention classification and coreference chain inference. Firstly, we predict the general class of each mention using an autoencoder, which incorporates contextual information about each mention, while at the same time taking into account the mention’s annotation complexity and annotators’ reliability at different levels. Secondly, to determine the coreference chain of each mention, we use weighted voting which takes into account the learned reliability in the first subtask. Experimental results demonstrate the effectiveness of our method in predicting the correct labels. We also illustrate our model’s interpretability through a comprehensive analysis of experimental results.

## introduction coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. however, it is time-consuming and expensive to collect the large amounts of data from expert annotators that are required to train high-performance coreference resolution systems. a rapid and costeffective alternative is to obtain labels through crowdsourcing @xcite . however, crowdsourced labels are often noisy. in the example in figure 1 , the mention it actually refers to the super lamb banana. however, different crowd annotators produced conflicting labels for this mention. we can also observe that the coreference annotation is more complex than classification and sequence labels. annotators have to determine an appropriate referent mention for some mentions. because the performance of supervised learning models is highly dependent on the quality of training data, the aggregation of these noisy labels, (i.e., the process of determining the label that is most likely to be correct) is important to obtain a high-quality training corpus. although label aggregation is a well-studied topic, most existing studies of natural language labelling tasks have only focused on aggregating classification or sequence labels. to the best of our knowledge, there is only one previous study @xcite that has investigated how to aggregate crowd-sourced coreference labels. in this paper, we propose a 2-step framework in which the aggregation task is broken down into two subtasks, i.e., mention classification and coreference chain inference. in the mention classification subtask, our model predicts the general category of a mention as shown in ## related work there have been many released coreference corpora in which the annotations were collected from a small number of in-house annotators such as experts @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . these annotators were assumed to be reliable. @xcite and @xcite attempted to collect coreference annotations from non-expert crowd annotators. even though crowd aggregation has been studied for many years, most existing studies have focused on aggregating classification labels @xcite @xcite @xcite @xcite @xcite @xcite or sequence labels @xcite @xcite @xcite @xcite . note that @xcite and @xcite also included contextual information. @xcite incorporated a classifier into their bayesian model. the classifier took an instance's representation as its input and then predicted an answer. the unsupervised latent dirichlet allocation topic model @xcite which can capture topics of words and documents was extended by @xcite to be able to handle crowdsourced noisy labels. however, these models can not be applied to the coreference annotation in a straightforward manner, because coreference labels are more complex and they are very different from to overcome the lack of a suitable aggregation method for coreference annotations, paun et al. (2018) proposed the mention-pair annotation model, which to the best of our knowledge, is the only study that attempts to address this challenge. they defined a graphical model and introduced a true label indicator for @xmath0, referent @xmath1, indicating whether or not the pair is correct. however, this model does not include contextual information, which may support the prediction of the correct labels, because the meaning of mentions usually depends on the context in which they occur. ## model we break down the aggregation of crowd-sourced coreference labels into two steps: mention classification and coreference chain inference, as illustrated in figure 2 . below, we describe our method in more detail. all the biases in linear layer parameters are omitted for simplification. figure 3 : per-category reliability in the method. each annotator's label is encoded as a vector which is described in section 3.1.1. ## results from ## conclusion and future work we proposed a two-step framework for aggregating crowd-sourced coreference labels. in the mention classification subtask, the encoder classifies each mention as belonging to one of the four general categories, i.e., dn, do, nr or pr. this encoder incorporates mention context, instance complexity and the annotator reliability at different levels (i.e., overall, per-category and per-instance). in the coreference chain inference subtask, we use the learned reliability to infer the coreference chains. experimental results demonstrate the effectiveness of our model. furthermore, our comprehensive analysis shows that the learned complexity and reliability are explainable, thus helping to explain how our model infers the correct label for each instance. lastly, an error analysis was carried out to understand the incorrect predictions. as future work, we will explore the challenges of solving other complex annotation tasks and how our model can be used and adapted for them.