Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and W ikipedia Texts (2023)

Abstract: Although several datasets annotated for anaphoric reference / coreference exist, even the largest such datasets have limitations in term of size, range of domains, coverage of anaphoric phenomena, and size of documents included. Yet, the approaches proposed to scale up anaphoric annotation haven’t so far resulted in datasets overcoming these limitations. In this paper, we introduce a new release of a corpus for anaphoric reference labelled via a game-with-a-purpose. This new release is comparable in size to the largest existing corpora for anaphoric reference due in part to substantial activity by the players, in part thanks to the use of a new resolve-and-aggregate paradigm to ‘complete’ markable annotations through the combination of an anaphoric resolver and an aggregation method for anaphoric reference. The proposed method could be adopted to greatly speed up annotation time in other projects involving games-with-a-purpose. In addition, the corpus covers genres for which no comparable size datasets exist (Fiction and Wikipedia); it covers singletons and non-referring expressions; and it includes a substantial number of long documents ( 2K in length).

## introduction many resources annotated for anaphoric reference / coreference exist; but even the largest such datasets, such as ontonotes @xcite , have limitations. the largest resources are still medium scale (e.g., ontonotes @xcite is 1.5m tokens, as is craft @xcite ). they only cover a limited range of domains, primarily news (as in ontonotes) and scientific articles (as in craft), and models trained on these datasets have been shown not to generalize well to other domains @xcite . 2 the range of anaphoric phenomena covered is also narrow @xcite . several proposals have been made to scale up anaphoric annotation in terms of size, range of domains, and phenomena covered proposed, including automatic data augmentation @xcite @xcite , and crowdsourcing combined with active learning @xcite @xcite or through games-with-a-purpose @xcite @xcite . however, the largest existing anaphoric corpora created using games-with-a-purpose (e.g., @xcite ) are still smaller than the largest resources created with traditional methods, and the corpora created using data augmentation techniques are focused on specific aspects of anaphoric reference. in order to use such approaches to create resources of the required scale in terms of size, variety and range of phenomena covered novel methods are required. the first contribution of this paper is the phrase detectives 3.0 corpus of anaphoric reference annotated using a game-with-a-purpose. this corpus has a comparable size in tokens (1.37m) to the ontonotes corpus @xcite , but twice the number of markables foot_2 . its annotation scheme also covers singletons and non-referring expressions; it is focused on two genres -fiction and wikipedia articles -not covered in ontonotes, and for which only much smaller datasets exist; and it includes a range of documents ranging from short to fairly long (14k tokens) enabling research in nlp on long documents @xcite . while ontonotes will remain a fundamental resource for the field in terms of size and languages it covers, we therefore hope that phrase detecbut the models trained on this genre have proven to have even worse performance on other domains. the second contribution of the paper is a new iterative resolve-and-aggregate approach developed to 'complete' the corpus by combining crowdsourcing with automatic annotation. only about 70% of documents in the corpus were completely annotated by the players. the proposed method (i) uses an anaphoric resolver to automatically annotate all mentions, including the few still unannotated; (ii) aggregates the resulting judgments using a probabilistic aggregation method for anaphora, and (iii) uses the resulting expanded dataset to retrain the anaphoric resolver. we show that the resolve-andaggregate method results in models with higher accuracy than models trained using only the completely annotated data, or the full corpus not completed using the method. ## background anaphorically annotated corpora a number of anaphorically annotated datasets now exist, covering a number of languages @xcite @xcite @xcite @xcite @xcite @xcite and turning anaphora / coreference into a very active area of research @xcite @xcite @xcite @xcite . however, only a few of these are genuinely large in terms of markables @xcite , and most are focused on news. corpora of comparable size exist for scientific articles (e.g., craft @xcite ), and substantially smaller ones for fiction (e.g., litbank (bamman et al., 2020) and phrase detectives 2 (poesio et al., 2019)), and wikipedia (e.g., wikicoref @xcite or again phrase detectives 2 @xcite ). but important genres such as dialogue are barely covered @xcite . there is evidence that this concentration on a single genre, and on ontonotes in particular, results in models that do not generalize well @xcite . existing resources are also limited in terms of coverage. most recent datasets are based on general purpose annotation schemes with a clear linguistic foundation, but especially the largest ones focus on the simplest cases of anaphora / coreference (e.g., singletons and non-referring expressions are not annotated in ontonotes). and the documents included in existing corpora tend to be short, with the exception of craft: e.g., average document length is 329 in preco, 467 in ontonotes, 630 in arrau, and 753 in phrase detectives. scaling up anaphoric annotation one approach to scale up anaphoric reference annotation is using fully automatic methods to either annotate a dataset, such as amalgum @xcite , or create a benchmark from scratch, such as knowref @xcite . while entirely automatic annotation may result in datasets of arbitrarily large size, such annotations cannot expand current models' coverage to aspects of anaphoric reference they do not already handle well. and creating from scratch large-scale benchmarks for specific issues hasn't so far been shown to result in datasets reflecting the variety and richness of real texts. crowdsourcing has emerged as the dominant paradigm for annotation in nlp @xcite because of its reduced costs and increased speed in comparison with traditional annotation. but the costs for really large-scale annotation are still prohibitive even for crowdsourcing @xcite . to address this issue, a number of approaches have been developed to optimize the use of crowdsourcing for coreference annotation. in particular, active learning has been used to reduce the amount of annotation work needed @xcite @xcite . another issue is that anaphoric reference is a complex type of annotation whose most complex aspects require special quality control typically not available with microtask crowdsourcing. games-with-a-purpose a form of crowdsourcing which has been widely used to address the issues of cost and quality is games-with-a-purpose (gwap) @xcite @xcite . games-with-a-purpose is the version of crowdsourcing where labelling is created through a game, so that the reward for the workers is in terms of enjoyment rather than financial. they were proposed as a solution for large-scale data labelling. a number of gwaps were therefore developed for nlp, including jeux de mots @xcite , phrase detectives @xcite , ontogalaxy @xcite , the wordrobe platform @xcite , dr detective @xcite , zombilingo @xcite , tileattack! @xcite , wormingo @xcite , name that language! @xcite or high school superhero @xcite . gwaps for coreference include phrase detectives @xcite , the pointers game in wordrobe @xcite and wormingo (kicikoglu et al., 2019), all deployed, and playcoref @xcite , proposed but not tested. however, whereas truly successful gwaps such as foldit have been developed in other areas of science @xcite , even the most successful gwaps for nlp only collected moderate amounts of data @xcite . in part, this is because the games used to actually collect nlp labels aren't very entertaining, leading to efforts to develop engaging designs such as @xcite @xcite . an interesting solution to this issue was proposed although not fully developed for wordrobe @xcite . this solution is a hybrid between automatic annotation and crowdsourcing: a combination of crowd and automatically computed judgments is aggregated to ensure that every item has at least one label. this solution wasn't properly tested in wordrobe, which only collected very few judgments and for a small corpus; and anyway the approach followed could not be applied to anaphora/coreference, due to the lack of a suitable aggregation mechanism for anaphora/coreference. in this paper we present the first true test of the idea by proposing a method for aggregating crowd and automatic judgments inspired by this idea, but using an aggregation method for anaphora, and truly tested on a dataset containing a very large number of anaphoric judgments. ## phrase detectives the human judgments used in our corpus were collected using the phrase detectives game-with-a-purpose foot_3 @xcite @xcite , designed to collect multiple judgments about anaphoric reference. game design phrase detectives doesn't follow the design of some of the original von ahn games @xcite , in that it is a oneperson game, and not timed; both competition and timing were found to have orthogonal effects on the quality of the annotation @xcite . points are used as the main incentive, with weekly and monthly boards being displayed. players play two different games: one aiming at labelling new data, the other at validating judgments expressed by the other players. in the annotation game, name the culprit, the player provides an anaphoric judgment about a highlighted markable (the possible judgments according to the annotation scheme are discussed next). if different participants enter different interpretations for a markable then each interpretation is presented to other participants in the validation game, detectives conference, in which the participants have to agree or disagree with the interpretation. every item is annotated by at least 8 players (20 on average), and each distinct interpretation is validated by at least four players. players get points for each label they produce, but especially when their interpretation is agreed upon by other players, thus rewarding accuracy. initially, players play against gold data, and are periodically evaluated against the gold; when they achieve a sufficient level of accuracy, they start seeing incompletely annotated data. extensive analyses of the data suggest that although there is a great number of noisy judgments, this simple training and validation method delivers extremely accurate aggregated labels @xcite @xcite . ## characteristics of the corpus the phrase detectives 3.0 corpus includes all the 805 documents originally uploaded in the game. in this section we highlight the main characteristics of the texts in this release, summarized in for comparison, we include in the appendix a short description of the previous release of the phrase detectives corpus, phrase detectives 2, released in 2019 @xcite . the new release the new release of the corpus, phrase detectives 3.0, is more than three times larger than the previous release of the phrase de-tectives corpus described in the previous section in terms of the number of tokens (1.4m) and markables (383k). (see 'all' row in ## resolve-and-aggregate the challenge to create a reliable corpus using crowdsourcing, multiple judgments are required to ensure a good coverage of correct answers, together with sufficient evidence to enable an accurate aggregation method @xcite to distill the correct answers from the noisy ones. the problem of collecting such large number of judgments is even more serious for long documents. annotating all anaphoric relations in long documents is challenging, partly due to the amount of time needed to complete the task, but also because the great number of entities makes it difficult for annotators to keep track of all the coreference chains. and indeed, the short documents in our corpus were completed much faster than the longer documents: the average length of the incomplete documents is 4k tokens, whereas for the complete documents is 850 tokens. thus in our corpus the rate at which judgments are collected from players, while substantial (over 1,000 judgments per day) is not sufficient to extract reliable labels in a reasonable amount of time, as discussed in section 3. ## resolving-and-aggregating results experiment setting for our experiments, we report the conll f1 scores as defined in (pradhan et al., 2012) in both singleton included and excluded settings, as well as non-referring f1 scores for non-referring expressions. we use the universal anaphora (ua) scorer @xcite ) that reports all the necessary scores. we trained the yu et al. ( 2020 ) system using most of its default settings. the only exception is that we always use the full context of the documents for training instead of choosing a random 1k tokens as done in @xcite . the default setting gives priority to the short documents as for each epoch, the full context of the short documents is always used, whereas for long documents only part of the documents is used. we establish three baselines, all using the same system @xcite with the same settings, but trained with different training sets. the first baseline is trained on the previous release. the second baseline is trained on train com-plete (complete documents aggregated by mpa without resolve-and-aggregate). the third baseline is trained on train full aggregated by mpa but without annotations from the coreference resolver. ## limitations the main limitation of this work is that the new release is still only twice the size of ontonotes in terms of markables. in ongoing work, we are developing a new platform to label a corpus twenty times the size of the current release. the new platform 5 combines more engaging games with activelearning like methods for allocating work to players more efficiently and according to their linguistic understanding @xcite . we hope that the new platform, in combination with the methods proposed here, will allow us to label the new and larger dataset much more quickly. a second limitation of the new release is that the markables in the corpus were automatically extracted; thus, the quality of the mentions is lower than in corpora in which they were hand-identified. the approach followed in these years has been to ask our players to signal issues; as a result, tens of thousands of markables were hand-corrected. however, this approach doesn't really lend itself to scaling up. thus, in our new platforms we are following a different strategy: asking our players to do the corrections themselves, by including also games to check other levels of linguistic interpretation. a third limitation, in particular in comparison with ontonotes, is that this release of the corpus only contains english documents, although a small amount of italian documents was uploaded in the game.