Pre-training Mention Representations in Coreference Models (2020)

Abstract: Collecting labeled data for coreference resolution is a challenging task, requiring skilled annotators. It is thus desirable to develop coreference resolution models that can make use of unlabeled data. Here we provide such an approach for the powerful class of neural coreference models. These models rely on representations of mentions, and we show these representations can be learned in a self-supervised manner towards improving resolution accuracy. We propose two self-supervised tasks that are closely related to coreference resolution and thus improve mention representation. Applying this approach to the GAP dataset results in new state of the arts results.

## introduction coreference resolution models cluster mentions by their referring entities. almost all such models rely on vector representations of mentions @xcite @xcite @xcite @xcite @xcite . the representations for all mentions are then compared (usually sequentially) and mention pairs judged to be most similar are considered coreferent. thus, the mention representation is a key component in modern coreference resolution models. indeed, it has recently been shown that improving this representation leads to improved resolution performance. for example, bert embeddings were used in @xcite and spanbert @xcite further improved performance. however, both bert and spanbert representations are trained on self-supervised tasks that seem quite distant from coreference resolution (e.g., masked-word-prediction in bert, and maskedwhole-span-prediction in spanbert). this suggests the possibility that unlabeled data can be used for further improving coreference resolution if we use self-supervised tasks that are more closely related to coreference resolution. motivated by the above, we ask: which selfsupervised tasks should be used to improve mention representation for coreference resolution. two recent attempts for pre-training coreference models have focused on tasks such as language modeling @xcite and maskedword-prediction for name resolution @xcite . here we propose self-supervision tasks that train the coreference model directly (rather than just the underlying bert), resulting in improved mention representations and resolution accuracy. we identify two signals in a text that are highly informative for coreference resolution and show how to use them for self-supervision. the first signal is that the same name can appear multiple times in a text, and these mentions very likely corefer. thus we can train mention representations to be similar for these mentions. the second signal is pronouns. since each pronoun is likely to refer to some mention, we optimize mention representations to maximize the accuracy of this prediction. we describe a training procedure for both these losses and show that together they result in new state of the art results on the gap coreference dataset. importantly this is a fairly small dataset, and thus our results demonstrate the power of unsupervised pre-training of mention representations. ## baseline model the coreference resolution task corresponds to extracting the set of mentions from a text and clustering these, such that clusters correspond to all mentions of a specific entity. as a baseline model, we use the work of @xcite , which builds on top of @xcite and uses spanbert @xcite . spanbert has the same architecture as bert @xcite but is trained with a different objective, where whole spans are masked and span boundary representations are optimized to predict all tokens of the masked span. this feature proves useful for coreference since, in many cases, entity mentions are spans of tokens, and span ranking models benefit from improved span representations. the input to the coreference model is a span representation r i for each mention i. in what follows, dependence on r i is implied from dependence on i. a scoring function s m (i) is defined to score whether a span i is a mention. only a portion of the top-scored mentions are kept for antecedent matching. an antecedent scoring function s(i, j) is defined to score whether j is an antecedent of i. using the pairwise function, for each span i, a distribution p (y i ) over antecedents is defined: ## pre-training process our goal is to propose an approach for pre-training mention representations on unlabeled data, such that they can be more readily used for coreference resolution. namely, the goal is that after pre-training, we can use the mention representation to train a coreference resolution system with relatively little labeled data. next, we propose two objectives for this pre-training process. most previous approaches to pre-training @xcite use only the bert model while pretraining on unlabeled data. here we propose to pre-train the coreference model in sec. 2. the motivation is that we want to directly train the mention alice and bob are friends. yesterday alice had an accident. when bob saw alice he was relieved. mask and bob are friends. yesterday alice had an accident. when mask saw table he was relieved. figure 1: an illustration of the name masking objective. on the top is the original sentence with colors corresponding to repeated names. bottom is the same sentence with some of the mentions replaced with random tokens and @xcite . the self-supervision task is to cluster the red and blue mentions using the coreference model.). ## fine-tuning after the pre-training process in sec. 3, we finetune the model on the gap dataset. unlike ontonotes, gap is partially labeled: only one pronoun and two names are labeled, even if additional entities exist in the text. partial labeling poses a challenge for coreference models that have a learnable mention detection phase, since the ground truth excludes mentions, and during training the model learns to falsely label their spans as nonmentions. to accommodate this, we change the baseline loss to consider only gold mentions, i.e., we optimize the log-likelihood of the correct antecedent filtered only for the gold mentions. we define it as l 1 (d) for a document d. we found it useful to additionally train for correct mention detection in the objective. we add a mention auxiliary loss l 2 (d) in the form of crossentropy on the predicted mention score s m . finally, we optimize: several works have set out to improve mention representations for coreference resolution @xcite @xcite . @xcite have refined the mention representation using attention over the antecedents of each mention. @xcite showed the entity equalization approach to represent each mention in a cluster via an approximation of the sum of all mentions in its cluster. employing @xcite to extract mention representations boosts coreference resolution accuracy @xcite . @xcite increased it even further using masked whole-span prediction. several works have explored pre-training for coreference resolution. the recent work of @xcite uses question answering as part of the model and can thus train on qa datasets. other works explored self-supervision for this goal. @xcite uses a language model objective to train a memory network, which can resolve coreference links. @xcite finds pairs of sentences with at least two distinct personal names such that one of them is repeated. one non-first occurrence of the repeated candidate is masked, and the goal is to predict the masked name, given the correct and one incorrect candidates. they collect examples with no more than two sentences, limiting the background context the model can extract. since only one name occurrence is masked and needs to be inferred, the model is not forced to resolve all person clusters. ye et al. (2020) use an approach similar to @xcite , but include a language modeling objective. @xcite use names and pronouns gender information to generate links between a pronoun and a name. again, the model needs to resolve a single coreference link for each example, instead of resolving multiple clusters. our proposed name masking is conceptually different since we do not try to predict names or single links, but rather use masking and random tokens to create hard coreference problems from the data. our proposed pre-training procedure generates rich examples with multiple clusters and mentions. this lets us train a coreference model directly rather than just a bert model. the above evaluation scheme does not use the fact that there are only three marked mentions in each snippet. there are however previous works @xcite that consider the goldtwo-mention task @xcite , where the locations of the gold names and pronoun are used during inference as well foot_0 . we will compare our results in both scenarios: detected-mentions, where models need to detect the mentions by themselves, and gold-two-mention. the metrics measured in this task are the overall f1, f1 on feminine and masculine examples, and bias defined as feminine f1 masculine f1 . we use the official scorer. 2 training: our experimental setup and code is built on top of the code in @xcite and spanbert @xcite . we pre-train on english wikipedia unlabeled text foot_2 for 700k steps on the objectives defined in 3 using spacy ner foot_3 for person names extraction. see masking strategy and model's hyperparameters in appendix a. ## results we report masculine, feminine and overall f1 and feminine f1 to masculine f1 bias @xcite . all fine-tuning results are averages of 5 runs. test set results for the detected-mentions scenario are shown in test set results for the gold-two-mention task are shown in fine-tuning on ontonotes: we explore another training setting of @xcite , where training is only on ontonotes @xcite and not gap. in this setting spanbert base and large yield an overall f1 of 85.76 and 87.5 on gap, respectively. our pre-training, followed by ontonotes training, improves these to 86.12 and 87.66. improvement is smaller than when training only on gap, since ontonotes is a much larger labeled dataset than gap, thus reducing the effect of pre-training. we can also compare our results to the recent work of @xcite . they also consider the setting of fine-tuning on ontonotes, and report an f1 of 87.5 on gap, as compared to our 87.66 in the same setting. however, the models are not completely comparable because @xcite pre-train on quoref and squad, whereas we pre-train on wikipedia. ## conclusion we proposed two self-supervision tasks to improve span representations of coreference resolution models. our approach directly optimizes the mention representations used by the coreference model, allowing it to be fine-tuned on relatively little data, with improved accuracy. our results demonstrate the potential of pre-training for coreference. we believe there is much potential for additional selfsupervision tasks and leave those for future work.