C oreferential R easoning L earning for L anguage R epresentation (2020)

Abstract: Language representation models such as BERT could effectively capture contextual semantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropriate fine-tuning. However, most existing language representation models cannot explicitly handle coreference, which is essential to the coherent understanding of the whole discourse. To address this issue, we present CorefBERT, a novel language representation model that can capture the coreferential relations in context. The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consistently on various downstream NLP tasks that require coreferential reasoning, while maintaining comparable performance to previous models on other common NLP tasks. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/CorefBERT .

## introduction recently, language representation models such as bert @xcite have attracted considerable attention. these models usually conduct self-supervised pre-training tasks over large-scale corpus to obtain informative language representation, which could capture the contextual semantic of the input text. benefiting from this, language representation models have made significant strides in many natural language understanding tasks including natural language inference @xcite , sentiment classification @xcite , question answering @xcite , relation extraction @xcite , fact extraction and verification @xcite , and coreference resolution @xcite . however, existing pre-training tasks, such as masked language modeling, usually only require models to collect local semantic and syntactic information to recover the masked tokens. hence, language representation models may not well model the long-distance connections beyond sentence boundary in a text, such as coreference. previous work has shown that the performance of these models is not as good as human performance on the tasks requiring coreferential reasoning @xcite , and they can be further improved on long-text tasks with external coreference information @xcite @xcite . coreference occurs when two or more expressions in a text refer to the same entity, which is an important element for a coherent understanding of the whole discourse. for example, for comprehending the whole context of "antoine published the little prince in 1943. the book follows a young prince who visits various planets in space.", we must realize that the book refers to the little prince. therefore, resolving coreference is an essential step for abundant higherlevel nlp tasks requiring full-text understanding. to improve the capability of coreferential reasoning for language representation models, a straightforward solution is to fine-tune these models on supervised coreference resolution data. nevertheless, on the one hand, we find fine-tuning on existing small coreference datasets cannot improve the model performance on downstream tasks in our preliminary experiments. on the other hand, it is impractical to obtain a large-scale supervised coreference dataset. to address this issue, we present corefbert, a language representation model designed to better capture and represent the coreference information. to learn coreferential reasoning ability from largescale unlabeled corpus, corefbert introduces a novel pre-training task called mention reference prediction (mrp). mrp leverages those repeated mentions (e.g., noun or noun phrase) that appear jane 𝒙 ! 𝒉 ! claire 𝒙 " 𝒉 " but 𝒙 # 𝒉 # [mask] 𝒙 @xmath0 presents 𝒙 % 𝒉 % against 𝒉 & 𝒙 & evidence 𝒙 ' 𝒉 ' presents strong 𝒙 !! 𝒉 !! s [mask] multiple times in the passage to acquire abundant co-referring relations. among the repeated mentions in a passage, mrp applies mention reference masking strategy to mask one or several mentions and requires model to predict the masked mention's corresponding referents. figure 1 shows an example of the mrp task, we substitute one of the repeated mentions, claire, with [mask] and ask the model to find the proper contextual candidate for filling it. to explicitly model the coreference information, we further introduce a copybased training objective to encourage the model to select words from context instead of the whole vocabulary. the internal logic of our method is essentially similar to that of coreference resolution, which aims to find out all the mentions that refer to the masked mentions in a text. besides, rather than using a context-free word embedding matrix when predicting words from the vocabulary, copying from context encourages the model to generate more context-sensitive representations, which is more feasible to model coreferential reasoning. we conduct experiments on a suite of downstream tasks which require coreferential reasoning in language understanding, including extractive question answering, relation extraction, fact extraction and verification, and coreference resolution. the results show that corefbert outperforms the vanilla bert on almost all benchmarks and even strengthens the performance of the strong roberta model. to verify the model's robustness, we also evaluate corefbert on other common nlp tasks where corefbert still achieves comparable results to bert. it demonstrates that the introduction of the new pre-training task about coreferential reasoning would not impair bert's ability in common language understanding. ## related work pre-training language representation models aim to capture language information from the text, which facilitate various downstream nlp applications @xcite @xcite . early works @xcite focus on learning static word embeddings from the unlabeled corpus, which have the limitation that they cannot handle the polysemy well. recent years, contextual language representation models pre-trained on large-scale unlabeled corpora have attracted intensive attention and efforts from both academia and industry. sa-lstm @xcite and ulmfit @xcite pre-trains language models on unlabeled text and perform task-specific fine-tuning. elmo @xcite further employs a bidirectional lstm-based language model to extract context-aware word embeddings. moreover, ope-nai gpt @xcite and bert @xcite learn pre-trained language representation with transformer architecture @xcite , achieving state-of-the-art results on various nlp tasks. beyond them, various improvements on pre-training language representation have been proposed more recently, including (1) designing new pre-trainning tasks or objectives such as span-bert @xcite with span-based learning, xlnet @xcite considering masked positions dependency with auto-regressive loss, mass @xcite and bart @xcite with sequence-to-sequence pre-training, electra @xcite learning from replaced token detection with generative adversarial networks and infoword @xcite with contrastive learning; (2) integrating external knowledge such as factual knowledge in knowledge graphs @xcite @xcite ; and (3) exploring multilingual learning @xcite @xcite or multimodal learning @xcite @xcite . though existing language representation models have achieved a great success, their coreferential reasoning capability are still far less than that of human beings @xcite . in this paper, we design a mention reference prediction task to enhance language representation models in terms of coreferential reasoning. our work, which acquires coreference resolution ability from an unlabeled corpus, can also be viewed as a special form of unsupervised coreference resolution. formerly, researchers have made efforts to explore feature-based unsupervised coreference resolution methods @xcite . after that, word-lm (trinh and le, 2018) uncovers that it is natural to resolve pronouns in the sentence according to the probability of language models. moreover, wikicrem @xcite builds sentence-level unsupervised coreference resolution dataset for learning coreference discriminator. however, these methods cannot be directly transferred to language representation models since their task-specific design could weaken the model's performance on other nlp tasks. to address this issue, we introduce a mention reference prediction objective, complementary to masked language modeling, which could make the obtained coreferential reasoning ability compatible with more downstream tasks. ## methodology in this section, we present corefbert, a language representation model, which aims to better capture the coreference information of the text. as illustrated in figure 1 , corefbert adopts the deep bidirectional transformer architecture @xcite and utilizes two training tasks: formally, given a sequence of tokens foot_1 @xmath1, x 2 , . . . , x n ), we first represent each token by aggregating the corresponding token and position embeddings, and then feeds the input representations into deep bidirectional transformer to obtain the contextual representations, which is used to compute the loss for pre-training tasks. the overall loss of corefbert is composed of two training losses: the mention reference prediction loss l mrp and the masked language modeling loss l mlm , which can be formulated as: ## experiment in this section, we first introduce the training details of corefbert. after that, we present the finetuning results on a comprehensive suite of tasks, including extractive question answering, documentlevel relation extraction, fact extraction and verification, coreference resolution, and eight tasks in the glue benchmark. ## ablation study in this section, we explore the effects of the whole word masking (wwm), mention reference masking (mrm), next sentence prediction (nsp) and copy-based training objective using several benchmark datasets. we continue to train google's released bert base on the same wikipedia corpus with different strategies. as shown in ## conclusion and future work in this paper, we present a language representation model named corefbert, which is trained on a novel task, mention reference prediction (mrp), for strengthening the coreferential reasoning ability of bert. experimental results on several downstream nlp tasks show that our corefbert significantly outperforms bert by considering the coreference information within the text and even improve the performance of the strong roberta model. in the future, there are several prospective research directions: (1) we introduce a distant supervision (ds) assumption in our mrp training task. however, the automatic labeling mechanism inevitably accompanies with the wrong labeling problem and it is still an open problem to mitigate the noise. (2) the ds assumption does not consider pronouns in the text, while pronouns play an important role in coreferential reasoning. hence, it is worth developing a novel strategy such as selfsupervised learning to further consider the pronoun. eclipse (meyer novel) [1] eclipse is the third novel in the twilight saga by stephenie meyer. it continues the story of bella swan and her vampire love, edward cullen. [2] the novel explores bella's compromise between her love for edward and her friendship with shape-shifter jacob black, ... [3] eclipse is preceded by new moon and followed by breaking dawn. [4] the book was released on august 7, 2007, with an initial print run of one million copies, and sold more than 150,000 copies in the first 24 hours alone. subject: new moon / breaking dawn object: twilight saga relation: part of the series subject: edward cullen / jacob black object: stephenie meyer relation: creator subject: eclipse object: august 7, 2007 relation: publication date noreen. reasoning over the above information, we could know that noreen's uncle trains the asthmatic boy. for the second example, it needs to infer that tippett is a composer from the second sentence for obtaining the final answer from the first sentence. after training on the mention reference prediction task, corefbert has become capable of reasoning over these mentions, summarizing messages from mentions in different positions, and finally figuring out the correct answer. for the third and fourth examples, it is necessary to know she refers to elena, and he refers to ector by respective coreference resolution. benefiting from a large amount of distant-supervised coreference resolution training data, corefbert successfully finds out the reference relationship and provides accurate answers.