P air S pan BERT : An Enhanced Language Model for Bridging Resolution (2023)

Abstract: We present PairSpanBERT, a SpanBERT-based pre-trained model specialized for bridging resolution. To this end, we design a novel pre-training objective that aims to learn the contexts in which two mentions are implicitly linked to each other from a large amount of data automatically generated either heuristically or via distance supervision with a knowledge graph. Despite the noise inherent in the automatically generated data, we achieve the best results reported to date on three evaluation datasets for bridging resolution when replacing SpanBERT with PairSpanBERT in a state-of-the-art resolver that jointly performs entity coreference resolution and bridging resolution.

## introduction bridging is essential for establishing coherence among the entities within a text through nonidentical semantic or encyclopedic relations @xcite . as demonstrated in example 1, local coherence is established via the implicit link between the bridging anaphor (prices) and its antecedent (meat, milk and grain). the task of bridging resolution, which involves identifying all the bridging anaphors in a text and linking them to their antecedents, is crucial for machine comprehension of the relations between discourse entities for various downstream applications, such as question answering @xcite and dialogue systems @xcite . the most successful natural language learning paradigm to date is arguably the "pre-train and finetune" paradigm, where a model is first pre-trained on very large amounts of data in a task-agnostic, self-supervised manner and then fine-tuned using a potentially small amount of task-specific training data in the usual supervised manner. this paradigm is ideally applicable to bridging resolution, where the amount of annotated training data is relatively small, especially in comparison to the related task of entity coreference resolution. in fact, by using spanbert @xcite to encode the input and fine-tuning it using bridging-annotated data, @xcite have managed to achieve the best results reported to date on two commonlyused evaluation datasets for bridging resolution, namely isnotes @xcite and bashi @xcite . a natural question is: how can we build upon the successes of this pre-train and fine-tune framework for bridging resolution? apart from achieving stateof-the-art results, @xcite show that bridging resolution performance deteriorates when spanbert is replaced with bert @xcite as the encoder. while it is perhaps not surprising that spanbert achieves better resolution results than bert given its superior performance on a wide variety of natural language processing tasks, it is important to understand the reason. recall that spanbert is an extension of bert that is motivated by entity-based information extraction tasks such as entity coreference resolution and relation extraction. these tasks typically involve the extraction of entity mentions, which are text spans. in order to learn span (as opposed to word) representations, spanbert is pre-trained with span-level masking and objectives. the key point here is that a pre-trained model tends to work better for a downstream task (which in our case is bridging resolution) if it is pre-trained with an objective that is in some sense related to the downstream task. motivated by this observation, we design a novel pre-training objective for bridging resolution that allows a model to learn the contexts in which two mentions are implicitly linked to each other. we subsequently use our objective to further pre-train spanbert in combination with its original objectives, yielding pairspanbert, a pre-trained model that is specialized for bridging resolution. note that an important factor that contributes to the success of pre-training is the sheer amount of data on which the model is pre-trained: since pretraining tasks are designed to be self-supervised learning tasks, a very large amount of annotated training data can be automatically generated, thus allowing the model to potentially acquire a lot of linguistic and commonsense knowledge. to enable our model to learn the contexts that are indicative of bridging, we employ a large amount of data that can be automatically generated either heuristically @xcite or via distance supervision using a knowledge graph. while the vast majority of existing bridging resolvers are evaluated in the rather unrealistic setting where gold mentions are assumed as input, we follow @xcite recommendation and evaluate our bridging resolver in both the (realistic) end-to-end setting, where we assume raw text as input, and the gold mention setting, where gold mentions are given. when replacing spanbert with pairspanbert in kobayashi et al's bridging resolver, we achieve the best results reported to date on three datasets for bridging resolution, isnotes, bashi, and arrau rst @xcite , in both evaluation settings despite the large amount of noise inherent in our automatically generated data. to our knowledge, this is the first work that reports end-to-end bridging resolution results on the arrau rst dataset. ## related work bridging resolution. the two sub-tasks of bridging resolution, namely bridging anaphora recognition and bridging anaphora resolution, have been tackled separately. one line of research has modeled bridging anaphora recognition as a part of the information status (is) classification problem where each discourse entity is assigned an is category, with bridging being one of the categories @xcite @xcite . in contrast, bridging anaphora resolution focuses on identifying the antecedents for gold bridging anaphors @xcite @xcite . there have been several studies addressing full bridging resolution, which involves recognizing bridging anaphors and determining their antecedents. these works include rule-based approaches @xcite , learning-based approaches @xcite , and hybrid approaches @xcite . a comprehensive overview of these approaches can be found in @xcite . recent studies have begun tackling bridging resolution and its sub-tasks in the end-to-end setting. for example, @xcite uses a combination of neural mention extraction and is classification models for bridging anaphora recognition. furthermore, @xcite proposes an approach of rephrasing bridging anaphors as questions and training question-answering models to directly extract antecedents from their previous contexts. finally, there are a few works that propose models for full bridging resolution in the end-to-end setting @xcite @xcite in the 2021 and 2022 codi-crac shared tasks on anaphora, bridging, and discourse deixis in dialogue @xcite . recently, @xcite conduct a systematic evaluation of bridging resolvers using different standard encoders, including bert @xcite and spanbert @xcite , in the end-to-end setting. enhanced pre-trained language models. bert @xcite , which is based on the transformer architecture @xcite , has recently attracted significant attention. researchers have proposed methods to enhance it for a wide range of downstream tasks. one line of research focuses on improving the masking schemes and the training objectives when pre-training models for tasks such as question answering and sentence selection @xcite @xcite . another line of work focuses on incorporating external knowledge into pre-trained models to solve knowledge-driven problems such as relation extraction @xcite . ## the current state of the art state-of-the-art results on isnotes and bashi are reported in @xcite , who extend yu and poesio's (2020) multi-task learning (mtl) approach to bridging resolution by (1) using spanbert to encode the input and (2) incorporating the predictions made by a rule-based resolver into the mtl framework. since we aim to create pairspanbert, which specializes span- ## pairspanbert next, we present pairspanbert, an extension of spanbert specialized for bridging resolution. to create pairspanbert, we use spanbert as a starting point and add a pre-training step to it that would enable the model to learn the contexts in which two mentions are implicitly linked to each 1 although spanbert is often viewed as an extension of bert, not everything in bert appears in spanbert. for example, while bert is pre-trained on the so-called next sentence prediction (nsp) task, spanbert is not. transformer encoder can it [mask] cause before masking t 1 t 2 t 3 t 4 t 5 t 6 t 7 can it severe cause food any5me restric5on [mask] any5me [mask] other from data that is automatically generated either heuristically or via distant supervision with the help of a knowledge graph. to do so, we will describe how we obtain automatically generated data (section 4.1), the masking scheme (section 4.2), and the pre-training task (section 4.3). ## conclusion we designed a novel pre-training task for bridging resolution using automatically annotated documents that contain noun pairs that are likely to be linked via implicit relations, and demonstrated that our newly pre-trained model, pairspanbert foot_4 , effectively captures bridging relations. on three commonly-used datasets for bridging resolution, our new resolver based on pairspanbert outperformed the previous state-of-the-art models and other strong baselines for full bridging resolution. in future work, we plan to apply pairspan-bert to other language processing tasks, particularly relation extraction tasks, since the noun pairs extracted from the syntactic structures and con-ceptnet are likely to have non-identical relations.