Subjectivity in the Annotation of Bridging Anaphora (2025)

Abstract: Bridging refers to the associative relationship between inferable entities in a discourse and the antecedents which allow us to understand them, such as understanding what “the door” means with respect to an aforementioned “house”. As identifying associative relations between entities is an inherently subjective task, it is difficult to achieve consistent agreement in the annotation of bridging anaphora and their antecedents. In this paper, we explore the subjectivity involved in the annotation of bridging instances at three levels: anaphor recognition, antecedent resolution, and bridging subtype selection. To do this, we conduct an annotation pilot on the test set of the existing GUM corpus, and propose a newly developed classification system for bridging subtypes, which we compare to previously proposed schemes. Our results suggest that some previous resources are likely to be severely under-annotated. We also find that while agreement on the bridging subtype category was moderate, annotator overlap for exhaustively identifying instances of bridging is low, and that many disagreements resulted from subjective understanding of the entities involved.

## introduction bridging is an anaphoric phenomenon where a newly introduced discourse entity is dependent on an associated, non-identical antecedent entity for interpretation. the term "bridging" refers to a discourse participant's construction of an implicature from the entity they are currently processing back to an antecedent entity @xcite . this associative relation can be triggered by a broad variety of linguistic mechanisms, including lexical partwhole relations (a house -the door) and implicit arguments (a murder -the victim). since the phenomenon was first commented on by @xcite , it has received a variety of theoretical treatments, including @xcite 's closely related notion of inferrables which centers information status as the key component in identifying anaphoric bridging relations. such theoretical divides have resulted in a number of different annotation formalisms varying in their definitions of bridging, as well as in their delineations of sub-varieties of bridging @xcite . while there has recently been some effort to harmonize bridging annotations across different corpora @xcite , the current landscape of bridging resources remains heterogeneous. the lack of consistency in and across bridging resources largely stems from their differing definitions for bridging, as well as the subjective annotator judgments that go into identifying instances of bridging. in this paper, we explore subjectivity in the annotation of bridging anaphora in order to understand how to account for that subjectivity and create more consistent annotations in future efforts. we examine three stages in the annotation process where annotators must make subjective judgments: (1) recognition of the bridging anaphor, (2) resolving back to its associated antecedent, and (3) identifying the subtype category of the bridging pair. to this end, we conduct an annotation pilot on the test set of an existing english corpus, gum (v10) @xcite . while the gum corpus includes bridging annotations, the annotation guidelines are underspecified and do not include bridging subtype annotations. this annotation pilot is a preliminary phase in the development a new bridging resource, gumbridge. for this effort, we develop a new classification system for bridging subtypes organized under 3 relation types: comparison relations, entity relations, and set relations, as well as an additional other category. we also create annotation guidelines for how to identify instances of bridging anaphor-antecedent pairs and how to classify them into subtypes. analyzing the results of this pilot, we find on the one hand that we are able to identify substantially more and denser attestation of bridging than suggested by several previous resources. in terms of subjectivity, we find moderate agreement for the selection of the bridging subtype category and for the selection of an antecedent for a given anaphor. however, the annotator overlap in the recognition of bridging anaphora is considerably lower, despite mostly plausible precision. we conduct a qualitative evaluation of the annotations from the pilot, and we find that subjectivity plays a role in each of the three annotator judgment stages listed above, especially for recall. we explore this role for each stage, and then give recommendations on how to structure the annotation of bridging anaphora in order to account for subjectivity in annotator judgment. ## background as mentioned above, there are a number of different annotation formalisms for bridging, all with somewhat different definitions of bridging as a phenomenon. in english, the evaluation of bridging resolution systems (systems which aim to automatically identify bridging anaphora and resolve back to their associative antecedents) is commonly conducted using the following three corpora: isnotes @xcite , bashi @xcite , and arrau rst @xcite . while arrau rst annotates bridging instances by identifying mention pairs that establish cohesion in text and then classifies then via a set of predefined semantic relations, isnotes and bashi annotate bridging anaphora based on the information status of entities, considering bridging to be a sub-variety of mediated information. the information status (is) of an entity refers to the extent to which the entity is accessible to the reader/hearer of a discourse @xcite . generally speaking, "new" information is unrecognized by the reader/hearer, while "given" information is recognized. "given" entities may be recognized by the reader/hearer for various reasons: the entity may have been previously introduced in the discourse (coreference), the entity may be accessible via generics/world knowledge, or, in the case of bridging, the referent of the entity may be inferred from a previous entity in the discourse. instances of bridging and generics/world knowledge are both considered "accessible" in that they are recognized by the reader/hearer when they are first introduced to the discourse, but only instances of bridging depend on an associative antecedent for comprehension. there are also a number of other existing bridging resources: in english, gum, scicorp (roesiger, 2016), corefpro @xcite , red (richer event descriptions, @xcite ; as well as in other languages: grain @xcite and dirndl @xcite in german, pdt @xcite @xcite in polish, to name a few. there have additional been efforts in areas closely related to bridging, such as @xcite , which puts forward a typology for classifying near-identity relations (nident) for coreference, and modjeska (2004)'s work on other-anaphora, which we now consider a subtype of bridging. we provide background on is-notes, bashi, and arrau rst, as they are commonly used in bridging resolution evaluation @xcite , and they illustrate diverging perspectives on identifying bridging instances. isnotes is a corpus of 50 wall street journal (wsj) documents from the ontonotes corpus @xcite annotated for fine-grained information status. isnotes distinguishes three main categories of is: new, old, and mediated. old information is that which known to the hearer and/or has been refereed to previously, while new information is introduced for the first time. mediated information has not been introduced before, but is not independently comprehensible, requiring either an inference from a previous mention or from general/real-world knowledge. within the mediated category, there are six subcategories, including bridging. the corpus contains 663 instances of bridging in the mediated/bridging category, and there are an additional 253 instances of comparative anaphora in the mediated/comparison category, which is considered a variety of bridging (~16.6 bridging instances per 1k tokens). @xcite report cohen's κ for annotator pairs, ranging ~0.6-0.7 for mediated/bridging, and ~0.8 for mediated/comparison. they note that the agreement for mediated/bridging is more annotator dependent relative to the other is categories. the bashi corpus is also annotated on top of 50 wsj documents from the ontonotes corpus, and it includes a total of 459 bridging pairs (~7.9 bridging instances per 1k tokens). @xcite introduces the contrast between referential bridging and lexical bridging, where referential bridging is a properly anaphoric relation (antecedent is required for the interpretation of the anaphor) and lexical bridging is a non-anaphoric semantic relation between two entities. the corpus specifically contains annotations only for referential bridging, not lexical bridging. the bridging instances in bashi have the subtypes definite, indefinite, and comparative anaphora. annotator agreement is reported for these categories individually and together. the joint agreement for identifying bridging pairs is 59.3%, with a higher rate for comparative anaphora at 71.4% and lower agreement for definite at 63.8% and indefinite at 42.3%. arrau is a multi-genre corpus covering a variety of anaphoric phenomena, composed of 4 subcorpora, each with its own annotation specifications. arrau rst is the largest sub-corpus, and also the one most used in evaluation for bridging resolution. it is composed of wsj news data, and it includes 3,777 bridging annotations (~16.5 bridging instances per 1k tokens). arrau's bridging annotation connects related mentions which establish "entity coherence" via non-identity relations, but as this casts a very broad scope, annotation is limited to a fixed set of semantic relations. the corpus uses an inventory of 9 bridging subtypes for annotation: possession, element-set, subset-set, anaphora marked with 'other', along with accompanying inverse relations of the previous, and an additional under-specified relation. the annotation schema and guidelines for bridging in arrau were extended from the gnome project @xcite . coders in the gnome project displayed high agreement (95.2%) in the choice of bridging subtype labels from its fixed set of relations, but low recall (22%) in unanimously identifying instances of bridging. limiting annotation to a predefined set of relations restricts the scope of bridging as a phenomenon, but also aims to increase consistency in the annotation. however, as has been noted in @xcite , annotating from predefined relations can also introduce false positives, in the case that an instance of a semantic relation is not actually a case of associative anaphoric reference that would constitute referential bridging. for instance, the case of europe -spain displays a meronomy relation, but it is not anaphoric because spain can be interpreted without reference to europe. annotating from an information status informed perspective aims to avoid such false positives, providing a more concrete linguistic criteria for identifying instances of bridging when compared to the notion of "entity coherence", and eliminating the need to only annotate a predefined set of relations for scoping reasons. however, this information status based approach also greatly widens the scope of what should be considered bridging, which in turn increases the influence of subjective judgment by annotators. as such, in order to forward an information status informed annotation perspective, we must develop means of dealing with additional subjectivity it produces. as we can see in ## annotation pilot the analysis on subjectivity in the annotation of bridging instances in this paper is conducted using the results of an annotation pilot for the creation of a new bridging resource called gumbridge. built on top of gum, an existing multi-genre corpus of english, gumbridge aims to unite aspects of currently existing formalisms: using an information status-informed view of identifying bridging instances (as in isnotes and bashi), followed by subtype categorization using a taxonomy of semantic relations (as in arrau). additionally, gumbridge aims to add genre diversity to the core english bridging resources, as isnotes, bashi, and arrau rst are all composed of wsj news data from more than 30 years ago, offering little to analyze in terms on genre diversity. while the development of this resource is still underway, an adjudicated version of the bridging annotations for the gumbridge test set (version 0.1) is released with this paper foot_0 . the details of this adjudication process are described in section 3.5. the guidelines for identifying instances of bridging (v0.1) are described in section 3.1, and the classification system for bridging subtypes (v0.1) is described in section 3.2. ## subjectivity in bridging annotation previous work on subjectivity in the development of linguistic data has heavily featured areas where annotator judgments can be highly variable, such as hate speech detection and sentiment analysis (e.g., @xcite ; @xcite ), though attention has also been given to tasks which seem more objective, such as part of speech annotation (e.g., @xcite ). several works discuss the paradigms for and implications of including subjective judgments in annotation efforts, rather than trying to eliminate all ambiguity @xcite . ultimately, the appropriate approach depends on the linguistic task at hand and what the researchers are hoping to achieve with the annotation effort. although detailed guidelines are provided to annotators in this paper's annotation pilot, subjective judgment is still an inherent part of the annotation of bridging instances, as annotators are making decisions based off their understanding of the implicit relationships that exist between entities in a discourse. as previously noted, there are three decision points in the annotating of bridging instances that can introduce subjective judgment: (1) recognition of the bridging anaphor, (2) identifying the corresponding associative antecedent, and (3) selecting the bridging subtype category of the pair. the sections below give examples to illustrate the unique considerations regarding subjectivity that are present at each of these annotation stages. ## conclusion in this paper, we examine the influence of subjectivity in annotator judgment on the various stages of annotating instances of bridging. we make this examination using the resulting annotations from a pilot to create a new resource for bridging annotations, gumbridge. we also release an adjudicated version of the bridging annotations for the preliminary test set of gumbridge (v0.1). in subsequent work, we plan to refine the guidelines and annotation procedure used in this pilot, which we will then use to annotate the remainder of the gum corpus (dev and train) to produce a full version of gum-bridge, as well as extending our annotations to gum's out-of-domain challenge test set, gentle (genre tests for linguistic evaluation, @xcite . as the time and effort required to manually annotate bridging limits the scalability of the annotation process, we will also investigate incorporating semi-automated methods, such as combining llms or other systems for bridging resolution with human correction in order to improve the efficiency of the process. in our development of gumbridge test (v0.1), we found that annotators' agreement on selecting the subtype of a bridging pair was moderate, but that it was more difficult to get the annotators to align on the identification of bridging anaphora. this indicates that recognition of bridging anaphora is the stage in the annotation process that is most vulnerable to the subjective judgment of annotators, and that should be given the most consideration when trying to account for annotator subjectivity. while some subjectivity arises from the inherent ambiguity of language in context, other aspects of subjectivity can be accounted for by providing guidelines on how to decide on preferable judgments when multiple options are available.