Improving Bridging Reference Resolution using Continuous Essentiality from Crowdsourcing (2022)

Abstract: Bridging reference resolution is the task of finding nouns that complement essential information of another noun. The essentiality varies depending on noun combination and context and has a continuous distribution. Despite the continuous nature of essentiality, existing datasets of bridging reference have only a few coarse labels to represent the essentiality. In this work, we propose a crowdsourcing-based annotation method that considers continuous essentiality. In the crowdsourcing task, we asked workers to select both all nouns with a bridging reference relation and a noun with the highest essentiality among them. Combining these annotations, we can obtain continuous essentiality. Experimental results demonstrated that the constructed dataset improves bridging reference resolution performance. The code is available at https://github.com/nobu-g/bridging-resolution .

## introduction the meaning of natural language texts is supported by cohesion among various linguistic units such as words, sentences, and paragraphs @xcite . analyzing cohesion is indispensable for capturing the semantic structure of natural language texts. among cohesion analysis tasks, predicateargument structure (pas) analysis and semantic role labeling (srl) have been actively studied @xcite @xcite . these tasks aim to find nouns that complement a predicate's essential meaning, such as who does/did what to whom. on the other hand, bridging reference resolution is the task of finding nouns that complement a noun's essential meaning. it is a special case of an anaphora resolution in which the anaphor and its antecedent have non-identical yet associated relations @xcite . in the above example, the roof is semantically insufficient by itself, and a house plays an essential role in complementing the meaning of the roof. here, the roof is called an anaphor, and a house is called an antecedent. the performance of bridging reference resolution is only 40-60%, while pas analysis and srl have reached 70-90% @xcite @xcite . one challenge of bridging reference resolution is the continuous distribution of the strength of the relation between nouns (we call the strength essentiality, hereafter). in the example (2), he, world swimming championships, and 100m breaststroke all modify record semantically but have different essentiality. the most essential information for record is what kind of event the record was set in, i.e., 100m breaststroke. although other phrases, he and world swimming championships, also complement the meaning of record, their essentiality is lower than that of 100m breaststroke. therefore, essentiality varies depending on noun combination and context and has a continuous distribution. although predicates and their modifiers also have essentiality, their continuity is less than that of nouns. predicates have syntactically required highly essential modifiers called arguments. for example, intransitive verbs always have their subject, and transitive verbs always have their subject and object. in contrast to arguments, less essential modifiers are called adjuncts. the argument/adjunct distinction is ambiguous, especially in prepositional phrases. thus the essentiality distributes continuously, like nouns. however, many of the modifiers are syntactically linked to the predicate. on the other hand, few nouns have such syntactic links, making it more difficult to distinguish between essential and non-essential due to many implicit modifiers. despite their continuous nature, existing datasets of bridging references have only a few coarse labels, such as essential, ambiguous, and optional @xcite . this fact suggests that there is a gap between the phenomenon of bridging reference and the annotations in existing datasets. this gap leads to performance degradation in bridging reference resolution. in this work, we utilize crowdsourcing to obtain annotations in which continuous essentiality is considered. crowdsourcing makes it possible to obtain multiple annotations for each example at a low cost. we asked crowd workers to select all nouns that have a bridging reference relation with a given noun. we also asked them to select the most essential one from the selected nouns. we assigned eight workers per example. considering the number of votes as essentiality between nouns, we collected annotations of essentiality on a 16-point scale. we used this method to create a corpus (crowd hereafter) consisting of about 3,900 documents. each document in crowd consists of three sentences, which add up to 11,700 sentences. we compared crowd with an existing corpus annotated with coarse labels by experts (expert hereafter). in the experiment, we trained bridging reference resolution models on crowd, expert, and the combination of them. the models trained on crowd or the combination always outperformed models trained only on expert, which demonstrated the effectiveness of using crowd as a training dataset. our general-purpose crowdsourcing interface is publicly available for further research. foot_0 our constructed dataset and training code are also publicly available. 2 ## existing corpora for bridging reference this section compares our dataset with existing corpora for bridging reference resolution. first, we introduce corpora for english bridging reference resolution, which is most actively studied, and then we describe japanese corpora, which we compare in this work, in detail. ## data construction with crowdsourcing in japanese, a noun pair which has a bridging reference relation can typically be connected by a genitive case "no." 5 in other words, when an anaphor noun b has a bridging reference relation with an antecedent noun a, "a no b" is a semantically valid noun phrase. although it is difficult for non-experts to judge whether two nouns have a bridging reference rela-tion, they can judge whether "a no b" is a valid noun phrase. we showed crowd workers a text in which one word (i.e., noun b) was underlined. we asked them to select all words (i.e., noun a) where "a no b" is semantically valid, based on the contexts. the noun as selected by the workers have continuous latent values of essentiality for the noun b. in order to obtain the continuous essentiality values, we adopt the following strategies: (1) we asked workers to select the most essential noun for the noun b; (2) for each sample, we assigned eight workers to obtain multiple annotations. we constructed the new corpus based on kwdlc @xcite (i.e., expert) in order to evaluate the quality of crowd workers' annotations. as shown in we plan to make the annotations publicly available in the future. workers agreed that the annotations will be used for academic research purposes in a non-personally identifiable manner. ## corpus evaluation in order to verify the quality of the constructed corpus (i.e., crowd), we compared it with the corpus with expert annotations (i.e., expert). for the quantitative evaluation, we define essentiality score for a noun a in crowd as follows. ## evaluation with bridging reference resolution in this section, we examine the effectiveness of crowd for improving bridging reference resolution performance through evaluation experiments. in the experiments, we use three kinds of corpora, expert, crowd, and the combination of expert and crowd, as the training data. we compare the performance of the models trained on each corpus. ## conclusion in the existing datasets of bridging reference resolution, the strength of relations between nouns was unnaturally represented by coarse-grained labels despite the continuous distribution of the strength. we focused on this gap and proposed a crowdsourcing-based annotation method to construct a dataset with more continuous annotations. we have developed a general-purpose interface for the data collection with crowdsourcing. this interface can be applied to crowdsourcing not only for bridging reference resolution but also for any relational analysis tasks. by training with our newly constructed dataset, we improved the performance of bridging reference resolution on a japanese standard benchmark dataset. moreover, the performance improvement was over 16 points, evaluated on the constructed dataset.