Evaluating Coreference Resolvers on Community-based Question Answering: From Rule-based to State of the Art (2022)

Abstract: Coreference resolution is a key step in natural language understanding. Developments in coreference resolution are mainly focused on improving the performance on standard datasets annotated for coreference resolution. However, coreference resolution is an intermediate step for text understanding and it is not clear how these improvements translate into downstream task performance. In this paper, we perform a thorough investigation on the impact of coreference resolvers in multiple settings of community-based question answering task, i.e., answer selection with long answers. Our settings cover multiple text domains and encompass several answer selection methods. We first inspect extrinsic evaluation of coreference resolvers on answer selection by using coreference relations to decontextualize individual sentences of candidate answers, and then annotate a subset of answers with coreference information for intrinsic evaluation. The results of our extrinsic evaluation show that while there is a significant difference between the performance of the rule-based system vs. state-of-the-art neural model on coreference resolution datasets, we do not observe a considerable difference on their impact on downstream models. Our intrinsic evaluation shows that (i) resolving coreference relations on less-formal text genres is more difficult even for trained annotators, and (ii) the values of linguistic-agnostic coreference evaluation metrics do not correlate with the impact on downstream data.

## introduction coreference resolution is the task of determining the expressions of the text that refer to the same entity. modeling coreference relations is a key step for understanding the meaning of the text that can benefit various tasks like machine reading comprehension @xcite , summarization 1 our code and coreference annotations on cqa datasets are publicly available at: https://github.com/ haixiachai/coref_cqa @xcite , and dialogue processing @xcite . the progress in coreference resolution is tailored to improve the performance on available coreference resolution datasets @xcite @xcite @xcite , but it is not clear how this progress translates to downstream applications. in this paper, we take a new perspective to directly evaluate the impact of coreference resolvers on a downstream task. first, we implement the extrinsic evaluation of coreference resolvers on the task of community-based question answering (cqa), in which the task is to select the correct answer given a question and a set of candidate answers. answers in cqa are often very long, and they contain multiple referring expressions in each answer. to do so, we use existing coreference resolvers for decontextualizing candidate answersi.e., replacing less informative nouns and pronouns with their most informative antecedent -so that the containing information in each individual sentence would be more standalone. to ensure that the resulting effects are not specific to a single dataset, domain, or downstream model, our settings cover multiple text domains and encompass several cqa methods. second, we provide coreference annotations on a subset of answers from two cqa domains to enable intrinsic evaluation of coreference resolvers on a downstream data. we evaluate several coreference resolvers from the rule-based system @xcite to the stateof-the-art coreference resolver @xcite using our extrinsic and intrinsic evaluation setups. the results of our extrinsic evaluation show that (i) rule-based system has a more positive and less negative impact on cqa compared to neural coreference resolvers, (ii) while there is a significant difference between the performance of the rulebased system vs. state-of-the-art neural model on coreference resolution datasets, we do not observe a considerable difference on their impact on cqa models. this means that intrinsic evaluation has to be accompanied by extrinsic evaluation, (iii) the impact of coreference resolution is different on various cqa methods. thus, we suggest to consider the overall impact on multiple cqa models in order to investigate the effect of a coreference resolver on cqa, and (iv) coreference resolvers are most beneficial when both training and test data are decontextualized, and the rule-based system has consistent impact on different domains of the data while the state-of-the-art neural models have a considerable different impact on different domains. our extrinsic evaluation results show that (i) resolving coreference relations on less-formal text genres -like ones in the stack exchange answers -is more difficult even for trained annotators, and (ii) the results of linguistic-agnostic coreference evaluation metrics do not correlate with the impact of coreference resolvers on downstream data. ## coreference for answer selection given a question, the task of answer selection is to find the correct answer among the set of candidate answers. we use answer selection datasets from community question answering (cqa). cqa questions are non-factoid and they often require answers with descriptions or explanations. therefore, cqa answers are long multi-sentence texts. with the length of answers, the use of less informative expressions like pronouns increases. this presents a challenge for answer selection methods that mainly rely on the lexical forms to compute the similarity of the candidate answers to the question. especially, when cqa data is collected by using a search engine or the answers to the similar questions for candidates, incorrect answers also have high lexical similarity with questions. using coreference resolvers for decontextualizing individual sentences in answer selection datasets makes correct answers more similar to the question and incorrect ones more dissimilar. q: do i need a uk visa to enter uk from ireland? a1: what is your nationality? according to the [uk] 1 government service information website (url), people from the countries who are mentioned in url would still need to acquire a visa to enter [the country] 1 . a2: data sharing means only that they share data, so while [the officers in [ireland] 6 ] 3 are able to see details of [your] 4 failed uk visa when [they] 3 process [[your] 4 irish visa] 5 , that doesn't mean [you] 4 will be refused to get [the visa] 5 to enter [the country] 6 . a1 and "get the visa to enter the country" in a2. these two text sequences can be easily discriminated given coreference information, i.e., "need to acquire a visa to enter uk" in a1 and "get your irish visa to enter ireland" in a2. ## extrinsic evaluation on cqa the following sections describe different components for the extrinsic evaluation of coreference resolvers using cqa. figure 1 shows the flow chart. ## intrinsic evaluation on cqa data we enable intrinsic evaluation of coreference resolvers on cqa data by annotating coreference relations on a subset of the cqa data. we annotate a subset of examples from the travel and cooking test sets. we use mmax2 @xcite for the annotations. 8 the annotations are done by six bachelor and master students with nlp background from the departments of computational linguistics and computer science. they received a minimal training for coreference resolution and the mmax2 annotation tool. travel cooking answers 389 558 max words/answer 319 283 coreference chains/answer 4.2 3.4 mentions/answer 14.0 12.3 while our agreement study shows a high interannotator agreement, we also perform a manual error analysis on the resulting annotations. 10 based on our analysis, annotating coreference relations in less-formal genres is more difficult than in the common genres in existing nlp datasets, e.g., news, and their error-free annotations would require expert linguists. 11 in particular, human annotations in travel contain more errors. this indicates that resolving coreference relations of the answers in the travel domain, which contains more nominal expressions, is more difficult than cooking. ## examined coreference resolvers we evaluate four different coreference resolvers. first, the stanford rule-based system @xcite that uses heuristic rules like string match for resolving coreference relations. there is a considerable gap between its performance and stateof-the-art coreference resolvers on the conll-2012 test set. however, it has a reasonable performance across different domains @xcite . second, deep-coref @xcite , which is a neural coreference resolver. deep-coref is a neural model that first extracts candidate mentions using syntactic information. for each candidate mention, it scores all preceding mentions to select the best scoring one as the antecedent. it also includes a dummy antecedent to determine nonanaphoric mentions, i.e., if the dummy antecedent has the highest score, the mention is non-anaphoric. third, e2e-coref @xcite that is an endto-end neural coreference resolver and the base model for the majority of state-of-the-art coreference resolvers since 2018. unlike deep-coref and last, bert-coref @xcite that is one of the most recent state-of-the-art coreference resolvers on the conll-2012 dataset. bert-coref is an extension of e2e-coref by replacing the bidirectional lstm encoder with spanbert encodings. concretely, we use the spanbert-large language model, which has a novel span masking pretraining objective that predicts the entire masked span instead of individual tokens. for the reported extrinsic and intrinsic evaluations, the supervised coreference resolvers are trained on the english conll-2012 dataset. ## discussion as mentioned in §7, there are many ways to incorporate coreference information in qa. in this work, we make it at the input-level by decontextualizing the input sentences. this makes the extrinsic evaluations efficient and enables evaluating any coreference resolvers on any downstream models and datasets. on the downside, the decontextualization results in unnatural sentences in some examples, which may negatively impact the downstream model. for instance, we observe that most coreference resolvers have a negative impact on sentence-bert in ## conclusions coreference resolution is an important step for text understanding. the main shortcoming of recent developments in coreference resolution is that they mainly target improving the performance in standard coreference datasets. however, coreference resolution is not an end-application and it is not clear how the progress in in-domain evaluations translates into downstream tasks performance. in this work, we enable direct extrinsic and intrinsic evaluation of coreference resolvers on downstream models and data, respectively. for the extrinsic evaluations, we use coreference resolvers for decontextualizing the input sentences in communitybased question answering (cqa) task. for intrinsic evaluation, we have annotated a subset of cqa data with coreference relations. our extrinsic evaluations suggest that (1) while there is a significant gap on the performances of state-of-the-art coreference resolver and the rule-based system on coreference datasets, the rule-based system has a more consistent and positive impact on cqa while the impact of the state-of-the-art model can considerably vary based on the domain of the downstream data, and (2) using coreference resolvers for decontextualizing both training and test datasets is more beneficial than decontextualizing the test data. our intrinsic evaluations suggest that there is a discrepancy between the rankings of existing coreference resolution evaluation metrics and the resulting rankings from the extrinsic evaluations. this suggests that intrinsic evaluation on conll should be accompanied by extrinsic evaluation to approximate the utility of the coreference resolvers for downstream tasks.