Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective (2024)

Abstract: It is increasingly common to evaluate the same coreference resolution (CR) model on multiple datasets. Do these multi-dataset evaluations allow us to draw meaningful conclusions about model generalization? Or, do they rather reflect the idiosyncrasies of a particular experimental setup (e.g., the specific datasets used)? To study this, we view evaluation through the lens of measurement modeling, a framework commonly used in the social sciences for analyzing the validity of measurements. By taking this perspective, we show how multi-dataset evaluations risk conflating different factors concerning what, precisely, is being measured. This in turn makes it difficult to draw more generalizable conclusions from these evaluations. For instance, we show that across seven datasets, measurements intended to reflect CR model generalization are often correlated with differences in both how coreference is defined and how it is operationalized; this limits our ability to draw conclusions regarding the ability of CR models to generalize across any singular dimension. We believe the measurement modeling framework provides the needed vocabulary for discussing challenges surrounding what is actually being measured by CR evaluations.

## introduction coreference is the broad phenomenon of multiple linguistic expressions referring to the same discourse entity (see examples in ## coreference coreference is itself an unobservable construct; therefore, measurements of coreference must be inferred from indicators, e.g., annotations of coreference in a dataset. in this section, we first describe the coreference datasets we consider in our analysis ( §3.1). we then describe discrepancies in how coreference is defined ( §3.2.1) and operationalized ( §3.2.2) within these datasets. ## methodology this section first provides a high-level overview of the experimental setup ( §4.1), and then describes the details of how the model evaluation is performed ( §4.1.2), as well as the models we consider ( §4.2). our goal is to understand how measurements of generalization might be affected by differing definitions and operationalizations of coreference across datasets. to this end, we empirically examine performance variations for types of coreference that differ in how they are conceptualized and/or operationalized across datasets. ## results to examine how model performance varies across different types of coreference and different datasets, we conduct an extensive disaggregated evaluation. summary results are presented in we observe significant performance variations for types of coreference that are defined differently across datasets. consider the coreference types highlighted in we also observe performance variations when there are operationalization differences across datasets constructed using the same definition of coreference. consider the case of ontogum in open-air markets, bookstores, and a [a bart station all [its]1 own]1 make rockridge another of many pleasant stops . . . so there's nothing too special about [the bed [itself]1]1, but it does have a very important function . . . such differences in how the same type of coreference are operationalized by dataset builders can further distort measurements of both model performance and model generalization. ## discussion one might find it intuitive that measurements of cr model performance and generalization are likely to be impacted by differences in how coreference is defined and operationalized across datasets; and yet, existing multi-dataset evaluations do not explicitly considered how these differences might impact the performance estimates and the conclusions we can draw about models ability to generalize. existing evaluations of model generalization across datasets mostly report accuracy using aggregate metrics @xcite @xcite . such aggregations, however, can obscure certain systematic errors @xcite . understanding of what exactly is being measured requires more explicit consideration of the measurement process. for this reason, we believe measurement modeling provides a useful framework for analyzing measurements in cr. our findings also highlight how relying on aggregated evaluations can obscure why models fail to generalize across datasets. we find this to be especially true in the case of coreference which involves multiple sub-phenomena that are inconsistently annotated across datasets. if we do not resolve inconsistencies and errors in existing datasets, it will remain difficult to accurately interpret performance evaluation results based on these datasets. ## conclusion we propose viewing the measurement of coreference, cr model performance, and cr model generalization from the perspective of measurement modeling. to do so requires clearly distinguishing the theoretical construct intended to be measured from its measurement via a measurement model. this allows one to more clearly consider what is being measured, and possible concerns with the validity of these measurements. through a disaggregated, out-of-domain evaluation of cr models, we have shown that models appear systematically limited in their ability to generalize to certain types of coreference that differ in how they are defined or operationalized across datasets. we showed that models ostensibly display limited generalization for types of coreference even when evaluated on datasets intended to use the same annotation guidelines. if the research goal is to develop cr models that generalize a consistent conceptualization across datasets, iteratively improving in-domain performance will have limited impact, as models' ability to generalize will inherently depend on operationalization differences between datasets and evaluation schemes. 5 measurement modeling as a blueprint. finally, our work provides a blueprint for evaluating outof-domain performance in a way that accounts for known inconsistencies in how coreference is defined and operationalized across datasets. in particular, our work highlights how explicitly considering the definition and operationalization of the construct being measured, as well as disaggregating performance results, can be used to better understand what we can learn from multi-dataset evaluations about models' ability to generalize. this is a necessary step in meaningfully evaluating models of contested constructs across multiple datasets.