A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution (2022)

Abstract: We propose a method for investigating the interpretability of metrics used for the coreference resolution task through comparisons with human judgments. We provide a corpus with annotations of different error types and human evaluations of their gravity. Our preliminary analysis shows that metrics considerably overlook several error types and overlook errors in general in comparison to humans. This study is conducted on French texts, but the methodology is language-independent.

## introduction coreference resolution is still one of the most challenging tasks in natural language processing. several metrics have been proposed to evaluate the task, each of them meant to rectify the weaknesses of the previous ones. however, neither their correctness nor their ability to reflect the real quality of algorithms is easily be provable from their mathematical definition. consequently, some additional tests should be conducted in order to confirm their pertinence. this work aims to compare the evaluation measures used for coreference resolution task with human judgments, i.e. to study them in terms of interpretability. more precisely, b-cubed @xcite , lea @xcite , ceafe and ceafm @xcite , conll-2012 (mela) @xcite , blanc @xcite and muc @xcite metrics will be analysed. ## related work although some properties of coreference resolution quality measures have already been studied in @xcite , moosavi (2020), @xcite and others, to the best of our knowledge, there are no works dedicated to the comparison between automatic measurements and human evaluation of performance for this task. however, very few similar studies were conducted in other domains. @xcite study the interpretability of machine learning models, in general, using application-grounded, human-grounded, and functionally-grounded approaches. foster @xcite describes an experience of evaluating a non-verbal behaviour of an embodied conversational agent. people were asked to choose the most appropriate talking head among the two generated using different strategies. then β inter-annotator agreement measure @xcite was calculated. in @xcite , the correlation between metrics for the dependency parsing task and human judgments was examined. several models were tested for different languages. the annotators had to choose the best of the two annotations predicted by two different models without knowing the correct option. the obtained results were normalised using spearman's ρ and compared with standard metrics. @xcite explore natural language generation (nlg) evaluation measures. the annotation process is organised as follows: an annotator should score an example using three likert scales from 0 to 6 based on informativeness, naturalness and quality criteria. the obtained results were normalised using spearman and intra-class correlation coefficients and compared with nlg metrics. considering these studies, for the present research, we will use an approach similar to @xcite , where the annotators evaluate a system on a likert scale. despite possible difficulties with the likert scale treatment (too many mid-point answers, a broad spectrum of responses for one question, etc.), this method seems more appropriate for our purposes. two main reasons make us choose this approach: (1) we do not test particular systems and, therefore, have no alternative annotations and (2) a scaled approach is more accurate and exact while evaluating a system. ## methodology this section is dedicated to the theoretical description of the methods used in the experiments within the scope of this study. ## experiments and results human evaluation analysis. since participation in this study was not rewarded and contained many ques-tions, it involved only 12 participants, 9 of whom were linguists and 8 of whom have already worked with coreference. the analysis of the three duplicated questions showed that no one answered at random among the annotators. krippendorff's alpha is rather low, so we supposed that some questions in our questionnaire raised more confusion among the respondents than others. therefore we eliminated the questions that contained more than three different answers from the annotators and computed the results only for the remaining simple questions. the total number of questions used in the main analysis is 97. we also decided to compute the inter-annotator agreement on a reduced scale from 0 to 4 points (0 → 0, 1 and 2 → 1, 3 and 4 → 2, 5 and 6 → 3, 7 → 4) and on the gravity (no errors -insignificant error(s) -significant error(s)) and elements importance (no errors -error(s) on peripheral element -error(s) on key element) scales. these agreements are presented in human-machine correlation analysis. in order to compare the obtained scores with human judgments, we calculated an average and a mode of human evaluations having previously transformed to a scale from 0 to 1. every metric was compared with the annotators' assessment on the standard scale, on the reduced scale and on the scale with errors gravity evaluation only. according to the data distributions, in general, the difference between a metric and humans is about 0.33. the averages of differences for all the examples are given in analysis by error type. in order to analyse the influence of a particular error type on a score, we train a linear regression model with the number of errors of each type as the input features and the reversed scores foot_2 as the outputs. all the input features were centered and reduced in order to obtain more stable results. the coefficients that were assigned to each input feature (and which correspond to one of the error types) during the training have been used as a ## discussion human evaluation analysis. ## conclusion this study aims to investigate the extent to which we may understand the results produced by the coreference resolution metrics. the preliminary results on the limited corpus show that metrics underestimate errors gravity compared to humans and add approximately 0.33 points to the final score on the scale from 0 to 1. however, these results need to be proven on a more significant number of annotators. this work's contribution consists in creating the corpus with various errors types and its annotation with the human judgments about the gravity of these errors, the proposal of the new automatic outlying annotator identification algorithm and the suggestion of a methodology of comparison of human evaluations with automatic metrics. all the code and corpus are available at https://github.com/ project178/coref-metrics-vs-humans . possible future work directions may consist in involving more people in the annotation process of the proposed corpus in order to verify the obtained results and in the development of a new metric that will take into consideration the identified shortcomings of the existing measures.