On reporting scores and agreement for error annotation tasks (2022)

Abstract: This work examines different ways of aggregating scores for error annotation in MT outputs: raw error counts, error counts normalised over total number of words (word percentage’), and error counts normalised over total number of errors (error percentage’). We use each of these three scores to calculate inter-annotator agreement in the form of Krippendorff’s alpha and Pearson’s r and compare the obtained numbers, overall and separately for different types of errors. While each score has its advantages depending on the goal of the evaluation, we argue that the best way of estimating inter-annotator agreement using such numbers are raw counts. If the annotation process ensures that the total number of words cannot differ among the annotators (for example, due to adding omission symbols), normalising over number of words will lead to the same conclusions. In contrast, total number of errors is very subjective because different annotators often perceive different amount of errors in the same text, therefore normalising over this number can indicate lower agreements.

## introduction manual error annotation is an increasingly investigated component of assessing the quality of automatically translated or otherwise generated text @xcite @xcite . error annotation can be construed as a word-span labelling task, and a variety of different error labelling schemes have been proposed @xcite @xcite @xcite . if annotators are instructed to assign a predefined error type to given word-spans, the task is called error classification @xcite @xcite . if annotators are asked only to mark the erroneous word-spans without assigning any particular error type, the task is called error marking @xcite . both for classification and marking, approaches differ in how they report the results from error annotation, i.e. how they convert the word-span labels obtained in the annotation process to aggregated, quantified results on the basis of which conclusions can be drawn. it is currently unclear how the choice of aggregation method affects conclusions, e.g. when comparing systems, or assessing inter-annotator agreement and reproducibility. this paper aims to investigate three widely used ways of aggregating error scores, namely (i) raw error counts, (ii) error counts normalised by total number of words (referred to in this paper as 'word percentage'), and (iii) error counts normalised by total number of errors (referred to as 'error percentage'). we carried out our experiments on a publicly available data set consisting of annotated machine translated outputs which was also used in previous work @xcite . ## related work in an early paper reporting an approach to systematic error classification in mt outputs, @xcite analyse several mt systems for two language pairs in order to obtain details about their weakest points. the results are reported as error percentages (in the above sense) for each defined error type in order to see which error types are predominant in each of the mt systems. since in this work different annotators evaluated different texts, it was not possible to report inter-annotator agreement in any manner. @xcite specifically address iaa for error classification in mt using the mqm 1 error scheme. iaa was calculated using both pairwise matching/accuracy between assigned labels on the word level as well as cohen's κ coefficient on the word level. although the work did not aim to evaluate mt systems, some overall results are reported: raw error counts as well as what might be called 'sentence percentage' -error counts normalised by the total number of sentences. it should be noted that the error counts did not take into account the number of words in the error span, but each span is counted as one error no matter how many words the span comprises. lommel et al. also investigated sentence-level agreement, which led to much better agreement since evaluators often identified the same error types in a sentence although on different spans. these differences in spans are mentioned as the main reason for low κ coefficients, apart from annotators' personal preferences. another work which deals with iaa (castilho, 2020) for error classification compares (a) iaa when evaluators see only isolated sentences, with (b) iaa when evaluators see larger portions of text in context (paragraphs, "documents"). since the goal was not analysis of mt systems, a simple scheme involving four error types was used. the tool used for error annotation allowed only marking on the sentence level, so that cohen's κ is also calculated on the sentence level. in addition, pearson's r calculated on error counts and percentage of matched error types are reported as well. since the focus of the work was fully on iaa, no error scores were reported. @xcite use the mqm error scheme for error classification to compare phrasebased and neural based systems for translating into croatian. they mention that there is no standard in reporting numerical results for error classification and discuss two possibilities: (1) counting only error spans (without taking the word span into account), (2) counting number of words in error spans and normalising this word count over the sentence length (what we call word percentage). they argue that although raw error counts provide useful information, they do not enable drawing statistically meaningful conclusions about the results. they further argue that different mt systems may generate different sentence lengths which would make comparison of raw error counts unfair. therefore, they decide to report the word percentage. as for iaa, cohen's κ on the sentence level was calculated and compared across different error types. they obtained higher agreement coefficients than @xcite and commented that the reason is calculating κ on sentence-level instead of word-level. @xcite perform error marking (annotation without classification into types) for use in a loss function in order to improve an mt system. therefore, no scores were reported, and iaa was reported as krippendorff's α calculated on raw error counts. the same approach, error marking and α calculated on error counts, was used by @xcite , together with the percentage of label matches (f-score). the error annotation results were reported in the form of word percentages. @xcite apply the mqm error scheme on large amounts of texts for two language pairs. they use raw span counts weighted with the error severity so that each minor error span is counted as 1 and each major error span as 5. a restriction of a maximum of five errors per segment is applied, and evaluators are instructed to choose the five most severe errors. as for iaa, for each rater freitag et al. report the score and the ratio over the average score of all raters. they also report pairwise agreement, and discuss that there is no obvious best way to compute it, especially since they take into account error severity. the chosen approach groups the scores of each rater into numeric intervals, and then compares these intervals. the authors report average, minimum, and maximum pairwise annotator agreements, but do not specify how exactly these agreements were calculated. a previous investigation @xcite ) also concentrated on iaa, with the focus on pairwise word level matches for different error types, and reported overall krippendorff's α calculated on error counts. the work reported in this paper was carried out on part of the same corpus, but with the following differences: (i) we introduce krippendorff's α and pearson's r calculated on three types of numerical scores, and (ii) we compare all iaa coefficients (including word overlap) and analyse the differences in depth. also, we choose a subset of the corpus for which we have annotations by four evaluators as a result of an earlier reproduction study @xcite , in order to have more variation. ## annotated data set our experiments were carried out on a subset of the publicly available annotated qrev data set. 2 the full set consists of english user reviews about qrev data set language pair @xmath0, comprehension imdb movies and amazon products translated into croatian and serbian by five different mt systems for each language pair. each text was annotated by two different evaluators. in the sub-set we used foot_2 (popović and belz, 2021), we had additional annotations obtained in a reproduction study @xcite ; therefore in this subset each text was annotated by four different evaluators. the sub-set consists of croatian mt outputs generated by three systems (amazon translate, microsoft bing and google translate). in total, 14 different evaluators, computational linguistics students and researchers as well as translation students fluent in the source language and native speakers of the target language, participated in the annotation of this sub-set. some descriptive statistics for the data sub-set we used are shown in the annotation of mt outputs was carried out in two stages. in the first stage, a group of annotators provided error marks (major, minor and none for correct words) according to two quality criteria: adequacy and comprehension. for both quality aspects, the evaluators were asked to concentrate on problematic parts of the text and to highlight them. for adequacy, they were instructed to highlight parts which entirely or partially change the meaning of the source text. for comprehension, they were asked to mark parts which are impossible or hard to understand. they were also asked to add omission tags "xx" whenever necessary. since it was an error marking task, i.e. not guided by any predefined error scheme, only by the quality criteria, the evaluators had more freedom in annotating errors than in error classification tasks. therefore, this annotation represents a descriptive type of evaluation @xcite which allows and encourages subjectivity. the annotators were also asked to distinguish between major and minor errors, however we did not use this distinction in the experiments reported here. this could be an interesting direction for future work. error types were assigned in the second stage by the first author of this paper, a computational linguistics researcher with expertise in translation, who analysed the marked errors and assigned error type labels according to their cause and/or origin. the error types were not predefined by any particular error typology, but defined on the fly, while looking at the annotated text. for some words, multiple error types were identified. some of these error types have small word spans (1-2 words) while others can involve a larger number of words, even the entire sentence. since the evaluation protocol in the first stage allowed free annotation, evaluators often perceived the same issue but marked different words. this freedom also allowed evaluators to express their individual stylistic preferences. moreover, it led to evaluators often marking up many consecutive words even in cases where only some of the words were actually part of an error. in some of these cases, the second-stage annotator was unable to identify an error type; to indicate this, such cases were tagged as none. ## aggregated error scores we investigated the following aggregated scores: 1. error count c(err_type) ## inter-annotator agreement we estimated inter-annotator agreement in the following three ways: ## results as a first step, we calculated iaa measures for all error types combined, taking into account only whether a word is tagged as an error or not, independently of the error type. we compared iaa measures for all aggregation methods except calculating coefficients on error percentage (count of a particular error type normalised over the total number of errors) because without distinguishing error types it would simply be 100% and therefore does not make sense. after observed certain tendencies in this first step, we calculated all values for each individual error type and analysed all observations in depth. ## analysis of differing agreement levels decreased agreement when using error percentage. for a large number of error types, calculating krippendorff's α and pearson's r on error percentage results in much lower agreement compared to using raw counts or word percentage. in order to explain this phenomenon, further analysis was carried out on these error types, and a systematic pattern was found: the total number of errors marked by different annotators often varies notably so that identical error counts become very different error percentages. in the second sentence (lower half of the table), all four annotators perceived one ambiguity error, but due to differences in perception of other error types (in this case presence and span of negation errors) the total number of errors, and therefore the error percentage, are different. this finding indicates that, regardless of which score is considered best for reporting the results of the analysis, error percentage is not suitable for calculating agreement coefficients. increased agreement when using word percentage. for some error types, coefficients calculated on word percentage are associated with higher agreement than those calculated on raw counts and error percentage. one of these error types is adequacy-related omission, which at first might look contradictory to the findings in the overall scores, where they contribute to decreased agreement by changing sentence length. however, increasing sentence length has another effect, namely 'smoothing' the number of omissions. an example can be in besides omission, such tendencies can be seen in a few other error types (verb form error for adequacy, noun phrase and source errors for comprehension). however, the analysis revealed that in those cases, there are no reasons related to the nature of the error type itself (as is the case for omissions). the only reason is that these error types often occur in sentences where lengths are different due to tokenisation changes and/or omission annotations, as mentioned in section 6.1. other differences in agreement. as previously mentioned, we carried out a more in-depth analysis of a few other kinds of differences in agreement related to certain error types. one phenomenon is that both word percentage and error percentage decrease the agreement for certain error types including gender, case (adequacy), named entity (comprehension), etc. we have also observed cases where word percentage notably decreases agreement while error percentage does not, e.g. for pos ambiguity (comprehension); in fact, conjunction error percentage even increases agreement (comprehension). nevertheless, these differences in agreement are not related to error type, but rather to the circumstances in which the majority of errors occur (variations in sentence length and total number of errors). moreover, many of the error types are relatively rare, which makes these effects even stronger. for example, conjunction errors are very rare, and almost all of them are marked by annotators which assign a higher total number of errors, too, so that error percentage is 'smoother' than raw error count. it should also be noted that while the results reported here indicate that error percentage is often associated with misleadingly low agreement, it is theoretically possible to find misleadingly high agreement (for example, if one annotator marks twice as many errors in total than another, but the proportions of error types are exactly the same, the agreement on error percentages will be perfect). ## conclusions this paper examined different ways of aggregating scores for error annotation in automatically generated text, more specifically mt outputs: we compared raw error counts, error counts normalised over total number of words (word percentage), and error counts normalised over total number of errors (error percentage), and the associated difference in inter-annotator agreement measures calculated on the different aggregated scores. while reporting each type of aggregated score as the evaluation result has its advantages depending on the goal of the evaluation, our experiments indicate that the overall best way to estimate inter-annotator agreement using such scores are raw counts. if the annotation process ensures that the total number of words cannot differ among the annotators (for example, due to adding omission symbols or separating punctuation marks), word percentage will lead to the same conclusions. in contrast, error percentage can be associated with misleading agreements for a number of error types, chiefly because the total number of errors is subjective as different annotators often perceive different numbers of errors in the same text. therefore, normalising one subjective number (error count for a particular type) by another subjective number (total number of errors) can notably influence the agreement.