Analysis of the Annotations from a Crowd MT Evaluation Initiative: Case Study for the S panish- B asque Pair (2024)

Abstract: With the advent and success of trainable automatic evaluation metrics, creating annotated machine translation evaluation data sets is increasingly relevant. However, for low-resource languages, gathering such data can be challenging and further insights into evaluation design for opportunistic scenarios are necessary. In this work we explore an evaluation initiative that targets the Spanish—-Basque language pair to study the impact of design decisions and the reliability of volunteer contributions. To do that, we compare the work carried out by volunteers and a translation professional in terms of evaluation results and evaluator agreement and examine the control measures used to ensure reliability. Results show similar behaviour regarding general quality assessment but underscore the need for more informative working environments to make evaluation processes more reliable as well as the need for carefully crafted control cases.

## introduction particularly since trainable neural automatic metrics took centre stage in the wmt metrics shared task in 2022 @xcite machine translation (mt) evaluation data sets annotated for quality are becoming essential to develop accurate models. if availing of parallel data with professional references was not difficult enough, we are currently faced with the need to collect data that is not otherwise produced, in other words, while parallel data could be gathered from previously published translations, a sentence-(or text-)level numeric quality assessment most usually needs to be generated for the specific task of metric training. this situation poses a particular challenge for low-resource languages which widens the gap between high-and low-resource scenarios. firstly, because pre-trained models such as comet @xcite might not include language-specific data for small languages and therefore quality predictions can be unreliable, and secondly, because collecting relevant annotated data requires a heavy investment. in this context, resorting to opportunistic data collections with crowd volunteers is increasingly tempting. this new scenario is yet an added reason to increase research efforts on evaluation design. more rigorous considerations of evaluation methodologies and design decisions emerged with claims of human and super-human parity of mt performance @xcite . researchers claimed that evaluations were not rigorous and pointed out issues such as raters' lack of translation expertise, the quality of reference translations, target language interference in source sentences and non-contextualised evaluations as aspects that skewed results in favour of mt contenders @xcite . reports of large evaluation initiatives and thirdparty reviews have shown that little by little evaluation approaches take into account some considerations @xcite and reference campaign such as the annual wmt share task have taken steps to follow best practices for reliable evaluations @xcite . adding to this, research on design-related topics are emerging, such as error methods to opti-mise test set configuration to reduce evaluation effort @xcite , classification schemes adapted to identifying critical errors in neural mt @xcite , document-level and context-aware agreement and effort @xcite , detection of postedited reference translations @xcite and differences between expert and nonexpert evaluators @xcite . several crowd evaluation initiatives have also been reported over the years @xcite even for lowresource languages @xcite that cover a number of design decisions. and yet, best practice and efficiency recommendation guidelines are limited and it is not uncommon that evaluation initiatives specially for low-resource scenarios lack the rigour that would benefit the outcomes the most. in this context, the current analysis is only a small step towards studying the characteristics of crowd-based evaluations within minority language communities. the remaining of this paper is divided as follows section 2 provides a brief description of the evaluation set-up from where the data set under study originated together with the obtained results; section 3 outlines the specific details of the evaluation set-up used to obtain a professional evaluation of the said set as well as the qualitative feedback collected on the task and reports a comparison of evaluation results and the agreement analysis between the crowd volunteers and the professional evaluator; section 4 examines the reliability of the control measures included in the set to identify outlier evaluators; finally, section 5 draws a number of conclusions from the study. ## description of the original evaluation initiative the data set studied in this work is the product of an evaluation initiative to obtain human assessments of mt for two low-resource languages, namely, basque and maltese @xcite . the authors aimed to collect sentencelevel direct assessments to test the potential improvement of the trainable comet metric with language-specific data. the resulting data set for the spanish-basque pair was kindly made available by the researchers for further analysis. 1 in this section, we briefly describe the evaluation setup used by the original research (for further details, see @xcite ) and report the overall results for later comparison. ## professional evaluation in order to explore the similarity between crowd and professional evaluators and their reliability, the author performed the same evaluation task for the complete set. she is a specialist in translation, native speaker of basque (accredited c2-level) and spanish, and has experience in mt evaluation. she will be referred to as the professional evaluator. while the feedback from a single professional is not necessarily indicative of the true annotations the sentence pairs should receive, it can be argued that it provides an educated guess that is consistent across the set to the extent this is possible in human judgement. ## discarding participants when evaluators are not asked to work on a minimum number of sentences, it becomes highly challenging to identify outliers because it is not possible to perform consistent comparisons. as an approximation to uncover unreliable participants, therefore not possible to calculate an individual agreement score. some works have included in the evaluation set sentence pairs for which the quality is known. it usually involves pairing source segments with artificially damaged translations as examples of bad quality output which should be assessed low, and it can also include source sentences paired with their reference translation as examples of excellent quality pairs. evaluators who fail to assess them as poor and good cases within a reasonable range are removed from the task and their contributions excluded from the collection. as described in section 2.2, this is precisely the approach taken by the researchers when running the evaluation task for the data set under study. we looked into these cases to check if these so-called control sentences allow to identify the crowd volunteers which agreed poorly with the professional evaluator. out of the 131 damaged sentences included in the data set, 18 were evaluated with a score of 50 or above by 8 crowd evaluators. out of those 8 evaluators, three have an icc score ranging between 0.75 and 0.80, that is, a good agreement with the professional evaluator; four have an icc between 0.61 and 0.72, a moderate agreement; and one of them a low agreement of 0.085, the lowest of all participants. two out of the three evaluators with a poor icc agreement with the professional evaluator did not assess any damaged translations. the one who did scored both cases shown with scores above 70 points. this might mean that the inclusion of damaged sentences and its implementation was not particularly accurate in this specific set to filter out deviant evaluators. to start, not all evaluators assessed damaged sentence pairs. out of the 44 participants, 36 assessed at least one case. eight were presented with one damaged sentence pair, 28 were presented with 2 to 11 damaged sentence pairs (pearson correlation between the total number of evaluations and damaged sentence evaluations is 0.95). out of them eight failed to pinpoint them as bad quality translations. only five evaluators with multiple damaged sentence pairs scored them 50 or above more than once. the professional evaluator assigned a low score to all the damaged sentences with scores ranging between 1 and 38. if we were to discard crowd annotations based on the assessments of damaged sentences, we would be discarding 373 annotations. if we decided to exclude the work of the 8 evaluators who let us briefly consider the approach used to create the damaged translations. according to the researchers, these were obtained by translating the source sentences with the three mt systems used to create the remaining data set and by replacing a random sequence of words with a sequence of another source sentence. this approach can result in different types of output, from poor quality target sequences mixed with source language sequences to very good quality target sequences mixed with the source language (note that the quality of the mt systems has been rated within an overall range of 61-78 points). evaluators trained in translation might be more aware of the importance of the text as a unit and clearly see that such sentences would be unacceptable for the great majority of contexts. yet this might not be the case for people without translation training. without a context to consider, it is possible that evaluators do not just penalise the translations for the presence of the source but also feel that they should provide positive points for the sequences with a good quality translation. depending on the length of the sentences and the proportion of source words, their location within the sentence and the amount of meaning contained in the correct target sequences, it is possible that some evaluators feel that they are being fair by providing a score above 50 to those translations even when they are fully aware of the truncated sequences. overall, the different results gathered for damaged sentences in this study might indicate that their current design is probably not the most favourable to serve as reliable control sentences. together with damaged translations, reference translations were also included in the data set as a control measure. in total, 86 sentence pairs with references were evaluated. out of the 44 evaluators, 21 were presented with this type of translations: six assessed one case and the remaining 15 assessed from two to 10 cases. out of the 86 reference sentences, 26 were evaluated with a score of 50 or below by 13 crowd evaluators. eight of them, which evaluated two or more of such cases, only assigned this score once, whereas the remaining five assigned a low score in multiple occasions. the professional evaluator assessed 11 of the 86 translations with a score of 50 or lower. out of those 13 evaluators, three have an icc score above 0.9, that is, an excellent agreement with the professional evaluator; 11 have an icc between 0.76 and 0.87, a good agreement; six have an icc between 0.61 and 0.74, a moderate agreement, and one of them a poor agreement of 0.085, the lowest of all participants. once again, two out of the three evaluators with a poor icc correlation with the professional evaluator did not assess any reference translations. the one who did scored the single case presented with a good score of 76, passing the test. this means that the inclusion of reference sentences was not useful in this specific set to filter out unreliable volunteers, on the contrary, by following this test, we would discard good annotations and keep outlier contributions. as was the case with damaged translations, not all evaluators assessed reference sentences. out of the 44 participants, 21 assessed at least one case. six were presented with one reference sentence pair, 15 were presented with two to ten (pearson correlation between the total number of evaluations and damaged sentence evaluations is 0.93). out of them 13 failed to pinpoint them as good quality translations. up to five evaluators with multiple reference sentence pairs scored them 50 or below more than once. if we were to discard crowd annotations based on the assessments of reference sentences, we would be discarding 632 annotations. if we decided to exclude the work of the 23 evaluators who were not presented with any reference translations, we would have to remove another 374 evaluations. this would leave us with a total of 180 evaluations, 15.18% of the total collected. if the reference translations would have served to accurately identify the outliers (based on the icc score), we would have only discarded 49 and 47 evaluations, 8.1% of the total evaluations collected. if we take both control measures into account and combine the performance information of the crowd volunteers, we can account for 37 evaluators out of the 44. if we exclude the work carried out by those who failed any of the tests, we would have to remove the contribution of 18 evaluators, that is, a total of 837 evaluations. let us briefly consider the case of reference translations. they were extracted from established sets or other bilingual data published as parallel corpora @xcite . the quality of reference translations included in test sets has often been questioned and so this was investigated further. if we consider the assessment of the professional evaluator, we see that a score of 50 or below was assigned to seven reference translations out of the 59 presented with scores ranging between 37 and 49. the scores assigned to the remaining references varied from 69 to 100. the range is even wider for crowd volunteers, between 51 and 100. this can be a clear indication that the quality of the reference translations was either not always at a professional level or could not be judged as such out of context. again, this might mean that carefully choosing high quality references and an evaluation set-up that allows to properly assess their quality is important in order to implement an efficient control measure for non-professional initiatives in particular. ## final remarks in this work we have explored an opportunistic evaluation initiative that targeted a low-resource language pair (spanish-basque) to study the impact of design decisions and the reliability of volunteer participants. a translation professional performed the same evaluation task carried out by volunteers. next, evaluation results and agreements were compared and the role of control measures that ensure evaluator reliability analysed. for the analysed set, we can conclude that the overall quality assigned to a mt system might not vary considerably when evaluated by crowd volunteers or a professional evaluator. it remains to be tested if sentence-level accuracy is also as reliable. in terms of task design, we gathered several issues to consider. task design is key in providing a working environment that will allow the evaluator to reduce the level of subjectivity and increase consistency. the feedback from a professional evaluator pointed at the benefit of (highly) contextualised sentences, meaningful evaluation categories, manageable complexity and topic specialisation, translation awareness and source sentence quality. in the same line, identifying outlier contributions seems key to guaranteeing a reliable annotated data set. this being the case, our analysis has demonstrated that while damaged and reference sentences might serve as measures to identify unreliable participants, attention must be paid to creating them. the resulting translations must be unquestionably poor/good so that alternative interpretations are ruled outthen again, it remains to be studied whether participants who properly assess control sentences that are too easily identifiable as poor/good translations will be able to accurately assess regular mt output quality. all in all, we must not forget that these remarks emerge from the analysis of a single data set and a particular crowd volunteer group. in fact, it would be interesting to study if there are commonalities among the characteristics of the crowd volunteer communities of minoritised languages (participant profiles, level of commitment, level of agreement with professional assessment, for example) and whether these are similar to the crowd participants of hegemonic languages. also, this work has explored design issues that are relevant in terms of translation assessment and reliability. however, further research into the real impact of more accurate and cleaner annotations on model training would also be beneficial to determine how rigid (or flexible) an evaluation set-up must be in order to yield useful annotations.