A fri MTE and A fri COMET : Enhancing COMET to Embrace Under-resourced A frican Languages (2024)

Abstract: Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).

## introduction recent advances in machine translation (mt) have focused on scaling multilingual translation models and evaluation data to hundreds of languages, including multiple under-resourced languages @xcite @xcite . however, measuring the progress made for these under-resourced languages accurately is difficult, since popular ngram matching metrics, such as bleu @xcite , meteor @xcite , and chrf @xcite , fail to capture semantic similarity beyond the lexical level @xcite @xcite . variants of these metrics have been developed when scaling to various languages such as spbleu @xcite ), but they often achieve worse correlation to human judgements @xcite when compared to embedding-based metrics like bertscore @xcite , and learned metrics such as comet @xcite . while embedding-based metrics are currently favored for evaluation in mt @xcite , the application of these metrics to under-resourced languages faces three challenges: (1) lack of highquality training and evaluation data significantly hampers the development of reliable metrics; (2) the complexity of the multidimensional quality metrics (mqm) framework @xcite presents a steep learning curve for non-expert bilingual evaluators, complicating the process of obtaining accurate human assessments; and (3) the limited language coverage of multilingual large language models such as xlm-r @xcite restricts their applicability to various lowresource languages @xcite . to address these challenges, recent work have utilized the direct assessment (da) scoring annotations @xcite collected by the organizers of wmt @xcite and leveraged the transfer learning capabilities of multilingual encoders to evaluate unseen languages @xcite . however, the dearth of evaluation data for under-resourced languages such as african languages still remains a significant hurdle in validating these methods. what is worse, as @xcite highlighted, the performance of these approaches is often unpredictable for languages that were not included in the pre-training phase of multilingual language models. in this paper, we address these challenges by enhancing the state-of-the-art comet evaluation metric @xcite to various under-resourced african languages. to overcome the scarcity of evaluation datasets, we create afrimte-a human evaluation dataset focusing on mt adequacy and fluency evaluation for 13 typologically diverse african languages. this is achieved through a participatory research methodology, ensuring a comprehensive and representative data collection process @xcite . in addressing the complexities inherent in the mqm framework, we develop a simplified version that aligns with the tenets of direct assessment (da) and is tailored specifically for non-expert evaluators, aiming to augment both usability and accessibility, thereby rendering the evaluation process more accessible to a wider spectrum of evaluators. finally, we develop the first comet model designed for mt evaluation for african languages. additionally, we introduce the first translation quality estimation (qe) model for african languages, which operates translation quality estimation without requiring reference translations, setting a new benchmark in the qe field @xcite @xcite . to summarize, our contributions are as follows: (1) we propose simplified mqm evaluation guidelines tailored for non-expert translators; (2) to support the application of our guidelines, we develop a specialized annotation tool; (3) we develop a high-quality human evaluation dataset focusing on machine translation adequacy and fluency for 13 typologically diverse african languages; (4) we establish benchmark systems for mt evaluation and quality estimation by employing transfer learning techniques from existing, well-resourced da data and utilizing an african-centric multilingual pre-trained language model; (5) to foster ongoing research in the domain of african machine translation evaluation, we will release all evaluation datasets, code, and models publicly. foot_0 ## afrimte: african machine translation evaluation dataset this section details the data and machine translation engines used for annotation, outlines our annotation guidelines and procedure, describes the data quality assurance process, and presents a quantitative analysis of the collected data. ## africomet: benchmark systems in this section, we will describe the development of our mt evaluation systems for african languages using afrimte. the primary objective of our modeling is to predict the normalized adequacy da score. our investigation centers around three key questions: (1) the feasibility of constructing an mt evaluation system that leverages transfer learning from other languages to african languages, (2) the impact of using african language-enhanced pretrained models for mt evaluation systems, and (3) the potential benefits of an additional mt evaluation dataset in african languages for modeling. our models are based upon the estimator framework @xcite , as illustrated in figure 7 of appendix a.1. in this architecture, the source (src), machine translation (mt), and reference translation (ref) are encoded separately using a multilingual encoder. the resulting word embeddings are passed through a pooling layer to create a sentence embedding for each segment. these sentence embeddings are then combined into a single vector and fed into a feed-forward regressor. the model is trained to minimize the mean squared error. we refer to this as "single-task learning" (stl). furthermore, we adopt a unified approach @xcite , which integrates the tasks @xmath0, mt>, <mt, ref>, @xmath1, mt, @xmath2, feeding all three inputs into the pre-trained model and uniformly distributing weight across the three sentence-level scores for the final score prediction for mt evaluation. we refer to this as "multi-task learning" (mtl). ## reference-free qe systems utilizing adequacy annotations within afrimte, we are also able to develop reference-free models that predict translation quality in the absence of reference translations, aligning with research advancements in translation quality estimation (qe) @xcite @xcite @xcite @xcite . our qe models adhere to the same estimator architecture as africomet, but excluding the reference translation from model inputs. both stl and mtl methods can be applied. however, different from applying mtl in mt evaluation, once the multitask model is trained, it strictly @xmath3, @xmath4, @xmath5. we choose afroxlm-r-l and infoxlm-l for comparison and train our qe models on "wmt others". 12 these models are validated and evaluated using adequacy dev and devtest sets within afrimte. we benchmark our qe systems against prompting gpt-4 with meta-prompt as shown in figure 8 of appendix a.1 and cometkiwi @xcite , which is trained on "wmt others" and leverages infoxlm-l as its encoder. when prompting gpt-4, we encounter certain challenges where the api sometimes fails to return the anticipated quality scores, likely attributable to two reasons: the inherent unpredictability of gpt-4's generative capability and its difficulty in identifying some extremely low-resource languages such as kikuyu, igbo and twi in the qe scenario. 13 we find that for eng-kik, eng-ibo, and eng-twi, error rates are markedly higher than the general trend, recorded at 26.2%, 7.5%, and 11.7%, compared to an overall error occurrence below 5% for other lps. therefore, for each lp, we implement a missing data imputation approach by assigning the mean of outputs from the remaining valid examples to those that get error responses to ensure consistency and fairness in our evaluation. finally, we introduce our benchmark qe systems for african mt: the afroxlm-r-l based stl and mtl models marked with ★ in ## additional evaluation additional evaluations have been conducted on the generalization of our africomet and africomet-qe systems to other datasets. please refer to appendix a.2, a.3 and a.4 for details. ## conclusion this study tackles the challenges of enhancing the comet metric for various under-resourced african languages. we simplify the mqm annotation guidelines for non-expert evaluators, create an mt evaluation dataset, afrimte, covering 13 typologically diverse african languages, and establish benchmark mt evaluation (africomet) and reference-free qe (africomet-qe) systems. our findings show the feasibility of employing transfer learning from well-resourced da data and an african-centric multilingual pre-trained encoder, afroxlm-r, for building mt evaluation and qe models for african languages.