MATEO : MA chine Translation Evaluation Online (2023)

Abstract: We present MAchine Translation Evaluation Online (MATEO), a project that aims to facilitate machine translation (MT) evaluation by means of an easy-to-use interface that can evaluate given machine translations with a battery of automatic metrics. It caters to both experienced and novice users who are working with MT, such as MT system builders, teachers and students of (machine) translation, and researchers.

## introduction due to the swift development of evaluation metrics for machine translation (mt) and the absence of up-to-date and user-friendly interfaces, this project aims to bridge the gap by joining together a diverse set of automatic, reference-based mt evaluation metrics, including both established and cuttingedge methods, into a single, easily accessible web interface. it is intended for researchers and practitioners in the social sciences and humanities (ssh) and beyond, also including mt developers and researchers, translation scholars, and experts in the fields of digital humanities and (computational) social sciences. furthermore, the tool can serve as an instructional resource for educators and students because it emphasises the importance of evaluating language resources. it improves the digital literacy of users: being able to easily evaluate machine-generated translations should make users aware not to blindly use mt systems but critically evaluate them for a task, topic, or domain at hand. mateo's web interface is open-source, foot_0 gplv3 licensed, and will be hosted at clarin.eu infrastructure. this project was kick-started with a sponsorship 2021 grant from the european association of machine translation. a substantial follow-up grant was acquired from the clarin.eu bridging gaps initiative. the secured funding accounts for halftime employment for one year at ghent university for the first author of this paper, who is the developer of this project. the project will end at the end of june 2023. ## related platforms similar platforms exist but they are either not maintained or do not provide all the functionality that we are interested in providing. in the past we made use of asiya online foot_1 for teaching mt classes, which provided similar functionality as we are aiming for but unfortunately the service does not work anymore. it also does not support more recent metrics which we would like to include. tilde mt also provides an interface to evaluate mt but it is limited to bleu. foot_2 mt-compareval @xcite is an open-source tool that is similar to our plans but it is rather dated when compared to the current, rapidly evolving landscape of mt evaluation metrics by only providing bleu, precision, recall and f-scores. 4 finally, mutnmt provides an interface to train mt systems in an educational setting but its evaluation methods are limited to bleu, ter and chrf. 5 3 progress mateo is currently in active development. below we describe the work that has been done and which next steps are planned. at the time of writing the beta version of the tool is available foot_5 , which will change considerably in the coming months after submitting this paper. the final version will be delivered in time for the eamt conference. underlying the interface, the tool currently makes use of a general purpose evaluation framework "evaluate" by hugging face for evaluating given machine translations. 7 as part of the ma-teo project, more mt evaluation metrics were added to that framework: nist @xcite , ter @xcite , chrf @xcite , character, charcut. other metrics such as bleu @xcite , bleurt @xcite , comet @xcite , bertscore @xcite were already present in the library and are included in mateo. in terms of the web interface, we have created a streamlit foot_7 website that contains information about the project, the metrics and supported languages, and that allows users to translate and evaluate single-sentence, multi-system machine translations. the translation engine is facebook's m2m model which we included so that users have access to an open-source multilingual baseline system without having to open other translation services. in terms of evaluation, this first version supports sacrebleu metrics (bleu, chrf, ter) bertscore, bleurt, comet. other metrics, as mentioned above, may be added for the final version. users get a bar-chart visualization of the evaluation scores of multiple systems and can download the results as an excel file. the first version of the tool was used in classes on mt at ghent university. students used ma-teo for assignments to improve their mt (evaluation) literacy in late december/early january. we discuss findings from their work in @xcite . they were also asked to give feedback about the usability of the tool which we will analyze in detail and incorporate in new versions of the tool. to complete the project, we have improvements planned, some of which are inspired by the students' feedback. importantly, file uploads for system-wide evaluations will be enabled, and the translation and evaluation components will be separated. the translation engine will be replaced by a more up-to-date model. also, the results of the wmt22 metrics shared task @xcite will be evaluated and promising metrics will be added to "evaluate" (if they are opensource), and ultimately also to the mateo interface. visualizations for edit operations as well as and different export options will also be included in the interface. finally, the tool will be hosted on clarin.eu infrastructure.