The OPUS - MT Dashboard – A Toolkit for a Systematic Evaluation of Open Machine Translation Models (2023)

Abstract: The OPUS-MT dashboard is a web-based platform that provides a comprehensive overview of open translation models. We focus on a systematic collection of benchmark results with verifiable translation performance and large coverage in terms of languages and domains. We provide results for in-house OPUS-MT and Tatoeba models as well as external models from the Huggingface repository and user-contributed translations. The functionalities of the evaluation tool include summaries of benchmarks for over 2,300 models covering 4,560 language directions and 294 languages, as well as the inspection of predicted translations against their human reference. We focus on centralization, reproducibility and coverage of MT evaluation combined with scalability. The dashboard can be accessed live at https://opus.nlpl.eu/dashboard/ .

## introduction the main motivation behind the opus-mt dashboard is to provide a comprehensive overview of open translation models. we focus on a systematic collection of benchmark results with verifiable translation performance and large coverage in terms of languages and domains. the landscape of machine translation (mt) is increasingly blurry and incomprehensible due to the growing volume of shared tasks and models published within the community. even with established events such as the conference on machine translation (wmt), a complete picture of translation performance is hard to obtain. in addition, large multilingual language and translation models push the language coverage making it difficult to keep an eye on the state of the art for particular language pairs. one additional problem is that most results reported in scientific and non-scientific channels come from selected benchmarks and model performance and are not explicitly verified by a careful replication study. in various cases, new models come with their own benchmarks and do not consider a wider evaluation across domains. training data is complicated to control and the danger of over-fitting to specific scenarios is apparent. figure 1 illustrates the substantial differences one can observe across benchmarks when comparing two competing models. our dashboard is an attempt to carefully provide a summary of results using an extendable collection of benchmarks with the largest language coverage possible accommodated with procedures to translate and evaluate in a standardized and consistent setup. the focus is clearly set on publicly available translation models as we want to emphasize translation results that we can replicate and verify from our own experience. the system is designed with the following requirements in mind: the implementation and all data files are available on github in public repositories and details about the components and implementations are given below. we start by a brief description of the background before discussing the collection of benchmarks and translation evaluations. the main features of the web interface are listed thereafter and we finish up with links to related work and an outlook into future developments. ## background and motivation the main motivation of the dashboard is related to our own initiative on building open translation models under the umbrella of opus-mt. the development of mt accelerated in recent years making it difficult to obtain a clear view on performance for individual language pairs. in contrast to related work, the dashboard is not intended to serve as a new mt evaluation service but rather as a central point of comparison between opus-mt and other publicly available models. user-provided translations are also supported as another point of reference but the focus is set on verifiable translation results produced by the system itself. opus-mt is based on opus @xcite , the major hub of public parallel data, the main ingredient for training modern translation models. it refers to an initiative to systematically train translation models on open data sets using the efficient nmt framework provided by marian-nmt @xcite . the project includes both bilingual and multilingual transformer models of different sizes with streamlined procedures to scale up language coverage and availability @xcite . opus-mt models are released with permissive licenses and can easily be integrated in existing workflows @xcite @xcite . currently, there are over 2,300 public models -release information is part of the dashboard. 1 the models cover over 290 languages and provide translations for 2,549 language pairs and 4,560 language directions. the sheer volume of opus-mt releases call for a systematic evaluation to monitor progress and support model selection in practical applications. ## collecting mt benchmarks mt benchmarks are developed for various tasks and domains and their distribution differs depending on the preferences of the original provider. in order to make it easier to systematically compare mt across available benchmarks, we collect known testsets in a unified and consistent format in a public repository foot_1 (opus-mt-testsets). the current collection covers benchmarks from translation tasks at wmt @xcite , tico19 @xcite , the tatoeba translation challenge @xcite , flores v1.0 @xcite , flores-101 @xcite and @xcite and multi30k @xcite . each benchmark may be comprised of several testsets (different years or dev and test sets). altogether we cover over 44,000 language directions with an average length of 1,945 sentences per testset. one important thing to note is that the multilingual benchmarks are typically english-centric in a sense that the original text has been translated from english to other languages. figure 2 illustrates the skewed distribution and we encourage suggestions foot_2 and developments of additional sources to change that picture. we sort benchmarks by language pair using iso-639-3 language codes and use a simple plain text format with utf-8 encoded data. translated segments (typically sentences) appear on the same line in separate files for the input text in the source language and reference translations in the target language. the file name corresponds to the benchmark name and the file extension specifies the language by its iso code. additional files may provide additional metadata such as domain labels or source information. we also add extensions about the writing script if necessary. here are a few examples from the collection: eng-deu/newstest2020.deu eng-deu/newstest2020.eng srp_cyrl-fin/flores200-devtest.fin srp_cyrl-fin/flores200-devtest.srp_cyrl currently, we do not support multi-reference benchmarks. instead, additional reference translations are stored as separate benchmarks. document boundaries may be specified by empty lines. other formatting information is not supported. we will extend the repository with additional test sets including ntrex @xcite , samanantar @xcite , iwslt @xcite and data from the masakhane project @xcite . ## systematic evaluation of public models we store the results of our systematic evaluation in three different public git repositories, depending on the type of model: (i) opus-mt models foot_3 , (ii) external models foot_4 , and (iii) user-contributed translations 6 . we emphasize a lean design avoiding the hassles of setting up and maintaining databases and additional services. each leaderboard repository follows the same structure and is divided into two main parts: (i) leaderboards for each benchmark and language pair and (ii) the scores for each individual model. file structures are organized accordingly and the setup makes it possible to easily scale the collection to a large number of models, benchmarks and language pairs. the inclusion of new evaluation benchmarks is also straightforward as we have separate files for each of them. the main file structure looks like this: scores/<@xmath0.txt models/<@xmath1.<@xmath2.txt source and target language ids (<src>, <@xmath3. supported metrics are currently comet @xcite ), foot_6 bleu @xcite and chrf @xcite with the two variants chrf++ @xcite and sentence-piecebased subword bleu @xcite . we use sacrebleu @xcite and unbabel-comet foot_7 to compute the results. we add the readily available chinese, japanese and korean tokenization features in sacrebleu and the flores-200 sentence piece model for subword splitting. models are sorted by provider (<@xmath4; and the model name (<@xmath5. for example, opus-mt models are grouped by languages they support, which may be a single language pair or pairs of language groups. besides benchmark-specific score tables, we also compile aggregated score tables for each language pair. those tables list an average score over several available benchmarks (avg-<@xmath6.txt) and the bestperforming model for each available benchmark (top-<@xmath7.txt). automatic makefile recipes are used to update those tables if needed. we keep separate tables for the three categories (opus-mt, external models, usercontributed translations) in each of the respective repositories. furthermore, the repository includes the procedures for translating and evaluating models with respect to the benchmarks collected as described in the previous section. the translations are systematically run with the same hyperparameters. these can be consulted in appendix a. the implementation streamlines the creation of batch jobs and enables a scalable approach that allows a straightforward update of leaderboards with new models and benchmarks. additional evaluation metrics can also be integrated by implementing appropriate recipes. the general workflow for evaluating models and updating score tables is divided into three steps: (i) translating and evaluating all benchmarks for all language pairs supported by a model, (ii) registering new scores to be added to existing leaderboards, and (iii) updating all affected leaderboards and sorting by score. gnu makefile recipes are used to properly handle dependencies. using revision control as the backend storage makes it possible to recover from errors and mistakes. we distinguish between internal and external models but the basic workflow is the same. we anticipate that the publication of this paper will attract some interest allowing us to harvest additional user-contributed translations. we report statistics in ## mt performance dashboard for the dashboard web-interface, we emphasize a lightweight implementation. in our system, we want to avoid a complex backend and heavy frontends and rather focus on lean and responsive functionalities. the interface has minimal requirements and basically runs with a standard php-enabled web server. data is automatically fetched from the opus-mt storage, github or the local file system. deployment requires no further installation or database setups. the frontend uses cookies and session variables to speed up the process but can also run without them. server-side caching is used to enable fast response time and no heavy graphics or animations are used that would slow down data transfer and client-side website rendering. ## related work in the current scenario of nlp, which is characterized by the increasing number of available models versus the constant lack of systematic documentation, von werra et al. (2022) identify three challenges: (1) reproducibility, the replicability of model performance, (2) centralization to avoid the duplication of workload and one single point of reference, and (3) coverage, the inclusion of a diverse set of metrics and languages, as well as pointers for efficiency, bias and robustness. one of the most well-known efforts on reproducibility is papers with code foot_9 (pwc), an opensource web-based platform that links papers with their implementations and includes many features, such as benchmark evaluations and information on the use of dataset. the huggingface repository also aims at solving the issue of reproducibility with their evaluation on the hub (von werra et al., 2022), a user-friendly platform that allows large-scale evaluation of any of their publicly available models. this feature holds significant potential, however, two primary concerns arise. firstly, the evaluation is not done systematically, as assessments are performed solely upon user request. secondly, although the openness of the hub to all users is a move towards democratizing ml, it also results in an absence of metadata for the uploaded resources that make it difficult to find suitable models for every dataset. yet another similar tool is dynabench @xcite , a research platform for dataset creation and dynamic model benchmarking for a wide range of nlp tasks. generic platforms mentioned above are useful but make it difficult to get a comprehensive overview of one specific task. mt has a long tradition of shared tasks and several systems have been developed to visualize and analyze benchmark results. the wmt matrix foot_10 was a dedicated platform for submitting and archiving submissions to the translation tasks at wmt. it provided a useful overview of the state-of-the-art in those tasks, but the system is down. mt-compareeval @xcite partially replaces that service with its wmt instance. 15 the system itself provides a modern tool for the analysis of mt output with various plots of automatic evaluation metrics and an interface for comparing translations with highlighted differences. the software is open source and can be deployed with the option to upload additional system translations, which is also used by the developers to host their own experimental results. 16 in contrast to the opus-mt dashboard, it does not implement the replication of translations to verify provided results. another open-source tool, compare-mt @xcite , has been developed to explore translation outputs. it supports a deeper analysis and comparison using detailed statistics of word-level and sentence-level accuracies, salient n-grams, etc. on the commercial side, there is intento, a language service provider that publishes a yearly report with an overview of current mt systems @xcite with a focus on commercial models. they provide an end-to-end mt evaluation service that comes with a cost and the yearly evaluation is not transparent and open. ## conclusions and future work the opus-mt dashboard implements a simple yet comprehensive interface for a systematic evaluation of public translation models. the main purpose is to provide an overview of opus-mt models and to relate their performance to other openly available models. the focus is on verifiable performance and a centralized evaluation procedure. the workflow and collection stress transparency and replicability and can easily be extended with new models and benchmarks. the current implementation is fully functional but we already work on several extensions. first of all, we would like to integrate more information about the model properties in the dashboard. important features are model size, inference time and computational costs that can be related to translation performance. additionally, we want to tag other important characteristics such as multilingual versus bilingual models. heatmaps for comparing multilingual model scores are also on our to-do list as well as better overviews of top-scores in multilingual benchmarks. we also want to integrate our geolocated visualization of language coverage implemented in opus-mt map. 17 finally, we are continuously working on the integration of new benchmarks and the systematic evaluation of available models. we look into other released models and their use for replicating benchmark results. we also continue to collect benchmarks and will integrate sentence-level scores while browsing through translation output. we may also connect to other systems like mt-compareeval for more detailed analyses.