MT - LENS : An all-in-one Toolkit for Better Machine Translation Evaluation (2025)

Abstract: We introduce MT-Lens, a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks, including translation quality, gender bias detection, added toxicity, and robustness to misspellings. While several toolkits have become very popular for benchmarking the capabilities of Large Language Models (LLMs), existing evaluation tools often lack the ability to thoroughly assess the diverse aspects of MT performance. MT-Lens addresses these limitations by extending the capabilities of LM-eval-harness for MT, supporting state-of-the-art datasets and a wide range of evaluation metrics. It also offers a user-friendly platform to compare systems and analyze translations with interactive visualizations. MT-Lens aims to broaden access to evaluation strategies that go beyond traditional translation quality evaluation, enabling researchers and engineers to better understand the performance of a NMT model and also easily measure system’s biases.

## introduction neural machine translation (nmt) models are typically evaluated using automated metrics that compare the model's translated outputs to one or more reference translations. recent neural-based evaluation metrics have demonstrated high correlations with human judgments @xcite @xcite replacing canonical overlap-based metrics @xcite . while automated metrics provide an essential means for assessing quality improvements and have become the way to go in evaluating state-of-the-art nmt systems, they only offer a general intuition of the model's overall performance and often lack interpretability @xcite . as such, the reliance on these metrics raises concerns about their specific error types that might affect the end-user experience, such as reinforcing gender bias @xcite , translationese or language mismatch @xcite . to address these problems, recent works have focused on developing inherently interpretable metrics for mt evaluation @xcite ) via token-level annotations that offer a more granularinsight at the segment level. however, such limitations necessitate more granular evaluation tools that can dissect translation outputs to identify and analyze specific error types and their impact on overall quality, thereby enabling mt engineers to make more informed decisions when evaluating translation systems. moreover, existing evaluation methodologies in state-of-the-art nmt models primarily focus on evaluating translation quality, frequently overlooking other equally critical evaluations like gender bias, added toxicity or robustness to misspellings. these biases and harmful outputs can have significant consequences for users @xcite , highlighting a need for evaluation strategies that go beyond quality to encompass a broader range of tasks. mt-lens seeks to address these critical gaps by providing a unified framework to test generative language models on a number of different machine translation evaluation tasks and providing a userfriendly interface to analyze the results. mt-lens is based on lm-eval-harness @xcite which has been widely used for evaluating llms in several natural language understanding (nlu) tasks. building upon this comprehensive framework, mt-lens extends the evaluation capabilities of lm-eval-harness for nmt. contributions of this framework are listed as follows: ## related work typically, automatic metrics like bleu @xcite are simply applied and reported at the corpus level. in recent years, the evolution of mt evaluation has seen the development of tools that offer more granular insights. mt-compareeval @xcite provides comparative analysis of segment-level errors, focusing on identifying differences in n-grams between two mt outputs. similarly, compare-mt @xcite offers a holistic analysis for pairs of mt systems, examining performance metrics such as n-gram frequency and partof-speech accuracy. mateo @xcite offers a friendly web-based interface for evaluating mt outputs on several evaluation metrics. mt-telescope, a newer platform developed by @xcite , not only supports segment-level analysis but also offers a web-based interface for better visualization of comparative performance between two mt systems. in addition, it enables focused analysis on phenomena like named entity translation and terminology handling. the last three mentioned tools include statistical significance testing through bootstrapped t-tests to ensure the reliability and statistical validity of their evaluations. importantly, these tools require users to upload the source sentences and system outputs to perform evaluations. more recently, towereval @xcite has been developed specifically for llms. this evaluation framework allows users to benchmark their models against a comprehensive suite of datasets for evaluating translation quality. it supports both generation and evaluation processes, allowing users to run inference and compute a variety of metrics such as bleu, comet @xcite , comet-kiwi @xcite , chrf @xcite and ter @xcite . it also allows for the creation of custom test suites and instructions. although the aforementioned works aimed to become the standard tools for evaluating nmt systems, they have not achieved widespread adoption within the mt research community. on the contrary, the lm-eval-harness library has gained significant popularity in the nlp community due to its extensibility, modularity, and ease of use. by building on this widely adopted framework, mt-lens bridges the gap between the versatility of lm-evalharness and the specific needs of mt evaluation, thereby addressing the limitations of prior works. mt-lens is largely inspired by previous frameworks but differentiates itself by being a unified framework where the user can easily run evaluations on the desired model and visualize the results in a user-friendly interface. it also supports a broader spectrum of mt tasks, including bias detection, toxicity evaluation, and robustness to character noise, in addition to traditional translation quality evaluation. ## building blocks mt-lens follows a similar evaluation methodology as lm-eval-harness for mt tasks. when evaluating a system, it follows a predefined sequence of steps which can be divided into five main blocks: ■ models, ■ tasks, ■ format, ■ metrics and ■ results. once the evaluation is completed the user interface will show a fine-grained analysis of the system with interactive visualizations. ## mt-lens ui the web user interface is organized into four main sections, each corresponding to a different mt task (figure 2 ). it is built in python using the streamlit framework 3 . in this section, we describe the tools implemented in the user interface of mt-lens and demonstrate its utility by evaluating two state-ofthe-art nmt systems: madlad-400-3b @xcite and nllb-3.3b @xcite , for the catalan-to-english translation direction. ## conclusion in this paper, we introduced mt-lens, a framework designed to address existing gaps in mt evaluation by unifying various evaluation strategies. mt-lens supports a diverse range of mt tasks, including traditional translation quality evaluation, gender bias detection, added toxicity, and robustness to character noise. by building upon the widely adopted lm-eval-harness library, mt-lens provides seamless integration for evaluating both nmt and llm-based models across various tasks. mt-lens also offers a platform designed to provide insights into system performance. we believe mt-lens has the potential to become the new adopted framework for evaluating nmt systems in the research community.