Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systems (2024)

Abstract: With the rapid advancement of machine translation research, evaluation toolkits have become essential for benchmarking system progress. Tools like COMET and SacreBLEU offer single quality score assessments that are effective for pairwise system comparisons. However, these tools provide limited insights for fine-grained system-level comparisons and the analysis of instance-level defects. To address these limitations, we introduce Translation Canvas , an explainable interface designed to pinpoint and analyze translation systems’ performance: 1) Translation Canvas assists machine translation researchers in comprehending system-level model performance by identifying common errors (their frequency and severity) and analyzing relationships between different systems based on various evaluation metrics. 2) It supports fine-grained analysis by highlighting error spans with explanations and selectively displaying systems’ predictions. According to human evaluation, Translation Canvas demonstrates superior performance over COMET and SacreBLEU packages under enjoybility and understandbility criteria.

## introduction as natural language processing (nlp) technologies evolve, the need for precise and detailed analysis of model outputs has become increasingly critical. despite significant advancements in translation models, a gap remains in the tools available for researchers to thoroughly evaluate and interpret these models' predictions. this issue is particularly acute in the context of translation research, where understanding the nuances of model errors and performance is vital for further improvements. translation model developers excel in creating sophisticated algorithms, but often face challenges when it comes to conducting fine-grained analysis of model predictions. moreover, tools designed to facilitate such analysis typically lack the flexibility and specificity required for detailed evaluation at the instance level. this disconnect underscores the necessity for an integrated solution that combines comprehensive model evaluation with user-friendly interfaces and advanced analytical capabilities. existing approaches to model evaluation often focus on high-level metrics such as bleu or comet scores, which, while useful, do not provide the granularity needed to identify specific areas of improvement like stylistic errors and incorrect word choices. moreover, the process of manually analyzing individual model predictions is time-consuming and prone to error. additionally, analyzing predictions in a language direction that translation researchers are unfamiliar creates a language barrier and hinders model improvement. as translation models continue to grow in complexity, the demand for a more sophisticated, streamlined approach to model evaluation and error analysis has never been higher. translation canvas addresses these challenges by offering a comprehensive toolkit designed specifically for translation researchers. it provides a dashboard that displays the distribution of common errors, instance-level performance and systemlevel performance for each model. this helps the user identify specific areas of improvement in their models, and identify gaps in model performances. the system also provides fine-grained analysis by displaying instances. the system visually highlights erroneous text spans and provides natural language explanation explaining the error. users can also construct complex search queries to filter instances and conduct targeted analysis. our evaluation shows that users find the system to be useful, enjoyable and as easy to use as command-line evaluation tools. our contributions are summarized as follows: in this paper, we present translation canvas and demonstrate how it meets the critical needs of translation researchers. we outline its key features, including instance-level analysis, error classification, and advanced search capabilities, and illustrate how these tools can be leveraged to gain deeper insights into model behavior. through detailed examples and use cases, we show how translation canvas transforms the process of translation model evaluation, making it more efficient, accurate, and insightful. ## related works recent years have seen a growing interest in developing tools and frameworks for comprehensive evaluation and analysis of nlp models, particularly in the domain of machine translation. these efforts aim to provide researchers with deeper insights into model performance, error patterns, and areas for improvement. many evaluation metrics have been proposed to measure, evaluate, and explain machine translation. these metrics include both automatic and human evaluation methods. automatic metrics, such as bleu @xcite , meteor @xcite , and rouge @xcite , measure the quality of translations by comparing them to human references, focusing on aspects like n-gram overlap and recall. more advanced metrics like bertscore @xcite , comet @xcite and sescore @xcite use learned techniques to assess translation quality by comparing sentence embeddings. instructscore @xcite ) leverages large language models to provide error classifications and explanations on an instance-level. human evaluation methods, such as the multidimensional quality metrics (mqm) @xcite framework, involve human judges assessing translation quality based on criteria like fluency and adequacy. these methods provide nuanced feedback but require effort from the model developer to interpret and process in order to make effective diagnosis of models. in addition, each metric must be evaluated and understood separately, making it harder to leverage the multiple metrics to identify core issues with models. some visual frameworks have been proposed to provide model developers with a unified interface to evaluate and debug models. explainaboard @xcite offers an explainable leaderboard for nlp tasks that provides fine-grained analyses of model performance. while explainaboard offers valuable insights, translation canvas builds upon this concept by providing more specialized tools for translation model analysis, an instance-level analysis of errors. mt-telescope @xcite provides an evaluation and visualization platform for machine translation systems that supports comparison of models, dynamically filtering content and visualizations that enhance model comparisons. although mt-telescope provides a fantastic platform for model comparison, translation canvas provides a more flexible content filtering system, allowing users to join requests to produce a complex search query. while mt-telescope is focused on model comparisons, translation canvas is flexible in the number of models it can compare, as well as having the option to analyze a model by itself. mateo @xcite provides a suite of evaluation metrics for machine translation, and visualization for model performance via a userfriendly web application interface. while mateo lets users easily evaluate their models on a wide variety of metrics, translation canvas provides users the ability to do fine-grained analysis with natural language error explanations, as well as an advanced search system. translation canvas builds upon these existing works by integrating comprehensive evaluation metrics, fine-grained error analysis, and an intuitive user interface specifically designed for translation researchers. our system uniquely combines features like advanced search functionality, model comparison dashboards, and instance-level analysis, addressing the need for a specialized toolkit in the field of machine translation evaluation. ## translation canvas translation canvas is implemented using python flask. it uses a flask backend and jinja templates for frontend with a duckdb connection. it is distributed over pip 1 and is available for use to everyone under the mit open-source license. ## use case here, we briefly showcase a use case for translation canvas to evaluate models and identify gaps between performance. for this use case, we use the anvita @xcite and gpt4-5shot @xcite machine translation models, submitted to the wmt 2023 general translation task @xcite . after extracting and evaluating the instances, we can start by comparing the anvita and gpt4-5shot instances at a system level. it is clear from the dashboard in figure 3 that gpt4-5shot has a far more favorable distribution of instance level instructscore (bottom right) and also makes far fewer errors of the type "incorrect translation is missing content from the correct translation". this implies that anvita is leaving out content from the translation at a much higher frequency than gpt4-5shot is. we can investigate this further by switching to an instance-level view. to filter the instances that we are interested in analyzing, we can use the search bar to search by error type and find only instances with error type "missing content", which will find all instances with those errors. in figure 2 , we see an instance with predictions from the 2 models. the text written in blue contains errors that we are looking for. when the hovering the mouse over the blue span of the text, we see the natural language explanation of the error. note that the error span for anvita is at the very end, and the explanation shows that anvita is missing content at the very end of the translation. this could help the model developer understand the weaknesses of anvita, such as possibly a tendency to truncate or omit information at the end of longer sentences. this insight could guide improvements to the model's handling of sentence endings or its ability to maintain context throughout longer translations. ## evaluation to assess the effectiveness of translation canvas, we conducted a user evaluation study with partic-ipants who have substantial expertise in machine translation and knowledge of existing mt metrics. our evaluation focused on two key aspects: instance-level analysis and system-level analysis. for instance-level analysis, translation canvas received high marks, with expert users rating it 4/5 for both enjoyability and usability. participants appreciated the highlight of error types and the quick analysis process. a particularly valuable feature was the elimination of the need for forward translation to understand unfamiliar languages, which even experienced users found beneficial. the system-level analysis features were also well-received by our expert evaluators, with enjoyability rated at 4/5 and usability at a perfect 5/5. participants found the graph presentations appealing and particularly valued the sorted error types, which saved time in fine-grained analysis. the support for multi-system analysis was highlighted as a key usability feature. we also benchmarked the time required for nonexperienced users, who have no prior knowledge of existing machine translation evaluation systems, to learn and use translation canvas compared to existing tools. our findings showed that these first-time users took an average of 10 minutes to learn and use translation canvas on a custom dataset. this was comparable to the time needed for sacrebleu @xcite (10 minutes) and faster than comet (15 minutes). these results demonstrate that translation canvas provides an intuitive interface accessible to users without prior mt evaluation experience. the system's ability to match or exceed the learnability of established tools, while offering more comprehensive analysis features, indicates an effective balance between advanced functionality and userfriendly design. this combination of accessibility and depth potentially addresses a significant need in the mt research community for tools that facilitate both rapid onboarding and sophisticated analysis. ## limitations and future work while these metrics provide valuable quantitative insights, they may not fully capture the nuanced aspects of translation quality that human expert evaluations could offer. the potential discrepancy between automatic and human evaluations underscores the need for a more comprehensive assessment approach. to address this limitation, we have implemented a re-ranking feature in the fine-grained analysis interface. this feature allows users proficient in both source and target languages to manually adjust the ranking of predictions according to their expert judgment. this user-driven re-ranking serves a dual purpose: it provides immediate value to users seeking more accurate rankings and generates valuable data for future improvements. the collection of this user-generated re-ranking data presents an opportunity for future work. we intend to leverage this dataset to refine our ranking algorithms and enhance our evaluation methodologies. by incorporating human expertise into our automated systems, we aim to bridge the gap between automatic metrics and human judgment, potentially leading to more robust and reliable evaluation techniques in machine translation research.