EVALIGN : Visual Evaluation of Translation Alignment Models (2023)

Abstract: This paper presents EvAlign, a visual analytics framework for quantitative and qualitative evaluation of automatic translation alignment models. EvAlign offers various visualization views enabling developers to visualize their models’ predictions and compare the performance of their models with other baseline and state-of-the-art models. Through different search and filter functions, researchers and practitioners can also inspect the frequent alignment errors and their positions. EvAlign hosts nine gold standard datasets and the predictions of multiple alignment models. The tool is extendable, and adding additional datasets and models is straightforward. EvAlign can be deployed and used locally and is available on GitHub.

## introduction translation alignment is the process of finding and linking translation equivalents between a text and its translations. it can be performed on different granularity levels. word-level translation alignment plays a key role in several nlp and digital humanities tasks such as statistical machine translation @xcite , cross-lingual transfer @xcite , classical language learning @xcite , dynamic dictionaries induction @xcite , word sense disambiguation @xcite and analyzing neural machine translation systems @xcite . the work on automatic translation alignment started 30 years ago when @xcite introduced the first statistical alignment models called ibm models. later, several tools and models such as giza++ @xcite and fast_align @xcite were developed based on brown's models with different improvements and optimization additions. with the recent advances in neural machine translation systems and the emergence of pre-trained multilingual transformer models @xcite , it has been possible to develop neural alignment models that significantly outperform the statistical models without needing extensive training datasets. performance evaluation of alignment models is essential, and many ground truth datasets have been developed for this purpose. precision, recall, f1, and alignment error rate (aer) are used as indicators of the alignment quality. although they are widely used, quantitative metrics have their limitations @xcite @xcite . thus, additional qualitative evaluation is required for a better understanding of the models behaviors. for this purpose, we introduce evalign, a tool for quantitative and qualitative evaluation of automatic alignment models that allows developers to estimate the quality of alignment models and get insights into their performance. with multiple visualization approaches and tailored views, the proposed framework helps researchers and developers working on automatic translation alignment models inspect their predictions with different gold standard data sets and compare their performance to other baseline and state-of-the-art models quantitatively and qualitatively. further, it supports nonexperts who want to employ alignment models in their research or business to explore different alignment models and their performance on texts in different languages to choose the suitable model for their purpose. evalign is available online foot_1 and the online demo hosts nine benchmark datasets and five alignment methods combined with four different embeddings models (20 models in total); evalign can be deployed locally, and users can add new datasets and import new models. ## related works employing visualization for exploring benchmark data sets, analyzing models' behaviour, and conducting qualitative evaluation is common practice in nlp. the language interpretability tool lit @xcite offers several interactive visualization techniques for a broad range of nlp tasks. @xcite supports visual and interactive performance comparison of deep learning models. summvis @xcite and summary explorer @xcite support qualitative evaluation for the summarization task. paper with code 3 platform allows to track state-of-the-art performance on benchmark datasets for different nlp tasks. vis-eval @xcite , asiya @xcite and mt-compareval @xcite allow for systematic comparison and evaluation of various machine translation models. visualizing word-level alignments was the aim of many tools such as ugarit @xcite and wa-continuum @xcite , which visualizes word alignment of automatically aligned sentences to facilitate their evaluation. @xcite introduced the parallel corpus explorer which supports exploring a word-aligned parallel corpus. to our best knowledge, evalign is the first system that allows researchers and practitioners to qualitatively evaluate the performance of alignment models on multiple gold standard datasets. ## automatic alignment models automatic translation alignment models can be categorized into three main categories: statistical models such as giza++, fast_align @xcite , and eflomal @xcite . they have been widely used and achieved state-ofthe-art performance until recently and are currently used as a baseline. however, the performance of the statistical models is governed by the availability of training corpora in the form of parallel sentences. neural models utilize neural machine translation models or multilingual transformer models to capture word-level translation alignment. different workflows are available. for instance, extracting alignment using embeddings-based semantic similarity by employing pre-trained and fine-tuned multilingual contextualized embeddings such as sima-3 https://paperswithcode.com/ lign @xcite , aweasome @xcite , xlm-align @xcite , and mirroralign @xcite . hybrid models combine statistical and neural models aiming for better performance, for instance, by using the output of statistical models as supervision to train neural models @xcite . ## implementation details we surveyed the automatic alignment papers published after 2019 ( regarding the alignment models, we selected embeddings-based sof tmax, entmax @xcite , argmax, itermax, and m atch (jalili sabet et al., 2020) with different contextualized embeddings, namely, mbert, xlm-r, fine-tuned mbert foot_2 , and xlm-align foot_3 @xcite . in addition to giza++, fast_align, and eflo-mal for the datasets de-en, en-fr, and ro-en. our selection was subject to the implementation availability and reproducibility. we used the default implementations provided by authors in their github repositories with the default parameters. the backend api is implemented using django and postgres database, while the visualization views are created with react js and d3.js. ## usage scenarios the framework offers a variety of usage scenarios that can be summarized as follows: gold standard quality control. visualizing the gold standard datasets using the grid view allowed us to inspect their accuracy and assess their quality. the analysis of the english-french (en-fr) dataset showed that the dataset contains several single or two token sentences, for which the alignment will always be correct (figure 9c-d ). moreover, some sentences occur more than once in the dataset, and that would affect the evaluation process since they increase recall and precision and consequently reduce aer (figure 9b , c , e ). the inspection showed that there are several sentences with plenty of possible alignments and few or no sure alignments (figure 9a , f ). comparing datasets' characteristics. users can see all hosted datasets on the datasets page with different statistics on the number of sentences, tokens, sure and possible alignments and their percentages. for instance, the english-french (en-fr) dataset has significantly more possible alignments than sure alignments (figure 15 ). this explains why all alignment models have the lowest aer on this dataset compared to all other datasets. the same applies to the romanian-english (ro-en) dataset since it only has sure alignments, which explains why the aer is always higher than the other datasets. comparing model performance with different configurations. as an example, we compare the performance of sof tmax with two different embeddings models,namely, mbert and a fine-tuned mbert, to estimate the improvement achieved with the fine-tuning process. in addition to comparing all quantitative metrics, the framework allows filtering sentences where model a outperforms model b. figure 2 shows that the fine-tuning enhanced the overall alignment accuracy and allowed to predict two more correct sure alignments and eliminate two incorrect ones. comparing quantitative metrics. the framework provides different options to compare the models performance using different quantitative metrics at corpus and sentence levels using bar chart and table views. the aggregated results in the table view (v5) reveals that the fine-tuned mbert achieved the best results in all datasets regarding aer. while itermax achieved the best recall on all datasets, argmax with fine-tuned mbert embeddings achieved best precision on 7 datasets and second best precision on 2 datasets. further, itermax with the fine-tuned mbert embeddings achieved the best phrase alignment accuracy on all datasets. the m atch algorithm generates more translation pairs than all other algorithms, and entmax with xlm-roberta embeddings generates always less translation pairs that all other algorithms. analyzing alignment errors. from the pie chart provided for the single dataset -single model view (figure 3 ), we can click on the red arc that represents the wrong alignment pairs to list all incorrect pairs produced by the model. our analysis of itermax with the fine-tuned mbert model on the german-english (de-en) dataset revealed the following: -the most frequent wrong pairs involve a punctuation mark in one or both languages. however, such issues can be avoided by adding constraints that prevent aligning punctuation to a word (figure 18 ). ## conclusion evaluating translation alignment models is a nontrivial task. qualitative evaluation is needed because quantitative evaluation metrics do not reflect the real quality of the alignment models due to many factors. for this purpose, we presented the framework evalign that supports quantitative and qualitative evaluation of automatic alignment systems. evalign hosts several evaluation datasets and various alignment models. it offers different visualization views and filtering functions to help users to investigate alignment datasets and models and conduct various quality analyses. moreover, we presented different usage scenarios that showcase the use and effectiveness of the tool. our analyses revealed that gold standard datasets, especially the german-english (de-en) and french-english (fr-en), which have been used almost in all related works on automatic alignment, contain plenty of errors and need to be revised and corrected by linguists and domain experts. in future work, we aim to incorporate morphological features such as pos, lemma and named entities to assess model performances and classify alignment errors. finally, we will keep the tool updated by adding new datasets and/or models, and we encourage researchers to send us the output of their new models to publish them on evalign. a short video demonstrating the tool is available on youtube https://youtu.be/hfii6xobktw