UTER : Capturing the Human Touch in Evaluating Morphologically Rich and Low-Resource Languages (2025)

Abstract: We introduce UTER, a novel automatic translation evaluation metric specifically designed for morphologically complex languages. Unlike traditional TER approaches, UTER incorporates a reordering algorithm and leverages the Sørensen-Dicse similarity measure to better account for morphological variations.Tested on morphologically rich and low resource languages from the WMT22 dataset, such as Finnish, Estonian, Kazakh, and Xhosa, UTER delivers results that align more closely with human direct assessments (DA) and outperforms benchmark metrics, including chrF and METEOR. Furthermore, its effectiveness has also been demonstrated on languages with complex writing systems, such as Chinese and Japanese, showcasing its versatility and robustness.

## introduction with the rise of machine translation systems, evaluating their quality has become a key challenge in numerous fields, particularly for under-represented and morphologically complex languages. assessing the quality of machine translations is of critical importance, especially in a context where translation systems are increasingly being adopted for languages that present specific challenges, such as complex morphology. however, traditional evaluation metrics, such as bleu @xcite , rely on simple n-gram matching with reference sentences, making them inadequate for capturing the morphological or syntactic variations characteristic of these languages. as highlighted by @xcite , these approaches face significant limitations in lowresource language environments, which are often marked by complex morphology. the authors rec-ommend using language models, which are better suited to capture linguistic and semantic nuances. nonetheless, language models are generally ill-suited for under-represented languages due to the lack of available data and limitations in contextual representations. this leads to a strong dependency on english and other dominant languages. moreover, their computational and training time requirements pose significant challenges in resource-constrained settings. to address these challenges, we propose a new lexical metric, universal-ter (uter), designed to enhance the translation edit rate (ter) by incorporating reordering mechanisms and a refined consideration of lexical and morphological variations. unlike traditional approaches, uter offers fast, accurate, and resource-independent evaluation. in this paper, we present uter's performance by comparing it against human direct assessments (da) from the wmt dataset, benchmark metrics such as chrf and meteor, and two alternative metrics. our results demonstrate that uter provides a reliable and accurate measure of translation quality, making it particularly valuable for morphologically complex languages. before delving into uter's performance, it is crucial to examine the specific challenges posed by under-represented and morphologically complex languages to evaluation metrics. an implementation of uter is publicly available as a python package on pypi: https://pypi . org/project/evalnlp/. ## challenges and limitations of metrics for morphologically rich and under-represented languages evaluating translations for low-resource languages presents unique challenges, largely due to the complexity of their morphological structures and the scarcity of representative linguistic data. these languages, often spoken within small communities and less influenced by non-native speakers, tend to retain elaborate morphological systems, including variations in suffixes, prefixes, and grammatical inflections @xcite . in such contexts, metrics based on lexical matching, such as bleu and ter, often lack the precision needed to capture the intricate morphological and syntactic nuances of these languages. this makes them particularly unsuitable for languages like finnish, kazakh, or xhosa, where minor morphological variations can result in significant discrepancies in scores, even when such variations are linguistically acceptable. to address some of bleu's limitations, the chrf metric @xcite was introduced, relying on character-level rather than word-level matches to better capture the morphology of complex languages. while chrf provides an improvement, it remains insufficient for evaluating translation quality with the accuracy of human assessments. recent approaches leveraging language models have sought to overcome these issues by capturing contextual and semantic information, thereby going beyond simple word matches. however, due to the limited availability of data for under-represented and morphologically rich languages, these models often lack sufficient examples for proper generalization @xcite . moreover, the absence of specific training data limits the accuracy of evaluations, as pretrained models are predominantly optimized for resource-rich languages. this creates a bias that prevents these models from delivering reliable evaluations for under-represented, morphologically rich languages @xcite . finally, the high computational and training time requirements add an additional hurdle, making these solutions costly and impractical in such settings. in summary, these limitations highlight the need for an alternative metric tailored to the specifici-ties of morphologically rich and under-represented languages. uter addresses this need by delivering results that align more closely with human judgments, without relying on advanced language models or large training datasets. ## limitations of ter the translation edit rate (ter) evaluates the quality of machine translations by measuring the minimal number of operations (insertion, deletion, substitution, and transposition) required to transform a translation into a reference sentence. its formula is as follows: total number of operations length of the reference sentence although ter is a widely used benchmark metric, it has several shortcomings that limit its ability to accurately reflect translation quality: dividing by the reference length amplifies penalties for additions in longer translations, while dividing by the translation length accentuates penalties for omissions in shorter translations. while this principle may be justified in asr, it remains unsuitable for translation evaluation. figure 1 shows how ter varies with normalization choice. • greedy approach to transposition calculation: this approach successively applies the most beneficial transpositions, those that minimize the number of insertion, deletion, and substitution operations, until no further improvements are possible @xcite . while this method allows for an approximate solution, recalculating the edit operations after each transposition results in high computational complexity. furthermore, it prioritizes local improvements without ensuring global optimality, which can lead to suboptimal choices and prevent reaching a more favorable configuration. finally, this strategy ignores the interdependencies between transpositions, where a different combination could further reduce the overall score. ## optimized evaluation of transpositions and lexical variations with uter to overcome the identified limitations of ter, we propose uter, an enhanced metric that integrates a reordering algorithm for detecting transpositions, accounts for lexical and morphological variations, and includes a weighting of edit errors. ## comparative evaluation and results analysis to assess the effectiveness of uter, we used the wmt22 workshop dataset @xcite , which provides translations, references, and human evaluation scores based on direct assessment (da) for 41 language pairs, with a total of 1.29 million lines. in order to test uter in a multilingual context and on morphologically complex languages, we focused our analysis on eight such languages: finnish, estonian, latvian, lithuanian, czech, kazakh, zulu, and xhosa, totaling 89,920 lines. among these, the last three are truly low-resource, adding an additional layer of complexity, while estonian and latvian can be considered mid-resource languages with moderate nlp support. uter was compared against benchmark metrics such as bleu, ter, meteor, and chrf, as well as alternative metrics like cder @xcite and ribes @xcite . this panel allows, initially, the validation of uter's relevance by comparing it with other metrics on languages with complex morphological characteristics. the results include, in addition to pearson and kendall correlations, the means and medians, in order to better reflect the overall performance of each metric. this approach is essential, as a high correlation does not necessarily guarantee a robust evaluation, as highlighted by @xcite and @xcite . correlation coefficients merely reflect whether two metrics vary in a similar direction, not whether their are close to human judgments. uter stands out for its balanced combination of correlation and overall performance, demonstrating its effectiveness for morphologically complex languages. to better visualize the results on a dataset of nearly 90k examples, the metric curves were plotted 4 with smoothing applied using a window corresponding to 1% of the data. the uter metric curve shows a close match with that of the reference human evaluations (human da), suggesting a superior ability to reflect human judgments compared to metrics such as bleu or ter. furthermore, the chrf metric shows acceptable performance, with notable consistency relative to human evaluations. in contrast, the ribes metric appears to tend towards overestimating results, which could limit its reliability. the figure 5 provides a closer look at the performance of uter compared to chrf in approximating human direct assessments. to assess the relevance of uter in distinct linguistic contexts, we decided to test its performance on cjk languages (chinese, japanese, korean), which present particular challenges due to their complex writing systems that make classical metrics such as bleu or ter often unsuitable, due to semantic ambiguities and segmentation variations @xcite @xcite . this analysis focuses on chinese and japanese, totaling 134,115 examples, and concentrates on the metrics that performed best on morphologically complex languages. despite being specifically optimized for morphologically complex languages, uter outperforms other metrics even on languages with complex writing systems such as chinese and japanese, as shown in figure 6 , although some areas for improvement remain. in comparison, chrf produces moderate results, while the cder and ribes metrics perform relatively poorly, highlighting their limited relevance in this context. finally, to complete the comparison between uter and chrf, we extended the analysis to the entire dataset, totaling 1.29 million lines, with 55% of translations being into english. the results show 7 that, while the overall performance of uter and chrf is moderate on this corpus, uter maintains better alignment in terms of consistency with human evaluations. ## conclusion in this paper, we presented uter, a new automatic translation evaluation metric specifically designed to better address the needs of low-resource and morphologically complex languages. unlike traditional metrics such as bleu or me-teor, uter integrates an optimized reordering algorithm and uses sørensen-dice similarity to effectively capture morphological and lexical variations. our comparative evaluation, based on a subset of 89,920 lines from the wmt22 dataset, showed that uter offers a closer match with human evaluations (da) compared to metrics like chrf, cder, or ribes, particularly for languages with complex linguistic structures. expanding the analysis to the entire dataset, which includes 1.29 million lines, with 55% of translations into english, we observed that, although overall performance is moderate on this corpus, uter continues to outperform chrf, thus confirming its robustness and ability to adapt to various multilingual contexts. in conclusion, uter positions itself as a reliable and versatile metric, offering a fairer evaluation for under-represented languages. future directions include integrating syntactic information to further refine this metric, particularly for languages with complex grammatical structures.