T okengram_ F , a Fast and Accurate Token-based chr F ++ Derivative (2023)

Abstract: Tokengram_F is an F-score-based evaluation metric for Machine Translation that is heavily in- spired by chrF++ and can act as a more accurate replacement. By replacing word n-grams with n-grams obtained from tokenization algorithms, tokengram_F better captures similarities between words.

## introduction machine translation (mt) is a subdomain of neural language processing (nlp) that is focused on the translation of one natural language to another, with the aim of producing natural-sounding sentences. to evaluate the quality of algorithmgenerated translations, evaluation metrics provide quantitative scores to objectively assess the accuracy of the model. machine-generated translations are compared to human-generated translations in different ways depending on the evaluation metric. in recent years, machine translation has seen a great deal of progress in terms of accuracy and fluency. however, there is still a need for more robust evaluation metrics that can effectively measure the quality of machine-generated translations. popular metrics in mt include bleu @xcite , which measures the overlap between sequences of words in the reference and generated texts; chrf @xcite , which is an f1 score at the character-level; chrf++ @xcite , which extends chrf with word n-grams; ter @xcite , which counts the erroneously-aligned words between the reference and the generation, and meteor @xcite , which takes into account synonyms and stems. more recent metrics rely on neural network architectures, such as comet @xcite and ms-comet @xcite . by using the similarity of vector representations of the generated translation and the reference translation, they provide state-of-the-art machine evaluation of translations at the cost of being expensive to train and compute. since its third instance in 2006, the workshop on statistical machine translation (wmt) has released an evaluation task to compare metrics each year. generated translations are usually ranked by humans, and the correlation coefficient between the human-performed ranking and the evaluation metric-performed ranking determines the quality of the metric. ## conclusion instead of using word n-grams as a basis for comparing a generated translation with a reference translation, tokengram_f utilises contemporary tokenization algorithms to accomplish this task. as a result, words that share common roots are deemed similar, regardless of whether they are exact matches or not. the results obtained from evaluating the token-gram_f metric on the wmt20, wmt21, and wmt22 datasets indicate that the use of tokens generated through the sentencepiece framework leads to improved performance compared to the use of traditional word-grams in the chrf++ metric. full results are displayed in appendix a. the to-kengram_f metric is a simple and efficient method for obtaining a reasonable correlation with human rankings, with the added benefit of requiring minimal training time to be applied to new languages. in the segment-level task of the wmt22 edition, tokengram_f managed to obtain better overall correlations than any other metric that could provide results for all language pairs, including ones that require extensive neural networks to operate. with the exception of two tasks, tokengram_f outperformed both chrf and chrf++ metrics. in conclusion, the tokengram_f metric is presented as a promising alternative for evaluating the quality of machine translations, as it offers a simple and efficient solution with above-average performance compared to other models. the findings of this study provide strong evidence of the potential of the tokengram_f metric as a valuable evaluation tool for machine translation. its combination of simplicity, efficiency, and adaptability make it an attractive alternative to existing metrics and a promising direction for future research in the field.