Vacillating Human Correlation of S acre BLEU in Unprotected Languages (2022)

Abstract: SacreBLEU, by incorporating a text normalizing step in the pipeline, has become a rising automatic evaluation metric in recent MT studies. With agglutinative languages such as Korean, however, the lexical-level metric cannot provide a conceivable result without a customized pre-tokenization. This paper endeavors to ex- amine the influence of diversified tokenization schemes –word, morpheme, subword, character, and consonants & vowels (CV)– on the metric after its protective layer is peeled off. By performing meta-evaluation with manually- constructed into-Korean resources, our empirical study demonstrates that the human correlation of the surface-based metric and other homogeneous ones (as an extension) vacillates greatly by the token type. Moreover, the human correlation of the metric often deteriorates due to some tokenization, with CV one of its culprits. Guiding through the proper usage of tokenizers for the given metric, we discover i) the feasibility of the character tokens and ii) the deficit of CV in the Korean MT evaluation.

## introduction for almost two decades, bleu @xcite has been a key driver of the development of machine translation (mt) and mt evaluation despite its blind spots. @xcite statistically support such trend, reporting that in the past decade, about 98.8% of research papers of acl under the title of "mt" regarded it as their prime evaluation metric. however much stern warnings we have got against its use @xcite , the fact that one of the most popular metrics besides it since 2018 is its stabilized implementation sacrebleu @xcite lets us ask ourselves if this rising metric is safe for all . the biggest strength of sacrebleu is that it reduces the influence of pre-processing scheme on the score computation that could have fluctuated otherwise upon any minor changes such as a type of tokenizers, a split of compound nouns, use of unknown tokens for rare words, or casing @xcite . by embracing the text normalizing step in the architecture, this automatic metric can provide more trustworthy evaluation scores. while it is gaining weight in the literature, its trust issue remains prominent in terms of agglutinate languages such as korean. languages of such typology by design require language-dependant tokenization to convey the morphological implications hardly expressible by whitespaces. presumably for that reason, sacrebleu specifies a customized tokenizer for some languages such as japanese. when assessing korean texts, therefore, the workshop on asian translation directs that the texts be tokenized by mecab-ko foot_1 before running any automatic metrics @xcite , but their correlation to human judgment has not been officially confirmed. in the context where korean is not capable of taking advantage of sacrebleu's protective layer, we shed light on the influence of varied pre-tokenization types on the human correlation of the given metric that features three surface-based metrics: bleu, ter @xcite , and chrf @xcite . with that information, we share empirical lessons for sacrebleu when applying it in the korean language in mt evaluation, some of which are summarized as such: on the segment level: similarly, concerning korean, @xcite found that spm unigram allowed their nmt model to attain a higher bleu score than simple bpe. while they mentioned that a smaller token unit was not always an answer in the case of korean, recent studies paid more and more attention to the sub-subword token unit called jamo, referring to consonants and vowels. 3 moon and okazaki (2020) introduced jamo-pair encoding, combining jamo with bpe. @xcite suggested a new division of jamo by sub-grouping it position-wise. they demonstrated that the model with such a word decomposition outperformed @xcite . we differ from the studies above in exploring the impact of tokenization on the mt evaluation. our keen interest is i) to observe how vulnerable this metric is to the agglutinative languages and ii) to find a way to ensure that the metric is in line with human perception in this regard. ## background this section describes the linguistic characteristics of korean as an agglutinative language. unlike most european languages, it features deeper layers and diversified decomposition. ## extra meta-evaluation as an extended work, we investigate the influence of pre-tokenization on other homogeneous automatic metrics: nltk-bleu foot_6 , gleu foot_7 @xcite , nist foot_8 , ribes @xcite , character @xcite , and eed @xcite . we compute the person correlation r of a total of nine metrics per tokenization on the segment and corpus level under the same environment. the results are provided in figure 5 through figure 8 in appendix d. ## limitations & future works we acknowledge some limitations this work has to embrace. first of all, the number of systems in question is small, which, in part, has led to an arguable conclusion on the corpus level. furthermore, all of the systems are online apis. second, while questioning the influence of token type on the agglutinative languages, we base our study solely on korean. it is of our future interest to probe into the consequence of token types in other comparable languages other than korean. we also intend to scale up the experiment by employing state-of-the-art nmt models. ## conclusion this paper analyzes the influence of diversified token units on the human correlation of sacrebleu on both segment and corpus levels when it comes to agglutinative languages such as korean by performing meta-evaluation with pearson correlation. we demonstrate that the pre-tokenization with a fitfor-all token type is not always an optimal choice in korean mt evaluation. we summarize some of the valuable lessons: