byt F : How Good Are Byte Level N-Gram F -Scores for Automatic Machine Translation Evaluation? (2025)

Abstract: Recently, chrF and chrF++ have become the preferred metric over BLEU for automatic n-gram evaluation of machine translation. Since they focus on character-level n-grams, it appears to have better correlations with human judgments for translating into morphologically rich languages compared to word-level metrics. However, for non-Latin languages with sub-character-level structures, we can go one step further namely bytes. To this end, we propose bytF to capture sub-character-level information, where we consider byte-level n-grams. Furthermore, we augment it to bytF+ and bytF++ where we consider character and word n-gram backoffs. On machine translation metric meta-evaluation datasets from English into 5 Indian languages, Chinese and Japanese, we show that bytF and its variants are comparable (give minimum difference) or significantly better (give maximum difference) correlated than chrF and chrF++ with human judgments at the segment level. We often observe that backing off to characters and words for bytF and to words for chrF does not have the highest correlation with humans. Furthermore, we also observe that using default n-gram values often leads to scores having poorer correlations with humans, indicating the need for well studied and tuned n-gram metrics for efficacy.

## introduction recently, chrfand chrf++ @xcite have become the preferred metrics for automatic n-gram evaluation of machine translation (mt) @xcite @xcite . compared to bleu @xcite , they focus on fine-grained character-level n-grams. as a result, they appear to have better correlations with human judgments for translating into morphologically rich languages. however, for non-latin languages with subcharacter structures, as shown in figure 1 for chinese, we can go one step further to evaluate the subcharacter-level structures, which are usually represented by bytes. this applies to a wide range of languages such as japanese and indian languages. to this end, we propose bytf, in which we consider byte-level n-grams instead of character-level n-grams that can be implemented with a single line code change. experimental results on wmt and indian mt meta-evaluation datasets show that bytf has a higher correlation (pearson and kendall tau) with human judgments at the segment level compared to chrf. we further extend bytf to bytf+/bytf++ where we incorporate characterand word-level n-gram backoffs to show that this further enhances correlations. ## related work we introduce commonly used mt evaluation metrics in section 2.1 and the recent trend of byte-level methods in section 2.2. ## proposed methods this section introduces our proposed bytf metric and the extended bytf+ and bytf++ variants. ## experimental setup we describe our datasets, language pairs and metaevaluation setup. ## results we now describe our results to determine whether byte-based metrics can be used to replace characterbased metrics. tables 1 and 2 provide the pearson and kendall tau correlations with human scores, along with the winning metric and the optimal configuration. for bytf and its variants, the configuration is given as a tuple a, b, c for byte, character and word n-gram values, respectively. similar for chrf and chrf++, the configuration is a, b for character and word n-gram values respectively. ## conclusion we proposed bytf, a byte-level n-gram evaluation metric that captures sub-character-level similarities for machine translation. we further augment bytf with character-and word-level backoffs as bytf+ and bytf++. our experiments show that they achieve higher correlations with human