e BLEU : Unexpectedly Good Machine Translation Evaluation Using Simple Word Embeddings (2023)

Abstract: We propose eBLEU, a metric inspired by BLEU metric that uses embedding similarities instead of string matches. We introduce meaning diffusion vectors to enable matching n-grams of semantically similar words in a BLEU-like algorithm, using efficient, non-contextual word embeddings like fastText. On WMT23 data, eBLEU beats BLEU and ChrF by around 3.8% system-level score, approaching BERTScore at −0.9% absolute difference. In WMT22 scenarios, eBLEU outperforms f101spBLEU and ChrF in MQM by 2.2%−3.6%. Curiously, on MTurk evaluations, eBLEU surpasses past methods by 3.9%−8.2% (f200spBLEU, COMET-22). eBLEU presents an interesting middle-ground between traditional metrics and pretrained metrics.

## introduction the machine translation field has improved significantly, with various metrics developed to measure translation quality. translation quality in human eyes is usually a delicate balance to convey meaning, style, tone, and other dimensions of text from one language into another with different idioms and concept ontologies. after all, translation is not only about translating words from one language to another literally, but ensuring that the core meaning behind is also accurately conveyed. traditional metrics, like the bleu score @xcite or chrf @xcite , have proven effective over last 20 year. however, there has been growing evidence that they have not kept pace with the performance of recent nmt and llm mt systems @xcite . bleu essentially computes a score based on string n-grams matches. one clear limitation of this approach is that it fails to recognize semantically similar words. for instance, in the eyes of bleu, the words (cat, kitty) are as different as (cat, book) or (fire, water). recent neural metrics, on the other hand, have explored the potential of leveraging pretrained language models for encoding entire sentences. these models either compare encoded sentences in a shared embedding space or employ a trained classifier to predict human scores, as demonstrated by @xcite ; @xcite ; @xcite . these methods are more capable of capturing semantic nuances. in this paper, we introduce ebleu, a metric designed to address the mentioned limitation of the bleu score while keeping the calculation as close to bleu as possible by using the word embedding similarities instead of string matching. by doing so, ebleu enhances the metric by recognizing semantically similar n-grams. our method relies on the meaning diffusion map to approximate ngram matching in a bleu-like algorithm. the core implementation leverages efficient, non-contextual word embeddings, such as fasttext embeddings. ## related work in machine translation, measuring quality is a balance of many potentially competing factors. the most prominent are language quality (fluency) and accuracy of meaning conveyed (adequacy). other factors may be critical in special scenarios. consider the conveyance of tone or cultural register in translated dialog (see for example registers in east-asian languages). or the conveyance of flow in a translated play (see some examples of translations of the greek epic iliad in @xcite . traditional automatic quality assessment methods, like bleu and meteor @xcite , rely on string matching against a reference. the more matches, the more a candidate captures of the intended meaning in the reference, as proxy for adequacy. while features like n-gram matching in bleu and explicit ordering penalties in meteor act as proxy for fluency. such metrics suffer from limitations inherent to their literal string matching core, which some try to mitigate (e.g. via lemmatization or synonym dictionaries). these limitations are clearer in light of the more complex and semantically rich language produced by recent neural mt systems and large language model mt systems. bertscore @xcite utilizes a similar idea to ours, matching contextual encodings of words in candidate/reference pairs. while it uses unigrams only, ebleu uses n-grams as well, and calculates token matches differently. other systems, like comet @xcite , are finetuned on human judgement scores for machine translation evaluation specifically. ## ebleu description we propose the following formulation for an embedding-based matching in the style of bleu precision from eq. ( 2 ). let x refer to the candidate sentence, and y refer to the reference sentence. now, given an asymmetric similarity function mdsim (a | b) from a with reference to b, we can define the following analogous values for "precision" and "recall": the final score is a weighted geometric average of the n-gram-based scores m n between candidate and reference, for @xmath0. ## conclusion 6.1 ebleu: between strings and neural eval in this paper, we introduced ebleu, a novel metric that adapts the bleu algorithm by adding embedding-based semantic understanding. by in- on wmt23, ebleu scores 89.3%, placing almost halfway between bleu, and comet, an especially finetuned model for mt evaluation. although ebleu lags behind the latest pretrained metrics, it presents an interesting approach for a simple semantically informed metric.