BLASER 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation (2024)

Abstract: We present BLASER 2.0, an automatic metric of machine translation quality which supports both speech and text modalities. Compared to its predecessor BLASER (Chen et al., 2023), BLASER 2.0 is based on better underlying text and speech representations that cover 202 text languages and 57 speech ones and extends the training data. BLASER 2.0 comes in two varieties: a reference-based and a reference-free (quality estimation) model. We demonstrate that the reference-free version is applicable not only at the dataset level, for evaluating the overall model performance, but also at the sentence level, for scoring individual translations. In particular, we show its applicability for detecting translation hallucinations and filtering training datasets to obtain more reliable translation models. The BLASER 2.0 models are publicly available at https://github.com/facebookresearch/sonar.

## introduction automatic evaluation of machine translation (mt) is difficult both because it requires good understanding of both source and target languages, and because the number of possible ways to express a thought in a language is often intractably large. although there are commonly accepted evaluation metrics, such as bleu @xcite , the search of the mt community for better metrics is still ongoing. traditionally, mt is evaluated by comparing the translations to the references with metrics like bleu/spbleu @xcite or chrf++ @xcite that estimate the proportion of common word or character n-grams in the texts. however, to evaluate translation accuracy, we would really prefer to compare the meanings (not the surface forms) of the translation and the source (not the reference). also, when the translation is in the speech modality, n-grams are no longer available, so the direct comparison is impossible. a workaround for this is to use automatic speech recognition systems and compare texts (e.g. asr-bleu by @xcite al. (2019) ), but asr introduces an additional layer of errors. a principled solution would be to use representations for the source and translation that abstract away from the form and the language, but preserve the meaning, and use an appropriate function to compare them. blaser @xcite is using laser3 embeddings @xcite as the representations, and a trainable function for predicting the human judgement of semantic similarity based on the source, translation, and reference embeddings. it was partially inspired by the comet model @xcite ) that later evolved into cometkiwi @xcite and xcomet @xcite . another notable reference is @xcite . all these metrics are based on text-only representations, and may not be easily extendable for speech or for texts in other languages. blaser seems to be the only mt evaluation model based on language-and modalityagnostic sentence representations, but it has been applied only for evaluating s2st outputs 1 . this paper describes the blaser 2.0 models, which build upon blaser by using better underlying text and speech representations based on sonar @xcite and extending the training data. it also introduces a referencefree model (blaser 2.0-qe), suitable for a more diverse set of evaluation tasks where a reference translation is not always available. to the best of our knowledge, this is the first non-cascaded foot_1 qe metric for speech translation that covers a wide range of languages. finally, this paper demonstrates the application of blaser 2.0 for tasks beyond evaluating speech translation models: hallucination detection and filtering mt training data. input sentence transformer encoder mean pooling sentence embedding input audio transformer encoder attention pooling sentence embedding cnn encoder source embedding e s translation embedding e t multilayer perceptron concatenation: e t , e s , |e t -e s | , e t ⊙ e s predicted score text sonar speech sonar blaser 2.0-qe ## conclusions blaser 2.0 is a multilingual and multimodal metric for translation evaluation and quality estimation. compared to its previous version, the second blaser has better coverage of modalities and supports reference-free evaluation. blaser 2.0 covers 202 languages in text and 57 languages in speech and provides well calibrated, interpretable and cross-lingually comparable scores. apart from evaluating translation models, blaser 2.0 is suitable for pointwise scoring of source-translation pairs, which enables its use for hallucination detection and training data filtering.