BLASER : A Text-Free Speech-to-Speech Translation Evaluation Metric (2023)

Abstract: End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASR dependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics.

## introduction speech-to-speech translation seeks to translate speech segments from one language into another. historically, it has been implemented and evaluated as a concatenation of three systems: automatic speech recognition (asr), machine translation (mt) and text-to-speech (tts) @xcite . in recent years, there has been increasing interest in end-to-end approaches @xcite . while end-toend s2st is becoming popular, researchers still rely on text-based metrics to evaluate model performance by automatically transcribing the generated speech segments @xcite . these cascaded metrics rely on asr systems, which for a given language may not have enough quality or may not even be available @xcite . they are also inappropriate for languages lacking standardized writing systems @xcite , like hokkien or algerian arabic. in this work, we propose the text-free metric blaser for s2st evaluation, sidestepping the dependency on asr systems. in particular, we use laser encoders that support multiple languages and modalities including text @xcite and speech @xcite . we use the laser encoders to directly embed speech segments into vectors and compute a score estimating the quality of generation. we then construct training and evaluation datasets from more than 40k human annotations, covering seven language directions (spanish↔english, french↔english, @xmath0, @xmath1, and @xmath2. we evaluate blaser on these datasets on the popular benchmark of @xcite . we also benchmark several strong asr-based metrics, e.g., asr-sentbleu (i.e., sentence-level asr-bleu @xcite ) and asr-comet (i.e., applying comet @xcite on asr outputs). there is a recent interest of supervised evaluation metrics that are trained on human quality scores @xcite . however, these human quality scores are precious and somehow limited or nonexistent, specially for low-resource languages. therefore, we propose both an unsupervised and a supervised version of blaser. the results show that on average both unsupervised and supervised blaser outperform their corresponding baseline metrics. in particular, blaser outperforms asr-comet significantly in five language directions and obtains comparable results in two other language directions. our analysis reveals that, while blaser can use both text and speech, encoding speech data give the most significant benefits. in addition, we show that replacing human-written source input and human-written reference with asr-generated ones hurts performance of text-based metrics, which motivates the use of modality-agnostic metrics as blaser. ## approach the underlying idea of our approach is to leverage the similarity between speech segments without requiring intermediate textual representations. compared to asr-based metrics, the advantage of blaser is that it is text-free. in particular, given the source input speech, the translated out-put speech of a s2st model, and the reference speech segment, respectively, we embed them into vectors h src , h mt , and h ref . these embeddings are combined and blaser predicts a score for each translation output, where higher scores suggest better translation quality. 2 the effectiveness of blaser depends on the quality of vector representations encoded from speech segments: it requires rich semantic information to be encoded in the speech embeddings. in this work, we use laser speech encoders @xcite , which we describe below. we note that our approach is generic and can be extended to other encoders. we study blaser under the unsupervised and the supervised settings, which allows it to exploit the information of human ratings, if available. ## experimental framework to show that blaser is useful both in its unsupervised and supervised form, we compare it to several baseline metrics. in this section, we describe the experimental framework for doing this comparison, including the evaluation data, the training and implementation of both baseline and proposed metrics and their evaluation. ## experimental results and analysis in this section we report the main results of our proposed metric blaser, on two different settings (unsupervised and supervised) and we compare it to widely used baseline text-based metrics. additionally, we report an analysis at various levels, including the impact of evaluating using different modalities and a qualitative inspection of several examples to observe scores of various metrics for particular examples. ## conclusion and future work we have introduced blaser, a text-free metric to evaluate speech-to-speech translation, which avoids the dependency on asr models required by popular text-based metrics currently used in s2st. we explored blaser in both unsupervised and supervised settings. experimental results in seven language directions show that blaser outperforms or is comparable to strong text-based metrics in terms of correlation with human scores at the sentencelevel. moreover, our metric is effective in zero-shot scenarios. as for future work, we want to explore the use of speech references generated by humans and the impact of synthesized references. we also want to evaluate blaser at the system-level with a much larger number of s2st systems, and explore different approaches to aggregate the sentence-level scores from blaser and we want to explore different speech and text representations as alternative to laser.