S peech QE : Estimating the Quality of Direct Speech Translation (2024)

Abstract: Recent advances in automatic quality estimation for machine translation have exclusively focused on written language, leaving the speech modality underexplored. In this work, we formulate the task of quality estimation for speech translation (SpeechQE), construct a benchmark, and evaluate a family of systems based on cascaded and end-to-end architectures. In this process, we introduce a novel end-to-end system leveraging pre-trained text LLM. Results suggest that end-to-end approaches are better suited to estimating the quality of direct speech translation than using quality estimation systems designed for text in cascaded systems. More broadly, we argue that quality estimation of speech translation needs to be studied as a separate problem from that of text, and release our [data and models](https://github.com/h-j-han/SpeechQE) to guide further research in this space.

## introduction recent progress in quality estimation (qe) @xcite makes it possible to automatically rate the quality of machine translation (mt) given only the input and output of an mt system. qe ratings have been found to correlate well with human judgments, sometimes as well as reference-based metrics @xcite @xcite . however, this work has focused on text translation. meanwhile, the rapid development of speech technology @xcite has expanded the use of speech translation (st) applications in daily life, thus increasing the need to predict the reliability of their output. this raises the question of whether quality estimation for st can be performed using a combination of state-of-the-art automatic speech recognition (asr) and text-based qe (text-qe or mtqe) methods. however, relying on a cascade of asr and text-qe systems presents two major issues: (1) the current top-performing st models directly translate the audio input into target language text without transcribing the audio, making it inefficient to run an additional asr system to generate an input for the text-qe module. (2) asr transcriptions of the audio input may not match the gold transcription, potentially misleading the text-qe system. hence, we hypothesize that end-to-end approaches might be better suited for this task. in light of these issues, we formulate the task of quality estimation for speech translation (speechqe or stqe, figure 1 ) and explore both cascaded and end-to-end (e2e) systems for this task (figure 2 ). while we rely on existing asr and text-qe modules for the cascaded system, we introduce a novel e2e speechqe model architecture to address the lack of a dedicated end-to-end system for this task. our design incorporates a pretrained speech encoder and a large language model (llm) to leverage their existing capabilities in extracting high-quality audio features and handling translation-related tasks. to conduct a thorough evaluation, we contribute an evaluation benchmark and training data for speechqe from diverse st outputs scored with reference-based metrics. results show that e2e models outperform the cascaded system based on a state-of-the-art (sota) asr module in correlation with both (1) human direct assessment ratings and (2) metric scores. additionally, our e2e model can detect error spans to some extent in a zero-shot fashion, though the best results are still achieved by cascaded systems with sota asr. qualitative analysis highlights the robustness of e2e models against wrong speech representation in score prediction, error span detection, and severity prediction. based on this evidence, we argue that speechqe should be studied as a distinct problem from text-qe. ## background quality estimation makes it possible to assess translation quality without reference translations, which is essential for practical use cases @xcite . qe signals can benefit end users by helping them decide how to rely on outputs in casual and high-risk settings alike @xcite . they can also benefit downstream tasks or enhance mt itself @xcite . the qe task has been framed in various ways, including predicting sentence-level quality ratings @xcite or word-level binary tags of ok/bad @xcite . while a wealth of methods have been developed for these tasks, recent work has shown the benefits of developing solutions to address them jointly. openkiwi @xcite streamlined qe by supporting both word-level tagging and regression toward a sentence-level score within a unified toolkit @xcite . it was further improved with a training recipe that better supports multilingual generalization @xcite . together with the development of learned metrics for reference-based evaluation @xcite , this set the stage for a single or family of models that flexibly rate the quality of mt output with or without access to a reference human translation @xcite with high correlations with human quality ratings @xcite . @xcite even integrates both sentence-level evaluation and error span detection capabilities while categorizing error spans, thereby enriching the quality measures. meanwhile, quality estimation for speech translation remains understudied. @xcite address the task of tagging each word in an st output as good or bad, using asr and mt features. their approach can be viewed as a cascaded speechqe system, which propagates a confidence score in a pipeline of asr and statistical machine translation (smt) modules. blaser2.0 @xcite produces a similarity score between a translation output and input, using sonar sentence-embeddings that can compare either speech or text @xcite . while this enables speechqe, this approach was initially designed for speech-to-speech translation @xcite , and was exposed to only a small amount of training data with quality labels. with advances in st technology and their growing use @xcite , there is a need for qe to support st scenarios where intermediate automatic speech recognition (asr) outputs are not available, along with new evaluations to correctly gauge the effectiveness of quality estimation in speech translation. ## speechqe: task and models we define the task of estimating the quality of speech translation (speechqe or stqe 2 ), before introducing our cascaded and e2e systems. in this work, we focus on predicting sentencelevel scores and measuring the correlation of reference ratings provided by humans or referencebased metrics @xcite . additionally, we will explore an error span detection task @xcite in section 5.4, to broaden the scope of qe beyond holistic numerical ratings. we refer to a reference-based metric as metric. given a reference target text r, an mt hypothesis h and optionally the mt source text t, the metric rates the quality of h as a score m: likewise, we refer to a text quality estimation system as text-qe. it produces an output score q given only a source text t and an mt hypothesis h. in the speechqe task (figure 1 ), given the source audio a and the translation hypothesis h, a system outputs the quality score q for this hypothesis: ## experimental settings in this section, we describe the construction of the speechqe benchmark as well as the configuration of the evaluated systems. ## results we first present our main results by comparing speechqe ratings with reference-based metrics ( § 5.1), then turn to using human ratings of translation quality ( § 5.2). we add the results of varying model sizes and architecture of the cascaded sys-tem. ( § 5.3). finally, we evaluate our models on a zero-shot error detection task ( § 5.4) and conduct a qualitative analysis of outputs ( § 5.5). we additionally evaluate and train our systems with out-of-domain settings (appendix a and b). ## related work recent work has explored how to inject additional modalities into a model pre-trained on a single modality. various configurations have been proposed to meet different demands including speech modality into text-llm @xcite , visual modality into text-llm @xcite , visual modality into speech foundation model @xcite @xcite , and audio-visual modalities into text-llm @xcite . when injecting the speech modality into text-llm, the main challenges are aligning long speech signals to corresponding text sequences with the same semantic contents, while avoiding overfitting to default training tasks like asr and st. several methods of compressing and aligning the speech and text sequence include the use of convolutional layer @xcite , ctc compression @xcite , and random downsampling @xcite . many mention the problem of task overfitting to homogeneous fixed instruction training on limited tasks. they suggest training on many diverse tasks @xcite or tuning on diverse speech instructions with tts-augmented instruction datasets @xcite . however, most of these works focus on asr, st, qa, and general instruction following within speech comprehension tasks @xcite . this paper initiates their application to the understudied speechqe problem. ## conclusion this work focused on the task of speechqe, evaluating the quality of speech translation using both cascaded systems and end-to-end systems. we developed an e2e speechqe model, proposing methods for corpus creation, training strategies, and architectural design. our findings indicate that e2e systems are generally better suited to estimate the quality of direct speech translation. additionally, we examined the error span detection task for st finding that e2e speech model transfer ability from text-based llm while cascaded systems with state-of-the-art asr still hold advantages in performance. we conclude that speechqe needs dedicated attention separate from text-qe, due to the growing use cases of st and the significant potential for further improvements in this field. quality estimation in the speech domain opens up a wide range of potential applications. in addition to the promise of helping people use speech translation systems more reliably in their daily lives, quality estimation can enhance speech translation itself, for instance by enabling prefix-toprefix quality estimation for re-translation and simultaneous speech translation. we contribute data, code, and models to support future work that broadens the scope of the translation-related tasks for the speech domain.