Effects of automatic alignment on speech translation metrics (2025)

Abstract: Research in speech translation (ST) often operates in a setting where human segmentations of the input audio are provided. This simplifying assumption avoids the evaluation-time difficulty of aligning the translated outputs to their references for segment-level evaluation, but it also means that the systems are not evaluated as they will be used in production settings, where automatic audio segmentation is an unavoidable component. A tool, mwerSegmenter, exists for aligning ST output to references, but its behavior is noisy and not well understood. We address this with an investigation of the effects automatic alignment on metric correlation with system-level human judgments; that is, as a metrics task. Using the eleven language tasks from the WMT24 data, we merge each system’s output at the domain level, align them to the references, compute metrics, and evaluate the correlation with the human system-level rankings. In addition to expanding analysis to many target languages, we also experiment with different subword models and with the generation of additional paraphrases. We find that automatic realignment has minimal effect on COMET-level system rankings, with accuracies still way above BLEU scores from manual segmentations. In the process, we also bring the community’s attention to the source code for the tool, which we have updated, modernized, and realized as a Python module, mweralign.

## introduction speech translation systems operate over a cascade of subtasks, including audio segmentation, speech recognition, and translation. each of these components introduces noise and error into the process. in recent years, some of these tasks have been combined, i.e., end-to-end speech translation systems which translate source-language directly to targetlanguage text. however, audio segmentation is still 1 pip install mweralign often treated separately. as discussed recently in @xcite , this creates a problem for the segment-level evaluation that is standard in machine translation. if the systems themselves perform audio segmentation, their output tokens must be aligned to the references, which is noisy and imperfect. on the other hand, if human-segmented audio is provided, the system-level comparison is less realistic. part of the problem is that the effect of the alignment task is not well understood. evaluations that do incorporate audio segmentation typically rely on a mwersegmenter @xcite , which uses a variant of levenshtein distance to align the system's output to a fixed set of segment-level references. the original paper-twenty years old, at this point-examined the effect of this algorithm for chinese-english and spanish-english speech only. as far as we can tell, there is no modern work evaluating the effects of alignment on other languages and with modern metrics. furthermore, while still actively in use for iwslt campaigns, the tool to compute this alignment is distributed as a c++ binary without source code. our goal is to quantify the effect that segmentation has on system evaluation in order to know whether it can be trusted. this paper updates (2005)'s original investigations in a number of ways. we we find that alignment imposes minimal costs to the accuracy of human rankings. when combined with comet22, correlation with human rankings sometimes helps, sometimes hurts, but is always far above computing bleu scores from the original, provided segmentations. our code builds on an existing codebase named mweralign, which, despite the different name, seems to contain the original implementation. we modernize and extend this code, wrapping it python via pybind11 @xcite , and publishing it on pypi. foot_0 ## related work the earliest work we are aware of for the speech alignment problem is @xcite . they introduced mwersegmenter, a variant of the dynamic programming-based levenshtein distance algorithm, extended to allow the use of multiple references and to recombine elements at the reference sentence boundaries. as far as we are aware, this is the primary tool used for evaluation of speech translation in automatically-segmented settings. the evaluation in matusov et al. focused on zh-en and es-en. since then, there have been at least a few other investigations of alignment quality. @xcite looked at alignment in a subtitling scenario; they introduced a new metric, suber, foot_1 a variant of the edit distance algorithm which incorporates block, line break, and timing information from subtitling metadata as constraints on the alignment. @xcite evaluated the correlation of automatic metrics with en-de human quality data under different alignment schemes, and recommended against the use of mwersegmenter with comet, since neural metrics are trained on complete sentences. despite this work, speech evaluation campaigns have continued to use pre-segmented data. in a recent survey, @xcite note that ignoring the complexities of speech segmentation means that speech systems are not evaluated in their proper real-world setting. one of the reasons that evaluation campaigns have continued to use pre-segmented data may because of lingering problems with alignment tools. first, it is not clear how to apply them to languages with different scripts and whitespace conventions. second, the existence of the source code is not widely known; iwslt has recently distributed only a compiled c++ binary, which requires separate data manipulation. minor hurdles like these can play a big role in pre-@xmath0. (k 1 = 1) @xmath1. (k 2 = 3) i conquered. (k 3 = 5) @xmath2. i saw. i won. ## aligning tokens to reference sentences this section introduces the as-wer algorithm @xcite , implemented in a publicly available tool, mwersegmenter. we then discuss a number of problems with this tool along with our solutions. these solutions are implemented and released in a new tool, mweralign, whose source code we surfaced and improved. ## evaluation on shorter segments the wmt24 was collected at the paragraph level. a consequence of this is that the segments are much longer and there are fewer boundary points for the system to navigate. to assure that this does not present an uncharacteristic picture, and for correspondence with matusov et al., we also evaluate on wmt22 @xcite data for chinese and for german (both directions). ## conclusion we have undertaken a modern investigation of word alignment for speech translation, testing it on a range of language pairs with full source, reference, system outputs, and-critically-human evaluations. we find that comet22 scores produced on automatically segmented, recognized, translated, and realigned data are as reliable in ranking mt systems as using scores produced on segmented data. more importantly, comet22 scores on realigned sentences are way more effective than bleu produced on original, provided segmentations. this suggests that speech translation can be evaluated with realignment of system outputs using unsegmented audio as input, addressing a problem raised by @xcite . we release our changes using the name of the codebase we found and improved, mweralign. for future work: segment. en-cs en-de en-es cs-uk en-ru en-uk en-ja en-zh ja-zh