Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality (2025)

Abstract: Machine Translation (MT) has achieved remarkable performance, with growing interest in speech translation and multimodal approaches. However, despite these advancements, MT quality assessment remains largely text-centric, typically relying on human experts who read and compare texts. Since many real-world MT applications (e.g., Google Translate Voice Mode, iFLYTEK Translator) involve translation being spoken rather printed or read, a more natural way to assess translation quality would be through speech as opposed text-only evaluations. This study compares text-only and audio-based evaluations of 10 MT systems from the WMT General MT Shared Task, using crowd-sourced judgments collected via Amazon Mechanical Turk. We additionally, performed statistical significance testing and self-replication experiments to test reliability and consistency of audio-based approach. Crowd-sourced assessments based on audio yield rankings largely consistent with text-only evaluations but, in some cases, identify significant differences between translation systems. We attribute this to speech’s richer, more natural modality and propose incorporating speech-based assessments into future MT evaluation frameworks.

## introduction reliable evaluation process is critical in the development and refinement of mt systems. mt evaluation (mte) often relies on both automated and manual measurement techniques. manual evaluation is always a preferred choice and provides a deeper understanding of system quality, while automatic evaluation metrics (aems) often serve as a proxy for human judgment @xcite . aems support reusable assessments, system comparison and rapid mt deployment. however, aems face several issues including their inability to handle contextual and cultural nuance, the dependency on reference translation, and domain-specific challenges. therefore, despite being time-consuming and expensive, human assessment is still a fundamental requirement for reliable evaluation. the annual conference on machine translation (wmt) is the primary forum for collecting human judgments to evaluate metrics and participating systems in its shared translation task each year. in early evaluation campaigns, 5-point adequacy and fluency ratings were gathered from participants as the primary evaluation metric @xcite . subsequent wmt campaigns adopted a ranking-based evaluation approach as the official metric @xcite , with rankings still collected from participants of the evaluation campaign. regarding fluency as a measure of mt output quality, @xcite argued that using a 1-100 continuous scale yields better inter-annotator consistency compared to a five-point interval scale. supporting this, @xcite found strong correlations between adequacy and fluency-based evaluations. these findings led wmt to replace relative ranking with adequacy-based direct assessment (da) on a continuous scale as the official metric @xcite . for into-english translation tasks, wmt frequently relied on crowd-workers for its human evaluation campaigns. crowd-based evaluations allow for a fast and cheap mt quality evaluations @xcite . when coupled with quality-controlled annotations, non-expert crowd assessments show better inter-annotator consistency @xcite . however, @xcite found that crowd-workers, compared to professional translators, were less capable of detecting subtle mt errors. studies by @xcite and @xcite also favored the use of professional translators over researchers or crowd-workers due to their ability to differentiate between human and machine translations. consequently, wmt revised its evaluation procedures to prioritize professional transla-tors over crowd-workers @xcite . despite its limitations, crowd-based assessment remains the most convenient choice for certain tasks, particularly monolingual da, which does not require human raters to have bilingual knowledge @xcite , making it easier to conduct. more recently, wmt performed evaluations using error span annotation (esa) protocol @xcite , which requires annotators to assign an overall score to each segment, similar to da and classify errors based on severity (e.g. major or minor). the human evaluation process has evolved over time; however, there is still no consensus on the best approach to evaluating translation quality @xcite . current mt evaluation metrics primarily considers text, despite the fact that many real-world mt applications involve spoken rather than written translation. most importantly, the recent emergence of pre-trained multimodal models @xcite has enabled support for direct speech-to-speech, text-to-speech and speech-to-text translation, however appropriate methods for evaluation for these systems are yet limited or borrowed from text-domain @xcite . we argue that speech, as a natural and expressive modality, can provide more reliable measures of mt quality. to support this claim, we propose incorporating text-to-speech (tts) technology into direct mt assessment, allowing for a direct comparison between text-only and speech-enabled evaluation approaches. our study collects human judgments for german-english translations from wmt shared task using crowd-workers hired via amazon mechanical turk. the evaluation consists of two conditions: (i) a text-only setup, replicating the conventional method where evaluators compare written mt output with a reference translation, and (ii) a text-audio setup, where evaluators listen to the mt output while reading the reference translation. we perform self-replication experiments and statistical significance tests to assess the consistency and reliability of the proposed method. a comparative analysis of these evaluation conditions yields two key findings. first, rankings derived from text-audio evaluations are broadly similar to the original evaluations but also show notable differences compared to conventional setups, with the audio-based method demonstrating a substantially greater ability to detect significant domain #segments avg. doc length conversation 462 6.8 ecommerce 501 18.5 news 506 14.5 social 515 15.6 ## results system rankings are calculated for each setup using filtered judgments-only those that passed the quality control criteria. quality control segments (bad_reference and ask_again) are excluded from the final system rankings. system rankings are based on the mean raw and standardized (z) scores. to compute the standardized score for each system, individual scores are first normalized using each worker's mean and standard deviation (as per equation 1). the standardized scores for all segments corresponding to a system are then averaged to obtain the system-level score @xcite . since hits are structured so that a single worker may assess multiple systems, standardising the scores helps mitigate individual biases and harmonise outputs across workers. the text-only results show increased correlation between the raw and standardised score, with few exceptions such as system lt22 and jdexploreacademy, which would have ranked better according to raw score. this close correlation suggests an even distribution of segments from different systems across workers and may also be attributed to the homogeneous nature of the task (text-only data). it is important to note that these rankings may not fully reflect actual system performance or align with the official wmt rankings, as we used a smaller set of judgments per system compared to wmt22, with the primary objective of investigating and comparing audio-based and text-based evaluations. in the multimodal scenario, the differences in system rankings between z scores and raw scores are more pronounced. based on the raw scores, a a direct comparison of the mean standardized scores across both tables reveals substantial differences in system rankings. for example, in the text-only evaluation, online-w outperforms promt based on standardized scores, whereas the multimodal evaluation ranks promt as the topperforming system. similarly, online-g is ranked sixth, below online-a, online-w, and online-y in the text-only setup, but is rated higher than these systems in the multimodal evaluation. for most other systems, rankings diverge by one or two places between setups, with the exception of human-b, which consistently ranks as the lowestperforming system in both evaluations. ideally, human-b (the human reference translation) should be the top-performing system. however, the results suggest that crowd-workers struggled to distinguish between human translations and mt outputs. this aligns with prior research suggesting that crowd-workers tend to favor literal, straightforward translations, resulting in lower rankings for human translations that deviate from the reference @xcite . ## conclusion we have presented our findings on integrating speech into human evaluation of mt quality. our experiments with crowd workers compared mt system rankings from text-only and speech-enabled evaluation setups. despite using basic tts tools and crowd workers, our study extends mt evaluation beyond traditional text-based assessments, highlighting the potential of audio-based evaluation to provide distinct insights into mt evaluation. as mt research increasingly embraces multimodal translation, our findings provide empirical evidence that text-only evaluation may be insufficient. beyond mt, this approach could benefit fields such as automatic dubbing, ai-assisted interpreting, and multilingual speech interfaces. overall, our study emphasizes the need for more holistic evaluation benchmarks that better reflect the complexity of real-world language use. our code foot_6 and collected human annotation data are freely available.