Context-Aware Monolingual Evaluation of Machine Translation (2025)

Abstract: This paper explores the potential of context-aware monolingual evaluation for assessing machine translation (MT) when no source is given for reference. To this end, we compare monolingual with bilingual evaluations (with source text), under two scenarios: the evaluation of a single MT system, and the comparative evaluation of pairwise MT systems. Four professional translators performed both monolingual and bilingual evaluations by assigning ratings and annotating errors, and providing feedback on their experience. Our findings suggest that context-aware monolingual evaluation achieves comparable outcomes to bilingual evaluations, and highlight the feasibility and potential of monolingual evaluation as an efficient approach to assessing MT.

## introduction mt evaluation has traditionally depended on comparing the translated text with its corresponding source text to assess the mt performance. however, several works in the area have explored the potential of monolingual evaluation @xcite @xcite @xcite . building on earlier studies suggesting that monolingual post-editors, even without access to the source text, can enhance mt output quality @xcite @xcite , schwartz (2014) examined the effectiveness of monolingual post-editing performed by domain experts in improving mt quality and reducing reliance on bilingual post-editing. the author reported that a subject-matter expert monolingual post-editor confidently corrected 87% of sentences based solely on the target text, with 96% of these corrections deemed appropriate following bilingual verification. while our work presented here differs in focus, exploring monolingual evaluation rather than post-editing, it highlights a complementary strength: the importance of context-awareness. the work of @xcite investigated various evaluation approaches with monolingual evaluation, such as the inclusion or exclusion of reference translations; the impact of additional contextual information (e.g., surrounding sentences); and the influence of displaying metadata to annotators. they concluded that monolingual evaluations, even when crowd-sourced, could effectively measure mt system performance. recent advancements in context-aware evaluation methodologies have introduced a deeper focus on the appropriateness and coherence of translations within broader textual contexts @xcite @xcite . these approaches challenge traditional evaluation methodologies that rely on isolated sentences, highlighting the limitations of sentencelevel assessments. by integrating contextual information, evaluators can better capture nuanced aspects of translation quality, such as consistency, discourse coherence, and appropriateness within the larger narrative. building on this, our study investigates the potential of monolingual assessments in the context, addressing the following research question: are monolingual assessments of mt comparable to bilingual assessments? to explore this, we examine two key scenarios: a) can context-aware monolingual assessments of a single mt output provide results comparable to bilingual assessments? b) can context-aware monolingual assessments of two different mt outputs (pairwise comparison) achieve comparability with bilingual assessments? the research question with the two scenarios are illustrated in figure 1 . a positive answer to the question would open potential benefits for facilitating the evaluation process. ## results this section presents the outcomes of the experiments, organized by the tasks performed. the first part, section 3.1, details the results from task 1, carried out in the monolingual modality, while section 3.2 focuses on task 2, conducted in the bilingual modality. the results for both tasks are reported across the two scenarios -single mt and pairwise mt -covering the overall ratings and error annotation. ## analysis of the results the aim of this evaluation is to compare the extent to which mt systems can be assessed monolin ## conclusion and future work this study, while limited in scope, highlights the potential of monolingual evaluation as a practical and effective alternative method for assessing mt performance. the results suggest that monolingual evaluation may be comparable to bilingual evaluation in assessing the performance of one mt output and in comparing multiple mt outputs -an approach commonly used in companies and academia alike. monolingual evaluation seems to be particularly effective when using contextual information, which reinforces the importance of assessing translations within context rather than isolated sentences. monolingual evaluation also seems to show a robust capability for error detection, with approximately 98% of errors identifiable by examining the target language alone. these findings suggest the potential usefulness of monolingual approaches as both a complementary and standalone alternative method for mt evaluation. additionally, findings from the survey suggest that monolingual evaluation may not significantly differ from bilingual evaluation in terms of translators confidence and satisfaction. notably, monolingual evaluation may be a faster task. while evaluators reported slightly higher effort -likely due to the need for rereading in the absence of a source text -this increased cognitive effort does not translate to longer task completion times. as a result, monolingual evaluations may offer a more time-and cost-efficient approach to mt assessment without compromising reliability. by suggesting the potential effectiveness of monolingual evaluation, this paper contributes to the field of mt evaluation and highlights potential benefits. one notable potential advantage is the expansion of the evaluator pool, as monolingual evaluations require proficiency in only one language, making it easier to find qualified evaluators, even for less commonly spoken language pairs. furthermore, the reliance on monolingual assessment aligns well with the needs of generative ai evaluation, where systems produce text in a single language. this suggests that monolingual evaluation could serve as a valuable framework for assessing text generation tasks beyond mt, including ai-generated content. you will be assessing translations at the segment level, where a segment may contain one or more sentences. each segment is aligned with a corresponding source segment, and both segments are displayed within their respective documents. annotate segments in natural order, as if you were reading the document. you may return to revise previous segments. please identify all errors within each translated segment, up to a maximum of five. if there are more than five errors, identify only the five most severe. if it is not possible to reliably identify distinct errors because the translation is too badly garbled or is unrelated to the source, then mark a single non-translation error that spans the entire segment. to identify an error, highlight the relevant span of text, and select a category/sub-category and severity level from the available options. (the span of text may be in the source segment if the error is a source error or an omission.) when identifying errors, please be as fine-grained as possible. for example, if a sentence contains two words that are each mistranslated, two separate mistranslation errors should be recorded. if a single stretch of text contains multiple errors, you only need to indicate the one that is most severe. if all have the same severity, choose the first matching category listed in the error typology (eg, accuracy, then fluency, then terminology, etc). please pay particular attention to document context when annotating. if a translation might be questionable on its own but is fine in the context of the document, it should not be considered erroneous; conversely, if a translation might be acceptable in some context, but not within the current document, it should be marked as wrong. there are two special error categories: source error and non-translation. source errors should be annotated separately, highlighting the relevant span in the source segment. they do not count against the five-error limit for target errors, which should be handled in the usual way, whether or not they resulted from a source error. there can be at most one non-translation error per segment, and it should span the entire segment. no other errors should be identified if non-translation is selected. -translation is missing content from the source.