Specification-Aware Machine Translation and Evaluation for Purpose Alignment (2025)

Abstract: In professional settings, translation is guided by communicative goals and client needs, often formalized as specifications.While existing evaluation frameworks acknowledge the importance of such specifications, these specifications are often treated only implicitly in machine translation (MT) research.Drawing on translation studies, we provide a theoretical rationale for why specifications matter in professional translation, as well as a practical guide to implementing specification-aware MT and evaluation.Building on this foundation, we apply our framework to the translation of investor relations texts from 33 publicly listed companies.In our experiment, we compare five translation types, including official human translations and prompt-based outputs from large language models (LLMs), using expert error analysis, user preference rankings, and an automatic metric. The results show that LLM translations guided by specifications consistently outperformed official human translations in human evaluations, highlighting a gap between perceived and expected quality.These findings demonstrate that integrating specifications into MT workflows, with human oversight, can improve translation quality in ways aligned with professional practice.

## introduction high-quality translation in professional settings requires more than a literal rendering of the source text. it must also fulfill a communicative purpose, which depends on factors such as the intended function, target audience, and the broader context of the original text @xcite . a single source text may yield different translations depending on these factors. these contextual factors are typically documented as translation specifications. a specification is a predefined set of conditions that guide the translation process, including purpose, audience, tone, style, and content priorities (iso17100:2015; jtf, 2018). they help translators make informed decisions and ensure that the translation meets user needs @xcite . without such guidance, translators may struggle to begin the process at all. specifications are also essential in translation evaluation. frameworks such as iso 5060 and the multidimensional quality metrics (mqm) emphasize specification-based assessment (iso5060:2024; @xcite . @xcite state that "translation quality can only be assessed in terms of whether or not a translation meets specified requirements and meets its communicative purpose." when specifications are absent or vague, evaluations tend to focus on surface-level features such as lexical accuracy or fluency, rather than on whether the translation achieves its communicative purpose. this perspective is also central to functionalist theories in translation studies, which hold that quality should be judged by how well a translation fulfills its intended purpose in the target context, rather than by equivalence with the source text @xcite . mqm is widely adopted in machine translation (mt) research, including the conference on machine translation (wmt), where it underpins human evaluation @xcite @xcite . its detailed error typology has contributed to translation evaluation. however, specifications are often treated implicitly, and the idea of translation as a goal-oriented process is not fully integrated into mt research. as a result, mt outputs often fall short in real-world applications where purpose and audience matter. this gap is increasingly problematic as industry clients now expect translations to serve specific business objectives @xcite . 1 in response, we propose a framework for specification-aware mt and evaluation. figure 1 outlines this framework, contrasting traditional mt pipelines with our approach, which incorporates contextual information and specificationbased evaluation. this approach reflects the functionalist view that translation should be guided by purpose, audience, and context. incorporating specifications into mt workflows and evaluation is essential for improving translation quality in professional domains. to support this claim, we present the theoretical foundation for understanding the role of specifications in section 3. we introduce a practical guide based on international standards such as iso 5060, iso 17100, mqm, and the jtf guidelines in section 4 (iso5060:2024; iso17100:2015; mqm, 2025; jtf, 2018). we apply this framework to a case study in section 5. our case study focuses on investor relations (ir) texts from 33 publicly listed japanese companies, where alignment with specifications is crucial. we compare five english translations: the company's official version, a proprietary mt output, a basic prompt-based llm output, a prompt-based llm output using specifications, and a prompt-based llm post-edit of the mt output. 2 we evaluate the translations through expert er-2 throughout this paper, llm refers to large language models guided by prompt-based customization. we use post-edit in a broad sense, including automated revision, and not limited to human editing as defined in iso 18587 @xcite . ## theoretical background while translation is often seen as producing an equivalent text in another language, the notion of equivalence has faced criticism in translation studies since the late 1970s. in response, functionalist approaches have gained prominence, viewing translation as a purpose-driven communicative act. skopos theory, a widely cited framework, holds that translations should be guided by their purpose. based on this view, we argue that translation specifications are essential for developing and evaluating mt systems that meet real-world goals. we begin with equivalence theory, which frames translation as reproducing the meaning or value of the source text, a view reflected in early mt systems and many current automatic evaluation methods (section 3.1). we then turn to skopos theory, a functionalist perspective aligned with our emphasis on translation specifications (section 3.2). finally, we discuss how specifications matter not only for translation but also for evaluation (section 3.3). ## a practical guide for specification-aware mt and evaluation we provide a brief overview of our practical framework for integrating translation specifications into both mt workflows and evaluation, where mt is performed using prompt-based llms. 9 a full version is available in appendix f. ## a case study in japanese-to-english translation of investor relations materials we present a case study to show how specificationaware translation can be applied using a promptbased llm. we compare it with human translation and non-prompt-based mt outputs, examining how each is evaluated through both human and automatic methods. this case study puts into practice the notions discussed earlier in section 3 and assesses the effectiveness of our approach in a realworld japanese-to-english translation task. figure 2 provides an overview of the case study. for llm, we use chatgpt via its public interface to simulate a scenario in which non-expert users, such as translators or corporate communications personnel, can control translation output through ## conclusion we demonstrate that translation specifications can improve mt quality and enable more targeted evaluation. we provide a theoretical rationale for the importance of specifications, drawing on skopos theory to support a functionalist perspective. based on this foundation, we outline a practical guide for specification-aware mt using llms, including prompt design, generation, and both error-based and subjective evaluation. in our case study, llm outputs guided by specifications received higher ratings than official translations, google translate, or unguided llm outputs. although comet scores favored google translate, they diverged from human evaluations. these findings suggest that specifications help llms produce more contextually appropriate translations that better align with communicative goals. the gap between human and automatic evaluations highlights the limitations of current metrics in capturing functional adequacy. through this work, we demonstrate the potential of specification-aware mt for professional, real-world use cases.