IIIT HYD ’s Submission for WMT 23 Test-suite Task (2023)

Abstract: This paper summarizes the results of our test suite evaluation on 12 machine translation systems submitted at the Shared Task of the 8th Conference of Machine Translation (WMT23) for English-German (en-de) language pair. Our test suite covers five specific domains (entertainment, environment, health, science, legal) and spans five distinct writing styles (descriptive, judgments, narrative, reporting, technical-writing). We present our analysis through automatic evaluation methods, conducted with a focus on domain-specific and writing style-specific evaluations.

## introduction neural machine translation has made significant strides and has achieved a level of quality that proves valuable in numerous everyday scenarios. nonetheless, various assessment methods for machine translation suggest that there is still ample room for enhancement. one such evaluation approach, geared towards identifying translation deficiencies in a more systematic manner, involves the utilization of test suites or challenge sets. unlike conventional evaluations that draw test sets from random everyday texts, test suites comprise sentences that are carefully curated or selected to assess the mt systems' competence in translating specific linguistic phenomena. in this context, we present the results obtained from applying these test suites, analyzing the performance of state-ofthe-art systems concerning numerous linguisticallydriven phenomena. these test suites were administered to 12 mt systems submitted during the 8th conference of machine translation (wmt23) @xcite for english-german language pair. we have developed a comprehensive test suite that encompasses five distinct domains (entertainment, environment, health, science, legal) and spans five different writing styles (descriptive, judgments, narrative, reporting, technical writing). the primary objective of the test suite is not to gauge a system's overall translation performance, as this aspect is already evaluated through manual assessment and various additional metrics within the primary shared task. instead, the test suite focuses on assessing the translational proficiency across diverse domains and writing styles. ## test suite details ## conclusion this paper provides a comprehensive overview of our evaluation of 12 machine translation systems designed for the english-german language pair, all of which were submitted to the shared task during the 8th conference on machine translation (wmt23). our evaluation comprises a robust and diverse test-suite covering five distinct domains and encompassing five diverse writing styles. we conduct our analysis through a combination of automated assessments and manual evaluations, with a particular focus on domain-specific and writing style-specific performance. based on our automatic evaluation, it is evident that both online-b and online-y consistently surpassed other mt systems in performance across a diverse array of writing styles and domains.