Linguistically Motivated Evaluation of the 2023 State-of-the-art Machine Translation: Can C hat GPT Outperform NMT ? (2023)

Abstract: This paper offers a fine-grained analysis of the machine translation outputs in the context of the Shared Task at the 8th Conference of Machine Translation (WMT23). Building on the foundation of previous test suite efforts, our analysis includes Large Language Models and an updated test set featuring new linguistic phenomena. To our knowledge, this is the first fine-grained linguistic analysis for the GPT-4 translation outputs. Our evaluation spans German-English, English-German, and English-Russian language directions. Some of the phenomena with the lowest accuracies for German-English are idioms and resultative predicates. For English-German, these include mediopassive voice, and noun formation(er). As for English-Russian, these included idioms and semantic roles. GPT-4 performs equally or comparably to the best systems in German-English and English-German but falls in the second significance cluster for English-Russian.

## introduction over the past few years, we have witnessed substantial advancements in machine translation (mt) alongside the rapid expansion of large language models (llms). these developments have brought translation quality up to par with human capabilities. however, these seemingly perfect translations might contain fine-grained linguistic errors that go unnoticed or get overlooked entirely in automated evaluation. a more structured approach to identifying linguistic issues in the outputs involves the use of test suites or challenge sets to systematically evaluate the system's performance on specific tasks. the current study focuses on providing a fine-grained evaluation of the translation proficiency of the latest generation of neural machine translation (nmt) against the latest generation of llms, exemplified by . one of the objectives is therefore to assess whether gpt-4, as an llm, excels nmt in managing specific linguistic phenomena. although our focus lies on gpt-4, we are aware that there might be other llms participating in the sub-task. in this context, we are presenting the results of the test suites analyzing state-of-the-art systems in terms of numerous linguistically motivated phenomena. these test suites foot_0 were applied to the mt systems submitted for evaluation at the 8th conference on machine translation (wmt23; @xcite across multiple language directions: german-english, english-german, and english-russian. this paper is structured as follows: section 2 goes through related work, whereas section 3 explains how the test suite was created and applied. section 4 outlines the setup of this year's experiment, whose results are detailed in section 5. section 6 concludes the paper with an outlook to future research. ## related work the origins of test suites can be traced back to the early days of machine translation in the 1990s @xcite @xcite . several researchers have adopted the use of test suites to achieve their goals. for instance, @xcite employed test suites to evaluate pronoun translation. other studies (e.g. @xcite compared different mt technologies, while @xcite explored their applicability in quality estimation methods. the machine translation test suite track has played a significant role in this context, leading to the creation of test suites focusing on specific translation-related phenomena. for example, the work by @xcite addressed the translation of morphologically complex words from german into english. additionally, semenov and bojar's research delved into documentlevel translation quality assessment. these test suites, however, focus on one or at most a few phenomena per test suite, including the works by @xcite 2020 ), @xcite . test suites, in conjunction with human evaluation, are also instrumental in assessing the quality of machine translation metrics @xcite . our approach enables a comprehensive analysis that spans over a hundred linguistic phenomena across three language pairs @xcite . it incorporates semi-automated human evaluation, combining efficiency with in-depth analysis. due to our participation in past shared tasks since 2018 @xcite , we are able to analyze the development of machine translation systems over the years. with the growing interest surrounding llms, researchers have been increasingly focused on evaluating gpt-'s performance in mt. for instance, the paper by @xcite concludes that chatgpt performs competitively with commercial translation products on high-resource european languages. a comprehensive evaluation across 18 languages of gpt models versus best-performing wmt-22 systems including human evaluations by @xcite supports the previous finding. other research explores these differences in terms of the literalness of translations produced by standard @xcite . @xcite have tested chatgpt for handling context-related linguistic phenomena such as coreference, terminology, etc. to show that it performed even better than other mt engines. this current paper also places a specific focus on evaluating gpt's performance compared to other systems in the shared task. ## experiment setup in this paper, we present the evaluation of 37 systems with our test suite. the systems were submitted to the news translation task of the eighth conference on machine translation (wmt23; kocmi et al., 2023): 13 systems for german-english, 12 systems for english-german, and 12 systems for english-russian. this year is the third time that the english-german systems are being evaluated with our test suite and the second time for the english-russian systems. every year, manual work is involved upon receiving the system translations as there are usually a number of translation outputs that are not yet covered by the existing rules in the database (the warnings). at the beginning of the evaluation process this year, there were on average 10.7 % of warnings for german-english, 15.6 % for english-german, and 70.6 % for english-russian. the english-russian test has grown significantly since last year and in comparison with the other sets had more new items that had not been evaluated before. it was also expected that english-german would have a higher amount of warnings than german-english as there were some new categories added to the english-german test suite. one annotator with extensive linguistic knowledge of the three languages conducted the manual evaluation of the warnings; problematic cases were discussed with several translation experts to exclude subjectivity. the manual evaluation took around three and a half weeks and involved around 55 person-hours. after the manual evaluation, there were on average 7 % of warnings left for german-english, 6.8 % for english-german, and 6.9 % for english-russian. as mentioned above, test sentences with at least one warning by one system were excluded from the analysis to achieve a fair comparison between the systems under inspection. as this year, we saw a lot of problematic outputs that could not be properly evaluated, this report deals with a significantly less number of test items than in the previous years. we suspect that some of these can be explained by possible models' hallucinations: a number of the mt outputs this year had some parts of the sentences repeated twice or parts of the test items were not translated at all or seemed out of place altogether. to illustrate, one unevaluated output was from the phenomenon intransitive-perfect "ich bin gerannt" ("i ran" or"i was running") that in the submission of lan-bridge (wu and hu, 2023) was rendered "i'm a manager". as a result, our analysis was conducted on 3234 (58.9 %) test sentences for german-english, 3109 (64.8 %) test sentences for english-german, and 909 (73.8 %) test sentences for english-russian. ## results all result tables can be found in the appendix. ## conclusions and outlook this paper presents a fine-grained, linguistically motivated test suite to evaluate machine translation outputs. the test suite was applied to evaluate and compare the outputs of 37 machine translation systems in three different language pairs: german-english, english-german, and english-russian. while the evaluation showed high scores for all language pairs, there was a clear drop in accuracy when dealing with structurally different languages, such as english and russian. for this language pair, gpt-4's performance falls in the second significance cluster. although we didn't observe a systematic significant difference between gpt-4 (5-shot) and other systems, it is important to highlight that gpt-4 shows competitive results in the context of our evaluation. this indicates that gpt-4, a general model, remains competitive in mt and sometimes performs better than some specialized nmt systems. nevertheless, many linguistic nuances still pose difficulties for these models, demonstrating the continuous need for study and improvement in the field of mt. in terms of linguistic coverage, the current test suite stands out as one of the most extensive available. the semiautomated approach offers a more effective, while still fine-grained analysis in comparison to a typical human evaluation. when paired with other automated metrics or mqm analysis, this method can be seen as a valuable addition offering deeper insights into translation quality. the test suite approach is also highly versatile, allowing for the analysis of various tasks performed by llms in different contexts.