Testing LLM s’ Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with C hat GPT (2025)

Abstract: This study investigates the capabilities of large language models (LLMs), specifically ChatGPT, in annotating MT outputs based on an error typology. In contrast to previous work focusing mainly on general language, we explore ChatGPT’s ability to identify and categorise errors in specialised translations. By testing two different prompts and based on a customised error typology, we compare ChatGPT annotations with human expert evaluations of translations produced by DeepL and ChatGPT itself. The results show that, for translations generated by DeepL, recall and precision are quite high. However, the degree of accuracy in error categorisation depends on the prompt’s specific features and its level of detail, ChatGPT performing very well with a detailed prompt. When evaluating its own translations, ChatGPT achieves significantly poorer results, revealing limitations with self-assessment. These results highlight both the potential and the limitations of LLMs for translation evaluation, particularly in specialised domains. Our experiments pave the way for future research on open-source LLMs, which could produce annotations of comparable or even higher quality. In the future, we also aim to test the practical effectiveness of this automated evaluation in the context of translation training, particularly by optimising the process of human evaluation by teachers and by exploring the impact of annotations by LLMs on students’ post-editing and translation learning.

## introduction as underlined by the famous quote attributed to yorick wilk: "more has been written about mt evaluation than about mt itself" @xcite . translation evaluation is an essential but highly challenging task. it relies primarily on two approaches. the first consists in assigning scores that reflect translation quality at different levels -be it a segment, a paragraph, a document, or a system. this type of metric is central to assessing machine translation performance. the second approach, more commonly used in the field of translation studies and translation training, although increasingly used in mt evaluation (see, e.g., @xcite ), involves annotating translations by identifying errors (i.e. words that need to be corrected to improve the translation) and categorising them according to an error typology. the high cost of human evaluation, whether in terms of time, technical expertise, effort, or financial resources, has driven researchers to explore ways to automate evaluation. while automatic metrics capable of approximating human judgments, with varying degrees of accuracy, have existed for quite some time and continue to improve @xcite , automating error annotation remains a significantly more complex challenge. until recently, it had attracted little attention and was even considered out of reach. the launch of chatgpt in 2022, and more generally the development of llms since the 2020s, opened up new possibilities for automating this second type of evaluation. llms, initially designed to produce fluid text in natural language, quickly drew the attention of the scientific community for their ability to perform complex tasks, for which they have not been explicitly programmed, taking advantage of their ability to model and manipulate language. numerous experiments have indeed highlighted the possibility of using llms to solve tasks simply by prompting them, i.e. by explaining in natural language how to solve the task at hand. in recent years, pioneering works, reviewed in section 2, have emerged, highlighting the potential of llms, notably chatgpt, to automate translation evaluation, reflecting a growing interest in integrating llms into tasks that, until now, have relied mainly on human intervention. our work follows this dynamic by investigating the possibility of prompting an llm (here chatgpt) to annotate mt outputs by identifying and categorising errors. however, we explore this possibility in a new direction: our experiments stand out by focusing exclusively on specialised translation (lsp, language for specific purposes), in opposition to previous research, which primarily considers general language. specialised translation has considerable economic implications, as it plays a crucial role in industries ranging from law and medicine to technology. specialised translation also raises additional challenges, both for human translators/evaluators and for mt systems. these include accurate processing of specialised terminology, phraseology, and the management of complex patterns inherent to specialised texts. we have carried out different annotation experiments using chatgpt with different prompts and two different mt systems frequently used by professional and non-professional translators alike (deepl or chatgpt). our goals with these experiments are: ultimately, by shedding light on this issue, our work aims to contribute to the creation of hybrid tools, where artificial intelligence and human expertise complement each other to promote more effective learning and teaching, and more accessible selfevaluation or teacher evaluation. the rest of this work is organised as follows. we will start by reviewing related works in section 2, before detailing the context and motivations of our work. we will then present our experimental setting in section 4 and the results of the different experiments we have carried out in section 5. ## related works in recent years, the nlp and translation community has shown interest in exploiting llms as translators, with promising results. many even imply that the future of mt is closely linked to llms and generative ai (cf. e.g. @xcite ; wang et al. (2023); he (2024); jiao et al. (2023); siu (2023) ). if llms are able to produce translations, they should also be able to assess translations and distinguish between high and low-quality translations. this assumption was the starting point for a group of researchers in nlp and translation studies who, in early 2023, began exploring the ability of llms to evaluate translations. ## context and goals in this work, we are conducting experiments aimed at assessing the capabilities of llms to evaluate the quality of translations using prompting only, but with different motivations and objectives than those of the works outlined in section 2. we are investigating whether llms can identify and categorise errors in a translation, and particularly in specialised translation. in light of the works described in section 2, it seems appropriate to focus solely on prompting and not consider fine-tuning, since prompt-based evaluation already delivers strong results and, more importantly, the number of languages and domains for which error-annotated corpora of mt exist (especially for lsp translation) is too small for model fine-tuning to be considered a relevant solution. in this context, given that our goal is to ask an llm, namely chatgpt, to identify and categorise errors, it is necessary to rely on an error typology that covers all the issues likely to arise in translations in order to ensure the effectiveness of this approach and the consistency of annotations. the typology we used is based on the mqm typology and on mellange (multilingual e-learning in language engineering) @xcite , an annotation framework designed for annotating translations in a translation training context. even if it is not the main objective of this work, the possibility of using llms to identify errors in students' translations offers many interesting prospects, whether for evaluation assistance or to help students in their learning. the modifications we have made foot_2 make it possible to adapt these two typologies to the evaluation of specialised translation. this includes, among other factors, a more granular categorisation for terminological errors, foot_3 in order to account for the complex and domain-specific nature of such texts. ## results as explained in section 4.3, we measure the ability of the different prompts considered to correctly identify translation errors by evaluating recall, precision and f 1 score on two corpora of translations generated by different mt systems. the results obtained are summarised in table the total number of gold errors (i.e. errors identified by an expert) in our corpus, the number of predicted errors and the percentage of labels that are correctly identified. figure 2 also shows the distribution of the various scores obtained to enable a more detailed analysis of the performance of the prompts. in the remainder of this section, we will detail the results achieved for each experiment. ## discussion results achieved with the full prompt for deepl mt annotations show satisfactory performances of chatgpt, with an average f 1 score of 0.707 and a fairly strong capability to identify and categorise errors (about 65%). in contrast to what has been claimed by the community (see e.g. lu et al. (2023) ), our experiments show that providing a detailed definition for each type of error in the prompt slightly improves the model's performance, in particular for error categorisation, which was below 50% with the short prompt and reached 65% with the definitions in the prompt. despite these encouraging results, however, the variability in scores, particularly with the short prompt and with chatgpt's mt outputs, suggests that chatgpt's effectiveness with these parameters is not consistent. indeed, a large number of texts in these two experiments have very low recall, precision and f 1 scores. finally, chatgpt shows considerable limitations in self-assessment, with significantly lower performance when annotating its own translations (f 1 score of 0.496). this result highlights a possible bias and the model's lack of ability to objectively evaluate its own productions, which justifies our motivations to set up annotation experiments with other open-source llms, as have @xcite and @xcite , demonstrating that comparable -or even better -performance can also be achieved with open-source llms. in order to further assess the relevance and usability of chatgpt's outputs, we provide an example of an individual annotation foot_13 performed by the model (see figure 3 ). this annotation by chatgpt clearly shows that the llm annotates the errors as instructed in the prompt, respecting the requested output format (sentence by sentence), giving the error span, the error category, the associated label and an explanation. however, we did not directly request explanations for each annotated error in our prompt, but the model seems to do so systematically. for these sentences, the explanations the model provides make sense. these explanations highlight the potential usability of these experiments with chat-gpt: since the llm provides explanations and possible solutions for each potential error, it seems reasonable to consider conducting other experiments in a real-life classroom setting using chatgpt's annotations (see section 7). this figure shows the sentence-by-sentence annotation performed by chatgpt, which identifies the error, categorises it, assigns a label and provides explanations and solutions for improvement. the initial output of chatgpt is in french, since the prompt provided is written in french. for the purposes of this article, we have translated it into english. ## conclusion this study explored the use of chatgpt for annotating mt outputs based on a customised error typology adapted to our specific needs in a specialised translation training setting. the annotations generated by the model were compared with reference human annotations to evaluate its ability to identify and categorise errors in a translation generated by deepl or chatgpt. initial results are encouraging, particularly with external machine translations, where chatgpt identified and categorised most errors with reasonable accuracy, in particular with the long prompt containing the definition for each type of error. however, its performance was far less reliable when evaluating its own translations. another key finding from our experiments is that, given the lack of a significant difference in error identification between the full and short prompts, it seems reasonable to suggest that the structure and degree of detail of the prompt does not have a major impact on chatgpt's performance in this annotation task. this could indicate that chatgpt is performing its annotations efficiently even without detailed instructions (without a definition for each error), relying more on its knowledge acquired during training rather than on the specifications of the prompt. however, this also suggests that while llms can rely on their pre-trained knowledge to identify errors, their ability to categorise these errors correctly benefits from clear, structured instructions and definitions. future work future experiments will extend this research to open-source llms, focusing on their potential to provide annotations of comparable or superior quality. these models, with greater transparency, will be evaluated not only for their accuracy and capabilities in annotating translations but also for their ease of integration into automated workflows for translation quality assessment. ultimately, our aim is to test the effectiveness of this automated evaluation by llms in a practical context of translation training. firstly, we intend to optimise the human evaluation process. specifically, with teachers annotating students' translations, we aim to examine whether the use of annotations generated by llms can reduce the cognitive effort associated with the annotation process. additionally, we intend to carry out experiments with translation students and test whether the use of llm annotations help them improve the quality of their mt post-editing. our aim is to test our prompt with other domains, notably earth and planetary science.