Translate your Own: a Post-Editing Experiment in the NLP domain (2024)

Abstract: The improvements in neural machine translation make translation and post-editing pipelines ever more effective for a wider range of applications. In this paper, we evaluate the effectiveness of such a pipeline for the translation of scientific documents (limited here to article abstracts). Using a dedicated interface, we collect, then analyse the post-edits of approximately 350 abstracts (English→French) in the Natural Language Processing domain for two groups of post-editors: domain experts (academics encouraged to post-edit their own articles) on the one hand and trained translators on the other. Our results confirm that such pipelines can be effective, at least for high-resource language pairs. They also highlight the difference in the post-editing strategy of the two subgroups. Finally, they suggest that working on term translation is the most pressing issue to improve fully automatic translations, but that in a post-editing setup, other error types can be equally annoying for post-editors.

## introduction in most, if not all scientific domains, academic communication and publication activities take place mostly in english @xcite . while sharing a common language can be viewed as a facilitating factor in many cases, it also generates tensions, frictions and inequalities @xcite , and hinders the exposure of science that is not discussed in english. furthermore, in non-english-speaking countries, it creates a linguistic barrier between the scientific community and the general public that can only amplify misunderstandings and doubts. these issues have motivated calls for changes as expressed in the "helsinki initiative". 1 among the natural language processing (nlp) community, this has motivated the acl 60-60 special initiative, foot_1 aimed at using automatic tools (speech recognition, machine translation (mt)) and resources (multilingual term lists) to help remove these barriers. in this paper, we report our attempts to use existing mt technologies to translate english scientific documents in the nlp domain into french. as has been well documented for the biomedical domain in the course of the challenges organised at the conference on machine translation since 2016 (see @xcite for the latest published edition), academic texts pose specific translation challenges, related notably to term translation and the generation of lexically consistent outputs. our main goal in this work is to evaluate the current state-of-the-art in mt for the translation of academic nlp texts with a view to using mt to aid nlp authors in the translation and postediting of abstracts in non-english languages. we base our evaluation on manually post-edited documents by two populations of post-editors: apprentice and well-trained professional translators on the one hand and nlp experts (academics) who are encouraged to post-edit their own articles on the other. the results of this pilot study will help us design and organise a large-scale experiment that will ultimately cover all scientific domains. the main questions we aim to answer are the following: (a) what is the effort needed for academics to post-edit automatic translations of texts in their domain of expertise? (b) can we measure the quality of the resulting translations? (c) can we see a difference between existing translation tools? and (d) what are the residual errors that still hinder the translation of academic publications? to answer these questions, we designed a postediting protocol aimed to facilitate the voluntary participation of academics in our domain and collected a set of more than 350 abstracts in the nlp domain, which were post-edited once or several times. a large subset of them are also associated with post-editor feedback on the types and severity of errors present. we analysed them in terms of the post-editing effort, measured using hter @xcite and studied them in terms of differences in post-editing patterns. we release the resulting corpus and the code for the post-editing interface for future use. foot_2 ## related work numerous challenges are faced when developing and adapting nlp models to scientific texts, including how to handle domain-specific terminology (including acronyms), and how to ensure coherence at the document level. in recent years, the development of such tools has been a growing area of interest for nlp researchers, with multiple models being published for the scientific and scholarly domains, e.g. scibert @xcite , pubmedbert @xcite , galactica @xcite and scholarbert @xcite . specifically for mt, there have been several initiatives, including the recent iwslt shared task on translating acl presentations @xcite . the project that is closest to our own is the cosmat project @xcite , whose aim was to develop a pipeline for integrating the translation of scientific documents into the hal foot_3 archiving platform for english-french translation. a few corpora are available for scientific document translation, covering different types of publications. the biomedical task at wmt, for instance, has produced parallel test sets for a number of years extracted from article abstracts from pubmed that are available in several languages (jimeno @xcite @xcite @xcite @xcite . the scipar parallel corpus of scientific texts @xcite is composed of master's and doctoral theses across several domains and in multiple languages. s2orc @xcite is also multi-discipline and contains monolingual english articles from semantic scholar. in the nlp domain, @xcite compiled and explored a large-scale comparable corpus of about 65k nlp papers from multiple sources, while @xcite focus on french, providing a monolingual corpus from the taln conferences. evaluating mt for scientific documents is challenging, particularly as standard metrics may well underestimate the impact of mistranslating scientific terminology if they are considered equal to other words. this is particularly the case for simple surface-based metrics such as bleu @xcite , but is also a currently unknown factor for other automatic metrics such as comet @xcite . according to the human evaluations of the wmt biomedical shared tasks, e.g. @xcite , term translation was one of the factors most impacting judgments of quality over other factors such as style and naturalness. another problem is the scarcity of parallel texts that can be used for reference-based evaluation. moreover, those that exist may not be perfect translations, either because there is no guarantee that two abstracts for the same paper in multiple languages were intended to be perfect translations or because the authors are non-native speakers of at least one of the languages. this therefore motivates alternative approaches to evaluation, including reference-less evaluation (for automatic evaluation, this would refer to quality estimation @xcite ) and human evaluation, through post-editing or error annotation for example. post-editing has previously been used as a means of evaluating mt quality, either through the time taken to render a text to an acceptable standard or (largely related) through the number of changes that were made, which is the basis for the hter metric ("human-targeted translation edit rate") @xcite . this task-based evaluation strategy is less costly both financially and in terms of effort on the part of translators, and can provide clues as to what types of er-rors are being produced by mt systems. it is also a realistic setting in many cases, including ours, where mt systems can be used to provide an initial translation of a text that the author can then modify. for example, the previously mentioned cos-mat project aimed to integrate such software into the publishing platform to facilitate the production of texts in multiple languages by the authors. ## data collection we collect a corpus of over 20k english nlp titles and abstracts that we translate automatically into french and of which a selection is then post-edited. basic statistics on the most common types of publications included are in ## nlp post-edit corpus in ## conclusion and future work in this paper, we report the results of a pilot study aimed at evaluating the quality of commercial mt systems for scholarly documents (abstracts) in the nlp domain (for @xmath0. this study explores a realistic scenario, where domain experts post-edit in their mother tongue their own texts (in english, supposedly their l2). we compare against the use of translators with a partial knowledge of the target domain to perform the same task. mt nlp experts post-edits translator's post-edits 1-we also release commute, a contrastive multilingual multimodal translation evaluation dataset, composed of ambiguous sentences and their possible translations, accompanied by disambiguating images corresponding to each translation. nous publions également commute, un ensemble de données d'évaluation de la traduction multimodale multilingue contrastive, composé de phrases ambiguës et de leurs traductions possibles, accompagnées d'images désambiguïsantes correspondant à chaque traduction. nous publions également le jeu de données commute (contrastive multilingual multimodal translation evaluation), composé de phrases ambiguës et de leurs traductions possibles, accompagnées d'images visant à leur désambiguïsation et correspondant à chaque traduction. nous publions également commute, un ensemble de données d'évaluation de la traduction multimodale multilingue contrastive, composé de phrases ambiguës et de leurs traductions possibles, accompagnées d'images désambiguïsantes correspondant à chaque traduction. 2-multilingual lexicalized constituency parsing with word-level auxiliary tasks lexicalized constituency parsing multilingue avec des tâches auxiliaires de niveau word tâches auxiliaires au niveau des mots pour l'analyse syntaxique en constituants lexicalisés multilingue analyse syntaxique de constituants lexicaux multilingues avec tâches auxiliaires au niveau des mots