P os E di O n: Post-Editing Assessment in P yth O n (2020)

Abstract: There is currently an extended use of post-editing of machine translation (PEMT) in the translation industry. This is due to the increase in the demand of translation and to the significant improvements in quality achieved by neural machine translation (NMT). PEMT has been included as part of the translation workflow because it increases translators’ productivity and it also reduces costs. Although an effective post-editing requires enough quality of the MT output, usual automatic metrics do not always correlate with post-editing effort. We describe a standalone tool designed both for industry and research that has two main purposes: collect sentence-level information from the post-editing process (e.g. post-editing time and keystrokes) and visually present multiple evaluation scores so they can be easily interpreted by a user.

## introduction post-editing of machine translation (pemt) is a very common practice in the translation industry. it has been included as part of the translation workflow because it increases productivity when compared with human translation @xcite and reduces costs @xcite without having a negative impact on quality @xcite . post-editors "edit, modify and/or correct pre-translated text that has been processed by an mt system from a source language into (a) target language(s)" @xcite . in the last few years, both research and industry have become very interested in neural machine translation (nmt) because it has produced very successful results in terms of quality, for example in wmt 2017 @xcite , wmt 2018 @xcite and wmt 2019 @xcite . given the overall performance of nmt, it is necessary to study all the potential this approach can offer to post-editing. one of the main problems is that automatic scores give a general idea of the mt output quality but do not always correlate with post-editing effort @xcite . many professional translators state that if the quality of the mt output is not good enough, they delete the remaining segments and translate everything from scratch @xcite . one of the main goals both of industry and research is to establish a correlation between the quality measurements of the mt output and translators' performance. research is especially focused on the effort this activity entails, mainly taking into account the temporal, technical, and cognitive effort @xcite . the use of tools that can log these three dimensions becomes a paramount challenge for research. professional translators usually use commercial products to translate and post-edit. in the 2018 language industry survey foot_0 conducted by euatc, elia, fit europe, gala and lindweb, sdl trados foot_1 was the most used product with more than half of the market quota, followed by memoq, 3 memsource, foot_3 wordfast, 5 and across. 6 however, these existing post-editing environments have a restricted availability and flexibility. as proprietary tools, they are difficult to modify and do not usually provide translator activity data that may be used to study post-editing effort. however, other open-source computer-assisted translation (cat) environments such as omegat, 7 have been modified and used for data collection @xcite . instead of trying to reproduce the working conditions of translators, which vary greatly among individuals, other tools establish controlled conditions in order to obtain non-biased data. for this purpose, translators use a post-editing tool that records the post-editing information, can be easily accessed from any platform and has an easy-to-use interface. in this paper we present posedion, a simple standalone tool that allows post-editing of mt output and records information of the post-editing effort (time and keystrokes) at sentence-level. it also includes multiple evaluation scores that the user can interpret easily to assess the post-editing process (such as edit distance, hbleu and hter). as it does not depend on any specific cat tool, it allows the collection of post-editing data in a controlled way. it can be used by professionals to assess the convenience of post-editing a certain mt output and by researchers to study post-editing effort. in section 2 we analyze some of the previous tools developed for this purpose. the tool and its mains characteristics are presented in section 3. in section 4 we describe the posedion analyzer, which is used to perform all the analysis, and section 5 includes the conclusions and future work. ## previous work in order to analyze the different components of post-editing effort, it becomes paramount to use tools that are able to log time, keyboarding, and other potential indicators of cognitive effort (e.g. gaze data). currently there is a proliferation of these tools @xcite , mainly because each research project has specific requirements. some of the tools developed focus more on pro-ductivity as part of an industry scenario. for example, the qualitivity foot_7 plugin can be added to sdl trados to measure post-editing effort. alternatively, taus developed dqf, 9 which can be used as a standalone benchmark or as an sdl trados plugin. there has also been eu-funded research to develop open-source workbenches to help improve quantitative measurements of effort (cas-macat foot_9 and matecat foot_10 ). other tools collect gaze data, which can be used to study post-editing effort. tobii pro lab is the commercial windows-oriented eye-tracking software that accompanies tobii eye trackers. it can calculate a variety of eye-tracking metrics and create visual representations of the data. another similar product is translog-ii @xcite , which is a windows-oriented program that records user activity data (uad), that is, all the keystrokes and gaze movements. it is meant specifically for translation process research (tpr) and it offers the possibility of further processing the data with the scripts included in the tpr database of the centre for research and innovation in translation and technology (critt tpr-db). even though these tools collect extensive information, they have specific and demanding settings which are not suitable for all experiments. some products devised for a specific experiment are not made available to the public afterwards @xcite . other tools focus on obtaining as much information as possible with an easy-to-use product. for example, transcenter @xcite is an open-source, web-based tool that allows users to carry out pe tasks and logs time and keyboard/mouse activity at a sentence level. another tool useful for quantitative investigations specifically designed for post-editing is pet @xcite . it can also be accessed from any platform, although it is based in java, which can sometimes be challenging for end-users who need to open the tool from their desktop computers. in addition to recording time and effort indicators at a segment level, pet also allows users to perform evaluation tasks on different customizable scales and criteria. the data file with all the information is saved in xml. however, it does not offer graphics or any other visual information with the results nor does it include an analyzer which produces multiple automatic metrics. ## posedion posedion is a post-editing tool developed mainly to collect information on different implicit and explicit effort indicators. it records time and keystrokes, and it also calculates some of the main indirect effort estimation measures (hter, hbleu and edit distance). it produces a file with the raw measurements but it also includes a results file with visually structured information that can be easily understood by any user. it was developed completely in python3 and it works in any platform which has python installed. translators tend to work from home with a great variety of platforms and devices, and do not always have the computer skills to solve any compatibility errors they may encounter with the tools they are about to use. a windows executable file is also available, which allows to run posedion without the need of installing the python interpreter. ## -----------------------------------------hter: hter mean 0.1611 hter std 0.1172 -----------------------------------------hbleu: hbleu mean 0.5303 hbleu std 0.2714 - posedion is able to generate graphics using the data, as the one shown in figure 8 created from the pruned hter values. the user can choose which data should be used to generate graphics. the results are stored in a tabulated text file, so they can be easily imported into any spreadsheet to perform further calculations. ## conclusions and future work in this paper we have presented posedion, a tool to perform evaluations of post-editing tasks, and its companion program posedion analyzer, which allows to user to easily analyze the data obtained with posedion. both programs are released under a free license (gnu gpl v3) and can be freely downloaded from the sourceforge page created for the project. 13 we plan to use this tool in several studies related to post-editing and to implement new features such as the evaluation of fluency and adequacy and an error mark-up tool. both programs are developed in python3 and they can be easily adapted and improved. as the data are stored as tabbed text files, they can be easily processed or imported into any spreadsheet program to perform further analysis or data visualization.