Comparing Post-editing based on Four Editing Actions against Translating with an Auto-Complete Feature (2020)

Abstract: This article describes the results of a workshop in which 50 translators tested two experimental translation interfaces, as part of a project which aimed at studying the details of editing work. In this work, editing is defined as a selection of four actions: deleting, inserting, moving and replacing words. Four texts, machine-translated from English into European Portuguese, were post-edited in four different sessions in which each translator swapped between texts and two work modes. One of the work modes involved a typical auto-complete feature, and the other was based on the four actions. The participants answered surveys before, during and after the workshop. A descriptive analysis of the answers to the surveys and of the logs recorded during the experiments was performed. The four editing actions mode is shown to be more intrusive, but to allow for more planned decisions: although they take more time in this mode, translators hesitate less and make fewer edits. The article shows the usefulness of the approach for research on the editing task.

## the experiment setting this section describes the set-up of a three-hour workshop with 50 translators, which was performed in january 2017. this included a twohour presentation and discussion on the conceptual framework of this project, followed by a one-hour hands-on experiment. this article summarises the experiment, which is described in detail in do @xcite . the setup of the experiment was exploratory. our main purpose was to test a novel interface in as many scenarios as possible in a short time, and to collect translators' impressions on it. in this sense, this could be framed as an extensive pilot test, even though there were no plans for a proper test to the novel interface. the use of edit distances to estimate operations performed by translators was one of the dimensions being tested during the experiments. as we wanted to collect data on the actual actions (the words that were deleted, inserted and replaced, besides those that were moved, together with a record of all the positions these edits were made on), we would need an interface that recorded those actions while they were being performed, not post-process estimations as edit distances are. it was also clear that this new interface would ask for conscious decisions from translators on using only the four actions. the tool developed during the study had usability issues, but it allowed us to explicitly ask translators to consider which words to delete, insert, move or replace, before performing any of those actions. this allowed us to gather information which would not be possible from an unconstrained work environment, in which translators could, for example, delete a whole mt suggestion, and write the whole translation, eventually retyping words already in the suggestion. ## discussion and conclusions following the aim of the project, we could confirm in the workshop that there is a research interest, from different points of view, in the approach of editing as being formed by four actions. first, it was confirmed by the participants in the surveys conducted during the workshop. although it was related to a disruption in work habits, translators admitted that this perspective made sense as a description of what they do when they edit mt text, at least from a technical perspective. as commented in the final surveys, the main advantage of this method, which is related to its intrusiveness, is the demand for translators to plan their editing strategy before applying it, thus bringing to light an often-unconscious decision process, forcing them to make more efficient actions. this efficiency, measured only as technical action data, is, naturally, meaningless if we aim at relating it to effectiveness, but it is relevant for the development of better forms of support in translation tools. as admitted from the outset, the four editing actions interface tested during the workshop was not appropriate for commercial settings, because of its lack of usability development. there are, however, potential research and pedagogical implications in this mode of work. the work mode based on the four editing actions gives visibility to the decision process in time-constrained and efficiency demanding work contexts, as is the case of pe. this study, namely in its analysis of the lack of correspondence between the estimations of edit distances and the actions employed by translators, also calls the attention to the complexity of editing work. this calls for a cautious use of metrics like ter as descriptors of processes, and to the need to study in closer detail the different levels of complexity in pe. our suggestion of an editing threshold that sets a boundary between levels in which editing can be easily described and others in which it cannot is a contribution to this type of study. the methodology applied in this study was adjusted to the objectives and constraints of the experiment. the amount of questions this exploratory approach raised meant that a choice had to be made between extensiveness or depth of statistical analysis. this approach enabled a rich discussion with the participants, on the two work modes and around concepts and practices that were novel, and to collect data for the comparison between the two modes, thus fulfilling the experiment objectives. the analysis of the results of our experiment exploited the weight of different dimensions of pe. the suggested editing threshold of 25% was the most frequent global average in the different levels of analysis, but relevant differences in editing intensity in different texts, by different users and using different work modes were also observed. we also saw how the number of edits and speed may be associated with different factors, text and work mode, respectively, but also how these two dependent variables helped us group users in four different clusters. this shows how the threshold may act as an important instrument to highlight varied degrees of complex editing in mt content. the experiment also showed that users were more consistent, and, in a certain sense, more efficient in pe mode, since they made fewer edits and returned fewer times to the same segment. this efficiency is contradicted by the fact that pe mode requires more time per edit, but we also observed improvements in speed of use of this work method. this highlights the usefulness of making the decision process more conscious, something which may be explored in pedagogic contexts. finally, this work showed how the four editing actions are useful features to describe and research editing work. the most frequent action employed by translators was replace, followed by delete, then insert and finally move, a result confirmed in similar experiments. insert and replace take longer to perform because they require typing. a tool that aims at supporting the actions performed by translators needs to act on all of these actions. this presents implementation challenges, not only at the data collection stage, but also regarding the usefulness and usability of the interfaces. for example, actions that require typing (like insert and replace) may be supported by auto-complete features, but delete and move only have an added value if they can appear previously as suggestions to the users, highlighting words which may have to be deleted and positions where words may be moved into. so, a combination of methods, flexibly adjusted to different degrees of editing, seems to be the most reasonable approach. although it focused on micro actions, this project allowed us to learn many things about the editing task. it has raised relevant questions related to technical effort, productivity, usability and usefulness of action support features, and on how to support specialised users. we suggest that more research around the editing actions and the editing threshold, besides new applications to support editing actions, should be pursued by translation technology researchers and developers.