MT for subtitling: User evaluation of post-editing productivity (2020)

Abstract: This paper presents a user evaluation of machine translation and post-editing for TV subtitles. Based on a process study where 12 professional subtitlers translated and post-edited subtitles, we compare effort in terms of task time and number of keystrokes. We also discuss examples of specific subtitling features like condensation, and how these features may have affected the post-editing results. In addition to overall MT quality, segmentation and timing of the subtitles are found to be important issues to be addressed in future work.

## introduction developments in machine translation (mt) in the last two decades have led to significant improvements in translation quality. the success and popularity of statistical machine translation (smt) systems were matched and eventually surpassed by neural machine translation (nmt). as quality has improved, the use of mt and post-editing (pe) has also increased in professional translation workflows. broadly, pe refers to the practice of using mt output as a raw version checked and corrected by the translator. the use of mt and pe has been found to increase productivity in various translation scenarios (e.g. @xcite . however, this workflow appears less common in the field of audiovisual translation (avt). for example, @xcite note that while specialised subtitling software with various functionc 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. this paper presents a pilot study carried out in november 2019 examining how the use of mt and pe in the subtitling workflow affects the work and productivity of subtitlers. in the study, 12 professional subtitle translators worked on a series of tasks in four language pairs (@xmath0, @xmath1, @xmath2, and @xmath3. they created interlingual (translated) subtitles for short video clips both with and without mt output. to assess productivity and effort, keylogging data were recorded during these tasks. task time and technical effort represented by keystrokes were compared between post-editing and translation from scratch. we first discuss related work on mt for subtitling and approaches to user evaluation of mtpe in section 2. the mt models and subtitle alignment are presented in section 3. section 4 outlines the user data collection, and section 5 presents the analysis of productivity measures. section 6 discusses observations on pe changes, followed by future work and conclusions. ## user pe data collection the subtitling tasks for productivity data collection were carried out in november 2019 at the premises of the finnish broadcasting company yle. in total 12 translators (3 per language pair) participated in the tasks: 8 in-house translators and 4 freelancers with experience of working for yle. the participants have between 4 and 30 years of professional subtitling experience in their language pair. only 2 stated they had previously used mt for subtitling, and 7 others had used mt for other purposes. the subtitling tasks were carried out using the subtitlers' preferred software (wincaps q4 or spot). to replicate their normal working environment, an external monitor and keyboard were provided, and they had access to the internet as well as terminology and other resources normally used in their work. process data were logged using inputlog @xcite , which records all keyboard and mouse activity. windows 10 screen recording software was used to capture video to support the analysis. pre-and post-task questionnaires were used to collect background information and participants' subjective assessment of the mt output and pe experience. after the tasks, a brief semi-structured interview was also carried out to collect more detailed feedback regarding problems in the workflow and the participants' views on potential improvements. in this paper, we focus on an analysis of the process data. subtitling tasks were carried out in 4 language pairs: @xmath4, @xmath5, @xmath6, and @xmath7. for each source language, six clips were selected from a dataset provided by yle. three clips were selected from unscripted european election debates, and three clips from semi-scripted lifestyle or cultural programmes. the individual clips were selected so that each clip (i) forms a coherent, self-contained section of the programme, (ii) is approximately 3 minutes long, and (iii) contains 30-35 subtitle segments. each participant completed a total of six tasks where they subtitled two clips "from scratch" without mt output, two clips using output from a sentence-level mt system, and two clips using output from a document-level mt system. the clips and mt outputs were rotated in a round-robin format so that each clip was subtitled once in each condition (no mt output, sentence-level mt output, document-level mt output) by a different participant. task order was also varied to minimise facilitation effect. the participants were instructed to produce subtitles that would be acceptable for broadcasting, and to use the resources they normally would for their work, but to not spend excessive time in "polishing" any given wording or researching information. no explicit time limit was given for each task, rather, the participants were instructed to work at their own pace. in the from scratch condition, the participants also created the segmentation and timing of the subtitles following their normal work process. subtitling templates are not used by yle for these content types. in the mtpe condition, the participants worked with output that was pre-segmented and timed based on the intralingual subtitles used as source text for the mt (see section 3.2). to assess productivity, the process logs were analysed using inputlog's analysis functions. the task time and the number of keystrokes logged were used as productivity measures. using inputlog filters, we focused only on task time and keystrokes in the subtitling software, excluding other activity such as internet searches for terminology or other information. based on the final subtitles produced, edit rate between the mt output and the final versions were calculated using hter @xcite and character @xcite . as pe of the subtitles involved also changes to the segmentation, e.g. adding or deleting frames and moving words between frames, subtitle segmentation was ignored and edit rates were calculated as document-level scores to focus on edits affecting the textual content. these measures were then compared between the tasks of creating interlingual subtitles from scratch and mtpe, as well as between pe of the sentence-level and document-level mt outputs described in section 3.1. ## comparison of subtitling productivity figure 2 shows a comparison of the average subtitling task time for subtitling from scratch and subtitling with mtpe. the topmost three bars show averages for post-editing the sentence-and document-level mt output and for translation from scratch across all language pairs, while the bottom pairs of bars show averages for pe (either mt output) compared to from scratch. on average, post-editing machine-translated subtitles (regardless of mt output) was slightly faster than creating subtitles from scratch. some differences can be seen between the language pairs: the largest difference in task times is seen in @xmath8, while the task times for @xmath9. no clear difference could be observed between the two different mt outputs, although on average postediting the sentence-level mt output appeared to be slightly faster. figure 3 shows a comparison of technical effort in terms of the average number of keystrokes used when producing subtitles. the topmost three bars show averages for post-editing the sentenceand document-level mt output and for translation from scratch across all language pairs, while the bottom pairs of bars show averages for pe (either mt output) compared to from scratch. on average, post-editing machine-translated subtitles (regardless of mt output) involved fewer keystrokes than creating subtitles from scratch. the reduction in the number of keystrokes is more pronounced than in the case of task times, and seen in all language pairs. again, no clear difference could be observed between the two different mt outputs, although on average post-editing the sentence-level mt output appeared to involve slightly less technical effort. although a detailed analysis of the types of keystrokes is not within the scope of this paper, some observations can be made regarding the distribution of keystroke types. intuitively, pe reduced the need for text producing keystrokes on average by 54% compared to from scratch, as the mt output provides some of the text needed. however, the number of text deleting keystrokes was 24% higher in pe, as correcting the output also involves removing words or characters. in the from scratch case, the participants needed to create and set the timing for each subtitle frame themselves, which requires keystrokes and/or mouse clicks. in mtpe, the mt output was already segmented and timed based on the intralingual subtitles used as source text, which reduced the associated keystrokes by approximately 32%, but the number of keystrokes shows that the participants found it necessary to change both the segmentation and timing. changes to subtitle segmentation are discussed in more detail below. to examine the number of changes between the mt outputs and final pe versions, edit rates were calculated using word-based hter and character-level character. in addition to the textual content of the mt subtitles, the participants edited both the segmentation of that content into subtitle frames and timing of the frames. on average, the participants increased the number of subtitle frames in the clips by 7% by splitting or adding frames. this tendency was particularly noticeable in @xmath13. @xmath14. comparing the timestamps of the original subtitle frames used for the mt output and the frames in the post-edited files, we observed that only 24% of the original timed frames had been retained in pe. for 27% of frames, either the in or out time had been changed, and for 49% both in and out time were changed. the intralingual subtitles used as source text were not translated as isolated subtitle frames but rather as sentences or longer passages and then aligned back to the frames (see section 3.2). however, the heuristics used for alignment were not always successful. in some cases, splitting a segment due to punctuation caused the next segment to become too long and started to push content into the following frames, causing the subtitles to fall out of sync with the audio. similar issues were also observed due to repetition in the mt output. it is also possible that the sync issues arising from incorrect segmentation may have lead the participants to also change the timing of subtitle frames. ## discussion of pe changes considerable variation in task times and numbers of keystrokes was observed between different participants. productivity gains were most evident for participants with the longest average task times overall. however, 5 out of the 12 participants were in fact slower in pe. two of them also used slightly more keystrokes, but three were slower despite using fewer keystrokes in pe. these findings are similar to other process studies both on subtitling and other text types (e.g. @xcite showing that potential productivity gains from mtpe vary, and that participants who are already fast benefit less. fewer keystrokes not necessarily leading to time saving has also been observed in other studies. while the number of keystrokes reflects the technical effort needed, it does not capture the amount of cognitive effort involved in recognising potential errors and deciding on necessary changes. the edit rates of different participants also vary. at the level of individual subtitlers, average hter scores range from 31.9 (@xmath15, participant c) to 84.8 (@xmath16, participant c). these edit rates are comparable to the hter scores reported by @xcite for different mt system outputs, genres and posteditors, which range from 27.8 to 82.7. in our study, the two participants with the highest average edit rates both worked on @xmath17, and the two with the lowest average edit rates on @xmath18, but differences are also evident within the same language pair. since the participants post-edited different mt versions, some variation may be explained by different output quality, but to some extent these differences may also reflect individual preferences. qualitative observations suggest that while some edits relate to clear mt errors, many are also caused by what appear to be preferential edits; for example, in the @xmath19, one participant accepts the translation "financial discipline" for the finnish talouskuri while another replaces it with "austerity". a possible factor affecting both productivity and number of changes is pe experience. the participants in this study had little prior experience with mt specifically for subtitling. the subtitlers' productivity and approach to the task may therefore have been affected by the fact that pe was unfamiliar and different from their normal work processes. as @xcite also note, psychological factors such as unfamiliarity and irritation with mt errors influence productivity. these factors may have also led to preferential and possibly unnecessary changes. more practice working with mt output and pre-segmented subtitles may affect their approaches, e.g. by reducing preferential changes, and increase productivity in this task. as noted in section 2.1, the spoken content of the videos and subtitles as a written representation of spoken language differ from each other. due to technical restrictions, condensation is common in subtitle translation, and may affect the edit rate to some extent. on the other hand, because the source text for the subtitlers consists of not only the written subtitles, but also the audiovisual context, they may make changes based on information in the audio or video of the clip being subtitled. an example of condensation through omission and paraphrasing can be seen in in contrast to condensation, the participants sometimes added content to subtitles. while some additions correspond to missing words in the mt output, others in fact involve content not present in the intralingual subtitles used as source text for mt. the intralingual subtitles themselves already involve some condensation and paraphrasing, and therefore do not match exactly the spoken audio. particularly in the swedish "lifestyle" clips, the intralingual subtitles appear to have been very condensed, and the participants post-editing @xmath21. these additions show one effect of the multimodal context: having the omitted information present in the audio led the participants to make additions that would have been unlikely or impossible if only the written subtitles had been available. subtitle translators are also affected by the visual context of the video. changes related to the visual context occur, for example, when the subtitler chooses to replace a pronoun with the referent seen in the video. an example of this appears in one of the @xmath22. the expression de ska kokas mjuka 'they should be cooked soft' in the dialogue is correctly translated in both mt outputs using the finnish pronoun ne 'they'. however, both participants post-editing mt output for this clip replaced the pronoun with hedelmät 'fruit', referring to the fruit being cooked. ## future work based on the experiment and user feedback, segmentation of the interlingual subtitle content into appropriate chunks is an important issue to be addressed, although using subtitle timing from preexisting intralingual subtitles was to some extent useful. potential directions for improving segmentation and timing could involve the use of time information to split the data into coherent blocks separated by significant breaks, and the integration of speaker information into the translation engines to segment subtitles into dialogue turns by leveraging speaker labels or diarisation output. multimodality can also play a crucial role in segmentation as visual and auditory cues may help in improving the division of verbal content into discourse units. we plan to implement an end-to-end system for subtitle translation and segmentation after @xcite , and investigate how well such a system could generate organic subtitles. multimodality may also be useful in optimising translation quality. augmenting subtitles with information from the visual and auditory modalities could help improve translation accuracy in general. for example, visual information could be helpful in resolving ambiguity. in future work, we will explore incorporating multimodal features in translation in connection with non-linguistic context for language grounding and disambiguation. a more detailed manual analysis of the types of pe changes made by the participants and their potential explanations (mt errors, subtitling conventions, or preferential changes) is currently underway. feedback collected from the participants is also being analysed for information regarding the user experience. a second round of user evaluations is also planned for 2020 to collect further data and assess the effect of the new developments of our mt approaches, and to give the participants more experience with post-editing subtitles. ## conclusion this paper presented a user evaluation pilot study of mt and post-editing for subtitles. based on an analysis of process data collected from 12 professional subtitlers in four language pairs, we presented a comparison of productivity in terms of task time and number of keystrokes when postediting mt subtitles vs translating from scratch. on average, our results indicate mtpe to be slightly faster and to involve fewer keystrokes than subtitling from scratch. however, considerable variation was observed between different language pairs and participants. we also discussed examples of specific subtitling features like condensation, and how these features may have affected the post-editing results. in addition to overall mt quality, the segmentation and the timing of the subtitles were found to be important issues to be addressed in future work.