Post-editing in Automatic Subtitling: A Subtitlers’ perspective (2022)

Abstract: Recent developments in machine translation and speech translation are opening up opportunities for computer-assisted translation tools with extended automation functions. Subtitling tools are recently being adapted for post-editing by providing automatically generated subtitles, and featuring not only machine translation, but also automatic segmentation and synchronisation. But what do professional subtitlers think of post-editing automatically generated subtitles? In this work, we conduct a survey to collect subtitlers’ impressions and feedback on the use of automatic subtitling in their workflows. Our findings show that, despite current limitations stemming mainly from speech processing errors, automatic subtitling is seen rather positively and has potential for the future.

## introduction machine translation (mt) is today widely adopted in most areas of translation and post-editing has been established as a professional practice, shaping the landscape of the translation industry. audiovisual translation (avt) is one area where mt has for long found limited success @xcite . among the main reasons are the inability of mt systems to deal with creative texts @xcite and the multimodality of the source, since the translation depends on visual, acoustic and textual elements @xcite . for subtitling, additional challenges are posed by the formal requirements of the target: subtitles should not exceed a specific length and should be synchronised with the speech @xcite . however, recent developments in neural machine translation (nmt) and speech translation (st) are paving the way for viable and usable (semi-)automatic solutions for subtitling. compared to solutions providing mt for subtitling, automatic subtitling tools do not simply translate human-generated source language subtitles, but incorporate automatic transcription of the speech, mt, automatic synchronisation (spotting) and segmentation of the translated speech into subtitles. altogether, these technologies come with the promise of reducing the human effort in the subtitling process, but, to date, automatic subtitiling has still to be put to test by the actual users. even though translators are fundamental for the advance of new technologies, their views are often not sufficiently considered @xcite . the study of subtitlers' perceptions of the technology they are interacting with can be beneficial for all stakeholders in the avt industry. furthermore, the inclusion of subtitlers in the process of technological change can alleviate their resistance to adopting technologies @xcite . developers can direct their implementation efforts in the right direction to provide user-friendly tools and interfaces @xcite , and avt trainers can identify necessary skills for teaching and training @xcite . a better understanding of subtitlers' interaction with technology can help define the rising profession of the subtitler post-editor @xcite , and establish metrics and standards to protect subtitlers against dropping rates and ensure fairness @xcite . in response to the challenges brought about by increasing technologisation, in this work we conduct a survey of subtitlers' perspectives on the developing paradigm of automatic subtitling. this survey is a timely contribution to take stock of this nascent technology and its implementation in the subtitling profession from the very beginning, while setting the stage for further developments. the survey focuses on the subtitlers' user experience when post-editing automatically-generated subtitles from and into different western european languages. it also aims at collecting feedback on the main issues and benefits of the technology, as well as on the impact of automatic subtitling on the subtitler's profession. based on qualitative and quantitative analysis of a survey questionnaire, we provide a participant-based evaluation of automatic subtitling and a comprehensive view of subtitlers' attitudes towards this new paradigm. our findings indicate that despite its current limitations mainly related to challenges in speech processing, automatic subtitling has potential and its benefits are already recognised by the users. based on the received criticisms, we provide a list of recommendations for future improvements in automatic subtitling tools, which we hope will serve as a guide for technology developers. we further release the questionnaire and responses to foster replication and reproducibility in automatic subtitling. 1 ## related work automatising subtitling has recently received growing interest. one research direction aims at controlling the generation of captions and subtitles based on particular variables and properties, such as genre @xcite , length @xcite or alignment between source and machine-translated subtitles @xcite . though relevant from the technological standpoint, this line of research has employed automatic metrics for the evaluation of mt and has not included subtitlers in the evaluation process. other studies have tested the usability of mt for subtitles by focusing on quality and productivity, mainly through the task of post-editing (pe). the human evaluation, however, did not always involve professional subtitlers. some studies used volunteers (c. m. de @xcite , native speakers @xcite or translators @xcite . nevertheless, subtitling requires special training and skills which 1 https://github.com/fatalinha/subtitlers-have-a-say native speakers or translators do not necessarily possess. larger scale evaluations involved professional subtitlers, but focused on machine translating human-generated source language subtitles. this setting has less challenges than automatic subtitling, since the source text is error-free and already compressed, while the spotting and segmentation are performed by a human. @xcite built an mt system between scandinavian languages, which was tested by professional subtitlers, and collected their feedback in a nonstructured way. the large-scale sumat project @xcite involved two professional subtitlers per language pair, who performed post-editing and rated their perceived pe effort. @xcite evaluated the productivity gains of their proposed english into spanish system with two post-editors, who were additionally asked to rank the adequacy, fluency and design of the subtitles. user feedback was collected in a nonstructured way, where subtitlers commented on the post-editing process and on their perception of mt in their workflows. lastly, @xcite performed a comprehensive human evaluation of their mt systems for scandinavian languages. the evaluation included the collection of product and process (keystrokes) data, as well as rich feedback based on a mixed methods approach using questionnaires and semi-structured interviews. our present study builds upon the work by @xcite by extending the feedback collection to a larger participant sample (22 compared to 12) working in a variety of western european language pairs. one main difference is the technology behind the generation of the target subtitles. in our study, respondents are asked to evaluate their user experience after post-editing subtitles generated through a three-step fully automatic process involving transcription, synchronisation and translation. on the contrary, in @xcite source subtitles were first obtained by a human (subtitle template), and then machine translated and aligned to the original frames. in addition, the subtitlers used their preferred subtitling software in the pe tasks. however, as the authors admit, the subtitling tools are not designed for mt post-editing (mtpe), and may therefore not be optimal for the task. our work has the benefit of evaluating the pe experience using a professional tool specifically tailored for post-editing automatically generated subtitles as a case study. the survey described in this paper was conducted in december 2021 and consisted in respondents filling in a questionnaire after having taken part in testing sessions of an automatic subtitling tool. ## discussion this study focused on subtitlers' user experience and perspectives on the task of post-editing automatically generated subtitles. our findings suggest a neutral to positive experience. even though there are those who still see no benefits from this new technology, automatic subtitling was welcomed with enthusiasm by many subtitlers, as an aid to save time and effort. as with studies on mtpe experience @xcite , subtitlers have expressed disfavour towards automatic subtitling in respect to technological flaws, but also acknowledged its positive aspects and expected technology to shape their profession in the near future. as for the dangers to the profession, most criticisms were not rooted in the fear of being outperformed by automatic systems, but rather in the effect of technology on the final product and market consequences @xcite . the positive aspects of technology can only be appreciated when combined with respectful and ethical professional and market practices. previous work reporting feedback of subtitlers focused on a setting where mt was applied to human-generated subtitles. the views of the subtitlers involved did not lead to auspicious conclusions in favour of the use of mt in subtitling. in spite of encouraging automatic evaluation scores, subtitlers were cautious in reporting productivity gains in @xcite , while in @xcite pe experience was rated as rather negative (2.37 on a 1-5 scale), with mt being useful only for simple and short sentences. an increase in productivity for simple sentences was reported in @xcite , where the two subtitlers rated their experience as fair. in @xcite the participants did not find pe particularly difficult but characterised it as negative or limiting and did not think mtpe increased productivity. similar criticisms were reported for mt quality in our study, with mt described as too literal, unable to properly translate spoken and figurative language. however, most subtitlers acknowledged that automatic subtitling makes their work faster and more efficient, especially when compared to old-style subtitling. the difference of our study compared to studies of mt for subtitling is the automatisation not only of the translation, but also of the technical aspects of spotting and segmentation. subtitlers recognised the importance of automatising these aspects, which are often characterised as tiresome and dull. by not focusing only on the translation but the automation of the technical aspects, automatic subtitling allows subtitlers to spare time and effort on the tedious part of the work (spotting and segmentation) and unleash their creativity in adjusting the final text. our study aimed at providing a broad view of subtitlers' perspectives, by complementing quantitative scores with open questions, attempting to cover several language pairs and a range of subtitler profiles. however, we acknowledge that the findings should be interpreted with some caution. questionnaire-based studies have a contextbound nature and may be affected by factors such as the system (quality, language), the participants (age, familiarity with technology) and the setting @xcite . therefore, some limitations should be considered when drawing conclusions. firstly, responses and user experience scores may have been affected by the language pair, due to differences in the subtitling quality depending on the asr and mt performance, despite keeping all other settings (videos, instructions) equal. still, we opted for not reporting results separately for each language pair, since the sample size per pair (2-3) would be too small to draw robust and generalizable conclusions on a per-language basis. second, even though we attempted to include a broad range of professional subtitler profiles, the group is not necessarily representative of the subtitlers' general population. for example, the respondents' age, a variable not collected in our survey, may affect their technological acceptance. moreover, their experience in subtitling, template translation and mtpe varies. we found in statistical tests that the only variable affecting the user experience is mtpe experience. subtitlers with less experience (<= 2 years) had significantly higher user experience scores than the more experienced ones. foot_2 it is possible that experts, already being used to a certain level of mt output quality and to their preferred interfaces, are less willing to change tasks and tools, while novices, having less consolidated working practices, are more open and less critical against new interfaces and workflows. accepting to take part in a task involving automatic subtitling already means the subtitlers were willing, curious or even familiar with the technology, and therefore may have been positively inclined towards automatisation in subtitling, contrary to many avt professionals (audiovisual translators europe, 2021). lastly, the interface used in pe has a great influence on user experience. we selected matesub as a typical instance of an automatic subtitling tool. however, the generalisability to other tools is not guaranteed. in an attempt to test whether previous experience with matesub had an effect on user experience, we separated the respondents in two groups based on their responses to the question how often do you use matesub in your subtitling jobs: regular users (often, sometimes) and occasional (never, rarely). we found that familiarity with the tool did not have an effect on the average user experience scores. 6 this shows that the tool is user-friendly, with a steep learning curve, and does not require extensive training. less user-friendly tools may negatively affect the post-editing experience. despite these limitations, this study presents a screenshot of the current state of the quickly evolving technology, necessary to drive implementation efforts in the right direction. ## conclusions in this work we presented findings on subtitlers' user experience and perspectives when postediting automatically generated subtitles, based on a survey questionnaire. subtitlers' experience was marked as neutral to positive. thematic analysis of the open questions showed that the main issues of automatic subtitling stem from failures in speech recognition and pre-processing, which result in error propagation, translations out of context, inaccuracies in auto-spotting and suboptimal segmentation. however, subtitlers acknowledge the positive sides of the technology, which are speed and reduction of effort, especially related to the technical aspects, as well as the provision of useful suggestions. we conclude that, despite current limitations, automatic subtitling tools can be beneficial for subtitlers, as long as improvements consider subtitlers' opinions, and ethical and professional standards are respected. we expect that as automatic subtitling tools mushroom, larger studies will be needed to explore different variables and monitor the progress in automatic subtitling.