A Case Study on Contextual Machine Translation in a Professional Scenario of Subtitling (2024)

Abstract: Incorporating extra-textual context such as film metadata into the machine translation (MT) pipeline can enhance translation quality, as indicated by automatic evaluation in recent work. However, the positive impact of such systems in industry remains unproven. We report on an industrial case study carried out to investigate the benefit of MT in a professional scenario of translating TV subtitles with a focus on how leveraging extra-textual context impacts post-editing. We found that post-editors marked significantly fewer context-related errors when correcting the outputs of MTCue, the context-aware model, as opposed to non-contextual models. We also present the results of a survey of the employed post-editors, which highlights contextual inadequacy as a significant gap consistently observed in MT. Our findings strengthen the motivation for further work within fully contextual MT.

## introduction as an innovation-driven company offering dubbing and subtitling services, zoo digital is dedicated to exploring assistive technologies to streamline our workflows. machine translation in particular is a promising tool for improving the efficiency of the (currently fully manual) translation of the transcribed video content during interlingual subtitling. our domain is characterised by specific challenges, both linguistic (preservation of style and function in dialogue) and practical (keeping within subtitle constraints, such as visual properties and considerations for the viewers' reading speed). we report on a case study where translation from scratch was replaced with post-editing machine translations of the source text. while such a formulation is far from new -mt has been consistently demonstrated to help reduce effort in the subtitling domain (c. m. de @xcite -previous studies have relied on off-the-shelf general-purpose neural machine translation (nmt) engines like google translate foot_0 . our work investigates two additional systems: base-nmt, a specialised engine trained on our data, as well its contextual version based on the mtcue architecture @xcite , whose training involves observing a vast range of metadata and document-level information. the study was carried out with the assistance of translation and post-editing professionals. hereinafter we refer as post-editors (pes) to those who were tasked with post-editing work, and as translators (hts) to those who were tasked with translation from scratch (fst). the campaign took place in a full-context multi-modal environment where the professionals had access to the video material and were able to directly jump to the segment corresponding to the utterance they were reviewing, as well as see the preceding and succeeding segments. a total of eight pes were employed, four for english-to-german (en-de) and four for english-to-french (en-fr) translation, and four hts, two per language pair. we measured the effort it took to post-edit or translate the tv series content and the number of specific translation errors observed by the pes. our findings highlight the necessity of tailoring mt engines to the target domain and motivate further work within leveraging contextual systems in dialogue translation. ## related work over the last few years, subtitle translation has been given a volume of attention: c. m. de @xcite , @xcite and @xcite observe that post-editing the outputs of an nmt system is a promising alternative to translation ex novo, reducing the temporal, technical and cognitive effort of both novice and professional translators and subtitlers. a survey among professional subtitlers detailed by @xcite , finds that professionals have a positive outlook on incorporating automatic components (such as mt) into their workflow, as they offer starting templates, reduce effort and can provide useful suggestions. however, some challenges in the automatic translation of subtitles remain unsolved @xcite , including the adherence to subtitle block limitations, which often necessitates shorter and paraphrased translations; lexical consistency, which involves translating the same terms across the text, as well as using vocabulary that maintains the cohesion and coherence of the text, aligns with the surrounding video or textual content, and conforms to standard language or industry conventions; lexical errors such as the translation of idioms and figurative language, and context-related inconsistencies. context-related errors in particular have been pointed out as the culprit in many works in mt that leveraged the opensubtitles corpus @xcite , a dataset of user-submitted subtitles and their translations. leveraging documentlevel information @xcite ), speaker's and interlocutor's gender identity @xcite and explicit extra-textual information @xcite has been found particularly useful in addressing this challenge. context is also useful during the manual post-editing procedure: @xcite show that such a setup decreases the cognitive load of student translators compared to a text-only scenario, suggesting as an explanation the dual coding theory, according to which the interactions between the verbal and non-verbal information enhances the translators' understanding of the material. this work employs mtcue @xcite , a multi-encoder transformer designed for contextual nmt capable of leveraging contextual signals such as film metadata and document-level information to improve translation quality, as well as enabling better control of phenomena such as speaker's gender and formality register. the mechanism for delivering context in the model involves converting the context fields into equal-sized vectors via sentence embedding. the resulting vector sequence is inputted into a distinct transformer encoder. additionally, we employ the context specificity evaluation method outlined in @xcite , which relies on the pointwise mutual information (pmi). in this method, pmi quantifies the degree of co-occurrence between tokens in a translation hypothesis and the respective context. ## experimental setup the primary objective of our case study was to investigate whether post-editing mt is a costeffective alternative to fst in our workflow, and to what extent domain-adapted training data and the utilisation of context have an impact in this area. guided by the availability of resources, we operated in two language pairs: en-de and en-fr and considered four versions of the text in each, including mt outputs from three systems: we also operated on the human translations of the test set (ref) approved for production. 3 . for both mtcue and base-nmt, we trained the models after @xcite , §4.1, in the over-lap setting which mimics a scenario with access to prior episodes of a tested series for training (a sample is presented in appendix f of that work). we operated on sentence-level translations, with mtcue using the context for each sentence in its dedicated space. ## results of automatic evaluation the automatic evaluation results (figure 2 ) suggest that mtcue was the best-performing system and google the worst-performing for both language pairs. interestingly, for en-de, the bleu and comet score differences varied in magnitude, to the point of comet judging all three systems as on par. a possible cause was the discrepancy in hypothesis length (the reference text uses 7.04 words per segment, base-nmt: 7.06, mtcue: 7.06, google: 8.29). since comet's calculation involves comparing sentence embeddings of the hypothesis and the reference, including more words or phrases in the hypothesis may lead to a closer similarity match, inflating the score even if the additional tokens are redundant or even harmful to quality. bleu does not have this problem as it is based on string matching @xcite . as per the pmi scores, the professional translations (ref) consistently exhibited the highest context specificity. however, mtcue was on par with this reference score in both cases and was consistently better than the other two systems. mtcue therefore shows promise at addressing the contextrelated issues in subtitle translation. ## results of the post-editing study this section analyses the results of the post-editing study: the translation errors ( §5.1), the post-editing effort ( §5.2), and finally, the post-campaign survey responses ( §5.3). due to the unprecedented nature of this work in the company, the professionals' contract allowed them to withdraw if they found the compensation insufficient for the requested work. at the midpoint of the campaign, two pes (pe.g1 and pe.g3) contacted the project manager to express concerns regarding the quality of the mt outputs, asserting that the task potentially required more effort than fst. to compromise, they proposed narrowing the scope of the remaining work to error identification and marking, without making the necessary corrections. this meant we would not obtain the effort metrics for the two pes. conse-quently, while the error analysis in §5.1 includes both language pairs, the effort analysis in §5.2 does not include results from pe.g1 or pe.g3. ## conclusions and future work we have presented a case study on post-editing mt of subtitles for tv series in a multi-modal scenario, with a focus on contextual mt. we found that the mt models custom-trained on dialogue required less post-editing effort than the one-size-fits-all google translate, potentially due to the overbearing literalness and stiffness of the latter system's outputs. we also found that some post-editors amended production-approved human translations at high rates, with hypervigilance about dealing with mt as a possible cause. our results did not determine a significant difference in post-editing effort between mtcue and base-nmt. however, the inclusion of context in mtcue yielded fewer errors in the style, context and fluency categories, motivating our future exploration of context-inclusive models. we further found that post-editing any mt output required four to six times less technical and temporal effort compared to fst, making it a promising costeffective venture. however, cognitive effort should be measured in future studies, given the exit survey sentiment that post-editing was sometimes harder and less interesting than fst. our future experiments will employ larger cohorts of pes and split them into groups who post-edit non-contextual and contextual inputs exclusively, so that clearer feedback can be collected, as well as to minimise the variance in effort.