Is 42 the Answer to Everything in Subtitling-oriented Speech Translation? (2020)

Abstract: Subtitling is becoming increasingly important for disseminating information, given the enormous amounts of audiovisual content becoming available daily. Although Neural Machine Translation (NMT) can speed up the process of translating audiovisual content, large manual effort is still required for transcribing the source language, and for spotting and segmenting the text into proper subtitles. Creating proper subtitles in terms of timing and segmentation highly depends on information present in the audio (utterance duration, natural pauses). In this work, we explore two methods for applying Speech Translation (ST) to subtitling, a) a direct end-to-end and b) a classical cascade approach. We discuss the benefit of having access to the source language speech for improving the conformity of the generated subtitles to the spatial and temporal subtitling constraints and show that length is not the answer to everything in the case of subtitling-oriented ST.

## introduction vast amounts of audiovisual content are becoming available every minute. from films and tv series, informative and marketing video material, to home-made videos, audiovisual content is reaching viewers with various needs and expectations, speaking different languages, all across the globe. this unprecedented access to information through audiovisual content is made possible mainly thanks to subtitling. subtitles, despite being the fastest and most wide-spread way of translating audiovisual 1 speaking of subtitles and their optimum length of 42 characters per line, we could not help but alluding to the book and series the hitchhiker's guide to the galaxy by douglas adams, where the number 42 is the "answer to the ultimate question of life, the universe, and everything", calculated by a massive supercomputer named deep thought for over 7.5 million years. in this work, we propose speech translation as an alternative to the template creation process. we experiment with cascade systems, i.e. pipelined asr+mt architectures, and direct, end-to-end st systems. while the mt system in the pipelined approach receives a raw textual transcription as input, the direct speech translation receives temporal and prosodic information from the source language input signal. given that several decisions about the form of subtitles depend on the audio (e.g. subtitle segmentation at natural pauses, length based on utterance duration), in this work we ask the question whether end-to-end st systems can take advantage of this information to better model the subtitling constraints and subtitle segmentation. in other words, we investigate whether, in contrast to text translation (where one mainly focuses on returning subtitles of a maximum length of 42 characters), in speech translation additional information can help to develop a more advanced approach that goes beyond merely matching the source text length. our contributions can be summarised as follows: ## analysis the higher scores for cpl and ter-br in section 4 suggest that the e2e system is better at modelling the subtitling constraints of length and proper segmentation. in this section we shed more light into this aspect by analysing factors which might be determining the system's behaviour in relation to the insertion of the break @xmath0. one question quickly arising is how the system can determine whether to insert a subtitle break @xmath1. since the maximum number of lines allowed per subtitle block is 2, a simple answer would be to alternate @xmath2. however, anyone having watched a film with subtitles is aware that subtitles can be two-liners or one-liners. coming back to the example in section 3.1, depending on the choice of break symbols (except for the last symbol which should always be @xmath3, there are two possible renderings of the subtitle: 10 00:00:31,066 --> 00:00:34,390 this kind of harassment keeps women from accessing the internet --11 00:00:34,414 --> 00:00:36,191 essentially, knowledge. and 10 00:00:31,066 --> 00:00:34,390 this kind of harassment keeps women 11 00:00:34,414 --> 00:00:36,191 from accessing the internet -essentially, knowledge. only the first rendering is acceptable because it satisfies the reading speed constraint but also corresponds to the speech rhythm, since the speaker makes a pause after uttering the word "internet". in this case, how can the mt system determine which type of break symbol to insert? we have mentioned in section 2.1 that the in and out times of a subtitle should follow the rhythm of speech. therefore, we expect that the end of a subtitle block, which in our setting is signalled by the break @xmath4, should correspond to the end of a speech act, a pause or a terminal juncture. on the other hand, line breaks inside the same subtitle block, which in our work correspond to the break @xmath5, have a different role. while line breaks can still overlap with pauses or signal the change of speaker, their function is to split a long subtitle into two smaller parts in order to fit the screen. the decision of where to insert a line break inside a subtitle block is determined by two factors: achieving a more or less equal length of the upper and the lower subtitle line and inserting the break in a position such that syntactic units are kept together. consequently, the insertion @xmath6. if the hypothesis above holds, the choice of whether to insert @xmath7. as a consequence, it is not a simple alternating procedure. to test this hypothesis, we compute the duration of the pause coming after each word in the source side of the must-cinema test set. to achieve this, we perform forced alignment of the transcript against the audio and subtract the end time of each word from the start time of the next word: pause @xmath8: i) pauses corresponding to positions @xmath9, ii) pauses corresponding to positions @xmath10. in pauses corresponding to the positions @xmath11. even if we take the most extreme cases (based on standard deviation), any pause above 0,37 seconds requires the insertion @xmath12. pauses corresponding @xmath13, but there is an overlap between the possible durations of the none and @xmath14. this confirms our hypothesis about the different roles of the two subtitle breaks. therefore, prosodic information is an important factor which can help the st system determine the subtitling segmentation according to the speech rhythm. this finding provides strong evidence towards a clear limitation of the cascade setting, where the raw textual transcription from the asr does not provide any prosodic information to the mt system. the mt system in the cascade setting is disadvantaged by the inability to: i) recover from possible asr errors, and ii) make decisions determined by factors other than text. with this knowledge, we analyse the breaks in the results of the two systems. in order to control for differences in translation, we select all sentences with at least 2 breaks and with the same number of break symbols (regardless of @xmath15, the output of the cascade and of the e2e. the resulting sentences are 137 for french and 158 for german. we calculate the accuracy of the type of break symbols for the two systems. for french the accuracy is 89% for the cascade and 93% for the e2e. for german the accuracy is 85% for the cascade and 88% for the e2e. this difference in accuracy suggests that the e2e is aided by the acoustic information and specifically by the pause duration in determining the correct break symbol. in the second example, although both systems have chosen the right type of break, there are differences in the actual positions of the breaks. the e2e inserts the first break symbol in the same position as in the source and reference (entreprise), while the cascade inserts it at a later position (expliquer), resulting in a subtitle of 46 characters, which is above the 42-character length limit. the cascade correctly inserts the second break (maître), as in the reference. here, the e2e copies the break position from the source sentence, which is in a different position compared to the reference (after the word be instead of the word master as in the reference). the e2e is faithful to the segmentation of the source language when it corresponds to the pauses of the speaker. the positions chosen by the cascade to insert the break symbols are before a conjunction (comment) and a preposition (du). contrary to the e2e, the cascade's decisions are based more on syntactic patterns, learned from the existing human segmentation decisions in the training data. the third example shows that prosody is important also for other factors related to the translation. the cascade, not receiving any punctuation, was not able to reproduce the question in the translation, while the intonation might have helped the e2e to render the sentence as a question despite using the wrong tense (saviez instead of savez). all in all, these examples confirm our analysis and once again indicate the importance of considering the intersemiotic nature of subtitling when developing mt systems for this task. ## conclusion we have presented the first speech translation systems specifically tailored for subtitling. the first system is an asr-mt cascade, while the second a direct, end-to-end st system. these systems allow, for the first time, to create satisfactory subtitles both in terms of translation quality and conformity to the subtitling constraints in the absence of a human transcription of the source language speech (template). we have shown that while the two systems have similar translation quality performance, the e2e seems to be modelling the subtitle constraints better. we show that this could be attributed to acoustic features, such as natural pauses, becoming available to the e2e system through the audio input. this leads to a segmentation closer to the speech rhythm, which is key to a pleasant user experience. our work takes into account the intersemiotic nature of subtitling by avoiding conditioning the translation on the textual source language length, as in previous approaches to nmt for subtitling, arriving to the conclusion that 42 is not the answer to everything in the case of nmt for subtitling. rather, key elements for good automatic subtitling are prosodic elements such as intonation, speech tempo and natural pauses. we hope that this work will pave the way for developing more comprehensive approaches to nmt for subtitling. mattia antonino di gangi, robert enyedi, alessandra brusadin, and marcello federico. 2019b. robust neural machine translation for clean and noisy speech transcripts. in 16th international workshop on spoken language translation (iwslt). mattia antonino di gangi, matteo negri, and marco turchi. 2019c. adapting transformer to end-to-end spoken language translation. in interspeech, pages 1133-1137. quoc truong do, sakriani sakti, graham neubig, and satoshi nakamura. 2016. transferring emphasis in speech translation using hard-attentional neural network models. in 17th annual conference of the international speech communication association (in-terspeech 2016), san francisco, california, usa. quoc truong do, sakriani sakti, graham neubig, tomoki toda, and satoshi nakamura. 2015. improving translation of emphasis with pause prediction in speech-to-speech translation systems. in 12th international workshop on spoken language translation (iwslt), da nang, vietnam. linhao dong, shuang xu, and bo xu. 2018. speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. in ieee international conference on acoustics, speech and signal processing (icassp). ieee. matthias eck and chiori hori. 2005. overview of the iwslt 2005 evaluation campaign. in international workshop on spoken language translation (iwslt). jonas gehring, michael auli, david grangier, and yann dauphin. 2017. a convolutional encoder model for neural machine translation. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 123-135, vancouver, canada. association for computational linguistics. panayota georgakopoulou. 2019. template files: the holy grail of subtitling. journal of audiovisual translation, 2(2):137-160. thanh-le ha, jan niehues, and alexander h. waibel. 2016. toward multilingual neural machine translation with universal encoder and decoder. in international workshop on spoken language translation (iwslt.