Simultaneous Speech Translation for Live Subtitling: from Delay to Display (2021)

Abstract: With the increased audiovisualisation of communication, the need for live subtitles in multilingual events is more relevant than ever. In an attempt to automatise the process, we aim at exploring the feasibility of simultaneous speech translation (SimulST) for live subtitling. However, the word-for-word rate of generation of SimulST systems is not optimal for displaying the subtitles in a comprehensible and readable way. In this work, we adapt SimulST systems to predict subtitle breaks along with the translation. We then propose a display mode that exploits the predicted break structure by presenting the subtitles in scrolling lines. We compare our proposed mode with a display 1) word-for-word and 2) in blocks, in terms of reading speed and delay. Experiments on three language pairs (en→it, de, fr) show that scrolling lines is the only mode achieving an acceptable reading speed while keeping delay close to a 4-second threshold. We argue that simultaneous translation for readable live subtitles still faces challenges, the main one being poor translation quality, and propose directions for steering future research.

## introduction the globalisation of business, education and entertainment, together with the recent movement restrictions, have transferred human interaction to the online sphere. the boom in online multilingual communication is setting new challenges for achieving barrierless interaction between audiences with diverse linguistic and accessibility needs. subtitles, as a key means for ensuring accessibility, have been adapted to respond to the challenge of timely communication, giving rise to live subtitling. live subtitling, whether intralingual (same language as the speech) or interlingual (different language than the speech), allows for obtaining subtitles in real time and is recently witnessing an upsurging demand in a range of occasions; from news and tv programmes, to online meetings, conferences, live events, shows and university lectures, live subtitles are bringing the world closer together. technology has always been a leading factor in live subtitling. live subtitles were initially obtained by means of keyboards or stenotyping, but automatic speech recognition (asr) gave rise to respeaking, a technique where the subtitler respeaks the original sound into an asr system, which produces the subtitles on the screen @xcite . although originally employed for intralingual subtitles, respeaking is gradually extending to produce interlingual subtitles, a task bearing close resemblance to simultaneous interpreting. still, the tediousness of the task and the scarcity of highly-skilled professionals for live subtitling call for a more pronounced role of technology for providing real-time access to information. these growing needs for access to multilingual spoken content have motivated researchers to develop fully automatic solutions for real-time spoken language translation @xcite @xcite @xcite . the new possibilities opened up by neural machine translation have led to improvements in automatic simultaneous speech-to-text translation (simulst). in simulst @xcite , the generation of the translation starts before the entire audio input is received, which is an indispensable characteristic for achieving low latency (translation delay) between speech and text in live events. the translation becomes available at consecutive steps, usually one word at a time. however, a display mode based on the word-for-word rate of generation of simulst systems may not be optimal for displaying readable subtitles. studies in intralingual subtitling have shown that a word-for-word display increases the number of saccadic crossovers between text and scene @xcite and leaves viewers less time to look at the images @xcite . for this reason, regulators, such as the uk office of communications, recommended displaying subtitles in blocks @xcite . however, this display mode is not ideal for live events since waiting until the block is filled before displaying the subtitle would extremely increase latency at the risk of losing synchronisation with the speaker. despite the existence of some applications of simulst, so far no work has explored its potential for live subtitling and how the delay in generation impacts the readability of the subtitles. given the boosting demand in live subtitles and previous studies on the readability of live subtitles, in this work we pose the following research questions: 1) can automatic simultaneous translation be a viable method for producing live interlingual subtitles? 2) what are the challenges of the generation mode of simulst systems for the readability of the subtitles? we first explore the performance of a direct simulst system on three language pairs (@xmath0, de, fr) in terms of translation quality and its ability of generating readable subtitles in terms of technical constraints, such as length and proper segmentation. second, we investigate two methods for displaying live subtitles, i.e. i) word-for-word and ii) blocks, and how the display mode affects their readability (reading speed and delay). thirdly, we propose scrolling lines, a mixed display method which takes advantage of the ability of our system to define proper line breaks and show that it leads to a more comfortable reading speed at an acceptable delay. lastly, we discuss challenges and recommendations for applying simultaneous translation for live subtitling. ## experimental setting data for our experiments we use must-cinema @xcite , an st corpus compiled from ted talk subtitles. this corpus is ideal for exploring display modes other than word-for-word because it contains subtitle breaks as special symbols. we conduct experiments on three language pairs, @xmath1, @xmath2. for tuning and evaluation we use the must-cinema dev and test sets. the text data were tokenized using sentencepiece @xcite with the unigram setting @xcite , trained on the training data with a 10k-token vocabulary size. the source audio was pre-processed with the specaugment data augmentation technique @xcite , then the speech features (80 log mel-filter banks) were extracted and cepstral mean and variance normalization was applied at global level. samples with a length above 30s were filtered out. the configuration parameters are the default ones set by @xcite . training settings our simulst systems are transformer-based models @xcite , composed by 12 encoder layers, 6 decoder layers, 256 features for the attention layers and 2,048 hidden units in the feed-forward layers. all models are based on a custom version of @xcite , having two initial 1d convolutional layers with gelu activation functions @xcite , but adapted to the simultaneous scenario as per @xcite . moreover, the encoder self-attentions are biased using a logarithmic distance penalty @xcite , leveraging the local context. training was performed with cross entropy loss, adam optimizer @xcite with a learning rate of 1e-4 with an inverse square-root scheduler and 4,000 warm-up updates. we set mini-batches of 5,000 max tokens and update the gradients every 16 mini-batches. the best checkpoint was selected based on the lowest cross entropy value on the must-cinema dev set. read/write actions of the wait-k policy are decided by means of a pre-decision module at bpe (token) level @xcite . the adopted pre-decision module is fixed, which triggers the decision process at every pre-defined number of frames. since a frame covers 10ms of the audio, an encoder state covers 40ms due to a 4x subsampling by the initial convolutional layers. since the average length of a word in must-cinema is 270ms, we consider 7 encoder states for a read/write action, which is the default parameter used by @xcite , leading to a window size of 280ms. in order to explore the quality vs latency compromise and to study the effect of the system latency on the delay of the subtitles, we experimented with two values of k, resulting in wait-3 and wait-5 models. for comparison, we also trained one offline st system per language. evaluation the evaluation focuses on two different aspects: 1) systems' performance and 2) display modes. for systems' performance, quality is evaluated with sacrebleu @xcite , which is computed on the st output containing the subtitle breaks. the latency of the system is evaluated with average lagging (al) @xcite adapted to the st scenario by @xcite . in order to test the ability of the systems to generate properly formed subtitles, we evaluate the conformity to the length constraint (len) as the percentage of subtitles having a length between 6 and 42 characters per line ted (2021). the display modes are evaluated in terms of reading speed and delay, as described in section 3.3. ## conclusions in this work we adapted simulst systems for the task of live subtitling, by forcing the systems to generate subtitle breaks. we showed that simulst systems are able to generate properlyformed subtitles. given this finding, we moved on to explore display strategies alternative to the word-for-word display, the established display mode in simulst. word-for-word display is sub-optimal for readability and comprehension @xcite . for automatically generated live subtitles, we found that it leads to an extremely variable reading speed, with some words lagging on the screen while the words towards the end of the utterance flushing through the screen. on the other hand, the display in blocks, which is the traditional mode for displaying offline subtitles, leads to a large delay and improves the reading speed only for simulst systems with a higher latency. our proposed display method, scrolling lines, is the only one achieving a comfortable mean reading speed below 21 cps, with around 80% of the subtitles having acceptable reading speed, while the delay remains along the 4-second threshold. as for the feasibility of simulst for live subtitling, there is still a long way to go in several directions. from a technical point of view, the principal issue is still the poor translation quality, which could benefit from advancements in tailored architectures. evaluation marks progress in any field, but we still lack robust evaluation methodologies, taking into account all dimensions of the target medium, both quality and readability. lastly, scholarly work is needed around user perception studies in automatic subtitling and live interlingual subtitling. technology is moving faster than research and user studies are key to ensure our implementational efforts are moving in the right direction. we hope our work has set the ball rolling for further research in automatising live interlingual subtitling.