ELITR Multilingual Live Subtitling: Demo and Strategy (2021)

Abstract: This paper presents an automatic speech translation system aimed at live subtitling of conference presentations. We describe the overall architecture and key processing components. More importantly, we explain our strategy for building a complex system for end-users from numerous individual components, each of which has been tested only in laboratory conditions. The system is a working prototype that is routinely tested in recognizing English, Czech, and German speech and presenting it translated simultaneously into 42 target languages.

## introduction with the tremendous gains observed recently in automatic speech recognition (asr) and machine translation (mt) quality, including methods of joint learning of both of the tasks, the goal of a practically usable simultaneous spoken language translation (slt foot_0 ) system is getting closer. in this paper, we introduce the slt system developed in the eu project elitr (european live translator foot_1 ) @xcite ) which aims at a distinct setting: real-time speech translation into many target languages. ## motivation in the current globalized world, meetings with participants from a very wide spectrum of nations are common. many multinational organizations, public or private, regularly run congresses and conferences where attendees do not have any language in common. interpretation is a must at such meetings and the cost of interpretation services consumes a considerable portion of the budget. the number of provided languages is then kept as low as possible, even in cases when some of the attendees are not sufficiently fluent in any of them. we primarily focus on the setting of such multinational congresses where one source speech needs to be translated into many target languages. while we are aware of the quality limitations of speech recognition and machine translation, we strongly believe that the technology has reached the level where it is becoming practically usable and related systems confirm that belief, see section 3 below. even if the automatic translation of recognized speech is not perfect, it can serve as a valuable supportive material. for instance, a czech attendee may have a fair knowledge of english and french, but may easily get lost due to pronunciation difficulties to follow, gaps in his or her grammar knowledge, general vocabulary or specific terminology. following live subtitles in mother tongue while listening to the foreign language could be of great help. some level of errors in the subtitles is acceptable if the subtitles are sufficiently simultaneous. our main goal is thus gist interpretation, i.e. live supportive translation of speech into text. within the elitr project, we focus on asr for english, czech, german, french, spanish and later russian and italian, and targetting the set of 43 languages spoken in member countries of eu-rosai, the association of supreme audit institu-tions of the eu and nearby countries. experimentally, we include also other languages based on available systems among the research partners in our project, e.g. hindi. the scientific motivation for our efforts is to find an approach that allows to assemble laboratory system components to a practically usable product and to document the problems on this journey. ## related systems live spoken language translation has been continuously studied for decades, see e.g. @xcite ; @xcite ; @xcite . recent systems differ in whether they provide revisions to their previous output @xcite @xcite @xcite , or whether they only append output tokens @xcite @xcite @xcite @xcite . @xcite were probably the first to allow output revision when they find a better translation. @xcite released a simpler setup as an open-source toolkit consisting of a neural speech recognition system, a sentence segmentation system, and an attention-based translation system providing also some pre-trained models for their tasks. @xcite evaluated only the quality of the output translations using bleu and wer metrics. @xcite proposed a new approach with a delay-based heuristic. the model decides to read more input (or wait for it) or write the translation to the output. @xcite introduced a simple wait-k heuristic: output is emitted after k words of input. both works are limited to simultaneous translation, i.e. they start from text and only simulate the speech-like input by processing input word by word. arivazhagan et al. (2020) combine industrygrade asr and mt and allow output revisions by re-translating the source from scratch as it grows to decrease the latency, providing acceptable translation quality at the price of a higher number of text revisions. ## elitr flexible architecture we always strive for the best performance for each considered language pair. with the perpetual com-petition in asr and mt research, it is not surprising that there is no universally best solution. the interplay of available data, underlying method, the actual implementation as well as its adaptability to the domain of interest requires different choices for different languages. furthermore, the top-performing components are often available only at universities or research labs, as more or less stable research prototypes. releasing any such system, let alone their combination so that they could be easily deployed by lay users is surely possible, but it would require considerable additional implementation resources. the elitr architecture @xcite tackles this integration problem by means of a distributed connection-based client-server application. research labs provide their components by connecting to a central point (the "mediator") which in turn uses these "workers" to satisfy users' stream processing requests. a technical benefit is that worker connection is issued from the secured networks of the labs so it usually does not run into firewall issues. ## system components all our workers, except recent online sequenceto-sequence asrs, have been described in our iwslt 2020 shared task submission @xcite . we briefly summarize them in following sections. ## evaluation we evaluate our systems in multiple ways: it is a challenge for future work. it can be achieved by speaker adaptation of the asr from a small sample of the speaker, by multi-lingual asr, and by collecting non-native speech training data, as ami corpus. for comparability across our project partners but also across external research labs, we publicly released a tool for evaluation, sltev foot_2 (ansari et al., 2021) and a test set. 4 the results of our currently best candidates on the testset are in it is important to realize that the evaluation for quality, latency and stability on a speech-to-text test set in lab conditions is necessary, but not sufficient for assessing the practical usability of the system. practical usability has to include the presentation layer (section 7) and tests in live sessions or rigorously controlled conditions. ## presentation techniques the last step in an slt system is the delivery of the translated content to the user. our goal stops at the textual representation, i.e. we do not include speech synthesis and delivery of the sound, which would bring yet another set of design decisions and open problems, see e.g. @xcite . we experiment with two different views for our text output, both implemented as web applications. the "subtitle view" is optimized toward minimal use of screen space. only two lines of text are available which leaves room either for e.g. a streamed video of the session or the slides, or for many languages displayed at once, if the screen is intended for a multi-lingual audience. the "paragraph view" provides more textual context to the user. ## conclusion we presented a complex system for live subtitling of conference speech into many target languages, composed of research prototype components but still serving in close-to-production setting. new and updated models and other components can be easily plugged in and tested in practice. as of now, we are at a good starting point for gradual model improvement and field tests. one of them is very likely to be the meta-forum 2021 but we are also searching for suitable events with more than one official communication language. demonstration videos from past sessions can be found in the blogposts at https://elitr.eu/ blog/ .