Operating a Complex SLT System with Speakers and Human Interpreters (2021)

Abstract: We describe our experience with providing automatic simultaneous spoken language translation for an event with human interpreters. We provide a detailed overview of the systems we use, focusing on their interconnection and the issues it brings. We present our tools to monitor the pipeline and a web application to present the results of our SLT pipeline to the end users. Finally, we discuss various challenges we encountered, their possible solutions and we suggest improvements for future deployments.

## introduction in april 2021, a european international organisation hosted an international congress for its members. while the event was originally planned to be in-person, the covid-19 pandemic meant that all the foreign participants connected remotely. the event was run in 5 languages (english, german, french, spanish and russian) covered by human interpreters. the remote audience spanned 51 countries with the total of 42 desired languages. our role was to provide live translation into these additional languages in text form. the technical backstage of the event operated in the standard in-person mode, with interpreters in their booths, following the main live video stream and providing or relaying interpretation as needed. this resulted in six audio channels being available, one for each language and one additional channel called "the floor" which always contained the speech of the current speaker regardless the language. the interpreters delivered their interpretation to the appropriate language-labelled channels. each of the interpreters was translating either from english to their assigned language, or vice versa. at any given moment, there was thus supposed to be exactly one source of english speech, either directly from the speaker or from one of the interpreters. the channel of the language spoken at the floor at a given point was silent (e.g. the german channel when german was coming from the floor, because the german interpreters' booth was busy providing the english interpretation to the english channel). while this arrangement caused some technical challenges, it also provided novel opportunities. the first challenge was to concurrently follow all the channels using portable equipment and the second challenge was to direct the correct audio channel to the respective speech processing server. the multiple input languages can make the setup interestingly robust: for example, when the speaker is speaking a non-english language, their speech can be automatically transcribed and then machine-translated to english. at the same time, an interpreter is providing human interpretation into english, which can be in turn processed by the english speech recognition system. we thus have two sources of english text and we can choose the better one, live, bypassing any processing hurdles at any of the paths. conversely, when the speaker is speaking english, the interpreters will be providing their assigned languages, perhaps captured in better sound conditions or better articulated. these languages can again be automatically transcribed and translated to english, serving as alternative sources if the main speaker is hard to follow for the technology. the paper is structured as follows: in section 2, we describe our hardware setup at the backstage. section 3 provides a complete picture of the processing pipeline from 6 input channels to 42 output languages. section 4 briefly summarizes the key building blocks, namely the speech recognition (asr) and machine translation (mt) systems used. these systems were cluster-based, run on premises of the individual research institutes contributing them, connected via the internet. section 5 presents our web-based solution for live display of the many translations. several members of our team were on duty to monitor all the components during the event and only one skilled "system operator" was present in person at the backstage. the operator's experience is provided in section 6 and some of the monitor tools at his disposal are described in section 7. section 8 summarizes the planned improvements of the setup and components and section 9 concludes the paper. ## lightweight hardware setup for multi-source speech processing building upon our previous experience with speech processing at live events, we knew that desk space would be a limiting factor and that at most one person would be admitted to take care of the system on site. aside from providing the translation service for the (remote) participants, we also needed to fully record the session for future analysis. the sound engineers facilitating the event itself did not have any recording equipment suitable for our purposes. our final solution consisted of one laptop (dell vostro 3583) running ubuntu 20.04 and two behringer u-phoria umc404hd external usb sound cards, each following up to 4 mono sound channels. to minimize the risk of losing the recording, the system setup was primarily geared towards recording. throughout the session, two arecord tasks were recording raw outputs of each of the external sound cards, producing two 4-channel pcm sound files sampled at 44 khz. any sound processing, be it for monitoring purposes or for the actual speech processing, was based upon these growing files. we avoided touching the software sound devices to prevent any software conflicts during the session. to monitor the incoming sound of any channel across the two recording devices, we simply followed the most recent additions to the respective file and selected the channel with ffmpeg: the added benefit of this file-based access to the live sound was that until the actual session was running, we could easily simulate live session by slowly copying data from a pre-recorded sound to mock "raw recording" file: the pv command limits throughput, simulating real-time growth, with the byte rate of 688k determined empirically. the dd @xmath0. when watching the "current" sound with tail -f, it is guaranteed that the processing starts aligned to the 4 channels in the file. such a simulation proved invaluable esp. immediately before the start of the live event. no tests of the components can ensure that the whole complex ensemble is running and ready for an immediate launch. ## slt pipeline description our "slt pipeline" consists of asr and mt systems and various components transforming and transporting data between them. the actual setup -which languages to follow, how to switch among them, which ones to use the source for the final translation -varies across events that we already took part in. here we focus on the particular setup of the international (remote) congress but our tools allow for a rather flexible configuration of the "wiring". the inputs to the pipeline are the audio sources: english and the five other languages. english audio is converted to text by an english asr system, while the other languages are converted to english text by the respective asr system and a subsequent mt system. there is still some room for system optimization by deploying multilingual asr systems. for mt, we already make use of multi-linguality (a single system trained to translate from any of a small set of languages into english). to achieve independent processing of each of the input languages, each audio source is processed by a separate asr and mt system. the central component of the pipeline is a selection tool. given the multiple variants of inputs (all converted to english text), the operator has the option to dynamically choose which one is currently most suitable for the translation into all the desired target languages, as discussed in section 6 below. then, the chosen english text source is fed to a single one-to-many multi-lingual mt system. in our case, the mt system translates the same input text from english to 41 target languages at once. finally, the selected english and all the translations are sent to the web application presenting the outputs to the users (section 5). it is worth mentioning that depending on the sound channel the user is following, they can observe bigger or smaller delay between the speech and the shown translation. for instance, a german user would most likely follow the german speech, but the automatic german transcript can actually be the result of the german interpreter producing english followed by english asr and english-to-german mt. there is a potential for improvement in this setup: for the five other non-english spoken languages, we could present their transcription to the users, instead of displaying the output of the one-to-many mt system. this setup would, however, emburden the system operator even more, because the final outputs of these 5 languages would be running on separate paths with independent risk of a crash, requiring independent monitoring. we thus opted for a more uniform approach which was easier to operate: a single input english translated to all languages at once. if any of the target languages stopped updating, the operator knew that all are affected and vice versa. see figure 1 for an overview of the data flow when processing non-english speech. ## asr and mt all asr systems we used followed the architecture proposed in @xcite for lowlatency online speech recognition. the modeling for all input languages are handled by the streaming sequence-to-sequence model proposed in @xcite with the use of a multi-domain speech dataset @xcite . the dynamic transcription mechanism in @xcite is adopted in all asr systems to achieve very low user-perceived latency. the mt systems into english are multilingual systems based on the transformer architecture @xcite . the system uses a re-translation strategy @xcite in order to reduce the latency of the mt, as opposed to streaming approaches such as @xcite . in the re-translation approach, incoming text from asr is translated afresh starting from the beginning of the sentence, or the end of the stable asr output, whichever is earlier. consequently, the new output of the mt system can rewrite, or "flicker" what has already been shown, leading to the question of how best to present this to the user (see section 5 below). following @xcite , we inject partial sentence pairs (prefixes) into the training data so that the system is better able to deal with this at runtime. similarly, the one-to-many system that translates out of english is a multilingual transformer, using the pseudo-word approach to identify the desired language @xcite . it is trained on 231m sentence pairs sampled from the opus collection @xcite and covers 41 target languages, including all official eu languages. the out-of-english systems also use re-translation and prefix training. to connect asr and mt, we deploy an nmt-based segmentation component which converts the asr output (all lower-case, no punctuation, with speech phenomena) into more standardized text by inserting punctuation, inferring capitalization and removing disfluency phenomena @xcite . in addition to improving the readability of the transcript, this component is necessary for mt, which is trained to expect orthographically correct (partial) sentences. ## presenting translations as mentioned above, our asr systems are gradually updating their outputs, not necessarily in the incremental fashion. similarly, the automatic segmentation can update the placement of punctuation symbols and as a result, the translations from the english-to-many mt systems can and will change over time. this situation (and the problems it brings when the space for the output translation is limited) is thoroughly explained by @xcite . in short, there is no easy and non-disturbing solution if an update changes some content which has already been scrolled away due to a small presentation space. eventually, a translated hypothesis becomes confirmed, meaning that the translation is final and will not change any more. while we could simply wait for the translation of a sentence to become confirmed and then display it to the users, it would introduce a needless and sometimes unacceptable delay. luckily, our setting allows us to use larger screen space than just the few lines of subtitles as @xcite consider. we use the term "paragraph view" for this. specifically, we developed a web-based interface for presenting the full live transcript of which the tail keeps changing. the web application is hosted on a web server, which receives the translated hypotheses from the mt system. the hypotheses are then published on a websocket, to which the browsers of end users connect. as the browser receives updates and finalized hypotheses from the websocket, it displays them to the end user. finalized messages are displayed in black and unconfirmed hypotheses are displayed in grey, so the user can distinguish between them. one important aspect of our setup is the relatively high number of available languages. while the presentation interface is flexible and can accommodate any number of languages, shown as columns, in practice showing too many languages leads to very narrow columns and, subsequently, the text scrolling too fast to be read. additionally, each user will be interested in following only a very small number of languages and will want them displayed close to each other. a simple table of language codes therefore allows the user to choose which languages get displayed. this choice of languages can also be preloaded by an argument to the tool's url entrypoint, allowing the event organizers to choose different default languages for different groups of users by spreading different versions of the link. based on our experience from several test sessions, we added the option for the operator to broadcast messages to users. there are many conditions of operation where some information from the operator would be very valuable for the spectators and would comfort them, such as "the show is delayed, stay tuned", "thank you for watching and we would like some feedback from you", or apologies for the current technical issues etc. we saw in practice that event organizers tend to choose very varied means and platforms of communication with the users, and the attention of users can also wander across them, so it is never certain where they would best notice. broadcasting these messages interleaved with the main content of the transcribed and translated speech is a unifying option here. it is important to note that these messages have to be delivered in all the supported target languages. to ensure the correctness of these messages, we collected a list of about 20 potentially useful english messages prior to the event. we translated them with our multi-target mt systems and asked many colleagues to review the automatic translations. for a few target languages, no native speaker of the language was available and the automatic translations remained unchecked. based on the experience at the event, 8 more messages were added, primarily explaining immediate failures or delays that we observed in the source stream. to differentiate them from the translations, these operator messages were displayed in bold. figure 2 contains an example of the messages displayed to the user. ## operator experience the operator was facing a challenging task when the pipeline was running during the live event. he had to monitor all individual parts of the pipeline, spanning from the sound input to the very final presentation in the web interface, and he was also selecting the current best variant of english source for the multi-target translation, as described in more details below. because our current pipeline still misses automatic language identification, the operator had one additional task: based on the floor sound which he was constantly following inform the system about any change of the language spoken at the floor. the system was then prepared to redirect "sound pipes" accordingly, so that each of the 5 asr system inputs always received its language, regardless whether it came from the original speaker (floor) or from an interpreter. ## pipeline monitoring tools the pipeline is complex and consists of many separate components. it can be only expected that something will break, sooner or later. thus, a set of tools monitoring the health and status of the pipeline was developed. coming back to the pipeline representation as a directed acyclic graph, it makes sense to monitor two parts of the graph structure: the health of individual components (vertices) and the data flow between them (edges). to monitor the individual components, the pipeline (as compiled to the bash script) saves (unix) process ids of all the components. then, a simple script regularly checks if processes with these ids are still running, showing the status of each component to the operator. this allows for cursory checks of which components of the pipeline, e.g. a client connected to an asr system, are up and running and which components have fully broken down. similarly, all intermediate component outputs and standard error outputs are duplicated to separate files via a modified tee which adds exact timestamps at the beginning of each saved line. this detailed (and now fully automated) logging proved essential both during the development as well as during the live event. these logs are recorded only on the operator's machine but it proved very useful to regularly upload them to a shared space where all technical team members could help investigating what is going on because the operator is generally fully occupied with other tasks. we are aware that there are server-based logging solutions but our approach is flexible, lightweight and does not need any external tools. in debugging, absolute timestamps in the logs are necessary when investigating why the pipeline crashed. this usually involves cross-checking many logs and timestamps are the best means of finding the culprit. in several occasions, we also made use of these timestamped logs to replay some problematic input, allowing us to debug only one isolated component. during live deployments, these log files are also monitored. the simplest approach taken at the reported event is to tail -f all the standard error outputs at once to see the latest errors of any component. for the intermediate output files (i.e. the data that are passed along each of the pipeline graph edge) we developed a new tool. this tool tracks the moving average of the time between output updates (simply checking for changes in log modification time). whenever the average extension time elapses with a suitable margin but no output is added, the operator is notified. this is a very flexible detection of situations where the components are running, but for some reason they stop or slow down outputting data. ## future improvements although the event went quite smoothly for us and there were no major technical issues or hiccups, we discovered some sore points and opportunities to improve. first, the task of the operator is quite demanding, as they have to constantly monitor the state of the pipeline, select the currently best-performing english text source and broadcast operator messages to the end users when necessary. while the operator already has some tools to monitor the health of the pipeline, they still have to juggle multiple monitoring tools. to alleviate this, we propose to use a web application as the control center for the operator. the server for the web application would live in the docker container, as described in section 3.1, along with the pipeline and it would provide an api to obtain the last few lines of each stored-output file, list of running processes and other necessary information. the operator would then be able to simply observe and control the pipeline in their web browser. another improvement could be automatically switching the english text source input of the english-to-many mt system based on asr confidence levels and other criteria. this would free the operator from having to constantly monitor the english text sources and judge which one is currently performing the best. however, the definition of these criteria would not be straightforward due to the different nature (and possible means of confidence estimation) of the components. the operator messages were pre-translated before the event and revised for quality. a nice addition would be the option to simply type a new operator message in english, let that sentence be translated by the english-to-many mt system and then broadcast it to the end users. ## conclusion we described our experience with running a complex system for spoken language translation aimed at substantially extending the set of provided target languages. from five official languages of the event, provided by human interpreters, we were able to cover 42 languages spoken in the participant's countries. we proposed a novel technique increasing the overall robustness of the system to technical or human failures, namely following multiple sources at once and dynamically choosing the current best one. while the technique was so far tested in its simplest form, switching between the sources manually, it helped us to navigate through partial system failures. at one occasion, our system was the only translation service available, because even human interpreters have given up processing the sound from a distorted remote call. for the future, we plan to improve the user interface for the operator. any means of automatic diagnostics, incl. recognition and translation confidence, would be highly desirable. we will also focus on more advanced techniques for combining multiple inputs.