S mar T erp: A CAI System to Support Simultaneous Interpreters in Real-Time (2021)

Abstract: We present a system to support simultaneous interpreting in specific domains. The system is going to be developed through a strong synergy among technicians, mostly experts on both speech and text processing, and end-users, i.e. professional interpreters who define the requirements and will test the final product. Some preliminary encouraging results have been achieved on benchmark tests collected with the aim of measuring the performance of single components of the whole system, namely: automatic speech recognition (ASR) and named entity recognition.

## introduction simultaneous interpreting is a very cognitively demanding task consisting in the execution of different processing sub-tasks in parallel. as an example, if we take the interpretation of numbers, a high error or omission rate is observed, especially in the case of interpreters working in isolation (without a booth-mate, as in remote simultaneous interpreting or rsi), ranging from 70% in the case of students to as much as 40% in the case of professional interpreters @xcite . as a further example, a study reported in @xcite shows that the number of disfluencies (i.e. hesitations) produced by interpreters is significantly higher than that produced by non interpreters, mainly due to the lexical richness of interpreters themselves. the smarterp project aims to develop a computer-assisted interpretation (cai) system to support the simultaneous interpreter, especially in the rsi modality, by addressing the entire workflow of the interpreting activity, from the preparation of specialised multilingual glossaries that will serve to feed and train the asr and ai built into the system and extract and propose terminology (e.g. named entities, numerals, etc) to assist the interpreter in real-time, to the post-event validation of new entries by the interpreter that will be fed back into the system to perpetuate a virtuous circle of generating and accumulating specialised knowledge for recurrent use by the interpreter/team of interpreters and the end-customer of the interpreting services. although the proposal of using both natural language processing (nlp) and asr technologies is not new for developing cai tools (see e.g. the works reported in @xcite for a good review) and, at the same time, there are projects, such as eabm foot_0 that aim to use extensively asr technology to create user-friendly interpreting interfaces, we believe that the strong synergistic effort produced in the smarterp project among nlp/asr experts, software developers and endusers, aimed at both defining the requirements and evaluating and refining the performance of the resulting cai system, can provide a significant step forward in the development of such tools. ## automatic transcription of audio one of the requirements of the asr systems used in the smarterp project is that they have to perform well on specific application domains. more precisely, the source language to be translated by the interpreter may contain a large number of technical terms and morphological variations that are usually not present (or occur with low frequencies) in "general purpose" training text corpora. the result is that a general purpose language model (lm) exhibits on in-domain data high values of both out-of-vocabulary (oov) word rates and perplexities, worsening the word error rate (wer) of the asr system that utilises it. to alleviate this effect we propose a procedure, described in section 2.2, that extracts from a given corpus the texts that are "closest", in some way, to a glossary of terms furnished by an interpreter. this one is assumed to contain most of the important words of the subject of a given interpretation session. then, taking advantage from previous experience for estimating the proficiency of second language learners (see @xcite ), we developed a procedure, summarised in section 2.2, to adapt a general purpose lm to the domain of each interpretation session. this way we are able to instantiate an asr engine specific to each interpretation session. note that this has a strong impact on the whole architecture of the system, since it requires to update, on demand by the interpreters, the lm of each asr engine. ## semantic interpretation once the transcripts are generated from the audio input, the role of the semantic interpretation module is to detect relevant entities that appear on these transcripts and that may be of interest for the interpreters. examples of such entities are those that may be difficult for them to translate during the interpretation session, such as terms that are very specific to the domain or numerical values, which are known to be hard to translate since they require an additional cognitive effort due to the transcoding exertion they require, etc. the main challenge in this context is that we are not dealing with a typical named entity recognition problem, where elements like persons, organisations, places, etc., need to be detected. that is, recognising the entity "united states" in the text and offering its potential translation into spanish "estados unidos" may not make much sense in the context of the whole system, since this is commonly a well-known term for interpreters. using an example of the dentistry domain, it is rather more useful for an interpreter to identify the noun "flap" and provide its translation into spanish ("colgajo"), or to identify a numerical value ("nineteen seventy six") and transform it into the arabic numeral (the year 1976) the interpreter will recognise and introduce in the interpreted speech (in the target language) with little or no effort. therefore, we need to talk about interpreter-relevant term recognition and their translation into the target language. to perform this type of task, the module is based on the usage of a layered set of multilingual general purpose, domain-specific and user-specific knowledge graphs, following best practices in the representation of multilingual linked data, as described in section 3.1. the translation of numerical entities is discussed in section 3.2. ## system architecture fig. 1 shows the block diagram of the integration between the following modules: -rsi technician console represents the web-based interface used by the technical staff managing an interpreting session. thanks to this interface, the technician is able to introduce in the system the audio and video flows of the conference speakers. on the other hand, rsi interpreter console represents the set of software consoles used by interpreters to visualise both the speakers' video and the materials (e.g. presentation slides) shared by them as well as to receive their audio. interpreters also use these consoles to manage the input and output audio channels for developing the necessary tasks during an interpreting session. they can switch the input channels from the technician one (i.e. floor) to the ones shared by other interpreters and modify their output language channel. in this console interpreters also see the ai tool output as explained below. -cloud asr represents the on-demand cloud service in charge of transcribing in real-time an audio input in different languages for producing a text output. its interface supports the receipt of a set of consecutive audio chunks (having a duration that will be optimised during the test phase of the smarterp project) extracted from an audio flow through a websocket. as a result the cloud asr system sequentially respond with the text transcription in the same websocket connection using a json document. -ai tool receives an audio transcription and generates a set of terms for helping interpreters to perform their job. its interface support the receipt of a set of consecutive json documents with a transcription through a websocket. as a result the ai tool generates the terms and sends them through the websocket interface when ready. the complete interaction flow between the modules can be summarised as follows: 1) using the javascript mediastream api, the interface asks permission to access web camera and microphone of the pc used by the technician. this generates a video stream and an audio stream. these streams are shared with the interpreters using licode foot_1 , an open source multi videoconferencing platform based on webrtc. 2) thanks to the audiocontext api, the interface extracts audio chunks from the audio stream and sends them to the cloud asr using a previously opened websocket. 3) cloud asr synchronously answers with the text transcription using the format described above. 4) after receiving the transcription, the technician interface sends each json object with the transcription to the ai tool using a second websocket connection previously created. 5) the ai tool process the transcriptions and asynchronously generates the terms for interpreters. these terms are sent to the rsi technician console through the websocket connection. 6) using licode data channel the technician console multicasts the terms generated by the ai tool to the consoles of the interpreters connected to the same session. these terms are displayed, without a significant delay, to the interpreters in the console. 7) audio output in the different available languages is sent to the assistants of the conference and to other interpreters in the same session. ## system evaluation as mentioned above, in smartterp we prepared benchmarks for the 3 languages of the project (english, italian, spanish) plus two important european languages, french and german. ## conclusions the smarterp project is an on-going innovation action funded by the eit digital aiming to develop a computer-assisted interpretation system to support the cognitively demanding task of simultaneous interpretation with state-of-theart language technology. to do so, the consortium, created to solve the many challenges the real-time constraints impose on the system, has obtained so far encouraging results. in particular: a) good performance on specific application domains by the asr systems thanks to a procedure that extracts from a given corpus the texts that are closest to a typical interpreters' glossary and adapts a general purpose lm to the domain of each interpretation session; and b) devising of a semantic interpretation module to detect relevant entities that appear on the asr transcripts and that may be of interest for the interpreters, such as named entities and terms that are very specific to the domain, or numerical values, which are known to be difficult to interpret since they require an additional cognitive effort due to the transcoding exertion they require.