A Multimodal Simultaneous Interpretation Prototype: Who Said What (2022)

Abstract: “Who said what” is essential for users to understand video streams that have more than one speaker, but conventional simultaneous interpretation systems merely present “what was said” in the form of subtitles. Because the translations unavoidably have delays and errors, users often find it difficult to trace the subtitles back to speakers. To address this problem, we propose a multimodal SI system that presents users “who said what”. Our system takes audio-visual approaches to recognize the speaker of each sentence, and then annotates its translation with the textual tag and face icon of the speaker, so that users can quickly understand the scenario. Furthermore, our system is capable of interpreting video streams in real-time on a single desktop equipped with two Quadro RTX 4000 GPUs owing to an efficient sentence-based architecture.

## introduction automated simultaneous interpretation (si) is promising for facilitating real-time cross-lingual communication. video streams have become a most popular form of communication nowadays because of the blossom of smart phones, video sites, chat apps and so on. therefore, there is an increasing trend towards applying si to video streams. spoken language such as conversations, discussions and debates are common in video streams where two or more speakers are involved. working out "who said what" is a natural path for people to understand these video streams. however, when applying conventional si, users are merely presented "what was said", that is, the translation of the transcripts from speech recognition. therefore, users must guess who the speaker of the source utterance of each translation was. given the following facts between the source utterances and translations, to address this problem, this paper presents a novel multimodal si system that presents users "who said what" from video streams. as illustrated by figure 1 , in the graphic user interface of our system, each record consists of the translation of an utterance which indicates "what was said"; a speaker tag, e.g., spk 0 , spk 1 , and a face icon which indicates "who said". users will be able to understand the video streams easily through reading these records. the main challenges for us to build the proposed multimodal si system are 1. how to recognize speakers from video streams? 2. how to maintain low latency for interpretation? to solve the first challenge, we develop a speaker predictor (sp). sp creatively adapts a multimodal speaker recognition approach that combines voice embedding @xcite , face proceedings of the 15th biennial conference of the association for machine translation in the americas orlando, usa, september 12-16, 2022. volume 2: users and providers track and government track i work well under pressure wonderful and what would you say are some of your weaknesses one of my biggest weaknesses is asking for help when i need it . i 'd like to do better at that i appreciate your honesty mister wang what can you tell me about some of your goals over the next few years my primary goal is to gain more work experience so a position like this would help me meet that goal i 'd also like to learn more about the different aspects of banking i think those goals are very smart (a) plain translations spk0 : i work well under pressure spk1 : wonderful spk1 : and what would you say are some of your weaknesses spk0 : one of my biggest weaknesses is asking for help when i need it . spk0 : i 'd like to do better at that spk1 : i appreciate your honesty mister wang spk1 : what can you tell me about some of your goals over the next few years spk0 : my primary goal is to gain more work experience spk0 : so a position like this would help me meet that goal spk0 : i 'd also like to learn more about the different aspects of banking spk1 : i think those goals are very smart (b) translations annotated with speakers embedding @xcite and active speaker detection @xcite . to solve the second efficiency challenge, we synchronize all the heavy multimodal computation with the sentence-based interpretation @xcite so that neural networks can run efficiently on gpus through processing large batches as input. the rest of this paper is organized as follows. first, section 2 describes the user interface of our system. then, section 3 presents the architectures of sentence-based multimodal si and multimodal speaker recognition. next, section 4 describes the implementation of each module in detail. after that, section 5 briefly analyzes the real-time capability of our system. furthermore, section 6 compares our system with related works on multimodal machine translation. finally, section 7 concludes this paper with a description on future works. ## user interface the user interface of our system is split into two parts as illustrated by figure 1 . the input video stream is displayed in the left part of the window, below which translations are presented in the conventional way as subtitles. the main output of our si system is displayed in the right part of the window which consists of speaker tags, face icons and translations. speaker tags are texts that follow the pattern of spk n, such as spk 0 and spk 1 , where n is a number assigned to each physical speaker based on the order of his or her appearance in video streams. because these tags are plain texts, in addition to being shown in the graphic user interface, they can also be used to generate textual transcripts for review or post-editing. speaker annotations can greatly improve the readability of the transcripts as illustrated by figure 2 . face icons are the face shots of the speakers when they are saying the corresponding utterances. these face icons have two merits. first, they allow users to double check whether asr word stream source-language sentence temporal range speaker tag face icon i'm smith nice to meet … sentence segmenter i'm smith 0.4s ~ 1.0s translator 私は スミスです (0.5s) (0.9s) (1.1s) (1.3s) (1.4s) speaker predictor spk1 translation audio stream figure 3: system architecture the speaker tags are correct or not. second, they allow users to predict the sentiments of the utterances from the facial expressions of the speakers. translations are translated sentences. our system performs sentence-based simultaneous interpretation. instead of waiting for a speaker to finish speaking, it detects the event that the speaker has finished a sentence @xcite . then our system translates that sentence and displays it to users. ## system architecture our system consists of an automatic speech recognition (asr) engine, a sentence segmenter, a speaker predictor and a translator as illustrated by figure 3 ). blue rectangles represent modules and red rounded rectangles represent data examples. arrows show the direction of data flow. the system takes a video stream as input, and generates the speaker tags, face icons and translations in an online manner. the system accomplishes this multimodal simultaneous interpretation task in four steps as follows. the word stream consists of words and their time stamps. for example, "i'm (0.5s)" means that the word "i'm" appears at the 0.5 seconds of the video stream. each active speaker is presented by a textual tag and a face icon. ## real-time capability our si system employs a parallel pipeline to integrate the main modules to meet the real-time requirement. we estimate the real-time capability of each module using workload percentage, which is calculated as, ## related works multimodal machine translation -the task of doing machine translation with multiple data sources -is a trending topic @xcite @xcite . a large volume of research effort has been dedicated to improving the translation quality through drawing information from modalities other than text @xcite @xcite @xcite . our interpretation system approaches the task of multimodal machine translation from a different angle. imaging when interpretating a video stream, the visual contents of the video stream will mainly fall into two categories, 1. the speakers; 2. the subjects of the speeches. our interpretation system focuses on the first category. it recognizes the speaker of each utterance, and then annotates the translation with the speaker, so that users can better understand the video stream despite translation latencies and errors. in contrast, the related works focus on the second category of contents so that users can get better translations. nevertheless, our interpretation system and the related works on multimodal machine translation are complement with each other. integrating our system with the related works will lead to very effective interpretation systems which can generate both well-annotated and high-quality translations from video streams. ## conclusion in this paper, we propose an automated multimodal simultaneous interpretation system to improve the user experience on interpreting video streams, and build an efficient implementation based on the sentence-based interpretation. our system has been tested on various video streams. the system works very well on some of the video streams and produces high-quality translations which are correctly annotated with the tags and face icons of speakers. however, our system performs poorly on some video streams which have difficult speeches. when the speech is not clear enough for the asr module to generate decent transcripts, the sentence segmenter will fail to produce sensible sentences, and then the whole system will perform poorly. therefore, in the future, we plan to address this problem through adding more audio and visual features into the sentence segmenter and the translator to improve the robustness of our system.