M eet D ot: Videoconferencing with Live Translation Captions (2021)

Abstract: We present MeetDot, a videoconferencing system with live translation captions overlaid on screen. The system aims to facilitate conversation between people who speak different languages, thereby reducing communication barriers between multilingual participants. Currently, our system supports speech and captions in 4 languages and combines automatic speech recognition (ASR) and machine translation (MT) in a cascade. We use the re-translation strategy to translate the streamed speech, resulting in caption flicker. Additionally, our system has very strict latency requirements to have acceptable call quality. We implement several features to enhance user experience and reduce their cognitive load, such as smooth scrolling captions and reducing caption flicker. The modular architecture allows us to integrate different ASR and MT services in our backend. Our system provides an integrated evaluation suite to optimize key intrinsic evaluation metrics such as accuracy, latency and erasure. Finally, we present an innovative cross-lingual word-guessing game as an extrinsic evaluation metric to measure end-to-end system performance. We plan to make our system open-source for research purposes.

## introduction as collaborations across countries is the norm in the modern workplace, videoconferencing is an indispensable part of our working lives. the recent widespread adoption of remote work has necessitated effective online communication tools, especially among people who speak different languages. in this work, we present meetdot, a videoconferencing solution with live translation captions. participants can see an overlaid translation of other participants' speech in their preferred language. currently, we support speech and captions in english, chinese, spanish and portuguese. the system is built by cascading automatic speech recognition (asr) and machine translation (mt) components. we process the incoming speech signal in a streaming mode, transcribe it in the speaker's language to be used as input to an mt system to decode in the listener's language during the call. we have very tight latency requirements to be able to provide good quality captions in a live video call. our framework has several features to enable a better user experience and reduce the cognitive load on the participants such as smooth pixel-wise scrolling of the captions, fading text that is likely to change and biased decoding of machine translation output @xcite to reduce the flicker of the captions. in addition, we present preliminary work towards identifying named-entity mentions in speech by interpolating a pronunciation dictionary with the asr language model. we present the key metrics to measure the quality of the captions such as accuracy, latency and stability (flicker in captions). our system provides an integrated evaluation suite to enable fast development through hill climbing on these metrics. in addition to these intrinsic metrics, we present an interesting online cross-lingual word guessing game, where one of the participants is given a word that they describe and the other participants have to guess the word by reading the captions in their respective languages. we present the following as the key contributions of our work: ## related work the area of simultaneous translation has attracted a lot of attention is recent years. the recent editions of the workshop on automatic simultaneous translation @xcite and the tutorial in emnlp 2020 @xcite provide us an overview of the state-of-the-art and challenges in live translation. recent technical advances include newer architectures such as prefix-to-prefix @xcite and adaptive policy methods such as imitation learning @xcite and monotonic attention @xcite . the solutions in this space are differentiated into speech-tospeech and speech-to-text, in the former there is a speech synthesis component. our work falls in the latter bucket, since we only display captions to the user. since we have access to separate audioinput channels (one per participant), we do not need speaker diarization @xcite . re-translation is a common strategy applied, wherein we translate from scratch every new extended source sentence, transcribed through the asr module as the participant speaks. the ability to modify previously displayed captions when new asr output is available can lead to flicker in displayed captions. following previous work @xcite , we measure this using the erasure metric and reduce it using biased beam search during mt decoding. to better capture our use case of short exchanges in a meeting scenario compared to long speech, we introduce additional metrics initial lag, incremental caption lag, mean word burstiness and max word burstiness. ## system description the overall system architecture is shown in figure 1 . it mainly consists of two components (1) frontend: runs on the user's computer locally (2) backend: runs on the server and consists of asr and mt modules that interact with the flask server. the modular architecture allows us to swap the asr and mt components from different service providers (such as google asr or in-house kaldi-based asr services). these are explained below. ## evaluation dataset: in order to evaluate our system in the most-appropriate deployment scenario, we construct an evaluation dataset based on meetings within our team. we have a total of 5 meetings, 3 meetings involving 7 participants and in english and 2 meetings involving 2 participants in chinese. the content of the meetings are daily work conversation and contains a total of 135 mins (94 mins english and 41 mins chinese). we manually transcribe and translate these meetings using a simple and uniform set of guidelines. the resulting dataset consists of 494 english utterances (∼ 11k words, translated into chinese) and 183 chinese utterances (∼ 9.8k characters, translated into english). we use this dataset to measure our intrinsic evaluation metrics. ## conclusion and future work we describe meetdot, a videoconferencing system with live translation captions, along with its components: ui, asr, mt, and captioning. we implement an evaluation suite that allows us to accurately compute metrics from the sequence of captions that users would see. we also describe a cross-lingual word game for a/b testing different captioning algorithms and conditions. our future work includes improved asr/mt, extrinsic testing, and an open source release. our overall goal is to provide a platform for developing translation captions that are accurate and "right behind you".