Towards Speech Dialogue Translation Mediating Speakers of Different Languages (2023)

Abstract: We present a new task, speech dialogue translation mediating speakers of different languages. We construct the SpeechBSD dataset for the task and conduct baseline experiments. Furthermore, we consider context to be an important aspect that needs to be addressed in this task and propose two ways of utilizing context, namely monolingual context and bilingual context. We conduct cascaded speech translation experiments using Whisper and mBART, and show that bilingual context performs better in our settings.

## introduction in this global era, it is becoming increasingly important for people from different countries/regions to interact with each other and have a mutual understanding. recent advancements in machine translation (mt) technologies have enabled us to communicate with people worldwide, especially in text. chat translation or dialogue machine translation @xcite supports such communications, which enables people who use different languages to have cross-language chats. speech translation (st) has also recently shown success (e.g., @xcite , especially in monologue translation (e.g., di @xcite . however, to the best of our knowledge, no study has focused on st of dialogues, which is an important aspect of language usage. in this study, we propose a new task: speech dialogue translation (sdt) aiming to mediate speakers of different languages. we consider bilingual dialogues where several people who speak in different languages talk with each other mediated by an st system. it is important to consider context in sdt because we need to consider context in different languages, which cannot be readily handled by current st systems that mainly focus on one translation direction. figure 1 shows an example of an stmediated dialogue between an english speaker and a japanese speaker. they are discussing some ideas, and the english speaker says, "what do you think about it?" the japanese speaker responds by saying the idea is naive, but without context it can be translated as "i think it's a bit sweet" because "甘い" has two meanings, sweet and naive. by utilizing dialogue context, the meaning of "甘い" becomes clear so that the utterance can be translated properly. for the proposed task, we construct the speechbsd dataset foot_0 based on an existing text dialogue corpus, bsd (bussiness scene dialogue) corpus @xcite . we collect audio of the bsd corpus through crowdsourcing along with speaker attributes. we conduct speech-to-text cascaded st experiments on the dataset. there are two mainstream methods for st, the cascade method @xcite where automatic speech recognition (asr) and mt are chained together, and the end-to-end method @xcite , where translations are directly predicted from speech. recent study @xcite suggests that the two methods are on par. we conduct cascade st experiments using whisper @xcite for asr and mbart @xcite for mt. we consider three settings for translation: without context, with monolingual context, and with bilingual context. the monolingual context is composed in the language the utterance to be translated is spoken, whereas the bilingual context is composed in the original language of the spoken utterances (see examples in figure 1 ). we show that translation with bilingual context performs better compared to the one without context by up to 1.9 bleu points in mt and 1.7 bleu points in cascade st with our settings. we also conduct a manual evaluation focusing on zero anaphora, a grammatical phenomenon where arguments of verbs are omitted when they are apparent from the context in japanese. we show that with bilingual context, the mt models can often predict zero pronouns correctly. ## related work although neural mt has greatly improved over the past few years, the translation of dialogues remains a challenging task because of its characteristics. @xcite summarizes the recent progress of dialogue mt and categorizes its issue into four categories, coherence, consistency, cohesion, and personality. the main approaches to address these problems include document mt (e.g., @xcite , usage of pretrained models (e.g., wang et al., 2020), and auxiliary task learning utilizing speaker information (e.g., @xcite . considering context in st is recently studied for the end-to-end approach @xcite . we point out that although not addressed in this work, considering context for asr is also an active research area (e.g., @xcite . in this work, we focus on the translation of speech dialogue. we use mbart, which performed best in a previous work of chat translation @xcite , and also consider utilizing context. ## speech dialogue translation (sdt) in sdt, there are several speakers who speak different languages with the help of a translation system. in this work, we consider m speak-ers {s m | @xmath0, 2, • • • , m } and 2 languages {l n | @xmath1, 2}. we consider a dialogue with t utterances @xmath2, • • • , u t ), where an utterance is u @xmath3, l n t , x t ). here, s m t is the speaker, l n t is the language spoken, and x t is the speech signal of t-th utterance. let y n t (@xmath4, 2) be text that has the same meaning as x t in language l n . the task of sdt is to generate translation y 2 t from speech signal x t when the source language is l 1 (or translation y 1 t from x t when the source language is l 2 ) for every utterance u t . ## speechbsd dataset we construct the speechbsd dataset to study sdt. it is based on the existing dialogue dataset in text, bsd corpus @xcite . we collect audio of all the sentences in the dataset along with speaker attributes (gender and homeplace) through crowdsourcing. ## considering context for sdt we propose two ways to consider context in sdt: monolingual context and bilingual context. first, for every utterance u t , an asr system is used to obtain transcripts y n t . the monolingual context is composed in the source language of the utterance to be translated. for example, in figure 1 , when translating the third utterance u 3 from japanese to english, as the source language of the utterance is japanese (l 1 ), the context (y 1 1 and y 1 2 ) is also composed in japanese. let the context composed in this way be y @xmath5. for monolingual context experiments, we use two translation models for each translation direction. the training objective of the mt model that translates from l 1 to l 2 is to maximize the follow-ing log likelihood foot_4 : similar objective l 2→1 can be derived when l 2 is the source language and l 1 is the target language. postprocessing is applied to extract y 2 t from the output that contains both y 2 <t and y 2 t . the bilingual context is composed of the original language of the spoken utterances. for example, in figure 1 , when translating the third utterance u 3 from japanese to english, the bilingual context on the source side is y 1 1 and y 2 2 , which involves both languages. the bilingual context on the target side is y 2 1 and y 1 2 . because there is no concept of source or target language in this case, let the source side utterance be y t , source side context be @xmath6, target side utterance be y t , and target side context be @xmath7. the mt model is trained with the following objective: postprocessing is applied to extract y t from the output. we consider constrained context with context size c in practice, which shows the number of previous utterances used for translation in addition to the utterance to be translated. more formal definitions of monolingual, bilingual, and constrained context are provided in appendix c. ## experiments 6.1 automatic speech recognition in sdt, asr has to handle bilingual inputs. we used a multilingual asr model whisper @xcite . the medium model with 12 encoder and decoder layers was used without finetuning. further details are provided in appendix d.1. we evaluated the performance of the speechbsd test set. for english the word error rate was 8.3 %, and for japanese the character error rate was 13.2 %. ## conclusion we presented a new task, sdt aiming to mediate speakers of different languages. we constructed the speechbsd dataset via crowdsourcing. we performed mt experiments utilizing context and showed its effectiveness. in the future, we plan to perform experiments in end-to-end st settings and sdt utilizing speaker attributes.