Code-switched inspired losses for spoken dialog representations (2021)

Abstract: Spoken dialogue systems need to be able to handle both multiple languages and multilinguality inside a conversation ( e.g in case of code-switching). In this work, we introduce new pretraining losses tailored to learn generic multilingual spoken dialogue representations. The goal of these losses is to expose the model to code-switched language. In order to scale up training, we automatically build a pretraining corpus composed of multilingual conversations in five different languages (French, Italian, English, German and Spanish) from OpenSubtitles, a huge multilingual corpus composed of 24.3G tokens. We test the generic representations on MIAM, a new benchmark composed of five dialogue act corpora on the same aforementioned languages as well as on two novel multilingual tasks ( i.e multilingual mask utterance retrieval and multilingual inconsistency identification). Our experiments show that our new losses achieve a better performance in both monolingual and multilingual settings.

## introduction a crucial step in conversational ai is the identification of underlying information of the user's utterance (e.g communicative intent or dialog acts, and emotions). this requires modeling utterance-level information @xcite , to capture immediate nuances of the user utterance; and discourse-level features @xcite , to capture patterns over long ranges of the conversation. an added difficulty to this modeling problem is that most people in the world are bilingual @xcite : therefore, progress on these systems is limited by their inability to process more than one language (english being the most frequent). for example, many people use english as a "workplace" language but seamlessly * stands for equal contribution switch to their native language when the conditions are favorable @xcite . thus, there is a growing need for understanding dialogs in a multilingual fashion @xcite @xcite . additionally, when speakers share more than one language, they inevitably will engage in code-switching @xcite @xcite @xcite : switching between two different languages. thus, spoken dialog systems need to be cross lingual (i.e able to handle different languages) but also need to model multilinguality inside a conversation @xcite . in this paper, we focus on building generic representations for dialog systems that satisfy the aforementioned requirements. generic representations have led to strong improvements on numerous natural language understanding tasks, and can be finetuned when only small labelled datasets are available for the desired downstream task @xcite @xcite @xcite . while there has been a growing interest in pretraining for dialog @xcite , the focus has mainly been on english datasets. thus, these works can not be directly applied to our multilingual setting. additionally, available multilingual pretraining objectives @xcite @xcite face two main limitations when applied to dialog modeling: (1) they are a generalization of monolingual objectives that use flat input text, whereas hierarchy has been shown to be a powerful prior for dialog modeling. this is a reflection of a dialog itself, for example, context plays an essential role in the labeling of dialog acts. (2) the pretraining objectives are applied separately to each language considered, which does not expose the (possible) multilinguality inside a conversation (as it is the case for code-switching) @xcite 1 . our main contributions are as follows: 1. we introduce a set of code-switched inspired losses as well as a new method to automatically obtain several million of conversations with multilingual input context in different languages. there has been limited work on proposing corpora with a sufficient amount of conversations that have multilingual input context. most of this work focuses on social media, or on corpora of limited size. hence, to test our new losses and scale up our pretraining, we automatically build a pretraining corpus of multilingual conversations, each of which comprises several languages, by leveraging the alignments available in opensubtitles (ops). 2. we showcase the relevance of the aforementioned losses and demonstrate that it leads to better performances on downstream tasks, that involve both monolingual conversations and multilingual input conversations. for monolingual evaluation, we introduce the multilingual dialogact benchmark (miam): composed of five datasets in five different languages annotated with dialog acts. following @xcite ; @xcite , we complete this task with both contextual inconsistency detection and next utterance retrieval in these five languages. for multilingual evaluation, due to the lack of code-switching corpora for spoken dialog, we create two new tasks: contextual inconsistency detection and next utterance retrieval with multilingual input context. the datasets used for these tasks are unseen during training and automatically built from ops. in this work, we follow the recent trend @xcite in the nlp community that aims at using models of limited size that can both be pretrained with limited computational power and achieve good performance on multiple downstream tasks. the languages we choose to work on are english, spanish, german, french and italian. 2 . miam is available in datasets @xcite https://huggingface.co/datasets/miam . 1 we refer to code-switching at the utterance level, although it is more commonly studied at the word or span level @xcite @xcite 2 although our pretraining can be easily generalised to 62 languages, we use a limited number of languages to avoid exposure to the so-called "curse of multilinguality" @xcite 2 model and training objectives notations we start by introducing the notations. we have a set d of contexts (i.e truncated conversations), i.e., @xmath0, c 2 , . . . , c |d| ). each context c i is composed of utterances u, i.e for da classification y i is the unique dialog act tag associated to u i . in our setting, we work with a shared vocabulary v thus ω i @xmath1. ## evaluation framework this section presents our evaluation protocol. it involves two different types of evaluation depending on the input context. the first group of experiences consists in multilingual evaluations with monolingual input context and follows classical downstream tasks (finch and choi, 2020; @xcite including sequence labeling @xcite , utterance retrieval @xcite or inconsistency detection. the second group focuses on multilingual evaluations with multilingual context. ## numerical results in this section, we empirically demonstrate the effectiveness of our code-switched inspired pretraining on downstream tasks involving both monolingual and multilingual input context. ## conclusions in this work, we demonstrate that the new codeswitched inspired losses help to learn representations for both monolingual and multilingual dialogs. this work is the first that explicitly includes code switching during pretraining to learn multilingual spoken dialog representations. in the future, we plan to further work on ops to obtain fine-grained alignments (e.g at the span and word levels) and enrich the definition of code-switching