Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking (2021)

Abstract: Recent progress in task-oriented neural dialogue systems is largely focused on a handful of languages, as annotation of training data is tedious and expensive. Machine translation has been used to make systems multilingual, but this can introduce a pipeline of errors. Another promising solution is using cross-lingual transfer learning through pretrained multilingual models. Existing methods train multilingual models with additional code-mixed task data or refine the cross-lingual representations through parallel ontologies. In this work, we enhance the transfer learning process by intermediate fine-tuning of pretrained multilingual models, where the multilingual models are fine-tuned with different but related data and/or tasks. Specifically, we use parallel and conversational movie subtitles datasets to design cross-lingual intermediate tasks suitable for downstream dialogue tasks. We use only 200K lines of parallel data for intermediate fine-tuning which is already available for 1782 language pairs. We test our approach on the cross-lingual dialogue state tracking task for the parallel MultiWoZ (English -> Chinese, Chinese -> English) and Multilingual WoZ (English -> German, English -> Italian) datasets. We achieve impressive improvements (> 20% on joint goal accuracy) on the parallel MultiWoZ dataset and the Multilingual WoZ dataset over the vanilla baseline with only 10% of the target language task data and zero-shot setup respectively.

## introduction in recent years, task-oriented dialogue systems have achieved remarkable success by leveraging huge amounts of labelled data. this technology is thus limited to a handful of languages as collecting and annotating training dialogue data for different languages is expensive and requires supervision from native speakers @xcite . to avoid having to create large annotated datasets for every new language, recent works focus on transfer learning methods which use neural machine translation systems @xcite , code-mixed data augmentation @xcite or large multilingual models @xcite . neural machine translation models incur additional overhead of training on millions of parallel sentences that may not be available for all language pairs. code-mixed data augmentation methods involve replacing individual words from the source language with the target language by using parallel word pairs found in a dictionary. however, a simple synonym replacement may not be sufficient as the tasks become complicated. in this paper, we focus on transfer learning via large multilingual models, which will allow us to extend models to languages with limited labelled training data. in techniques that use multilingual models, a task-specific architecture uses this pretrained model as one of its components and then is trained with task data from a high resource language (see fig. 1 ). it is then evaluated directly or with some labelled examples in a different language. the use of intermediate fine-tuning, which is fine-tuning a large language model with a different but related data/or task and then fine-tuning it for the target task has shown considerable improvements for both monolingual and cross-lingual natural language understanding tasks @xcite . but, it is relatively under-explored for multilingual dialogue systems. in this work, we demonstrate the effectiveness of using cross-lingual intermediate fine-tuning of multilingual pretrained models to facilitate the development of multilingual conversation systems. specifically, we look at cross-lingual dialogue state tracking tasks, as they are an indispensable part of task-oriented dialogue systems. in this task, a model needs to map the user's goals and intents in a given conversation to a set of slots and values - ## intermediate fine-tuning for dialogue tasks in this section, we will provide details about the training data used for different intermediate tasks, explain existing and proposed intermediate tasks, and detail their integration into the end task. ## experiments we experiment with the recently released parallel multiwoz dataset @xcite and the multilingual woz dataset @xcite . as the datasets vary in difficulty and languages, we choose a different amount of target training data and dialogue state tracking architectures for both of them. we briefly provide their description and discuss the results obtained with our methods. ## analysis we analyse the outputs from the state tracker and design choices for the intermediate tasks. we also provide insights into the difficulty of conducting zero-shot transfer learning using the sumbt architecture for the multiwoz dataset. ## conclusion we demonstrated the effectiveness of cross-lingual intermediate fine-tuning of pretrained multilingual language models for the task of cross-lingual dialogue state tracking. we experimented with existing intermediate tasks and introduced two new cross-lingual intermediate tasks based on the parallel and dialogue-level nature of the movie subtitles corpus. our best method had significant improvement in performance for the parallel multiwoz dataset and multilingual woz dataset. we also demonstrated the data efficiency of our methods. our intermediate tasks were trained on a generic dataset unlike the related high resource tasks used in @xcite . as opensubtitles is available for 1782 language pairs, we speculate that using these cross-lingual intermediate tasks will be effective for languages where a collection of large training datasets for dialogue tasks is not feasible. we speculate that this setup can be useful for crosslingual domain transfer too -when such benchmark becomes available for dialogue tasks. we hope that our method can serve as a strong baseline for future work in multilingual dialogue.