Zero and Few-Shot Localization of Task-Oriented Dialogue Agents with a Distilled Representation (2023)

Abstract: Task-oriented Dialogue (ToD) agents are mostly limited to a few widely-spoken languages, mainly due to the high cost of acquiring training data for each language. Existing low-cost approaches that rely on cross-lingual embeddings or naive machine translation sacrifice a lot of accuracy for data efficiency, and largely fail in creating a usable dialogue agent. We propose automatic methods that use ToD training data in a source language to build a high-quality functioning dialogue agent in another target language that has no training data (i.e. zero-shot) or a small training set (i.e. few-shot). Unlike most prior work in cross-lingual ToD that only focuses on Dialogue State Tracking (DST), we build an end-to-end agent. We show that our approach closes the accuracy gap between few-shot and existing full-shot methods for ToD agents. We achieve this by (1) improving the dialogue data representation, (2) improving entity-aware machine translation, and (3) automatic filtering of noisy translations. We evaluate our approach on the recent bilingual dialogue dataset BiToD.In Chinese to English transfer, in the zero-shot setting, our method achieves 46.7% and 22.0% in Task Success Rate (TSR) and Dialogue Success Rate (DSR) respectively. In the few-shot setting where 10% of the data in the target language is used, we improve the state-of-the-art by 15.2% and 14.0%, coming within 5% of full-shot training.

## introduction while dialogue agents in various forms have become commonplace in parts of the world, their lack of support for most human languages has prevented access to the benefits they provide for much of the world. commercial virtual assistants for example, only support a handful of languages, as extending their functionality to each new language is extremely costly, partially due to the need for collecting new annotated training data in that language. in recent years, several non-english taskoriented dialogue (tod) datasets have been created; they are either collected from scratch such as ri-sawoz @xcite and crosswoz @xcite , paraphrased from synthetic sentences by crowdworkers such as bitod @xcite , or manually translated from another language @xcite . all of these approaches are laborintensive, expensive, and time-consuming; such investment is unlikely to be made for less widely spoken languages. cross-lingual transfer, i.e. using training data from other languages to build a dialogue agent for a specific language, seems especially appealing. an emerging line of work has employed machine translation of training data, and multilingual pre-trained neural networks to tackle this task @xcite @xcite . however, work in tod cross-lingual transfer has for the most part, focused on understanding the user input, namely dialogue state tracking (dst) and natural language understanding (nlu). other necessary parts of a dialogue agent like policy and response generation have mostly remained unexplored. in this paper, we present a methodology for building a fully functional dialogue agent for a new language (e.g. english), by using training data in another language (e.g. chinese) with little to no additional manual dataset creation effort. we found that despite prior efforts to improve modeling for existing tod datasets, the dialogue representation used as input to these models, e.g. full dialogue history in natural language @xcite , is sub-optimal, especially when the training data is either scarce or created automatically us-ing noisy machine translation. we propose a new distilled representation to fix the shortcomings of current representations. we also found that previously proposed entity-aware translation technique @xcite to be inadequate. our proposed technique effectively combines entity-aware neural machine translation with text similarity classifiers to automatically create training data for a new language. this paper explains all the ingredients we found useful, and motivates their use by performing extensive ablation studies. the contributions of this paper are: 1. a new state-of-the-art result for the bitod dataset in both few-shot and full-shot settings on english according to all of our 6 automatic metrics, including an improvement of 14.0% and 2.9%, respectively, in dialogue success rate (dsr). in fact, using our distilled representation, our few-shot model trained on only 10% of the training data, achieves similar results to the previous sota model trained on 100% training data. 2. the first dialogue agent created in the zero-shot cross-lingual transfer setting, i.e. starting from no training data in the target language. our agent achieves 71%, 62%, 40%, and 47% of the performance of a full-shot agent in terms of joint goal accuracy (jga), task success rate (tsr), dsr, and bleu score, respectively. 3. a concise dialogue representation designed for cross-lingual tod agents. the distilled dialogue representation works well with our new decomposition of agent subtasks, making significant improvements possible. 4. an improved methodology for automatic translation of tod training data. we adapt and improve an existing entity-aware machine translation system that localizes entities @xcite , extend it to agent response generation, and equip it with a filtering step that increases the quality of the resulting translations. ## distilled tod agent our methodology includes a dialogue task decomposition and a distilled dialogue representation that are tailored to cross-lingual tod agents. in this section we describe these two components. we follow the end-to-end task-oriented dialogue (tod) setting @xcite where a user converses freely with an agent over several turns to accomplish his/her goal with all of its constraints (e.g. "book a restaurant that is rated at least 3."). in each turn, the agent must access its database if needed to find the requested information (e.g. find a restaurant that satisfies user constraints), decide on an action (e.g. to present the information to the user or to ask follow-up questions) and finally respond to the user in natural language based on the action it selects. ## automatic dialogue data translation given a training dataset for one language, we automatically generate a training set in the target language we are interested in. this problem has been studied in the context of nlu for questions @xcite @xcite and for dialogues @xcite @xcite . one challenge is that the translated dataset should refer to entities in the target language. thus, @xcite proposed to first use cross-attention weights of the neural translation model to align entities in the original and translated sentences, then replace entities in the translated sentences with local entities from a target language knowledge base. our initial experiments showed that applying this approach directly to end-to-end dialogue datasets does not yield good performance, especially for response generation. thus, we adapted and improved this approach for dialogues as discussed below. ## results and discussion we first show how our distilled representation affects the performance of an agent in a full-shot setting. we then evaluate our proposed techniques on cross-lingual settings with varying amounts of available training data. ## conclusion this paper shows how to build a dialogue agent in a new language automatically, given a dialogue dataset in another language, by using entity-aware machine translation and our new distilled dialogue representation. the performance can be further improved if a few training examples in the target language are available, and we show that our approach outperforms existing ones in this setting as well. on the bitod dataset, our method achieves 3.9% and 2.9% improvement in tsr and dsr, respectively, over the previous sota in full-shot setting, and 15.2% and 14.0% in a 10% few-shot setting, showing the effectiveness of our approach. more importantly, training on translated data and only 10% of original training data comes within 5% of full training. we have implemented our methodology as a toolkit for developing multilingual dialogue agents, which we have released open-source. our proposed methodology can significantly reduce the cost and time associated with data acquisition for task-oriented dialogue agents in new languages.