Multilingual Paraphrase Generation For Bootstrapping New Features in Task-Oriented Dialog Systems (2021)

Abstract: The lack of labeled training data for new features is a common problem in rapidly changing real-world dialog systems. As a solution, we propose a multilingual paraphrase generation model that can be used to generate novel utterances for a target feature and target language. The generated utterances can be used to augment existing training data to improve intent classification and slot labeling models. We evaluate the quality of generated utterances using intrinsic evaluation metrics and by conducting downstream evaluation experiments with English as the source language and nine different target languages. Our method shows promise across languages, even in a zero-shot setting where no seed data is available.

## introduction spoken language understanding is a core problem in task oriented dialog systems with the goal of understanding and formalizing the intent expressed by an utterance @xcite . it is often modeled as intent classification (ic), an utterance-level multi-class classification problem, and slot labeling (sl), a sequence labeling problem over the utterance's tokens. in recent years, approaches that train joint models for both tasks and that leverage powerful pre-trained neural models greatly improved the state-of-the-art performance on available benchmarks for ic and sl @xcite . a common challenge in real-world systems is the problem of feature bootstrapping: if a new feature should be supported, the label space needs to be extended with new intent or slot labels, and the model needs to be retrained to learn to classify corresponding utterances. however, labeled examples for the new feature are typically limited to a small set of seed examples, as the collection of more annotations would make feature expansion costly and slow. as a possible solution, previous work explored the automatic generation of paraphrases to augment the seed data @xcite @xcite . in this work, we study feature bootstrapping in the case of a multilingual dialog system. many large-scale real-world dialog systems, e.g. apple's siri, amazon's alexa and google's assistant, support interactions in multiple languages. in such systems, the coverage of languages and the range of features is continuously expanded. that can lead to differences in the supported intent and slot labels across languages, in particular if a new language is added later or if new features are not rolled out to all languages simultaneously. as a consequence, labeled data for a feature can be available in one language, but limited or completely absent in another. with multilingual paraphrase generation, we can benefit from this setup and improve data augmentation for data-scarce languages via cross-lingual transfer from data-rich languages. as a result, the data augmentation can not only be applied with seed data, i.e. in a few-shot setting, but even under zero-shot conditions with no seeds at all for the target language. to address this setup, we follow the recent work of @xcite , which proposes to use an encoder-decoder model that maps from structured meaning representations to corresponding utterances. because such an input is language-agnostic, it is particularly well-suited for the multilingual setup. we make the following extensions: first, we port their model to a transformer-based architecture and allow multilingual training by adding the desired target language as a new input to the conditional generation. second, we let the model generate slot labels along with tokens to alleviate the need for additional slot projection techniques. and third, we introduce improved paraphrase decoding methods that leverage a model-based selec-tion strategy. with that, we are able to generate labeled data for a new feature even in the zero-shot setting where no seeds are available at all. we evaluate our approach by simulating a crosslingual feature bootstrapping setting, either fewshot or zero-shot, on multiatis, a common ic/sl benchmark spanning nine languages. the experiments compare against several alternative methods, including previous work for mono-lingual paraphrase generation and machine translation. we find that our method produces paraphrases of high novelty and diversity and using it for ic/sl training shows promising downstream classification performance. ## related work various studies have explored paraphrase generation for dialog systems. @xcite showed that generating sentences from a continuous latent space is possible using a variational autoencoder model and provided guidelines on how to train such a generation model. however, our model uses an encoder-decoder approach which can handle the intent and language as categorical inputs in addition to the sequence input. @xcite explored a variety of controlled paraphrase generation approaches for data augmentation and proposed to use conditional variational autoencoders which they showed obtained the best results. our method is different as it uses a conditional seq2seq model that can generate text from any sequence of slots and does not require an utterance as an input. @xcite propose a transformer-based conditional variational autoencoder for few shot utterance generation where the latent space represents the intent as two independent parts (domain and action). our approach is different since it models the language and intent of the generation that can be controlled explicitly. also, our model is the first to enable zero-shot utterance generation. @xcite generate paraphrases for seed examples with a transformer seq2seq model and self-label them with a baseline intent and slot model. we follow a similar approach but our model generates utterances from a sequence of slots rather than an utterance, which enables an explicitly controlled generation. also the number of seed utterances we use is merely 20 for the few shot setup unlike around 1m seed para-carrier phrase pairs in @xcite . several other studies follow a text-to-text ap-proach and assume training data in the form of paraphrase pairs for training paraphrase generation models in a single language @xcite @xcite . our approach is focused towards generating utterances in the dialog domain that can generate utterances from a sequence of slots conditioned on both intent and language. @xcite showed that an interpretationto-text model can be used with shuffling-based sampling techniques to generate diverse and novel paraphrases from small amounts of seed data, that improve accuracy when augmenting to the existing training data. our approach is different as our model can generate the slot annotations along with the the utterance, which are necessary for the slot labeling task. our model can be seen as an extension of the model by @xcite to a transformer based model, with the added functionality of controlling the language in which the utterance generation is needed, which in turn enables zero shot generation. using large pre-trained models has also been shown to be effective for paraphrase generation. @xcite for instance show the effectiveness of using gpt-2 @xcite for generating text from tabular data (a set of attributevalue pairs). our model, however, does not rely on pre-trained weights from another model such as gpt-2, is scalable, and can be applied to training data from any domain, for instance, dialog domain. beyond paraphrase generation, several other techniques have been proposed for feature bootstrapping. machine translation can be used from data-rich to data-scarce languages @xcite . cross-lingual transfer learning can also leverage use existing data in other languages @xcite . if a feature is already being actively used, feedback signals from users, such as paraphrases or interruptions, can be used to obtain additional training data @xcite . ## proposed method we want to augment existing labeled utterances by generating additional novel utterances in a desired target language. in our case, existing data consists of feature-unrelated data (intents and slots already supported) spanning all languages and featurerelated data, which is available in a source language but is small (few-shot) or not available (zero shot) in other languages. for generation, we first extract transformer encoder slot embeddings intent embeddings language embeddings fromloc.city_name toloc.@xmath0.city_name to o montreal b-toloc.city_name the intent and slot types from the available data. we then generate a new utterance by conditioning a multilingual language model on the intent, slot types and the target language. we refer to utterances that have the same intent and slot types as paraphrases of each other since they convey the same meaning in the context of the slu system. ## experimental setup we evaluate our approach by simulating few-shot and zero-shot feature bootstrapping scenarios. ## conclusion in this paper, we proposed a multilingual paraphrase generation model that can be used for feature bootstrapping with or without seed data in the target language. in addition to generating a paraphrase, the model also generates the associated slot labels, enabling the generation to be used directly for data augmentation to existing training data. our method is language agnostic and scalable, with no dependencies on pre-trained models or additional data. we validate our method using experiments on the multiatis++ dataset containing utterances spanning 9 languages. intrinsic evaluation shows that paraphrases generated using our approach have higher novelty and diversity in comparison to cvae seq2seq based paraphrase generation. additionally, downstream evaluation shows that using the generated paraphrases for data augmentation results in improvements over baseline and related techniques in a wide range of languages and setups. to the best of our knowledge, this is the first successful exploration of generating paraphrases for slu in a cross-lingual setup. in the future, we would like to explore strategies to exploit monolingual data in the target languages to further refine the paraphrase generation. we would also like to leverage pre-trained multilingual text-to-text models such as mt5 @xcite for multilingual paraphrase generation in the dialog system domain.