Improving Zero-Shot Multilingual Text Generation via Iterative Distillation (2022)

Abstract: The demand for multilingual dialogue systems often requires a costly labeling process, where human translators derive utterances in low resource languages from resource rich language annotation. To this end, we explore leveraging the inductive biases for target languages learned by numerous pretrained teacher models by transferring them to student models via sequence-level knowledge distillation. By assuming no target language text, the both the teacher and student models need to learn from the target distribution in a few/zero-shot manner. On the MultiATIS++ benchmark, we explore the effectiveness of our proposed technique to derive the multilingual text for 6 languages, using only the monolingual English data and the pretrained models. We show that training on the synthetic multilingual generation outputs yields close performance to training on human annotations in both slot F1 and intent accuracy; the synthetic text also scores high in naturalness and correctness based on human evaluation.

## introduction in multilingual dialogue systems, natural language generation is used to generate utterances in various languages, using as input semantic frames, which contain a representation of the user intent together with relevant information or entities related to said intent @xcite . past works that generalize dialogue systems to multilingual settings often made two unrealistic assumptions about the data availability of any new dialogue domain @xcite @xcite @xcite : (1) first assumption is that a large set of monolingual data has already been annotated. (2) some in-domain text or annotated data in the target languages are already available for the purpose of transfer learning. neither assumption holds in all cases @xcite . to overcome these challenges, we utilize knowledge-grounded pre-training (kgpt) (chen et al., 2020) -a pretrained data-to-text generation model that was shown to be effective in overcoming data scarcity, as the model is capable of representing the inductive biases required to encode structured data such as the slot-value pairs (frames). in few-shot settings, we can exploit kgpt's pretrained knowledge to obtain in-domain text labels for a large set of unlabeled frame sequences. these text labels are then converted to multiple languages with the use of bilingual translation models as teacher models, as inspired by past works on sequence-level knowledge distillation @xcite @xcite . in this way, we perform zero-shot cross-lingual transfer for all 6 languages. we use mbart @xcite as the multilingual student model to acquire stronger bilingual knowledge from the translation teacher models from tiedemann and thottingal (2020). we leverage a two-step distillation process where we first derive a large synthetic english dialogue data from the english seed data, then generalize it to multilingual data by using the bilingual translation models to produce synthetic text labels. finally, we perform rounds of iterative knowledge distillation following the process of the expectationmaximization algorithm for further improvements. this work makes the following contributions: ## approach summary in our setting, we have (1) a seed english dataset s which consists of k labeled pairs, and (2) the full set of unlabeled frame sequences u where |u | ≫ @xmath0. the goal is to create labeled samples in all target languages consisting of the frame sequences (x ) with their corresponding texts (y).