LINGUIST : Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging (2022)

Abstract: We present LINGUIST, a method for generating annotated data for Intent Classification and Slot Tagging (IC+ST), via fine-tuning AlexaTM 5B, a 5-billion-parameter multilingual sequence-to-sequence (seq2seq) model, on a flexible instruction prompt. In a 10-shot novel intent setting for the SNIPS dataset, LINGUIST surpasses state-of-the-art approaches (Back-Translation and Example Extrapolation) by a wide margin, showing absolute improvement for the target intents of +1.9 points on IC Recall and +2.5 points on ST F1 Score. In the zero-shot cross-lingual setting of the mATIS++ dataset, LINGUIST out-performs a strong baseline of Machine Translation with Slot Alignment by +4.14 points absolute on ST F1 Score across 6 languages, while matching performance on IC. Finally, we verify our results on an internal large-scale multilingual dataset for conversational agent IC+ST and show significant improvements over a baseline which uses Back-Translation, Paraphrasing and Slot Catalog Resampling. To our knowledge, we are the first to demonstrate instruction fine-tuning of a large-scale seq2seq model to control the outputs of multilingual intent- and slot-labeled data generation.

## introduction conversational agents typically rely on large quantities of labeled training data to understand user requests through intent classification and slot tagging (ic+st) @xcite . such data is plentiful for existing usage patterns (although costly to annotate), yet scarce for new intents/slots and new languages. a growing trend to address this problem is to generate synthetic training data, e.g. via paraphrasing, back-translation (bt), slot replacement, and example extrapolation (ex2). @xcite 2020; input: <@xmath0, [3 snow ] [5 tomorrow ] </@xmath1, [2=country , [3=condition_description , [4=city , [5=@xmath2, 2022 ] in [4 gyeongju ] </@xmath3: 1. give the [1 national wildlife refuge ] forecast for [3 snow ] [5 tomorrow ] 2. can i get the [3 snow ] forecast for [1 lake tahoe ] [5 tomorrow ] ? 3. i want to know if it will [3 snow ] [5 tomorrow ] at [1 mount rainier state park ] . figure 1 : linguist uses an instruction prompt to generate data with both user-requested slot values ("snow") and model-generated values ("*"). this model has not seen any training data for getweather intent, or for the slot tag geographic_poi: it was fine-tuned only on the other 6 snips intents. @xcite . in this work, we propose a novel data generation method called language model instruction tuning to generate annotated utterances for intent classification and slot tagging (linguist). our linguist method addresses several important gaps in the existing literature: (1) controlling the generated data to include specific slot types and values, (2) cross-lingual and multilingual data generation, and (3) ability to leverage the intent and slot names to inform the generation. the key to these achievements is our design of a novel instruction prompt (figure 1 ), consisting of natural language descriptions for the desired model outputs. we first fine-tune a large pre-trained seq2seq transformer @xcite model to learn how to generate annotated utterances following the prompt instructions. then, for a novel intent or slot with only a few or even zero training examples, we apply the model to generate similar data, which we add to the training set for an ic+st model. we demonstrate the effectiveness of linguist on three datasets by showing substantial improvements over strong baselines. (i) on a 10-shot novel intent setting with english snips @xcite , linguist improves over back-translation and ex2 by +1.9 points absolute on ic and +2.5 points absolute on st. (ii) on cross-lingual ma-tis++ @xcite , linguist out-performs the best machine translation plus slot alignment reported by @xcite points in st across 6 languages, while matching performance on ic. (iii) finally, to demonstrate the success of our method on a real-world conversational agent system, we apply linguist on an internal dataset containing hundreds of intents and slot types across 4 languages, and show large improvements over a baseline which uses back-translation and paraphrasing. we also show linguist can generate ic+stannotated data from zero examples, using only the natural language intent and slot label names. linguist achieves 80.0 ic recall and 56.9 st f1 score on new snips intents despite never seeing a single example for the new intents. to our knowledge, linguist is the first system capable of generating ic+st-annotated data in this setting. ## related work large-scale language models (llms) such as gpt @xcite and alexatm 20b @xcite excel at performing novel tasks with only a few examples via in-context learning, i.e. without requiring any model parameter updates. for example, @xcite apply gpt-3 to generate variations of examples from a given class. @xcite generate both text and class label together via gpt-3, towards eliminating the need for human labeling. pre-training then fine-tuning of seq2seq models was introduced in english bart and multilingual mbart @xcite . t5 and mt5 @xcite extended the idea by framing more downstream tasks as text-to-text. flan @xcite introduced instruction tuning, where a large-scale seq2seq model is fine-tuned on instruction prompts from a variety of tasks, in order to generalize to new tasks without any further parameter updates. prior work also explores conditioning generation on intent and slot labels. @xcite train a conditional language model on a mixture of annotated and unannotated text, allowing to sample novel annotated utterances. @xcite train a seq2seq model from interpretationtext pairs, applying variational auto-encoders for more diversity. @xcite expand on this, exploring different sampling strategies, adding more variety by shuffling slot names, and examining the behavior where a new intent is introduced with limited training data. @xcite extend this to the multilingual setting. generative insertion transformers @xcite generate carrier phrases for a target intent and containing specific entities. a limitation of all these approaches is that the trained model cannot generalize to novel intents and slots at inference time. generative conversational networks (papangelis et al., 2021) are trained via reinforcement learning to generate annotated data from seed examples, studying english ic+st and other tasks. the closest relative of linguist is example extrapoloation (ex2) @xcite , which generates annotated ic+st data using a seq2seq model and provided seed examples. we compare lin-guist and ex2 in more detail in section 3.3. a widely used paraphrasing method is back-translation (bt), i.e. translating text from one language into another "pivot" language, then back again. @xcite extract paraphrases directly from parallel corpora. sennrich et al. (2016) and edunov et al. (2018) use bt for machine translation and xie et al. (2020) for data augmentation on classification tasks. other approaches directly target the paraphrasing task: prakash et al. (2016) learns an lstm model by supervised training on a paraphrase corpus, whereas kumar et al. (2020) use an unsupervised denoising task, in both cases only using text and not covering slot labels. cho et al. (2019) explore paraphrasing via semi-supervised learning. a different approach to data augmentation is token replacement: seqmix (zhang et al., 2020) replaces tokens with the nearest neighbor in the embedding space, dai and adel (2020) replaces slots with synonyms, or mentions from other instances of the same label. easy data augmentation (wei and zou, 2019) includes synonym replacement, random insertion, random swap, and random deletion for classification task. zheng et al. (2021) benchmark the success of llms on few-shot settings. 3 linguist data generator model linguist provides three key innovations compared to prior data generation work: (1) controls for slot types and values (either user-supplied or model-generated) to include in the outputs; (2) multilingual and cross-lingual generation; and ## experimental setup this section describes the datasets, tasks, ic+st model, baseline gata generation methods, and metrics that we use to evaluate linguist. ## conclusion and future work we introduced linguist, a novel method for annotated data generation, via fine-tuning a largescale pre-trained multilingual seq2seq model. our method generalizes to new intents and slots in challenging few-shot, zero-shot, and cross-lingual settings, which we have shown on three datasets. in future work, we wish to explore ways to improve the generation output, e.g. human-in-theloop and reinforcement learning. we would also like to include more controls in the prompt such as text style, and explore generation for multi-turn dialogues, and more complex and nested semantics.