N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking (2022)

Abstract: Augmentation of task-oriented dialogues has followed standard methods used for plain-text such as back-translation, word-level manipulation, and paraphrasing despite its richly annotated structure. In this work, we introduce an augmentation framework that utilizes belief state annotations to match turns from various dialogues and form new synthetic dialogues in a bottom-up manner. Unlike other augmentation strategies, it operates with as few as five examples. Our augmentation strategy yields significant improvements when both adapting a DST model to a new domain, and when adapting a language model to the DST task, on evaluations with TRADE and TOD-BERT models. Further analysis shows that our model performs better on seen values during training, and it is also more robust to unseen values. We conclude that exploiting belief state annotations enhances dialogue augmentation and results in improved models in n-shot training scenarios.

## introduction task-oriented dialogue (tod) agents are the nextgeneration user interface and are slated to replace browsing static websites. however, a key bottleneck in fielding such agents practically concerns adapting to new domains with few available data. in the light of this dependency in ample amounts of annotated data, data augmentation is growing in importance @xcite . most augmentation methods in natural language processing (nlp) target written forms of text -passages, news articles, etc. -which operate with word-or sentencelevel permutations of the original text data, synthesizing new text @xcite @xcite @xcite . these methods do not exploit the structure 1 cnrs@create ltd, 1 create way, #08-01 create tower, singapore 138602 of conversational data in its entirety. we study augmenting task-oriented dialogues, a specific form of conversational data. a tod is a form of conversation where the aim is to accomplish a task through exchanges between a user and an agent, accounting for the user's preferences. within tod, dialogue state tracking (dst) is a fundamental task, which aims to detect these preferences in a given dialogue. for this task, each pair of utterances in a dialogue is annotated with slot-label and slot-value pairs (cf. figure 1 : traindestination: "cambridge") and a belief state. here, a belief state can be equated as an attribute-value store that gives the final values of each slot label (attribute) after an utterance. there have been several attempts to augment conversational data in the literature. @xcite up-sample the data through word or sentence level modifications, following standard text augmentation techniques in nlp such as synonym substitution, back-translation, or paraphrasing. @xcite perturb embeddings of single utterances and decode similarly functioned synthetic utterances. @xcite create an end-to-end pipeline that finds the utterances with similar dialogue functions and trains a paraphrasing model. coco @xcite ) trains a conditional user-utterance generation model, then generates synthetic turns by modifying belief states using a rule-based system and conditioning the model on the modified belief state. @xcite create a working graph of tod datasets where each edge is a dialogue act and create synthetic dialogues by traversing alternative paths; however, their framework requires user acts to work with. critically, none of the above techniques exploit the belief state annotations of tods within an n-shot scenario. in contrast, dialogue belief state annotations guide our approach to an effective n-shot augmentation method. we observe that the belief state identifies the specific slots that each turn-pair discusses. as such, belief states can be used as a proxy to represent dialogue function. for example, after exchanging two turn-pairs that serve the same dialogue function in separate dialogues, coherency in both dialogues should be preserved, if discounting necessary changes to slot values (figure 1 ). motivated by this, we delexicalize and store each turn-pair with their dialogue function to effectively construct new dialogues from scratch. we evaluate our framework with multiwoz, a multi-domain dialogue dataset @xcite . each of its 10,000 dialogues is annotated with its turn belief states, system acts, and turn slots. we experiment using both the previous state-ofthe-art (sota) recurrent trade @xcite model and the transformer-based tod-bert @xcite ) model. our framework significantly increases n-shot performance, both when adapting a dst model to a new domain and when adapting a language model to the dst task. a fine-grained analysis of evaluation results reveals that models finetuned on synthetic data become robust to previously unseen slot values, and recognize seen values better. the latter aspect accounts for the majority of the performance gain. ## method our method leverages a simple hypothesis, visualized in figure 1 : that the function of a pair of turns in a dialogue can be defined by its slots, and its interactions with its previous and next turn-pairs. the example has two turn-pairs: s a from dialogue a and s b from dialogue b. the turn-pair belief states that precede both s a and s b are composed of the same set of slot labels. the same holds for the belief states of turn-pairs following s a and s b . thus s a and s b have the same function in the dialogue. we hypothesize the interchange of these pairs of turns (after changing the values according to the parent dialogue state) maintains a coherent dialogue. our observations on the multiwoz dataset showed that this is true to a large extent for taskoriented dialogues because the belief state history represents the ongoing topic, and slot labels of the next turn give hints about the system acts. our framework implements this hypothesis in three steps. in step 1 ( § 3.1), we create turn-pair templates by delexicalizing each pair (replacing slot values with their respective slot label), then storing each template with the previous, current, and next pair's belief states (cf. figure 2 ). we also mine a dictionary of possible slot labelvalue pairs to be used in filling generated templates. combining these pairs constrained such that two consecutive pairs' dialogue functions do not break coherency. we do this combination in a breadthfirst manner, best visualized as a tree where each node is a turn-pair template, and every string of nodes from root to leaf is a dialogue template (cf. figure 3 ). finally in step 3 ( § 3.3), we create final synthetic dialogues by filling the slot labels in the dialogue templates (cf. figure 4 ) using the mined dictionary. ## conclusion our framework showcases a distinct approach to dialogue augmentation, where, unlike other studies, we apply the modification not on a datum/sample experiments on multiwoz dataset using both the trade and tod-bert models suggest that our framework consistently improves the performance of the base-model it is applied to. this is true both when adapting the model to the dst task from scratch and also when adapting a model pretrained on dst task to a new domain. the performance boost behind our augmentation framework comes mostly from performance increase on seen values during training although it also makes the model more robust to unseen values. showing that our framework consistently improves the few-shot performance over the dst task we believe it can open doors for many other tod tasks in limited data scenarios. we showcase two synthetic dialogues generated with our framework, cf. the second dialogue on the other hand combines slots from domains: train (from two different dialogues) and hotel (from another third dialogue). although both dialogues seem coherent in shape, the latter has a redundancy where the system request the day information after the user already stated it. this is because of a missing annotation where the train-day slot in the belief state of the third turn is missing. these kinds of annotations are unavoidable but negligible because it recaptures a misunderstanding by the agent which is observed in real dialogues frequently.