Controllable Generation of Dialogue Acts for Dialogue Systems via Few-Shot Response Generation and Ranking (2023)

Abstract: Dialogue systems need to produce responses that realize multiple types of dialogue acts (DAs) with high semantic fidelity. In the past, natural language generators (NLGs) for dialogue were trained on large parallel corpora that map from a domain-specific DA and its semantic attributes to an output utterance. Recent work shows that pretrained language models (LLMs) offer new possibilities for controllable NLG using prompt-based learning. Here we develop a novel few-shot overgenerate-and-rank approach that achieves the controlled generation of DAs. We compare eight few-shot prompt styles that include a novel method of generating from textual pseudo-references using a textual style transfer approach. We develop six automatic ranking functions that identify outputs with both the correct DA and high semantic accuracy at generation time. We test our approach on three domains and four LLMs. To our knowledge, this is the first work on NLG for dialogue that automatically ranks outputs using both DA and attribute accuracy. For completeness, we compare our results to fine-tuned few-shot models trained with 5 to 100 instances per DA. Our results show that several prompt settings achieve perfect DA accuracy, and near perfect semantic accuracy (99.81%) and perform better than few-shot fine-tuning.

## introduction dialogue systems need to faithfully produce utterances that realize multiple types of dialogue acts (das), such as providing opinions, making recommendations, or requesting information. in the past, natural language generators (nlgs) for dialogue have been trained on large parallel corpora that map from a domain-specific meaning representation (mr) that specifies the desired da and semantic attributes to an output utterance. the nlg must faithfully generate utterances that realize the style and form of the da, and all of the specified attributes, as shown by the reference utterances in attributes and values (name [call of duty: advanced warfare], rating @xcite , developer [sledgehammer games], esrb [m (for mature)]) give_opinion call of duty: advanced warfare must be one of the best games i've ever played. sledgehammer games always nail their m-rated games. recommend since you seem to love m-rated games developed by sledgehammer games, i wonder if you have tried call of duty: advanced warfare. inform developed by sledgehammer games, call of duty: advanced warfare is targeted at mature audiences and has overall very positive ratings. previous work on semantically-controlled nlg has focused on improving semantic accuracy @xcite @xcite @xcite . however, obviously an nlg for dialogue needs to faithfully realize the da as well as the semantic attributes. however, previous work has neither controlled for nor evaluated da accuracy. we speculate that this is because many nlg training sets, such as e2e, weather, webnlg, wikibio, dart and totto, only include inform das @xcite @xcite @xcite . yet nlg training sets for spoken dialogue include many types of das, e.g. the viggo corpus has 9 das @xcite , the rnnlg corpus provides 13 das @xcite , multi-woz has 34 das @xcite , and topical chat was automatically labelled with 11 das @xcite . we present a few-shot pbl framework that overgenerates and ranks nlg outputs and achieves high accuracy for both semantic attributes and das. we develop high accuracy da classifiers for three domains and use them to define 6 ranking functions that combine estimates of da probability with measures of semantic accuracy. we also compare a combination of prompt formats, prompt sampling methods, and da representations. several prompt templates take the novel approach of treating da control as a textual style transfer (tst) problem @xcite . for completeness, we report results for few-shot fine-tuned models trained with 5 to 100 instances per da. our contributions include: the results demonstrate large performance differences across prompt styles, but show that many prompts achieve perfect da accuracy, and semantic accuracy as high as 99.81% with only 10 examples, while 100-shot per da fine-tuning only achieves 97.7% semantic accuracy, and 80.6% da accuracy. ## related work this paper applies few-shot pbl to the task of controllable generation of das using an overgenerateand-rank nlg framework. the overgenerate-andrank paradigm for nlg has primarily used two methods for ranking: (1) language model probability @xcite ; and (2) ranking functions trained from human feedback @xcite @xcite . we extend this framework by applying it in the context of pbl, by using da probability in ranking, and by comparing many ranking functions, including beyond-bleu and bleu baselines @xcite . we know of only a few previous studies on controllable generation of das in the context of dialogue systems, each of which has only focused on one or two types of das. obviously, tasks like question generation (qg) aim at controllable generation of questions @xcite but research on qg is not focused on trying to control the generation of questions as opposed to other types of das. however, some work has focused on controlling questions in dialogue, e.g. @xcite learned a latent representation of questions from a labelled corpus and then used this as a prompt prefix to control question generation. @xcite finetuned a persona chat model and tested decoding methods that controlled question frequency, but did not guarantee a question on a particular turn. other work has focused on dialogue acts like opinions and recommendations. for example, @xcite curated opinionated utterances from user reviews that had been marked with exclamation points, and then used the exclamation points as a way to control the production of exaggerated opinions. @xcite used token supervision to control the production of recommendation as opposed to inform dialogue acts where recommendation das stated that a particular restaurant was the best and then justified the recommendation with attributes from the mr. @xcite used pbl with similar prompts to control the expression of big 5 personality types @xcite , rather than dialogue acts. it is well known that data-to-text nlgs based on fine-tuned llms are prone to semantic errors @xcite , thus previous work has focused on methods for ensuring semantic correctness. this includes automatically augmenting the training data @xcite , modifying the input representation @xcite , using rankers or classifiers or decoding methods that identify semantically accurate or acceptable candidates @xcite @xcite @xcite . previous work on few-shot pbl for semantically-controlled nlg has not attempted to control da accuracy @xcite , and has not used an overgenerate and rank approach, resulting in lower semantic accuracies than we report here. much previous work on few-shot nlg has investigated few-shot finetuning rather than few-shot pbl. previous work on the viggo, tv and laptop corpora @xcite @xcite supports direct comparison to our work, but is not few-shot, does not rank outputs or use pbl. few-shotwoz trains a model called sc-gpt on a 400k data-to-text corpus, and then tests transfer learning with only 40 or 50 fine-tuning examples @xcite . other recent work develops methods for augmenting fewshotwoz using synthetic data or by self-training and shows improvements in semantic accuracy and bleu score. the few-shotwoz corpus includes many types of das but none of this previous work includes an evaluation of nlg da accuracy. previous work on few-shot finetuning in the weather domain used 300 examples in fine tuning, and also explored different ways of textualizing the mr @xcite , but did not attempt to control das, develop ranking functions, evaluate da accuracy, or use instructions such as our novel definitional prompts and the templates for tst tasks. @xcite achieve an 85% reconstruction accuracy, while our best prompt/llm combinations achieve 99.44% perf score for viggo, 99.57% perf for tv and 99.47% perf for laptop, a similar metric to reconstruction accuracy, with only 10 examples. ## automatically ranking nlg outputs we start by providing a mathematical formulation of our problem. when generating from a da representation, a high-quality response should: (1) manifest the specified da; (2) have no missing or incorrect mentions of the attributes; (3) hallucinate no additional attributes; and (4) be fluent. thus the generated utterance y, conditioned on an input x composed of da d and attribute values a, can be formulated as @xmath0, a). the conditional likelihood of y given the mr can then be decomposed using bayes rule into the product of three probabilities: p(y|d, a) = p(d|y, a) * p(a|y) * p(y) the term p(d|y, a) is the da probability given the generated utterance y and the semantic attributes a. the term p(a|y) represents the semantic accuracy. the term p(y) is the unconditional probability of the generated utterance, which is commonly used as a measure of fluency. below, we show how we compute estimates of these terms at generation time, and then explain their use in the ranking functions. dialogue act classifier. the term p(d|y, a) requires highly accurate da classifiers to use in automatic ranking. we fine-tuned two classifiers using pre-trained bert-base-uncased on huggingface. we discovered that even though the viggo, laptop and tv training corpora are good size @xcite , producing high accuracy classifiers required us to modify the training data. 1 we originally trained the viggo classifer with the original viggo training set, when we applied this classifier to the generated outputs, we noticed many cases of low confidence classification. a qualitative analysis of the data showed that many generated outputs did not actually fit into the original viggo ontology, which is not surprising, given that the training data for an llm would have included many different types of das. to increase the viggo classifier performance, we introduced an "other" class of dialogue acts, doubly annotated another 1000 viggo nlg outputs by hand, and added them to the original training set. final results are shown in the second classifier was trained using the complete rnnlg corpus with all 4 domains to maximize classifier domain transfer. when we tested it on the rnnlg test set, we discovered that several classes had low f1. examination of the confusion matrix showed that the recommend and inform das were highly confusable, so we created a new type of da we call "describe" by combining their training sets. the final results for for the rnnlg classifiers is shown in we provide these da classifiers along with additional human-labelled model outputs so that other researchers can duplicate our setup. 2 the resulting classifiers achieve average f1s over .97 for all three domains. ## experimental overview figure 1 provides an overview of the experimental architecture. given a set of da representations for a domain, we sample prompt examples from the original training sets while varying the number of samples. we then textualize the da representations in the sample to look more similar to the llms free-text training data. the samples are then fed through the 8 prompt formats in prompt formats. llms are typically trained on far more monologic data than dialogue, and will have rarely, if at all, seen examples of data-to-text nlg @xcite @xcite . while there are llms trained on dialogue such as dialogpt @xcite , and semantically-controlled dialogue data such as kgpt @xcite , and sc-gpt @xcite , there are clear benefits to using a general llm. previous work also shows that without specific dialogic data, many llms do well on nlg for dialogue @xcite . here, we test the hypothesis that performance can be improved by using prompt formats that make the data-to-text task look more like the llm's textual training data. ## results experimental roadmap. we first experiment with viggo over all the experimental settings from section 4 using jurassic-1 jumbo, a 175b autoregressive transformer-based llm with a different depth-width tradeoff than gpt3 @xcite . all experiments set top @xmath2, and @xmath3.7 based on pilot experiments. we compare prompting to few-shot fine-tuning using 5, 25, 50 and 100 examples per da sampled from the training data. we test the 8 prompt formats in we then test the best settings from viggo on the laptop and tv corpora @xcite with results in ## conclusion and future work here we apply an overgenerate-and-rank nlg approach and and provide the first experiments using automatic ranking functions that optimize both da and semantic accuracy in few-shot promptbased nlg. we test and compare a combination of prompt formats, sampling methods, and da representations. we test prompts used for textual style transfer (tst) by treating das as styles to be controlled. we also create novel prompts that provide definitions of das, for completeness, we fine-tune few-shot models and compare them with the fewshot results. the results show that several prompting styles achieve perfect da accuracy, and that few-shot methods can achieve semantic accuracy as high as 99.81% with the right ranking function, while 100-shot fine-tuning achieves 97.7%, and performs much worse on da accuracy (80.6%). our contributions include systematic experimentation with different ways of textualizing mrs, providing instructions to the llm, and ranking outputs. our results also show that formulating the data-to-text task as textual style transfer using pseudo-references yields the highest performance. we achieve sota semantic accuracy with only 10 prompt examples with our best prompt styles, and achieve the surprising results that a ranking function that prioritizes da correctness results in higher semantic accuracy. limitations and risks one limitation arises from the challenges of prompt-engineering: it is impossible to tell whether another prompt format could perform better, e.g. with smaller llms like gpt-neo, where we get poor comparative results. another limitation is the need for a high-accuracy da classifier that works well on out-of-domain model outputs. we address this limitation by releasing our classifiers. another possible limitation is the use of the overgenerate and rank approach in real-time. in future work we plan to use the high quality (ranked) generated data, to fine-tune a smaller real-time language model, without the need for overgeneration. another limitation arises from the comparison to few-shot fine-tuning -there are many ways to fine tune and many representations of the mrs, so it is possible that some other method of fine-tuning would lead to better fine-tuning results @xcite . our main goal here was to show that with a small-number of examples, using reasonable assumptions, few-shot fine-tuning performs worse than pbl. a potential risk of using llms is the possibility of disinformation, often called hallucinations. control of hallucinations is an active area of research. one of the challenges is that it is very difficult to automatically identify them. here we experiment with ranking functions for better control of hallucinations, hand-label hallucinations and characterize them. another potential risk of our work is that some of our dialogue acts like recommend and suggest could be used, in an application context, to persuade a user to buy something. in this context, it is even more important to ensure that the system is not providing false information to users.