Knowledge-Grounded Dialogue Act Transfer using Prompt-Based Learning for Controllable Open-Domain NLG (2024)

Abstract: Open domain spoken dialogue systems need to controllably generate many different dialogue acts (DAs) to allow Natural Language Generation (NLG) to create interesting and engaging conversational interactions with users. We aim to create an NLG engine that can produce a variety of DAs that make substantive knowledge-grounded contributions to a conversation. Training such an NLG typically requires dialogue corpora that are labelled for DAs, which are expensive to produce and vulnerable to quality issues. Here, we present a prompt-based learning approach to transfer DAs from one domain, video games, to 7 new domains. For each novel domain, we first crawl WikiData to create Meaning Representations that systematically vary both the number of attributes and hops on the WikiData Knowledge Graph. The proposed method involves a self-training step to create prompt examples for each domain followed by an overgeneration and ranking step. The result is a novel, high-quality dataset, Wiki-Dialogue, of 71K knowledge-grounded utterances, covering 9 DAs and the Art, Movies, Music, Sports, TV, Animal, and Boardgames domains, whose combined DA and semantic accuracy is 89%. We assess the corpus quality using both automatic and human evaluations and find it high. The corpus is found to be safe, lexically rich, and large in vocabulary, when compared to similar datasets.

## introduction open domain spoken dialogue systems need to be able to controllably generate many different dialogue acts (das) in order to create interesting and engaging conversational interactions with users. for example, they should be able to ask questions of different types, inform the user of facts and express opinions, make recommendations and suggestions, and confirm what the user said. moreover, using knowledge to ground das supports taking the initiative to drive the conversation forward, and has been shown to help avoid hallucinations in generated outputs @xcite @xcite . we aim to create a natural language generation (nlg) engine that can produce a variety of das and make substantive knowledge-grounded contributions to a conversation. a synthetic example of a conversation about music with a variety of das is shown in dialogues like this require a semanticallycontrolled nlg that also controls the expression of das. in order to create such an nlg, training data consisting of dialogue corpora with utterances labeled with das, such as multi-woz @xcite , or a parallel corpus of domainspecific da representations and reference utterances, such as viggo or rnnlg @xcite , is typically needed. such training data is typically collected via crowdsourcing, making it expensive to produce, and vulnerable to quality issues @xcite . as a result, existing corpora primarily use the inform da @xcite @xcite @xcite or cover a limited number of topics @xcite @xcite . however, recent work using prompt-based learning (pbl) with llms shows that pbl can be used to create high quality dialogic datasets @xcite @xcite . for example, @xcite pbl to enlarge the da labelled viggo video games and rnn laptop and tv corpora @xcite . here we extend their overgenerate-and-rank nlg method to transfer the das from the viggo corpus onto a set of 6 wikidata domains and a we first specify the constraints on different types of viggo das for each domain, then crawl wiki-data to create meaning representations (mrs) consisting of sets of kg triples for each da type and domain. example mrs can be seen on the left side of ## related work most knowledge-grounded dialogue corpora are based on crowdsourcing utterances matching different types of mrs. one motivation for generating corpora using llms is that crowdsourcing is expensive, and crowdsourced corpora can be noisy, requiring extensive filtering or additional annotation to ensure accuracy. for example, @xcite states that up to 40% of the utterances in the e2e corpus either omitted information that was present in the mr or contained additional information @xcite , while qian et al. state that 70% of the dialogues in multiwoz contained dialogue state (semantic) annotation errors, which had to be corrected @xcite . web-nlg @xcite also contained mismatches between the mrs and the crowdsourced utterances that have been corrected. 1 we show in section 4 that there are fewer semantic errors in wiki-dialogue than in similar crowdsourced corpora. the most similar corpus to wiki-dialogue is webnlg, a multi-domain corpus that has been used for nlg challenges @xcite @xcite . webnlg is based on the dbpedia kg @xcite , and covers 19 domains, with utterances realizing the mrs collected via crowdsourcing. the english dataset contains about 17,000 triple sets and 45,000 crowdsourced texts. however, the only da in webnlg is inform, and dialogues consisting of only inform utterances are unnatural @xcite @xcite , while wiki-dialogue provides da diversity with nine das @xcite . previous work by @xcite created the opendialkg dataset by crowdsourcing humanhuman dialogues consisting of 15k utterances for the music, movies, sports and books domains. each utterance corresponds to either a one or twohop fact in the freebase kg @xcite and the kg paths are hand-annotated on each utterance. this corpus contains different types of das, but there are no da annotations. the schema-guided dialogue (sgd) dataset consists of over 20k annotated multi-domain, taskoriented conversations between a human and a virtual assistant @xcite . these conversations target interactions with services, such as travel, spanning 20 domains. the dialogues are generated in two steps: first a simulator automatically generates a dialogue, given a task, in the form of a sequence of das whose semantic values are filled by queries to freebase. the das used by the simulator are distinct for the system and the user, with 10 system das and 11 user das. then crowdworkers are tasked with paraphrasing each dialogue act/mr combination in a dialogue to ensure naturalness of the utterance realizations. the topical chat corpus consists of 235k utterances from 8 domains, but differs from other knowledge-grounded corpora in that the knowledge is represented by sentences which are automatically aligned with the dialogues. das were automatically labelled on this corpus using the 11 das from the iso da standard with an f1 of 0.54 @xcite . the da labels in wiki-dialogue are much less noisy: see other knowledge-grounded nlg corpora with rich sets of das have focused on specific domains. the viggo video games corpus contains 9 das @xcite , the rnnlg corpus encompasses 13 das for domains such as laptops, tvs, hotels, and restaurants @xcite , and mul-tiwoz offers 34 task-oriented, domain-specific das, for the restaurant, hotel, attraction, taxi, train, hospital and police domains @xcite . recent work by @xcite on controlling das in nlg for task-oriented dialogue introduces diacttod, a model based on learning latent das from pre-existing datasets, achieving state-of-theart performance on multiwoz @xcite . however, the evaluation diacttod is based on a benchmark set and automatic evaluation metrics, which do not evaluate da accuracy. we use a fine-tuned classifier that filters for da accuracy as one step in the automatic ranking of possible responses. we also apply both automatic and human evaluation, showing that we achieve average da accuracies of .98 for one-hop and .89 for multi-hop. other work on creating synthetic data for nlg has focused on creating whole dialogues or augmenting existing corpora @xcite @xcite . one of the main challenges with synthetic dialogue generation is producing high quality outputs without human evaluation. here we show that the quality of synthetic dialogue data can be improved using a cycle of self-training, along with an overgenerate-and-rank step that uses a da classifier and semantic accuracy estimates. we build on previous work by ramirez et al. by using their definitional prompt style, ranking function, and da tagger. we extend their approach by transferring the das used for controlled generation to 7 new domains and incorporating a self-training step required to bootstrap high quality generation outputs for completely novel domains. examples and conduct a second round of overgenerate-and-rank. ## results and corpus quality evaluation the wiki-dialogue corpus includes more than 71k utterances (∼50k for one-hop and ∼21k for multihop). we compare them using two metrics: bbleu @xcite , and dac. after self-training (st), the bbleu scores show a significant improvement (paired t-test, @xmath0.05), except for the music domain. overall for dac, self-training improves some domains but not others. however, the dac is consistently high across all domains, with values ranging from 0.97 to 0.99 after self-training. ## conclusion and future work this paper describes and provides a novel 71k utterance corpus called wiki-dialogue, covering 9 das and 7 kg based domains that are useful for both social conversation and task-oriented dialogue. the corpus includes both one and multi-hop sets of kg triples, and the mrs vary from a single triple for some request dialogue acts up to 8 triples for inform das. our novel method involves a self-training step to create prompt examples for novel domains, fol-lowed by an overgenerate-and-rank step, and we show that these two steps combined drastically improve the quality of the corpus. we assess the quality of the corpus with both automatic and human evaluation and find that the quality is high. we hand annotate for hallucinations and semantic errors and find their frequency to be lower than reported values for crowdsourced corpora such as e2e and multiwoz @xcite @xcite . we also check the corpus for safety using canary and find that only 0.22% of the utterances are flagged as needs_intervention, while a manual inspection of these suggests that there are no safety issues with them. in a comparison with similar corpora such as webnlg and opendialkg, we observe that although webnlg cover more domains wiki-dialogue is lexically richer and has a larger vocabulary. while webnlg has only inform das, wiki-dialogue covers 9 das, providing a complementary and unique resource to the dialogue community. future work should explore how the wiki-dialogue corpus can be used to train an nlg for dialogue. in preliminary experiments, a subset of wiki-dialogue was used to fine-tune a 3b multidomain nlg engine, which was tested in athena, a real-time amazon alexa prize system, with human users of amazon "let's chat" (yue fan and wang, 2023). the deployment used a cross-domain universal dialogue policy based on the nine wiki-dialogue das. an example of this policy is shown in the music conversation in figure 1 . ethical considerations. there are several potential risks with such an nlg. llms introduce the possibility of disinformation, often called hallucinations, whose control is an active area of research. one of the challenges is that it is very difficult to automatically identify them. here we experiment with ranking functions for better control of hallucinations, hand-label hallucinations and characterize them. another potential risk is that some of the das, like recommend and suggest, could be used in an application to persuade a user to buy something. a appendix domain mr text verify_attribute sports venus williams rating okay. venus williams start time 1994. venus williams domain athelete. you said you think venus williams is okay. do you think she is an average athlete compared to other players who started in 1994? movies avengers: endgame review score good. avengers: endgame cast member robert redford, taika waititi. avengers: endgame domain movies. i remember you thought avengers: endgame was good. do you usually enjoy movies with robert redford and taika waititi? music kings of the wild frontier good. kings of the wild frontier i know you like kings of the wild frontier. do you like other songs from the kings of the wild frontier album? tv the crown review score average. the crown genre biographical film, drama television series. the crown domain tv. i know you said the crown had an average review score. do you feel the same way about other biographical films and drama television series? art raphael rating excellent. raphael genres portrait, portrait painting. raphael movements italian renaissance, high renaissance. you rated the portrait painting by raphael as excellent. would you say you generally have a strong liking for italian renaissance and high renaissance portrait paintings? animals carolina dog good with other dogs friendly to other dogs. carolina dog good with children kid-friendly. carolina dog domain animals. i know you said that carolina dogs are friendly to other dogs and kid-friendly. are you generally interested in animals that are friendly to other dogs and kid-friendly? board games a game of thrones average rating good. a game of thrones genre fantasy. a game of thrones domain board game. you said a game of thrones was good. do you usually like fantasy board games? recommend sports jake paul sport boxing. jake paul height 1.83. jake paul domain athelete. have you heard of jake paul? he's a professional boxer who stands at 1.83 meters tall. movies elvis review score good. elvis producer gail berman, catherine martin, baz luhrmann. elvis domain movies. have you seen elvis, produced by gail berman, catherine martin, and baz luhrmann? it has a good review score. music money for nothing rock music littlest things mark ronson have you heard of the rock song money for nothing produced by mark ronson? tv peaky blinders review score good. peaky blinders creator david caffrey. peaky blinders domain tv. have you seen peaky blinders from david caffrey? it's got a really good review score. art the haywain triptych creator hieronymus bosch. the haywain triptych location museo del prado. have you seen the haywain triptych by hieronymus bosch at the museo del prado? animals staffordshire bull terrier trainability very easy to train. staffordshire bull terrier domain animals. i think you'd really be interested in the staffordshire bull terrier, which is very easy to train. have you heard of it before? board games heroquest publisher games workshop. heroquest genre fantasy. heroquest domain board game. have you ever heard of heroquest, the fantasy board game from games workshop? give_opinion sports ronaldo sport association football. ronaldo rating good. ronaldo domain athelete. i think ronaldo is a great footballer. he has consistently put in strong performances for his teams and is known for his skill and passion for the game. movies scooby-doo review score average. scooby-doo director raja gosnell. scooby-doo mpa film rating pg. scooby-doo domain movies. scooby-doo is an average movie, but i think it's worth watching for the great direction of raja gosnell and the pg rating. music y.m.c.a. rating excellent. y.m.c.a. producer jacques morali. y.m.c.a. record label casablanca records. y.m.c.a. domain songs. y.m.c.a. by the village people, produced by jacques morali and released by casablanca records, is an excellent song. tv gunpowder review score average. gunpowder screenwriter ronan bennett. gunpowder domain tv. gunpowder was an average show, but i appreciate the work of the screenwriter, ronan bennett, for making the story interesting. art henri de toulouse-lautrec rating good. henri de toulouse-lautrec genres animal painting, portrait. i think henri de toulouse-lautrec is a good painting. the combination of animal painting and portraiture in this piece is quite engaging. animals fin whale popularity very popular. fin whale weight 1,800kg -70,000kg 4,000lbs -150,000lbs. fin whale domain animals. i'm really impressed by the fin whale with its huge weight range of 1,800kg -70,000kg (4,000lbs -150,000lbs). board games twilight struggle average rating excellent. twilight struggle maximum number of players 2. twilight struggle domain board game. i highly recommend twilight struggle. it's an excellent board game with a maximum number of players of 2, making it perfect for a competitive game night. request_attribute sports athletes sports teams. what sports team does your favorite athlete play for? movies movies review score none. movies domain movies. what rating would you give to movies that you have watched recently? music songs performer. who is your go-to artist for music? tv show screenwriter none. show domain tv. what is your favorite tv show and its screenwriter? art paintings genres. what types of painting genres resonate with you the most? animals animal good with other dogs. animal domain animals. is your favorite animal good with other dogs? board games board game genre. board game domain board game. what board game genre do you find the most enjoyable?