Structured Chain-of-Thought Prompting for Few-Shot Generation of Content-Grounded QA Conversations (2024)

Abstract: We introduce a structured chain-of-thought (SCoT) prompting approach to generating content-grounded multi-turn question-answer conversations with a pre-trained large language model (LLM). At the core of our proposal is a structured breakdown of the complex task into a number of states in a state machine, so that actions corresponding to various subtasks, e.g., content reading and utterance generation, can be executed in their own dedicated states. Each state leverages a unique set of resources, including prompts and (optionally) additional tools, to augment the generation process. Automatic evaluation shows that SCoT prompting with designated states for hallucination mitigation can increase agent faithfulness to grounding documents by up to 16.8%. When used as training data, our open-domain conversations synthesized from only 6 Wikipedia-based seed demonstrations train strong conversational QA agents. In out-of-domain evaluation, for example, we observe improvements of up to 13.9% in F1-score against ground truth over target domain gold data when the latter is augmented with our generated examples.

## introduction despite enormous advances in large language model (llm) capabilities in recent years, their notorious propensity to hallucinate, i.e., generate text that are factually inconsistent with existing knowledge, remains an open issue @xcite @xcite . of particular interest to us in this paper are what openai (2023) characterize as closed-domain hallucinations, whereby models fail to generate text that can be supported by a given document even when they are explicitly instructed to do so @xcite @xcite . despite being powerful generators, llms can thus be unreliable readers, especially in the absence of extensive instruction tuning. figure 1 : a multi-turn qa conversation grounded in a document. if the document does not have an answer to a user query, the agent acknowledges so in its response. here we focus on closed-domain hallucination in pre-trained llms in the specific context of conversational question answering (qa) @xcite @xcite , and its mitigation through inference-time augmentation. concretely, given a document, we want to generate a multi-turn qa conversation between a user and an agent, in which the agent's responses to the user's queries must be grounded in the document. the requirements also crucially include for the agent to be able to determine if the given document has an answer to a query, and refrain from giving a memorized or made-up answer if not. figure 1 shows an example. compared to a single-turn setting @xcite , multi-turn qa provides a more useful backdrop for studying llm hallucinations, as an increased number of diverse user queries on a shared topic is likely to induce agent hallucinations more often than a single qa turn. we propose novel algorithms for this task ( §3) that adhere to the general notion of language model (lm) augmentation @xcite @xcite , whereby a set of actions -each leveraging its own dedicated tools and resources -are executed in an interleaved manner to solve a complex problem. for example, besides generating utterances, we execute intermediate tasks that aim to answer questions such as: "is this user query answerable from the given document?" or "where in the document is the answer?" all subtasks are performed using few-shot in-context learning (icl) @xcite , for which we utilize task-specific instructions, exemplars and (optionally) supporting models. our proposed actions and their execution sequences can be collectively represented as a state machine ( §2), whose state transitions define our different algorithms ( §3). we refer to this approach as structured chain-of-thought (scot) prompting, which, like ordinary chain-of-thought (cot) prompting @xcite , computes a final output through a set of careful reasoning steps, but unlike cot, distributes those steps across designated states of a state machine. one key advantage of the approach stems from the simplicity of its individual actions, which enables us to successfully prompt relatively small and open-source llms to generate high-quality conversations. leveraging open-source models such as fal-con-40b @xcite and flan-ul2-20b @xcite and a small set of wikipediabased exemplars that we create by hand ( §3) -the example of figure 1 is one of them -we generate open-domain qa conversations from wikipedia passages using our algorithms, and evaluate them both intrinsically and extrinsically ( §4). in intrinsic evaluation, we directly examine the quality of the generated conversations, including their faithfulness to the grounding document and their overall accuracy relative to pseudo-references provided by a high-performance instruction-tuned mixtral-8x7b-instruct-v0.1 @xcite . automatic evaluation using lexical and semantic overlap metrics shows that our proposed mechanisms for mitigating agent hallucination do indeed reduce it by up to 16.8%, improving overall accuracy of agent utterances by as much as 7.7%. in extrinsic evaluation, we train agents with our generated data to answer questions in multi-turn qa conversations, and evaluate them against gold labels using automatic lexical overlap metrics. we report experiments with few-shot icl and supervised fine-tuning (sft) of agents on two conversational qa datasets: doqa @xcite and quac @xcite . in icl evaluation, our synthetic data -generated from only 6 seed demonstrations and with a relatively small llmoutperforms human-labeled data, including target domain gold data. we also observe strong performance in sft evaluation, which includes training conversational qa agents only with synthetic data as well as augmenting existing target domain gold data with it. for example, augmenting with our open-domain synthetic data improves agent performance over using only target domain gold data by an absolute 10-14%. in summary, the following are our main contributions: (i) we present novel structured chain-ofthought prompting methods with lm augmentation for generating document-grounded qa conversations using pre-trained llms; (ii) in automatic intrinsic evaluation, our proposed augmentations for hallucination mitigation help the llm agent remain considerably more faithful to the given document; (iii) in automatic extrinsic evaluation on grounded conversational qa datasets, our generated conversations demonstrate strong standalone performance as well as the ability to effectively augment target domain gold data. ## preliminaries we introduce our state machine for scot prompting in this section, and discuss the alignment of its states to different stages of generating a documentgrounded multi-turn qa conversation. further implementation details are provided in §3. let d be the set of all documents and c n the set of qa conversations ⟨q i , r i ⟩ n @xmath0, where each q i is a user utterance (a query) and r i is a corresponding agent utterance (a response). our task is to map an input document @xmath1 our proposed algorithms prompt specific sets of state transitions in the state machine of figure 2 to generate a single utterance pair (q i , r i ); a full conversation c n is generated by repeating the process generate user utterance identify answer sentences determine if question is answerable generate agent utterance from answer sentences 𝑢𝑢 𝑎𝑐 𝑠𝑠 𝑎𝑢 user utterance question is answerable/ unanswerable answer sentences agent utterance generate agent utterance from document generate agent utterance from document and question answerability decision n times. there are four states in the state machine, each corresponding to an individual action: the current user query q i is classified as answerable or unanswerable from d: this state captures the notion that the assessment of the answerability of a user query can be kept separate from generating a response for it, which provides modularization and added flexibility. for example, a different set of resources can now be leveraged for question answerability classification, such as a classifier trained on existing qa data. in state ss, the current query q i is mapped to relevant sentences in d: state au can be reached from any of the other three states; the following are to be noted: (i) if au is reached via a @xmath2, then the input document d * = d; (ii) if au is reached via an @xmath3, then d * may contain only the sentences of d deemed relevant to q i in ss, or alternatively special symbols marking those sentences; and finally, (iii) if au is reached via an @xmath4, then this step is deterministic whenever q i is deemed unanswerable in ac, in which case r i takes the form of a pre-defined no answer text. our methods can each be completely specified using (i) the state transitions it executes in the state machine, and (ii) the resources, e.g., models and associated prompts, that it utilizes in its different states, as we detail in §3. ## methods we implement five algorithms that execute three unique sequences of state transitions. what follows is a description of the sequences along with how the different algorithms implement them. document: title: harry potter s1: harry potter is a series of … s2: the novels chronicle the life of … s3: the main story arc concerns … … eod conversation: user says: tell me about harry potter agent says: harry potter is a … user says: what happens in the story? are there sentence(s) in the above document that can help answer this last user query? yes: s3 document: title: … s1: … … eod conversation: user says: … agent says: … … user says: … are there sentence(s) in the above document that can help answer this last user query? here is a document broken down into its title and text: title: harry potter text: harry potter is a series of … eod what follows is a conversation between a user and an agent. the user asks a number of different questions about the topic of the document. the agent answers those based solely on the content of the document. if the document does not contain information that can answer the user's query, then the agent says "sorry, i can't find an answer in the document." here is the conversation: user says: tell me about harry potter. eou agent says: harry potter is a … eoa … user says: are there any movies based on the series? eou agent says: sorry, i can't find an answer in the document. eoa here is a document broken down into its title and text: title: … text: … eod what follows is a conversation between a user and an agent. the user asks a number of different questions about the topic of the document. the agent answers those based solely on the content of the document. if the document does not contain information that can answer the user's query, then the agent says "sorry, i can't find an answer in the document." here is the conversation: user says: … eou agent says: … eoa … user says: … eou agent says: the two differ in their implementation of ac, for which one utilizes the same llm and the other leverages a separate classifier fine-tuned on preexisting qa data. separate prompts containing both answerable and unanswerable exemplars are used for the two types of models, which we show in figures 5 and 6 of appendix a. the above methodology enables us to study the reading and generation aspects of our task separately and in a controlled manner, and better understand where pre-trained llms need the most assistance. for example, we can explore the following questions directly: (a) is it useful to break down the task into reading (ac, ss) and generation (uu, au) stages where scot prompting can be utilized? and (b) in the reading and reasoning states of ac and ss, can a pre-trained llm perform the corresponding tasks by itself, or is a different set of tools needed? ## experiments we write 6 simple wikipedia-based qa conversations with a total of 32 user turns and an equal number of agent turns; 20 of the user queries are answerable and 12 are unanswerable from their respective passages. the example of figure 1 is representative of the distribution; other titles include "table tennis" and "evolution". we then prompt llms with these as exemplars to synthesize new data from additional wikipedia passages. in this section, we first analyze and evaluate our generated datasets intrinsically. to assess their practical utility, we then train conversational qa agents with each and evaluate those agents on different test sets. as stated before, each generation algorithm is uniquely specified by its three components: (a all evaluations reported in this section are automatic, where we compare generated texts algorithmically with references such as human-written utterances or grounding documents. while human assessment of the generated texts is almost always more accurate at the level of individual texts, we opt for large-scale and extensive (both intrinsic and extrinsic) automatic evaluation instead, as long multi-turn conversations are expensive to annotate at scale for the results to be statistically reliable. moreover, the grounded nature of our task inherently limits the extent to which an accurate response can differ from its ground truth. ## related work chain-of-thought (cot) prompting @xcite showed that llms can be few-shot prompted @xcite to produce intermediate reasoning steps, or "thoughts", which can improve their performance. prompts were created manually and were application-specific (see @xcite ). chain-of-verification (cove) @xcite adopted a structured approach to cot for answer verification and error correction, splitting the thought process into verification planning and execution steps that utilize their own prompts, finally producing a rewritten response with reduced hallucination. also related is the recitation-augmented cot approach of @xcite , which recites related knowledge stored in the llm's parameters before answering a question. other structured cot prompting work include react @xcite and rewoo @xcite , which leverage llm-powered agents to solve complex tasks through interleaved steps, with use of external tools such as search engines. the proposed approach draws inspiration from the above studies, but aims to solve the complex task of multi-turn content-grounded conversation generation, focusing on the key associated requirements of determining question answerability and knowledge selection. more importantly, we produce synthetic data that can train agents for direct inference, removing the need for computationally expensive cot prompting during inference. the cot precursor scratchpad @xcite ) showed that models can also be fine-tuned to execute thought processes in order to solve a task in a step-by-step manner, and the benefits of generating synthetic cot data for training. it did not, however, use prompting and was limited to relatively close-ended problems. constitutional ai @xcite used a two-step cot process (critique and revise) to generate corrected responses. like this work, and unlike scratchpad, it removed the cot and kept only the final response to generate the synthetic data. unlike our work, constitutional ai focuses on reducing harmfulness and uses this step as part of a bigger process that includes reinforcement learning. finally, it is worth noting that many prior works relied on closed-source models such as gpt-3.5 foot_1 , palm @xcite , codex @xcite and lamda @xcite , and are thus not reproducible. here we only use opensource models whose weights are publicly available, both for generation: falcon @xcite and flan @xcite , and for eval-uation: mixtral @xcite , llama-2 @xcite and falcon. ## conclusion we introduce a structured chain-of-thought (scot) prompting approach to generating multi-turn content-grounded conversations and empirically show that high-quality synthetic data can be produced from only six human-written seed conversations. designated states for hallucination mitigation and the use of supporting tools enable our methods to generate agent utterances that are highly faithful to grounding documents. used as training data, our generated conversations train highperformance models as evaluated on out-of-domain test sets, successfully augmenting target-domain human-labeled data. future work will explore more complex conversational settings, e.g., multidocument grounding and response generation for harder, more ambiguous user utterances.