Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models (2023)

Abstract: Existing dialogue models may encounter scenarios which are not well-represented in the training data, and as a result generate responses that are unnatural, inappropriate, or unhelpful. We propose the “Ask an Expert” framework in which the model is trained with access to an “expert” which it can consult at each turn. Advice is solicited via a structured dialogue with the expert, and the model is optimized to selectively utilize (or ignore) it given the context and dialogue history. In this work the expert takes the form of an LLM.We evaluate this framework in a mental health support domain, where the structure of the expert conversation is outlined by pre-specified prompts which reflect a reasoning strategy taught to practitioners in the field. Blenderbot models utilizing “Ask an Expert” show quality improvements across all expert sizes, including those with fewer parameters than the dialogue model itself. Our best model provides a ~10% improvement over baselines, approaching human-level scores on “engingingness” and “helpfulness” metrics.

## introduction dialogue systems based on pre-trained language models (plms) can be easily tailored via finetuning to exhibit particular characteristics, such as empathy @xcite and emotion @xcite ). however, it has been previously observed that such models tend to produce vacuous "fallback" responses when presented with unfamiliar situations (e.g., extraneous @xcite ). for instance, we observe that fine-tuned blenderbot @xcite models have a propensity to use the response, "do you have any hobbies?" as a substitute for furthering the conversation in helpful ways when the situation becomes too complicated. for goaldirected dialogues, where the discourse should consistently move towards a desired resolution or effect @xcite , frequent reliance on such i am so worried about the coming exams. what should i do? i know you can do it! ask an expert seeker seeker i am so worried about the coming exams. what should i do? what is the emotional status of the seeker? the seeker feels worried. why is the seeker worried? the seeker feels worried about performance in upcoming exams. figure 1 : the proposed method of consulting the expert, where the dialogue model interactively obtains advice from the llm via prompting (e.g. gpt3). without the aid of expert knowledge and reasoning, dialogue models are less able to generate useful and engaging responses. we hypothesize that the use of fallback responses may stem from the model being unable to formulate a more suitable reply in the absence of appropriate knowledge of the situation. in this study, we propose a framework called "ask an expert" to enhance dialogue responses through on-the-fly knowledge acquisition. our approach involves integrating dialogue models with an external "expert" by the following tenets: (a) the expert is a large language model (llm) which is available both during training and inference, (b) the act of soliciting information from the expert itself takes the form of a dialogue, which can span multiple turns in order to identify relevant information and strategies, and (c) the knowledge is integrated into the dialogue model via the context. recently many efforts have sought to utilize text as an api to chain together multiple models to perform complex tasks @xcite . our approach differs in that the model interaction takes place within the optimization loop, and thus allows the dialogue model to learn to selectively choose which advice to incorporate, and when use it. we apply "ask an expert" to the domain of mental health support (mhs) systems. mhs is notable in being one of many domains in which practitioners are formally trained to follow specific discourse strategies @xcite . we incorporate an mhs strategy into the model via a series of handcrafted prompts, which are designed to shape the expert conversation to reflect the inner monologue of a human expert (figure 1 ). the resulting conversation is then provided in a structured way as conditioning context to the dialogue model. we perform human evaluations on the models following the method of acute-eval @xcite to assess the system on six dimensions, including the ability to both have general conversations and provide helpful suggestions. we find models with reasoning processes significantly outperform the baseline model (without reasoning) in providing constructive suggestions and sharing similar experiences while remaining engaging and empathetic. contributions of this work are as follows: ## ask an expert the architecture we propose, ask an expert, consists of a dialogue model, and a separate expert model. in this work the expert is a (presumably larger or specialized) llm. the key distinction between ours and other work which uses additional knowledge acquisition in dialogue systems is that ours takes the form of another dialogue, in which we utilize prompts to guide the expert towards providing important reasoning to guide the dialogue system's response. the dialogue model is trained to optimize dialogue quality while working together context conversation seeker: whenever we have family gathering, my aunts and uncles would brag about how much their children make. i have higher degree but will only make half of their salary so i feel bad. supporter: so, you feel that your family is judging you for your earning potential? seeker: yes, my parents won't say it to me but they never show they're proud either. ## training dialogue models to evaluate the effect of incorporating our knowledge acquisition procedure into a state-of-the-art dialogue model, we train the following: blenderbot for mental health with reasoning (bbmhr) this is a model utilizing the ask an expert architecture as applied to mental health support systems, fine-tuned on the reasoning processes that are collected through prompting as described in section 3.1. at training time, seeker utterances and associated reasoning processes that we collected from llm expert models are concatenated as inputs. at inference time, we modify the parlai framework to allow communications between the dialogue model and the llm experts to get ad-hoc reasoning annotations. like bbmh, bbmhr is also fine-tuned in a multi-task style on both blend-edskilltalks and esconv (with reasoning) for the same purpose. all models are fine-tuned with parlai framework @xcite using blenderbot-bst 2.7b @xcite ) as the initial model foot_2 . both bbmh and bbmhr are trained on 4 tesla v100 gpus for 96 hours. to be noticed, we train multiple bbmhr models with reasoning processes from different llms. in the following, bbmhr + llms denote the dialogue model with reasoning processes from the specific llm (e.g. bbmhr + gpt1 denotes the bbmhr model with reasoning processes from gpt1). ## discussion what are the advantages of utilizing llms for strategic reasoning? goal-oriented dialogue systems not based upon llms often rely on inferring dialogue states to carry out only meaningful conversations, and thus significantly rely on the definition of the task and an ontology of possible dialogue trajectories @xcite . this makes the systems brittle and open to catastrophic errors when the dialogue breaks significantly from the categories of the ontology. llms show similar ontological knowledge and planning ability in many domains, but are more flexible. as language models, interfacing with llm experts is as straightforward as establishing a short goal-oriented conversation, and incorporating their responses into the dialogue model via the model's context is similarly easy. in that sense, utilizing llms greatly reduces the efforts defining a complicated ontology and dialogue state tracking module by providing necessary reasoning power and knowledge. why not use gpt-3 directly for dialogue generation? is the dialogue model still necessary when there is an expert model? our results ( deploying ask an expert? a natural restriction in the ask an expert is that it requires the expert to be present at inference time and during deployment. if a motivation of ask an expert is to allow dia-logue models to be deployed on simpler hardware, having a large expert model limits its usefulness in such situations. however, recent advancements in technology, such as chatgpt and bard, offer api services that facilitate convenient access to expert knowledge. furthermore, software tools like langchain efficiently manage prompts, computations, and knowledge, presenting an alternative to local deployment of extensive expert models. another scenario that imposes limitations on the adoption of ask an expert pertains to certain domains where the system must be deployed locally to uphold privacy concerns, such as mental health systems aiming to safeguard patient data. in such instances, relying on external api services becomes less feasible. however, it is not always necessary to utilize all the knowledge of large expert models. and for specific domain use cases, such as mental health, it is unlikely that the full size of the model is indispensable. given the effectiveness of our approach, in future work we would like to explore the extent to which the expert model can be distilled @xcite into models which are able to run locally on consumer-grade hardware. ## conclusion in this work we propose the "ask an expert" framework for building more robust dialogue systems using external knowledge obtained via prompt-based conversations with llm "experts". the prompts are designed to elicit a step-by-step expert analysis of the current discourse context, intended to mimic the inner monologue of a human professional counselor, and provide it at each turn to the dialogue model. as the expert consultation process occurs both during training and inference time, the dialogue model itself can learn useful strategies for flexibly incorporating the advice of the expert. we have shown in both human and automatic evaluations that the addition of such reasoning knowledge results in models which are more suggestive, helpful, and engaging than comparable baseline models which do not consult the expert. our result supports the hypothesis that current dialogue models often fail to implicitly learn effective goal-oriented strategies from dialogue data alone, and provides evidence that combination with other models may help alleviate current shortcomings.