Cue- C o T : Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLM s (2023)

Abstract: Large Language Models (LLMs), such as ChatGPT, greatly empower dialogue systems with strong language understanding and generation capabilities. However, most of the previous works prompt the LLMs to directly generate a response based on the dialogue context, overlooking the underlying linguistic cues about the user status exhibited in the context. Such in-depth dialogue scenarios are challenging for existing LLMs to figure out the user’s hidden needs and respond satisfactorily through a single-step inference. To this end, we propose a novel linguistic cue-based chain-of-thoughts (Cue-CoT), which enhances the LLMs inference with an intermediate reasoning step to find cues exhibited in the dialogue, aiming to provide a more personalized and engaging response. To evaluate the approach, we build a benchmark with in-depth dialogue questions, consisting of 6 datasets in both Chinese and English, targeting 3 major linguistic cues during the conversation: personality, emotion, and psychology. We conducted experiments on the proposed benchmark with 5 LLMs under both zero-shot and one-shot settings. Empirical results demonstrate our proposed Cue-CoT method outperforms standard prompting methods in terms of both helpfulness and acceptability on all datasets.

## introduction large language models (llms), or foundation models @xcite , especially after the appearance of chatgpt 1 , recently revolutionize the paradigm of various natural language processing (nlp) tasks, including dialogue response generation tasks @xcite . however, most existing llm-based studies directly feed the user query or dialogue content to the llm for generating a response with a preceding prompt, making the responses stereotypical and tedious, especially for in-depth dialogue questions @xcite . on the contrary, it is widely acknowledged that dialogue contexts generally convey a lot of information about the user status in addition to the pure semantic information from a linguistic perspective @xcite @xcite . specifically, the linguistic cues underlying dialogue context have been shown to be an effective means of revealing the emotions @xcite , personality traits @xcite , psychological characteristics @xcite , and other relevant information of users @xcite . consequently, recognizing and understanding these cues exhibited in the context of dialogues becomes crucial to comprehend user intentions and status @xcite . by doing so, a dialogue system can generate responses that align with the user's expectations, meet their unique needs and preferences, and ultimately strive towards constructing a human-like conversational experience @xcite . inspired by the emergent capabilities of llms on reasoning over texts @xcite , while previous efforts mostly investigate in-context learning @xcite or chain-of-thought prompting schemes @xcite ) on text classification @xcite and question answering @xcite @xcite , we take advantage of llms to reason for dialogue @xcite , which, to our best knowledge, has not been thoroughly investigated yet. specifically, we design a linguistic cue-based chain-of-thoughts (cue-cot), consisting of two variants: o-cue cot and m-cue cot in which the former one outputs intermediate reasoning re-user: if we only rely on free online teaching, can we learn the violin? what about the piano? system: yes, but with a conscious learning spirit and patient practice….. user: can we learn violin and piano from scratch? ## related work chain-of-thought prompt. following the initial chain-of-thought prompting @xcite , lots of works spring up aim to improve different parts of original reasoning processing, including auto-cot @xcite , self-consistency @xcite , active prompt @xcite , automatecot @xcite . besides that, a further line of work studies in-context learning @xcite as its efficiency and effectiveness with llms as backbones in which the key of it is to select informative demonstrations to prepend the input as additional information to get better results @xcite . to find the best demonstrations and unleash llms' power, @xcite propose to retrieve examples that are semantically similar to a test query sample while some works utilize uncertainty @xcite or diversity @xcite to refine and evaluate the selected examples. also, few works @xcite focus on the intermediate reasoning steps, and they use the reasoning complexity @xcite , i.e., chains with more reasoning steps, making the effective demonstration. dialogue system. most of the previous work develops personalized @xcite @xcite , emotional @xcite @xcite @xcite , empathetic @xcite @xcite dialogue system in isolation, rather than seamlessly blending them all into one cohesive conversational flow @xcite . a common approach is to predict the emotion or persona from a pre-defined set and generate the response in a multi-task manner @xcite @xcite @xcite . besides that, lots of work notices these linguistic cues underneath text by directly predicting them independently as a classification task @xcite @xcite . distinguishing from these previous works, we regard different aspects of cues as part of user status and prompt the llms to reason user status exhibited in the dialogue context, aiming to generate more helpful and acceptable responses for users. ## method in this section, we introduce more details about our method and how we select demonstrations under the few-shot setting. ## datasets collection in order to evaluate the performance of proposed cue-cot to reason different user statuses, we collect six datasets in terms of personality, empathy, and psychology, in both chinese and english. personality. previous works found that the content and style of a user's inquiry can provide indirect insights into their personality traits @xcite . for instance, an individual with a tendency towards anxiety may ask for advice on how to alleviate nervousness before an upcoming job interview, phrasing the question as follows: "what strategies can i employ to reduce my anxiety and perform well in tomorrow's interview?". since the public @xmath0, user status 1 response 1 > < dialogue context 2, user status 2 response 2 > < dialogue context n, user status n response @xmath1. ## experiment in this section, we have conducted a comprehensive experiment to compare the performance of three prompting methods: standard prompting, o-cue and m-cue cot in the benchmark under both zeroshot and one-shot settings foot_3 . ## analysis in this section, we conduct an extensive analysis with the backbone as chatgpt using m-cue cot because of its superior performance in both chinese and english 14 . ## conclusion in this paper, we build a benchmark to evaluate the helpfulness and acceptability of responses generated by current llms, considering three major linguistic cues of user statuses. we then propose a cue-cot to trace the status of users, decomposing the response generation into multiple reasoning steps. experimental results demonstrate the superior performance of our method on 6 datasets under both zero-shot and one-shot settings. we hope the release of our work can shed some light on the evaluation and development of llms. we left chain-of-thought tuning and instruction tuning in our future work.