Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models (2024)

Abstract: Large Language Models (LLMs) have achieved remarkable performance in objective tasks such as open-domain question answering and mathematical reasoning, which can often be solved through recalling learned factual knowledge or chain-of-thought style reasoning. However, we find that the performance of LLMs in subjective tasks is still unsatisfactory, such as metaphor recognition, dark humor detection, etc. Compared to objective tasks, subjective tasks focus more on interpretation or emotional response rather than a universally accepted reasoning pathway. Based on the characteristics of the tasks and the strong dialogue-generation capabilities of LLMs, we propose RiC (Reasoning in Conversation), a method that focuses on solving subjective tasks through dialogue simulation. The motivation of RiC is to mine useful contextual information by simulating dialogues instead of supplying chain-of-thought style rationales, thereby offering potential useful knowledge behind dialogues for giving the final answers. We evaluate both API-based and open-source LLMs including GPT-4, ChatGPT, and OpenChat across twelve tasks. Experimental results show that RiC can yield significant improvement compared with various baselines.

## introduction large language models (llms; @xcite @xcite @xcite have made rapid advancements in recent years and have achieved excellent performance on various objective tasks, including open-domain question answering @xcite , mathematical reasoning @xcite , and code generation @xcite , etc. despite the success, research on llms in subjective tasks @xcite @xcite is still underexplored, as examples identify whether the sentence "joseph is very kind" is a paraphrase of the metaphoric sentence "joseph has the heart of a lion". the second sentence suggests that joseph possesses qualities associated with a lion, while the first sentence only mentions kindness. the answer is false. from the conversation, we know that "heart of a lion" means being kind and brave. the answer is true. i heard joseph has the heart of a lion. really? what do you mean by that? shown in recent methods based on chain-of-thought (cot) style prompting have improved the reasoning abilities of llms, showing promising results on tasks such as commonsense and mathematical reasoning @xcite @xcite . however, compared to these objective tasks, we found that such methods are not particularly effective on subjective tasks. as an example shown in figure 1(b) , the reasoning pathway does not comprehend the metaphorical expression in "joseph has the heart of a lion" well, sarcasm detection which statement is sarcastic? (a) yes, the game is over after the second half. 43.10 ♢ (2) @xcite (b) yes, the game is over after the first half. ## method the overall pipeline of the proposed ric can be divided into three steps. ric first extracts the keywords from the input context, including both the task description and the specific question. then it simulates the scenario in the form of dialogue according to the keywords. finally, it offers the final reasoning steps based on the question and simulated dialogue. formally, given a subjective task with a description illustration of the unified prompting and response /* input */ metaphor recognition: identify whether the sentence "joseph is very kind." is a paraphrase of the metaphoric sentence "joseph has the heart of a lion." (t , q) first, extract keywords from the question. (p 1 ) then, according to the keywords, construct a scenario for the question in the form of dialogue. (p 2 ) finally, according to the question and conversation, reason and give the final answer. select from true or false. (p 3 ) /* response of m */ keywords: (k) joseph, heart of a lion, kind. scenario: (s) sarah: i heard joseph has the heart of a lion. amy: really? what do you mean by that? sarah: i mean he is very kind and brave, just like a lion. reasoning: (r) from the conversation, we know that "heart of a lion" means being kind and brave. the answer is true. t and a specific question q, our goal is to let the llm m solve the question q. ## analyses and discussions in this section, we conduct a series of analyses to probe the reason behind the effectiveness of the ric method. we first investigate the effectiveness of keywords and dialogue ( § 5.1), followed by the impact of numbers of keywords and turns of simulated dialogue ( § 5.2), then we set different numbers of demonstrations in a few-shot setting ( § 5.3) and compare the length of response for different methods ( § 5.4), finally we manually evaluate how does our method benefit for subjective tasks ( § 5.5). ## conclusion we introduce ric (i.e., reasoning in conversation), a tuning-free method to enhance the ability of llms to solve subjective tasks through dialogue simulation. the core motivation of the proposed ric is to better leverage the useful information from human conversation based on the advanced dialogue generation ability of current llms. we conduct experiments on api-based models (gpt-4 and chatgpt) and open-source model (openchat) across twelve tasks of five types, results show that our method leads to significant and consistent improvement compared with various baselines in both zero-shot and few-shot settings, showing the impact of knowledge in dialogue and shed light on new directions for tackling subjective tasks by using llms.