The Art of SOCRATIC QUESTIONING : Recursive Thinking with Large Language Models (2023)

Abstract: Chain-of-Thought (CoT) prompting enables large language models to solve complex reasoning problems by generating intermediate steps. However, confined by its inherent single-pass and sequential generation process, CoT heavily relies on the initial decisions, causing errors in early steps to accumulate and impact the final answers. In contrast, humans adopt recursive thinking when tackling complex reasoning problems, i.e. iteratively breaking the original problem into approachable sub-problems and aggregating their answers to resolve the original one. Inspired by the human cognitive process, we propose SOCRATIC QUESTIONING, a divide-and-conquer style algorithm that mimics the recursive thinking process. Specifically, SOCRATIC QUESTIONING leverages large language models to raise and answer sub-questions until collecting enough information to tackle the original question. Unlike CoT, SOCRATIC QUESTIONING explicitly navigates the thinking space, stimulates effective recursive thinking, and is more robust towards errors in the thinking process. Extensive experiments on several complex reasoning tasks, including MMLU, MATH, LogiQA, and visual question-answering demonstrate significant performance improvements over the state-of-the-art prompting methods, such as CoT, and Tree-of-Thought. The qualitative analysis clearly shows that the intermediate reasoning steps elicited by SOCRATIC QUESTIONING are similar to humans’ recursively thinking process of complex reasoning problems.

## introduction the art of socratic questioning is important for critical thinkers and excellence of thought. what socratic adds is systematicity, depth, and a keen the kinetic energy theorem states that the net work done on an object equals its change in kinetic energy. in this case, the net work done is done by the frictional force. the work done by friction can be calculated as @xmath0, where l is the landslide length and @xmath1. write the equation as: μ * m * g * cos(θ) * @xmath2. solve that equation, @xmath3. ## related work prompting large language models with the scaling of both modal size and corpus size, large language models (llms) such as gpt-3 @xcite and chatgpt (openai, 2022) have exhibited emergent abilities, including prompting @xcite , in-context learning @xcite , and commonsense reasoning @xcite . one notable example of emergent abilities is the chain-of-thought (cot) @xcite which steers large language models to resolve complex problems by guiding them to produce a sequence of intermediate steps before giving the final answer. self-consistency chain-of-thought (sc-cot) @xcite improves naive cot by sampling multiple reasoning paths and selecting the most consistent answer. sc-cot is based on the assumption that given a complex reasoning problem, multiple reasoning paths can lead to the unique correct answer. tree-of-thought (tot) @xcite proposes to break the thinking process into small steps and at each step, the language model deliberately decides a set of next steps to try. ## experiment setups language-only tasks we leverage chatgpt as the llm for qa, qg, and qa2h modules, and provide detailed prompts for each module in appendix k. we evaluate socratic ques-tioning on several complex reasoning tasks, including the physics and chemistry tasks in massive multitask language understanding (mmlu) @xcite , mathematical tasks in math @xcite , and logical reasoning tasks based on logiqa @xcite . we adopt several state-of-the-art prompting methods as baselines, including standard prompting (sp) that directly prompts chat-gpt to answers a question with a few in-context examples. chain-of-thought (cot) @xcite , self-consistency chain-of-thought (sc-cot) @xcite , and tree-of-thought (tot) @xcite . following previous studies @xcite , we use exact match to measure the accuracy for all language-only tasks. more details for the baselines, evaluation metrics, and evaluation datasets are discussed in appendix c.1. multimodal tasks we use blip2-flan-t5-xl as our visual perception module. we leverage chat-gpt (openai, 2022) for factual/visual question generation and factual question answering and gpt-3 (gpt-3-davinci-003) for visual question answering foot_0 , motivated by the observation that chatgpt tends to be excessively cautious and neutral, and avoids answering some questions. we provide detailed sample prompts for each module in appendix k. we evaluate socratic questioning on several visual question answering datasets, including vqa-v2 @xcite , ok-vqa @xcite and aok-vqa @xcite , and compare our approach with several baselines, including blip-2 @xcite and. pica @xcite . more details for implementation, baselines, and datasets are discussed in appendix c.2. for evaluation, we employ the conventional vqa accuracy metric @xcite to measure the performance. to alleviate stringent penalization for minor discrepancies between predicted answers and ground truth, we normalize the answers by converting plural forms to singular forms and changing the tense of verbs to present tense. in addition, to address the conventional metric's limitation due to synonyms and expression differences, we design semantic-based accuracy by employing chat-gpt to evaluate the correctness of the predicted answers @xcite . we provide chatgpt with the visual question, the predicted answer and the ground-truth answer, and ask if the ground-truth answer and the predicted answer can support each other. if the answer is "yes", we treat the predicted answer as correct. we show the exact prompts used for chatgpt in appendix k.8. ## conclusion we present socratic questioning, a novel divide-and-conquer fashion algorithm that is inspired by human's recursive thinking processes. socratic questioning consists of a top-down reasoning phase that decomposes a complex problem into simpler sub-problems and a bottom-top phase where the solutions to the sub-problems are recursively returned and used to solve the original problem at higher levels. extensive experiments on four challenging language-only tasks and the few-shot vqa task validate the effectiveness of our socratic questioning. moreover, qualitative analysis demonstrates our approach can effectively elicit intermediate reasoning steps and consequently yield a correct final answer while enjoying transparency and interpretability.