Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task (2023)

Abstract: Large language models (LLMs) have shown incredible performance on many tasks such as dialogue generation, commonsense reasoning and question answering. In-context learning (ICL) is an important paradigm for adapting LLMs to the downstream tasks by prompting few demonstrations. However, the distribution of demonstrations can severely affect the performance, especially for challenging classification tasks. In this paper, we propose the concept of task-level thinking steps that can eliminate bias introduced by demonstrations. Further, to help LLMs distinguish confusing classes, we design a progressive revision framework, which can improve the thinking steps by correcting hard demonstrations. Experimental results prove the superiority of our proposed method, achieving best performance on three kinds of challenging classification tasks in the zero-shot and few-shot settings. Besides, with task-level thinking steps, automatically generated chain-of-thoughts (CoTs) bring more competitive performance.

## introduction large language models (llms) have shown incredible performance on many tasks such as dialogue generation, commonsense reasoning, and question answering @xcite . with the new paradigm of in-context learning (icl) @xcite , llms directly adapt to downstream tasks without updating parameters by prompting with few demonstrations. @xcite found that elaborating the reasoning steps in demonstrations can significantly stimulate the complex reasoning ability of llms, which is called manual chain-of-thought (cot). zero-shot-cot @xcite enables llms to generate thinking steps automatically using the simple but efficient prompt "let's think step by step.", as shown in figure 1 @xcite . more recently, plan-andsolve prompt @xcite extends it with @xcite vs. prompt with automatic task-level thinking steps "let's first understand the problem, devise a plan, and solve the problem step by step." to address the issue of missing reasoning steps and "extract relevant variables and their corresponding numerals" to address the issue of calculation errors. inspired by this, we propose the concept of task-level thinking steps for challenging classification tasks as shown in figure 1 (b), which helps llms to clarify tasks and remind them of overlooked issues. to this end, we design a progressive revision framework, which automatically generates thinking steps by llm agents and then progressively revise the steps by correcting hard demonstrations. for classification tasks, it is widely investigated that the demonstrations distribution, including content, label, and even the order, can severely affect the performance @xcite @xcite . therefore, many works focused on selecting high-quality and representative demonstrations, such as selecting similar demonstrations @xcite or maintaining the class balance between demonstrations @xcite . however, the best distribution depends on the task, and bias is inevitable. our proposed task-level thinking steps are helpful in reducing the bias introduced by demonstrations. ## conclusion although llms have shown incredible performance on many tasks, we pointed out that many classification tasks, such as multifaceted analysis of subjective text, fine-grained classification, and domain-specific classification tasks are still chal-lenging. therefore, we proposed the concept of task-level thinking steps and verified the robustness for biased demonstrations. to further enhance the classification capabilities of llms, we designed a progressive revision framework, which purposefully improves the thinking steps based on feedback from hard demonstrations. experimental results proved the superiority of our proposed method. besides, with task-level thinking steps, we found that the automatically generated cots are more reasonable and effective.