Can Large Language Models Reason About Goal-Oriented Tasks? (2024)

Abstract: Most adults can complete a sequence of steps to achieve a certain goal, such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented tasks, or simply tasks in this paper, one must use sequential reasoning to understand the relationship between the sequence of steps and the goal. LLMs have shown impressive capabilities across various natural language understanding tasks. However, prior work has mainlyfocused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well LLMs can perform on more complex reasoning tasks like sequential reasoning is not clear. In this paper, we address this gap and conduct a comprehensive evaluation of how well LLMs are able to conduct this reasoning for tasks and how they scale w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context examples, varying complexity of the sequential task). Our findings reveal that while Chain of Thought (CoT) prompting can significantly enhance LLMs’ sequential reasoning in certain scenarios, it can also be detrimental in others, whereas Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally, we discover that an increase in model size or in-context examples does not consistently lead to improved performance.

## introduction large language models (llms) have transformed natural language processing (nlp), achieving groundbreaking performance across an array of tasks, primarily due to their capacity for (incontext) zero-shot and few-shot learning @xcite @xcite . this prowess in task adaptation arises from their ability to "prompt"-essentially conditioning the models on limited examples or explicit task descriptions, and responding appropriately @xcite . the potential for models to adapt to tasks with limited to no exposure, especially without requiring extensive fine-tuning, is a testament to their potential and may be a step towards artificial general intelligence @xcite . the ability to logical reasoning is one of the most intriguing capabilities of llms, which has been explored in various studies, including the evaluation of their grasp of commonsense knowledge @xcite @xcite @xcite . although their performance on intuitive and single-step tasks is exemplary, their efficacy on tasks requiring multi-step reasoning, particularly tasks that simulate human system 2 foot_0 cognitive functions, has remained a challenge @xcite @xcite . this aspect of reasoning is vital, especially for goal-oriented tasks where the order and sequence in which actions are taken is crucial to the successful completion of the task. yet, in goal-oriented tasks, understanding and reasoning about a sequence of steps is critical. a disruption in the order of these steps can help, complicate or even nullify the task's objective. for example, in an effort to minimize speed in a certain task, such as preparing a soup in the kitchen, one must consider whether reordering certain steps is acceptable or by doing so the recipe (the goal) would be damaged. in the soup-making example, this could mean measuring, chopping and doing all preparation work-mise-en-place-before any cooking actually begins, which, oddly enough few recipes actually include as an explicit step but seems to not only speed up the overall cooking experience but lead to fewer later-step errors that would have otherwise resulted from inadequate inter-step time. we are hence drawn to consider how well the recent advances in llms translate to the system 2-type of reasoning, which we call sequential reasoning, necessary working with goal-oriented tasks. recent innovations, like the chain of thought prompting (cot) @xcite , provide a promising solution to this reasoning challenge. instead of relying on standard question-answer exchanges, cot feeds llms with sequential reasoning examples, facilitating the model to map out a logical reasoning path. alongside cot there is an emerging technique known as tree of thoughts (tot) prompting @xcite . tot extends cot's linear reasoning by allowing llms to explore multiple reasoning paths simultaneously, forming a tree of potential thoughts. this approach enables deliberate planning and exploration in problem-solving, where each thought is generated or solved independently. moreover, there is an emerging interest in their inherent zero-shot reasoning skills @xcite . novel approaches, such as zero-shot-cot @xcite , have demonstrated that by simply prompting models with an instruction like "let's think step by step", llms can autonomously derive a plausible reasoning pathway and arrive at logical conclusions. such findings not only underline the untapped potential of llms but also underscore their ability to mimic higher-level cognitive functions like generic logical reasoning @xcite . this is the first study that pushes this inquiry further, to evaluate llms' potential as logical reasoners for goal-oriented tasks, and investigate if the aforementioned claims for enhanced capability under certain prompting strategies hold true when used under the framework of sequential reasoning. using adapted versions of the youcook2 dataset @xcite and the crosstask dataset @xcite with varied sequence permutations, we probe the extent to which llms can discern and reason about the logical continuity of steps, especially when disruptions in their order are introduced (fig. 1 ). ## methodology sequential tasks can be largely divided based on their properties, complexity, and dependence on previous steps. in this study, we focus on goaloriented tasks -tasks that are directed towards achieving a particular objective, often encapsulated within a sequence of actions that must be executed in a specific order. figure 1 : illustration showcasing a permuted goaloriented task, specifically for preparing miso soup. on the left, the original recipe sequence is displayed, and on the right, the same recipe steps are shown in a permuted order. the example is from the youcook2 dataset. ## conclusion in this work, we adapted and utilized the youcook2 and crosstask goal-oriented datasets to contain varied levels of step sequence permutations in order to analyze how large language models respond to disruptions of logical order. we discover that cot prompting strategies can significantly augment models' sequential reasoning capacities in some cases. however, it also unexpectedly harms reasoning performance under certain conditions. moreover, tot reasoning approaches prove less effective on perturbed goal-oriented tasks, while increases in provided in-context examples seems to improve model outcomes, but not across all cases. we also discover a "u-shaped" scaling behaviour, where llms with significantly less parameters perform better than one of their larger counterpart, in one of our tasks. in total, while recent strategies can bolster goaloriented reasoning, the models seem to have a fragile understanding of the complex dependencies in multi-step procedures, frequently overlooking logical flaws in permuted sequences. however, performance gains under simpler permutations indicates reasoning capability may rapidly improve alongside advances in scale and prompting. our analysis provides a methodology for continued investigation as models evolve on this challenging reasoning frontier. this study contributes to a deeper understanding of the scalability and adaptability of llms in complex reasoning tasks.