Can LLM s Reason in the Wild with Programs? (2024)

Abstract: Large Language Models (LLMs) have shown superior capability to solve reasoning problems with programs. While being a promising direction, most of such frameworks are trained and evaluated in settings with a prior knowledge of task requirements. However, as LLMs become more capable, it is necessary to assess their reasoning abilities in more realistic scenarios where many real-world problems are open-ended with ambiguous scope, and often require multiple formalisms to solve. To investigate this, we introduce the task of reasoning in the wild, where an LLM is tasked to solve a reasoning problem of unknown type by identifying the sub-problems and their corresponding formalisms, and writing a program to solve each sub-problem, guided by a tactic. We create a large tactic-guided trajectory dataset containing detailed solutions to a diverse set of reasoning problems, ranging from well-defined single-form reasoning (e.g., math, logic), to ambiguous and hybrid ones (e.g., commonsense, combined math and logic). This allows us to test various aspects of LLMs reasoning at the fine-grained level such as the selection and execution of tactics, and the tendency to take undesired shortcuts. In experiments, we highlight that existing LLMs fail significantly on problems with ambiguous and mixed scope, revealing critical limitations and overfitting issues (e.g. accuracy on GSM8K drops by at least 50%). We further show the potential of finetuning a local LLM on the tactic-guided trajectories in achieving better performance. Project repo is available at https://github.com/gblackout/Reason-in-the-Wild.

## introduction large language models (llms) demonstrate strong capabilities in solving a wide variety of reasoning problems that involve different formalisms such as formal logic, math, graph, and commonsense reasoning. as an example, recent llms achieve above 90% (figure 1 ) accuracy on math we find llms' accuracy drops significantly when tasked to solve problems with a non-trivial program (w/ prog.+); this further decreases when tasked to reason in the wild, where the tasks are blended with other contexts and their reasoning types are unknown (w/ hyb). ## related work program-aided llms. recent research has improved llms' reasoning capabilities with the help of programs, achieving better performance on math @xcite @xcite and logic @xcite @xcite @xcite reasoning tasks. these works are "hardwired" to solve a specific type of reasoning problem, and do not explicitly model the interactions with the environment as a trajectory, limiting their in-the-wild applications. trajectory-based llm agents agent-like llms models such as react @xcite explicitly model the interaction as a trajectory, and several works study its potential in solving reasoning problems. the most prominent ones in this space are fireact @xcite , tora @xcite , and eurus @xcite , which collect problem-solving trajectory data from gpt models in solving various known reasoning tasks and train local models, showing they can achieve better performance after fine- tactic name: predicate_logic_z3 problem type and tactic: this tactic builds a formal logical model using predicate logic using z3 lib … code template: action space: #a# plan input: the problem given functionality: give a plan for the question, include a sketch of the solution and libs to be used output: text description of the plan, code snippets #a# build fol model input: the problem given functionality: build the fol system … output: the main() function with z3 code #a# revise code … #a# aggregate and answer … import z3 def check_model(solver): … def main(): @xmath0.solver() <your @xmath1: example of a predicate logic tactic. ## reasoning in the wild the task of reasoning in the wild considers an llm agent solving reasoning problems by writing programs and interacting with an environment defined by a tactic. similar to prior agent-like llm frameworks @xcite , the problemsolving process consists of a series of thoughts, actions, and observations. at each step, the llm agent f llm generates a thought t that serves as a chain-of-thought (cot) context for its action, and an action a which carries out the actual action in the environment: this could be outputting a plan in free text (i.e., plan), generating a program (i.e., write program), or returning an answer (i.e., answer). then it receives observation o from the environment. the series of @xmath2, a, o⟩ (1) , ⟨t, a, o⟩ (2) , ...] forms a problem-solving trajectory e that will be included as prompt for the next round of interaction. different from the prior work, we extend this problem-solving trajectory by explicitly applying a tactic to specify and monitor the agent's behavior. formally, let @xmath3, t 2 , ...} be a tactic, and each tactic is defined by @xmath4, a, o} (figure 3 ). here, x t is the tactic description, which is a passage describing what this tactic is suitable for and how it is generally used; @xmath5, a 2 , ...} is the action space defining the allowable actions in this tactic, for example, plan and write program. each action is defined by its input, functionality, and output @xmath6, a func , a out }; and o is the tacticcorresponding observer that parses, monitors, executes the agent's action, and finally provides observations o. a complete tactic is shown in §a. let q be a reasoning problem and f llm be the llm agent, the agent solves the problem by performing two levels of reasoning: (1) at the top level, a routing trajectory is formed that picks the subproblems and the corresponding tactics; (2) at the second level, the chosen tactic is carried out, forming a "sub-trajectory" for subproblem solving. as shown in figure 4 : 1. routing: given the problem q, the agent executes the routing tactic, that is t r , where it extracts the subproblem q sub from q, and identifies the best tactic t to solve it by checking all the tactic descriptions included in t r and reflecting on the past experience e (where e (0) = ∅). at every (i)-th iteration: and then repeat step 1 until the answer action is called in the routing trajectory, where the final answer to problem q is aggregated and returned. by explicitly applying and monitoring the tactics and actions, we (1) make it easy to control the agent's behavior, defending against potentially undesired behaviors; (2) gain deeper insight into the llms, by tracing if it uses the right tactic for the right subproblems; (3) make it possible to create a challenging hybrid problem that requires multiple tactics to solve with fine-grained metrics. ## data generation to enable evaluating (and training) llms on our task, a large dataset of problem-solving trajectories over diverse reasoning problems needs to be created. to do so, we select several existing datasets and generate the trajectories with the specific tactic and icl prompts and select those successful ones with non-trivial programs (introduced shortly) to include in our dataset, rewild. tactics and datasets. we manually create a pool of diverse tactics, each corresponding to the following datasets: (1) math tactic for gsm8k; (2) logic tactic for folio, which is a logic-grounded natural language inference (nli) dataset; (3) graph tactic for proscript @xcite , which is a dataset containing graph-like steps for achieving certain goals such as "opening a bank account"; and (4) general program tactic for reclor, which is a commonsense reasoning dataset. to generate trajectories, we manually created examples and included them as icl prompts to guide the generation using a mixture of three models: gpt4-turbo-0409, gpt-4o, and claude-3-opus. more details in §b. post-processing. we filter the generated trajectories with the following steps: (1) we filter those that do not write any programs or those with programs that failed to run. this indicates the model "shortcuts" the process by directly outputting the answer. (2) a more severe shortcutting happens with reclor dataset for all the llms we tested. we refer to this as "trivial programs": reclor problems are typically ambiguous in scope and do not fit into any existing formalisms are known to be trained by these llms. when llms are "forced" to generate programs, they often generate programs that "hardcoded" the answer and put cot free-form reasoning in the comments. we filter them by labeling a set of programs and using them as icl prompts of an llm classifier (examples and details are provided in §b). in experiments, we show this phenomenon is widely present and causes a significant performance drop. ## experiments settings. we evaluate seven most powerful api llms to date listed in figure 5 (excluding claude-3-opus due to its high api cost), and two of our models tiger-pj-8b and tiger-ipj-8b on the standalone and hybrid problems of rewild. all api llms are given up to five trajectories as icl prompts (detailed settings in §d). ## conclusion we present reasoning in the wild, a task that challenges llms' reasoning capabilities in solving ambiguous and mixed-in-scope problems, where we create dataset rewild. our experiments reveal critical issues of the existing llms and show that they can be alleviated with fine-tuning, that is tiger-8b. during the creation of rewild, we used a combined approach of manual labeling and auto filtering. it is likely the trajectories, especially for reclor data, still contain "trivial programs". during evaluation, we use two different icl prompts for problem-solving trajectories to balance between cost and performance. while the total length of the message can go to 10k long, it does not exhaust all the context window length. that said, it is likely by providing more fully detailed trajectories, the api llms can achieve better performance than those reported in the paper.