THREAD : Thinking Deeper with Recursive Spawning (2025)

Abstract: Large language models (LLMs) have shown impressive capabilities across diverse settings, but still struggle as the length and complexity of the context increases. To address this challenge, we propose Thinking Recursively and Dynamically (ThReaD). THREAD frames model generation as a thread of execution that, based on the context, can run to completion or dynamically spawn new threads. By spawning, threads can offload work (e.g., thinking, retrieving information) to child threads, which only return tokens needed for the parent thread to do its work. We apply THREAD in the settings of LLM task solving and question answering, where the dynamic threading allows the model to recursively decompose the given task or question into progressively simpler sub-problems that can be solved by separate child threads. We test THREAD, implemented using a few-shot learning approach, on diverse benchmarks for agent tasks and data-grounded question answering. THREAD achieves state-of-the-art performance with GPT-4 and GPT-3.5 on these benchmarks, including ALFWorld, TextCraft, and WebShop, along with two new benchmarks, DataCommons QA and MIMIC-III ICU QA. In addition, THREAD outperforms existing frameworks by 10% to 50% absolute points with smaller models, including Llama-3-8b and CodeLlama-7b.

## introduction large language models (llms) have shown success in diverse settings @xcite , but their performance degrades as context length and complexity grows @xcite @xcite . this constraint limits their efficacy in settings that require more work @xcite than can fit into a concise line of correspondence to philip schroeder at pschro@mit.edu. source code is available at https://github.com/ philipmit/thread . thread is a general framework where model generation is treated as a thread of execution that, based on the context, can independently run to completion or dynamically spawn new threads in a recursive fashion. when a thread spawns a child, the child generates conditioning on context that derives from the parent's token sequence. spawning child threads allows work, such as internal thinking or interacting with an external environment, to be completed on behalf of the parent, without directly adding to the parent's context. child threads return only the information (tokens) needed for the parent to complete its work. in effect, spawning enables the model to dynamically adapt the amount of work or intermediate computational steps used to produce different parts of its token sequence. the synchronization and spawning mechanisms of thread can vary based on the setting. figure 1 shows an example of applying thread in a synchronous setting, where a parent waits for a child in a form analogous to thread.join() in multithreaded programming. in this example, a parent thread pauses generation until the child execution completes and the child returns output tokens that are appended directly to the token sequence of the parent before the parent proceeds generation. thread improves the flexibility of model generation, allowing the model to adapt, through recursive spawning, the amount of work it does based on the given problem, without overextending its context. importantly, this unfolds with limited need for explicit rules or hard coded logic. instead, thread relies on the model's ability to infer the appropriate continuation given the context of each thread at the time of execution. finally, thread is agnostic to the type of sequence. depending on the underlying model, thread can be applied in varied settings, including multi-modal applications, and fulfill varied purposes (e.g., carrying out calcu- thread 0 ├─ thread 1 │ ├─ thread 1.1 │ └─ thread 1.2 ... ... in the example with join synchronization, when a thread spawns a child, it pauses generation until feedback is returned. child threads generate starting from context derived from their parent's token sequence. when the child completes, it returns output tokens (colored bars), which are added to the context of the parent before it continues generating. ϕ and ψ are functions that control information flow from parent to child and from child to parent, respectively, by defining the tokens that are propagated based on the thread's token sequence. in this paper, we consider thread in the settings of agent tasks (figure 2a ) and question answering (figure 2b ). in these settings, thread enables the llm to recursively decompose a task, or question, into progressively simpler sub-problems that can be solved by child threads in a compartmentalized manner. a child thread infers the specifications of its sub-problem from its parent's token sequence. if needed, child threads can troubleshoot their sub-problem, and spawn their own threads, all without distracting their parent. we test thread, using a few-shot learning approach, on benchmarks consisting of question-answering and agent tasks. for these problems, we apply thread with join synchronization, allowing parent threads to see feedback from child threads before defining their next step. for example, in the agent setting, if the model is given the task of cleaning a bowl and putting it on the countertop, the main thread may spawn a child with the sub-task of finding a bowl. this child can then spawn further threads for handling the complexities of navigating to and checking different locations for the bowl, with each thread returning only the information needed for its parent to proceed with its sub-task. based on the feedback with regard to finding the bowl, the main thread can spawn a new thread to re-attempt the sub-task or spawn threads to execute the remaining sub-tasks (i.e., washing the bowl, putting the bowl on the countertop). thread has several advantages over existing frameworks in these settings. to start, thread addresses limitations of methods that require the model to solve the problem in one line of context @xcite @xcite ) by allowing the model to dynamically offload work by spawning child threads. further, unlike methods for task decomposition @xcite @xcite @xcite , thread enables the model to adapt its decision-making in real time as it receives feedback from each step, as depicted in figure 2 . existing methods either do not allow for plan adaptation @xcite or only adapt plans by calling pre-existing sub-task handlers @xcite or by adding more sub-steps within the existing plan when a sub-task fails @xcite . finally, thread provides a more unified framework compared to these methods, which require separate prompting mechanisms for separate planner and executor modules. instead, thread can be implemented with the same few-shot prompt used for every thread at every step of the task completion. we evaluate thread on agent tasks and datagrounded question answering. thread significantly outperforms prior methods, achieving stateof-the-art performance with gpt-4 and gpt-3.5 on alfworld, textcraft, and webshop, along with two new benchmarks, datacommons qa and mimic-iii icu qa. thread also shows success with smaller models, outperforming prior methods by 10% to 50% absolute points across the benchmarks with llama-3-8b and codellama-7b. ## thread framework thread frames model generation as an execution thread that, given the context, can run to completion or spawn new threads. a thread consists of generating with a model, g, given a specific starting context, c. for the main thread, c is the initial seed context, c 0 (e.g., context provided by a user). for each child thread, c derives from the parent's token sequence. a thread continues until it meets some termination criteria, such as a stop token. ## applying thread for agent tasks and question answering in this paper, we test thread in the settings of question answering and agent tasks with llms, where thread enables the model to dynamically decompose the given problem into simpler subproblems that are completed by separate threads. we apply thread using an in-context learning approach, where the same few-shot prompt is used for every thread at every step of the task completion or question answering. since these problems benefit from the model adapting to feedback as it defines its next steps, we apply thread with the join synchronization described above. thread spawning and termination based on special stop tokens. we implement a spawning mechanism using a special stop token, ω listen , which pauses the thread generation and marks the start of the output the thread expects from the child. the context for the child thread is defined by ϕ based on the token sequence of the parent that occurs before ω listen . a thread ends when it generates the end token, ω end . as exemplified in figure 2 , we @xmath0. implementing thread with few-shot learning. we leverage a few-shot learning approach to implement thread. the few-shot examples are comprised in a single prompt with examples of successful spawning and problem solving at different thread depths, including how the token ω listen should be used. as a result, the approach does not require explicit rules to define when new threads should be spawned and how parent threads should respond to feedback from a child. instead, these mechanisms are all implied by the examples provided in the few-shot prompt. changing the spawning and parent-child dynamics simply requires modifying the prompt. all threads are provided the same few-shot prompt for every step of the problem completion. thus, thread only requires generating a single prompt for a given setting. this prompt, q, is prepended to the context for each thread, g(q + c + y ), forming the full context from which the thread will generate. defining ϕ and ψ. as described above, the functions ϕ and ψ control information flow from parent to child and from child to parent, respectively. the function ψ returns the result from a child thread based on its full token sequence. as depicted in fig -ure 2, we implement ψ as a function that returns the tokens included in the print statement at the end of the child thread's token sequence. in addition, ψ appends the @xmath1. we implement ϕ as a function that extracts the last line of the parent's token sequence to create the context for the child thread. to improve the ability of parent and child threads to efficiently organize, use, update, and propagate shared information, we leverage the coding-based skills of the model to define variables that represent pieces of information that are important for the given problem. for example, if the task is to find an item on an e-commerce site that has certain attributes, the model can define a variable, such as obj_attributes, and spawn a child to search for items with obj_attributes instead of having to list all of the attributes. then, when ϕ processes the token sequence from the parent, it instantiates the variables, allowing the child to see, for example, the full list of attributes. we also test thread implemented without using these variables, with results shown in the appendix. threads performing actions in an environment. threads can listen for feedback when performing an action in an environment the same way they listen for feedback from a child thread. they simply generate tokens that represent the action and then produce the token ω listen to listen for feedback from the environment. the action, a, is then executed in the environment, e, and the output, @xmath2, is appended to the thread's token sequence the same way it is done for the feedback from a child thread (algorithm 2). the thread can then continue generating based on the feedback. ## experiments we evaluate thread on 5 benchmarks for agent tasks and question answering using large and small models, including gpt-4, gpt-3.5, llama-3-8b, llama-2-7b, and codellama-7b. for each benchmark, we use the same prompt for all models. we include details regarding models, prompts, and experiments (including ablations) in the appendix. examples of the prompts are shown in appendix g. details regarding the qa benchmarks are provided in appendix f. we release all code and data at https://github.com/philipmit/ thread . ## conclusion we introduce thread, a general framework in which model generation is treated as a thread of execution that can dynamically offload work, such as thinking or retrieving information, by spawning new threads. thread enables a model to adapt, through recursive spawning, the amount of intermediate work it uses to produce different parts of its token sequence. we apply this framework in the settings of agent tasks and question answering. we show that thread, implemented using a fewshot learning approach, achieves state-of-the-art performance on diverse benchmarks.