S tructu T hink: Reasoning with Task Transition Knowledge for Autonomous LLM -Based Agents (2025)

Abstract: Decision-making tasks have highlighted fundamental challenges in grounding decisions within real-world contexts. Traditional decision knowledge utilization methods often struggle to effectively integrate structured decision constraints, limiting their ability to decompose high-level tasks, maintain logical consistency, and adapt to dynamic environments. To bridge this gap, we introduce StructuThink, a knowledge-structured reasoning framework that enhances LLM-based agents with explicit decision constraints. Specifically, we propose the Task Transition Knowledge Graph (TTKG) that learning decision knowledge in embodied scenarios. Leveraging this knowledge, we propose the StructuThink framework, comprising a subtask chain constructor for grounding natural language instructions and a constraint-based executor for adaptive and consistent decision-making. We validate StructuThink across multiple benchmarks, including ALFWorld and WebShop, where it achieves higher task success rates (improving by up to 7%) and more efficient action sequences (requiring up to 15% fewer steps) than baseline methods. Our approach enables LLMs to more effectively ground decision-making in domain-specific scenarios, enhancing both interpretability and reliability, thus paving the way for more reliable and adaptable decision-making systems.

## introduction large language models (llms) exhibit strong language understanding and generation abilities, making them promising foundations for autonomous agents in complex environments @xcite . however, their decision-making capabilities remain limited due to the lack of domainspecific training and the disconnect between highlevel linguistic reasoning and low-level environmental affordances @xcite , 2023). this gap often results in failures to decompose tasks or generate context-aware actions @xcite . to mitigate this, researchers have explored incorporating scene knowledge into llms @xcite , which can be categorized as perceptual knowledge (e.g., object types, spatial relations) and apperceptive knowledge (e.g., commonsense, affordances, intentions). in this work, we focus on integrating the decision-related aspects of apperceptive knowledge to guide task execution more effectively. existing efforts to inject apperceptive knowledge into llms primarily fall into two categories: unstructured and structured representations @xcite . unstructured knowledge (e.g., natural language instructions, expert demonstrations) provides flexible and diverse representations, typically learned via in-context prompting @xcite . however, the limited con-text window of llms @xcite , combined with the logical vagueness of natural language, often leads to decision hallucinations @xcite and reasoning inconsistencies @xcite , making such methods unreliable for complex decision-making. in contrast, structured knowledge leverages explicit formats like logic rules or graphs to represent task states and dependencies @xcite , offering better logical clarity and grounding. for example, arigraph @xcite constructs semantic-situational memory graphs, while knowagent @xcite constrains action spaces using logical rules. yet, these approaches face a trade-off: methods like pddl provide adaptability via real-time feedback but require costly expert-defined logic @xcite , whereas lower-cost tools (e.g., policy scripts, knowledge graphs) are easier to build but lack dynamic adaptation. thus, how to efficiently incorporate structured decision knowledge into llms without sacrificing flexibility remains an open challenge. inspired by hierarchical task networks (htns), we propose a method that utilizes subtask knowledge to construct a decision constraint graph, which imposes low-cost constraints on llm-agent decision-making. this approach aims to bridge the gap between high-level natural language understanding and grounded task execution. to achieve this, we address the following three key questions: q1. how can decision knowledge be leveraged to generate task-completable constraints from natural language instructions? q2. how can llms be guided to accomplish tasks by adhering to constraint-based task chains? q3. how can decision knowledge be generalized across scenarios by learning from expert trajectories? specifically, our key contributions are as follows: ## methods we draw inspiration from hierarchical task networks (htns), which decompose complex tasks into structured subtasks and capture the transition relationships among them. we observe that interactive task scenarios often exhibit an inherent hierarchical structure, where the task space can be abstracted as a sequence of subtasks connected by transition dependencies. leveraging the strong semantic understanding capabilities of llms, we propose to extract subtask-level decision knowledge-i.e., the subtask units and their transitions-from expert trajectories. these are stored in a task transition knowledge graph (ttkg), where nodes represent subtasks described in natural language, and edges capture learned transi-tion knowledge between them. during decisionmaking, ttkg is first used to ground high-level natural language instructions into executable subtask chains. these chains then serve as dynamic constraints to guide the llm-agent's step-by-step decisions, ensuring that its behavior aligns with task structure and prior expert knowledge.