Semantic-Aware Action Space Compression via LLM - DRL Synergy for Efficient Task-oriented Dialogue Policy Exploration (2025)

Abstract: The flexibility of natural language significantly expands the action space in task-oriented dialogue systems, causing inefficient exploration and slow convergence in deep reinforcement learning (DRL)-based policy optimization. Pre-trained large language models (LLMs), with world knowledge and semantic understanding, offer promising solutions. To this end, we propose LLM-Guided DRL via Semantic-Aware Action Pruning (LLMSAP), a novel framework that synergizes pretrained LLMs with DRL. LLMSAP leverages the world knowledge and contextual understanding of LLMs to guide decision-making via an action feasibility assessment. Instead of requiring LLMs to directly generate optimal actions due to their limited precision in sequential decision tasks, LLMSAP employs a lightweight action pruning mechanism. Specifically, LLMs act as action filters, rapidly eliminating semantically implausible or low-potential actions from multi-turn dialogue context, allowing the DRL agent to focus exploration on a refined candidate subset. This two-stage framework (“prune-then-optimize”) avoids extensive LLM fine-tuning while preserving the decision-making precision of DRL. Experiments on multiple benchmarks verify the effectiveness of LLMSAP.

## introduction task-oriented dialogue (tod) systems achieve user goals via multi-turn interactions @xcite , where dialogue policy (dp), as the core component, selects appropriate actions to steer the conversation. deep reinforcement learning (drl) has emerged as the dominant approach for dp optimization due to its strength in sequential decision-making @xcite . however, natural language flexibility induces exponentially large action spaces @xcite . the resulting high dimensionality and semantic ambiguities in actions not only hinder * corresponding author efficient exploration @xcite but also destabilize policy optimization through biased value estimation @xcite , degrading drl's generalization in complex tod scenarios. recent advances in large language models (llms), which acquire powerful world knowledge and multi-turn semantic awareness through massive corpus pretraining, offer promising solutions to the above challenges @xcite . studies show that llms could deeply model implicit semantic relationships in dialogue contexts and infer relevant system action sets accordingly @xcite . this capability offers a theoretical foundation for developing semantic-aware action pruning modules, enabling semantically-guided drl exploration in expansive action spaces. despite their potential, deploying llms as endto-end decision-makers for dialogue policy guidance faces dual challenges: (1) untuned llms lack alignment with tod-specific reward signals, making it difficult to optimize long-term action rewards @xcite ; 2) large-scale finetuning for specific tasks is hindered by the scarcity of annotated tod data and high computational costs @xcite . to bridge these gaps, we propose llm-guided drl via semantic-aware action pruning (llm-sap), a two-stage "prune-then-optimize" framework. it first leverages llms to assess action feasibility based on dialogue context, pruning semantically inconsistent or low-potential actions to compress the action space. then, drl performs finegrained exploration over the pruned action subset, optimizing action sequences via long-term reward maximization. this design avoids costly llm finetuning while harnessing its role as a semantic filter, balancing drl policy optimization accuracy with exploration efficiency. to the best of our knowledge, this is the first study to integrate llm with drl to dialogue policy optimization. in summary, our contributions are threefold: ## related work in drl research, llms typically serve as information processors, reward designers, decision-makers, or generators @xcite . nevertheless, existing studies predominantly concentrate on gaming environments, exhibiting scant exploration of tod systems. due to fundamental differences in task specifications (discrete vs. continuous action spaces) and interaction patterns (turn-based vs. game dynamics), gaming approaches are ill-suited for tod. this paper investigates analogous llmas-decision-maker methods in related fields, categorizing them into action decision-making and action guidance paradigms, to inform our exploration of llm-drl integration for tod. for decision-making, recent advances have explored the use of llms through two main paradigms: policy initialization and sequence modeling-based decision making. the policy initialization paradigm utilizes pre-trained llms to provide strong priors for drl @xcite . in contrast, the sequence modeling paradigm reframes decision-making as a conditional generation problem, typically implemented via decision transformers @xcite . however, these approaches share a fundamental limitation: dialogue policy learning is a long-term task, and llms without task-specific fine-tuning often fail to generate an optimal sequence of actions @xcite . for action-guiding, llms do not generate actions directly, but act as guides, producing a condensed set of candidate or expert actions. @xcite proposed the instructrl framework, which utilizes pre-trained llms to generate a priori strategy distributions based on linguistic cues to guide policy learning. however, instructrl relies on llms to generate strategy distributions directly, which has significant limitations in long sequence tasks due to cumulative bias propagation and failure to maintain temporal consistency in action sequences. meanwhile, drl agents are subordinate and lack the autonomy to correct policy deviations, which may lead to unstable convergence or even failure, especially in complex and dynamic dialogue tasks @xcite . in summary, direct policy initialization or action generation with llms suffers from limited adaptability and long-term optimization in tod scenarios. in contrast, llmsap utilizes llms for semantic-level action pruning, avoiding their limitations in long-horizon decision-making while retaining the strengths of drl, thus significantly enhancing efficiency in complex dialogue tasks. ## methodology llmsap comprises two stages: (1) llm-driven semantic action pruning: leveraging the powerful semantic reasoning and world knowledge of pre-trained llms, this stage filters out semantically inconsistent or low-potential actions, thereby significantly reducing the effective action space; ## experiments we conducted experiments on three datasets from the microsoft dialogue challenge platform @xcite : movie ticket booking, restaurant reservation, and taxi booking. each domain has its domain-specific intents, slots, and labeled dialogues, and the statistics are shown in ## conclusion this study introduces the llmsap architecture, a novel framework that synergizes llms with drl to enhance exploration efficiency in task-oriented dialogue policy optimization. the framework initiates by conducting semantic interpretation of the ongoing dialogue context and action space, where an llm is employed to eliminate redundant or semantically inconsistent actions, thereby generating a streamlined and high-purity action subset. drl is subsequently utilized to execute exploration and policy refinement within this pruned action space. cross-domain experiments spanning multiple scenarios reveal that llmsap surpasses standalone drl and llm methods in both exploration efficiency and convergence velocity. its consistent performance across diverse llm variants further underscores its robust generalizability. to the best of our knowledge, this is the first study to integrate llm with drl to dialogue policy optimization.