R e S p A ct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents (2025)

Abstract: Large language model (LLM)-based agents have been increasingly used to interact with external environments (e.g., games, APIs, etc.) and solve tasks. However, current frameworks do not enable these agents to work with users and interact with them to align on the details of their tasks and reach user-defined goals; instead, in ambiguous situations, these agents may make decisions based on assumptions. This work introduces ReSpAct (Reason, Speak, and Act), a novel framework that synergistically combines the essential skills for building task-oriented “conversational” agents. ReSpAct addresses this need for agents, expanding on the ReAct approach. ReSpAct framework enables agents to interpret user instructions, reason about complex tasks, execute appropriate actions and engage in dynamic dialogue to seek guidance, clarify ambiguities, understand user preferences, resolve problems, and use the intermediate feedback and responses of users to update their plans. We evaluated ReSpAct with GPT-4 in environments supporting user interaction, such as task-oriented dialogue (MultiWOZ) and interactive decision-making (Alfworld, WebShop), ReSpAct is flexible enough to incorporate dynamic user feedback and addresses prevalent issues like error propagation and agents getting stuck in reasoning loops. This results in more interpretable, human-like task-solving trajectories than baselines relying solely on reasoning traces. In two interactive decision-making benchmarks, AlfWorld and WebShop, ReSpAct outperforms strong reasoning-only method ReAct by an absolute success rate of 6% and 4%, respectively. In the task-oriented dialogue benchmark MultiWOZ, ReSpAct improved Inform and Success scores by 5.5% and 3%, respectively.

## introduction instruction-following is a fundamental capability for intelligent agents operating in real-world environments. recent works such as @xcite @xcite figure 1 : respact is a framework for task-oriented conversational agents that allows agents to ask questions, request feedback, and adapt their strategies based on user input. effective conversational agents go beyond clarifying ambiguities-they actively collaborate with users by offering alternative suggestions, providing status updates, and following up on requests to ensure alignment with user goals. for example, when asked to "go to the kitchen and bring me the pan," an agent can confirm which pan is needed if multiple options exist, suggest alternatives if the desired pan is unavailable, and update the user on progress. similarly, when tasked with "arrange a trip to hawaii," the agent can verify key details, propose travel options based on preferences, and keep the user informed throughout the process. this dynamic interaction enables agents to adapt to evolving user needs, ensuring tasks are @xcite . we show only the task-solving trajectories generated by the model (act, thought and speech) and the environment (obs). existing reasoning and decision-making approaches for language agents augment the agent's action space with a language model, allowing the agent to generate free-form thoughts in natural language that help contextualize and reason about the task at hand. by alternating between task-solving actions and language thoughts, these agents can perform multi-step reasoning and compose useful information for solving complex tasks. however, such frameworks do not explicitly incorporate user interaction and feedback into the agent's reasoning process. in real-world scenarios, dynamic engagement with users is critical not only for clarifications and guidance but also for addressing incomplete task specifications, exploring alternative solutions, and achieving user-defined goals. in this paper, we propose respact, a framework for task-oriented conversational agents that allows the agent to actively engage with users through dialogue actions. by introducing a new action space for user interaction, the agent can work collaboratively with users through free-flowing active dialogue, and incorporate user responses into its evolving context. this human-in-the-loop approach enables the agent to leverage user insights, adapt to user preferences, and refine its task-solving strategy based on user input. the respact framework, as illustrated in fig. 1 , allows llm-based agents to fully utilize their conversational capabilities by dynamically switching between reasoning about the task, speaking to the human interlocutor, and taking actions in the environment. figure 2 shows an example from alf-world setting, contrasting react with respact interactions. in the second turn, the agent cleverly asks the user the possible location of a cloth, making the task easier for itself. unlike static settings @xcite , respact incorporates a range of conversational styles, well beyond asking clarifying questions, in dynamic interactive settings detailed in our key contributions are as follows: • we introduce the respact,a novel framework to dynamically integrate reasoning, decisionmaking, and conversational capabilities, building upon and expanding the react method. ## related work logical reasoning in language models often involves breaking down complex inputs into intermediary steps to achieve a final goal, as shown by @xcite and its variants @xcite @xcite . however, these methods are prone to error propagation, where mistakes in earlier steps compound as the sequence length increases @xcite . iterative refinement methods @xcite @xcite aim to address these issues but often neglect the critical role of human feedback. respact mitigates this by enabling agents to engage in dialogue with users, seeking feedback and guidance to prevent cascading errors. llms have also been adapted for decisionmaking tasks, serving as high-level policy models in robotics @xcite @xcite and excelling in text-based environments like web navigation @xcite @xcite . techniques such as react @xcite integrate reasoning and action, while some approaches incorporate limited dialogue for decision-making @xcite . unlike these, respact seamlessly integrates reasoning, action, and dialogue, enabling agents to fluidly transition between these modes for more effective decisionmaking in complex, interactive environments. previous works @xcite @xcite highlight that communication skills enhance autonomous embodied agents' reliability by leveraging human knowledge in collaborative tasks. approaches like @xcite @xcite improve question-asking in static and embodied settings respectively. respact extends these works by creating a unified framework for reasoning, speaking, and acting in dynamic, interactive settings. while @xcite , focuses on proactive information-gathering to resolve initial ambiguities before decision-making, respact agent's conversations go beyond that and demonstrate their utility in multiple task settings. recent work in conversational systems has explored using llms in task-oriented dialogues (tod) through fine-tuning @xcite @xcite and in-context learning @xcite . @xcite examines instruction-finetuned llms in multi-turn dialogues, while @xcite use prompting schemas to build autonomous agents. however, these approaches struggle to interpret instructions, resolve ambiguities, and act appropriately. ## respact: reason + speak + act in interactive settings consider a setup where an agent can interact with an environment to perform tasks and achieve specific goals. when the agent operates in these environments, at each time step t, it receives an observation o t from the environment, where o @xmath0. then it executes an action a t based on its policy π, where a @xmath1. the policy π is a function that maps the agent's current context c t to an action a t . formally, we can define this policy as π : @xmath2. the context c t encapsulates the relevant information available to the agent at time step t, including the current observation and the history of previous observations and actions: as highlighted in @xcite , learning the optimal policy can be challenging, especially when the mapping from the context to the appropriate action is highly implicit and requires extensive computation. in complex real-world scenarios, the agent may need to reason about the task, consider multiple factors, and handle incomplete or ambiguous information. therefore, augmenting the agent's action space â = a ∪ l allows it to think by taking action ât ∈ l from the language space. these actions represent the agent's internal thought process or reasoning and can implicitly affect the environment by influencing the agent's actions, but they do not have any explicit external effect on the environment. the user can visually inspect these thoughts for interpretation but cannot influence them. in respact, we redefine these actions ât ∈ l from the language space, such that they are not only useful for the internal reasoning of the agent, but can also be used for external communication by generating dialogue actions for conversing with users. when the agent selects a dialogue action u @xmath3; u ⊂ l, it generates a natural language utterance to communicate with the user, such as asking a clarifying question, requesting feedback, or engaging in dialogue. the user response r t is then added to the agent's observations, allowing the agent to incorporate explicit feedback, clarification, and information from the user into its evolving context c t to inform subsequent thoughts, actions, and dialogue. engaging in dialogue also allows the agent to explain its reasoning, build rapport, and gain insights from the user's domain knowledge. the agent can share its current thoughts and future plans, e.g., "based on [context], i'm considering @xcite . what do you think of this direction?", allowing the user to provide feedback and steer the agent's task-solving process. by alternating between environment actions a, language thoughts l, and dialogue actions u, the agent interleaves tasksolving reasoning with targeted human interaction. the dialogue history becomes an important part of the context for language thoughts and environment actions, allowing human feedback to shape the agent's task-solving trajectory over multiple thought-action-observation steps. ## experimental setup in our experiments, we evaluate respact across multiple task-oriented decision-making environments, employing a human-in-the-loop approach to demonstrate its versatility. the agent is tested on multi-step tasks in common household environments using alfworld @xcite , tasked with making reservations in the multiwoz dialogue setup @xcite , and instructed to purchase products in webshop @xcite . we use react as a baseline for comparison, a reasoning-only approach. for these experiments, we focus on frozen gpt models, particularly gpt-4o @xcite , which is prompted with few-shot exemplars. these exemplars guide the model in generating a mix of domain-specific actions, free-form reasoning ("thoughts"), and dialogue actions interleaved throughout task execution. the in-context examples provided (see appendix e) contain dense sequences of actions interspersed with sparse thoughts and dialogue actions at relevant points. to scale our experiments, we implement a user simulator in each environment, which plays a critical role in replicating user interactions. ## conclusions respact framework enables dynamic, contextaware interactions that extend beyond basic command-response exchanges. by fostering meaningful dialogue, this framework allows ai agents to not only explain their decision-making processes but also adapt their actions in response to user feedback, transforming them into truly "conversational" agents. such capabilities are crucial for creating more intuitive, trustworthy, and effective ai assistants that can operate in complex, real-world scenarios. one can also incorporate stateful policies in respact for higher precision, such as asking to confirm all arguments of reservations before finalizing them, or using a particular api for action depending on the current state, similar to following a dialogue flow. this is important for policy alignment of llms for task-completion.