Multi-step or Direct: A Proactive Home-Assistant System Based on Commonsense Reasoning (2025)

Abstract: There is a growing expectation for the realization of proactive home-assistant robots that can assist users in their daily lives. It is essential to develop a framework that closely observes the user’s surrounding context, selectively extracts relevant information, and infers the user’s needs to proactively propose appropriate assistance. In this study, we first extend the Do-I-Demand dataset to define expected proactive assistance actions in domestic situations, where users make ambiguous utterances. These behaviors were defined based on common patterns of support that a majority of users would expect from a robot. We subsequently constructed a framework that infers users’ expected assistance actions from ambiguous utterances through commonsense reasoning. We explored two approaches: (1) multi-step reasoning using COMET as a commonsense reasoning engine, and (2) direct reasoning using large language models. Our experimental results suggest that both the multi-step and direct reasoning methods can successfully derive necessary assistance actions even when dealing with ambiguous user utterances.

## introduction with recent advancements in language understanding capabilities enabled by large language models (llms), task-oriented domestic robots and systems that assist users based on natural language instructions are becoming a reality. such systems interpret language (=user utterances) and map them to system actions that the system should perform, then determine an action plan based on feasible capabilities and surrounding situations @xcite @xcite . the task of mapping language instruction to system actions is known as natural language understanding (nlu), and it has long been a central challenge in this field @xcite . in nlu tasks, the user's intent is assumed to be explicit, and the primary focus has been on how to call the appropriate system action api in response to the user's utterance @xcite . however, real-world user utterances are often more diverse and ambiguous @xcite @xcite . in such cases, the system must infer the user's potential needs or implicatures from the utterance and proactively propose supportive actions @xcite . for example, when the user says, "thank you for the meal," it can be inferred that the user has finished eating. ideally, the system should proactively suggest actions, such as "shall i clear the plates on the table?," before the user explicitly makes a request. several challenges arise when the system tries to infer the user's potential needs from their utterances. a user's potential needs can vary greatly depending on the situation, and few proactive actions are universally acceptable to all users. the system must consider several action candidates based on commonly expected or socially appropriate actions, and select the most suitable one tailored to the users @xcite . moreover, when considering the appropriate proactive action in response to an ambiguous user utterance, it is often crucial to clarify the underlying assumptions behind the utterance. in this process, multi-step reasoning based on commonsense reasoning, such as comet, is practical @xcite @xcite . however, multi-step reasoning without a clearly defined goal can require many inferences to reach an appropriate conclusion, and it requires a method for re-evaluating the diverse hypotheses generated during this process. in response to these challenges, a method known as chain-of-thought (cot) has recently gained attention @xcite . cot involves providing llms with examples of rea-soning steps, allowing them to perform inference passes @xcite . when using cot to directly estimate the user's potential intent, explicit inference rules are not provided; however, the llm can suggest plausible reasoning paths leading to the inferred conclusion. compared to traditional multi-step reasoning approaches, cot is expected to reduce computational cost regarding the number of inferences. in this study, we develop a framework of proactive home-assistant robots by leveraging different reasoning methods. we extend the existing do-i-demand dataset @xcite by annotating robots' support actions that are generally expected, i.e., commonsensically appropriate, by most people in response to specific user situations within a domestic setting. using this dataset, we build an action reasoning system of proactive actions that robots should take. we compare two approaches: multi-step reasoning based on comet and direct reasoning based on llm, which are prompted by chain-of-thought (cot) examples. experimental results indicate that each method has its strengths and limitations, suggesting that integrating these approaches may improve accuracy in a proactive home-assistant system. ## conclusion in this study, we developed a framework for estimating appropriate support actions for domestic assistive robots, based on ambiguous user utterances and contextual information, to realize reflective/considerate actions that users expect from such systems. we first re-evaluated and extended the existing do-i-demand dataset. by relabeling the original single reflective/considerate action with a set of actions that are widely agreed upon as appropriate by multiple annotators, we enabled more flexible evaluation. our dataset reflects the real-world assumption that user expectations are not always uniquely determined by utterances. we also proposed two distinct reasoning approaches to predict such reflective/considerate actions. the multi-step reasoning method used a commonsense inference system (comet) to generate related events from the utterance and infer actions. the direct inference approach using llms directly predicted actions by jointly processing the utterance and contextual information. experimental results demonstrate that both proposed methods outperformed the similarity-based baseline. the multi-step reasoning method outperformed gpt-3.5, which uses a knowledge base comparable to comet. when we use the multi-step reasoning approach, it is crucial to carefully select the knowledge fed into the reasoning process, determine which relations to use, and design effective pruning strategies for generated event candidates. the selection method may also be improved through enhanced similarity measures. since the information in the real world that needs to be attended to varies depending on the target action, future work should explore methods that better use knowledge specifically related to the action candidates.