Learning and Reasoning for Robot Dialog and Navigation Tasks (2020)

Abstract: Reinforcement learning and probabilistic reasoning algorithms aim at learning from interaction experiences and reasoning with probabilistic contextual knowledge respectively. In this research, we develop algorithms for robot task completions, while looking into the complementary strengths of reinforcement learning and probabilistic reasoning techniques. The robots learn from trial-and-error experiences to augment their declarative knowledge base, and the augmented knowledge can be used for speeding up the learning process in potentially different tasks. We have implemented and evaluated the developed algorithms using mobile robots conducting dialog and navigation tasks. From the results, we see that our robot’s performance can be improved by both reasoning with human knowledge and learning from task-completion experience. More interestingly, the robot was able to learn from navigation tasks to improve its dialog strategies.

## introduction knowledge representation and reasoning (krr) and reinforcement learning (rl) are two important research areas in artificial intelligence (ai) and have been applied to a variety of problems in robotics. on the one hand, krr research aims to concisely represent knowledge, and robustly draw conclusions with the knowledge (or generate new knowledge). knowledge in krr is typically provided by human experts in the form of declarative rules. although krr paradigms are strong in representing and reasoning with knowledge in a variety of forms, they are not designed for (and hence not good at) learning from experiences of accomplishing the tasks. on the other hand, rl algorithms enable agents to learn by interacting with an environment, and rl agents are good at learning action policies from trial-and-error experiences toward maximizing long-term rewards un-der uncertainty, but they are ill-equipped to utilize declarative knowledge from human experts. motivated by the complementary features of krr and rl, we aim at a framework that integrates both paradigms to enable agents (robots in our case) to simultaneously reason with declarative knowledge and learn by interacting with an environment. most krr paradigms support the representation and reasoning of knowledge in logical form, e.g., prolog-style. more recently, researchers have developed hybrid krr paradigms that support both logical and probabilistic knowledge @xcite @xcite . such logical-probabilistic krr paradigms can be used for a variety of reasoning tasks. we use p-log @xcite in this work to represent and reason with both human knowledge and the knowledge from rl. the reasoning results are then used by our robot to compute action policies at runtime. reinforcement learning (rl) algorithms can be used to help robots learn action policies from the experience of interacting with the real world @xcite . we use model-based rl in this work, because the learned world model can be used to update the robot's declarative knowledge base and combined with human knowledge. ## related work research areas related to this work include integrated logical krr and rl, relational rl, and integrated krr and probabilistic planning. logical krr has previously been integrated with rl. action knowledge @xcite has been used to reason about action sequences and help an rl agent explore only the states that can potentially contribute to achieving the ultimate goal @xcite . as a result, their agents learn faster by avoiding choosing "unreasonable" actions. a similar idea has been applied to domains with nonstationary dynamics @xcite . more recently, task planning was used to interact with the high level of a hierarchical rl framework @xcite . the goal shared by these works is to enable rl agents to use knowledge to improve the performance in learning (e.g., to learn faster and/or avoid risky exploration). however, the krr capabilities of these methods are limited to logical action knowledge. by contrast, we use a logicalprobabilistic krr paradigm that can directly reason with probabilities learned from rl. relational rl (rrl) combines rl with relational reasoning @xcite . action models have been incorporated into rrl, resulting in a relational temporal difference learning method @xcite . recently, rrl has been deployed for learning affordance relations that forbid the execution of specific actions @xcite . these rrl methods, including deep rrl @xcite , exploit structural representations over states and actions in (only) current tasks. in this research, krr-rl supports the krr of world factors beyond those in state and action representations, e.g., time in navigation tasks, as detailed in section 4.2. the research area of integrated krr and probabilistic planning is related to this research. logicalprobabilistic reasoning has been used to compute informative priors and world dynamics @xcite for probabilistic planning. an action language was used to compute a deterministic sequence of actions for robots, where individual actions are then implemented using probabilistic controllers @xcite . recently, human-provided information has been incorporated into belief state representations to guide robot action selection @xcite . in comparison to our approach, learning (from reinforcement or not) was not discussed in the abovementioned algorithms. finally, there are a number of robot reasoning and learning architectures @xcite @xcite , which are relatively complex, and support a variety of functionalities. in comparison, we aim at a concise representation for robot krr and rl capabilities. to the best of our knowledge, this is the first work on a tightly coupled integration of logical-probabilistic krr with model-based rl. ## background we briefly describe the two most important building blocks of this research, namely model-based rl and hybrid krr. ## krr-rl framework krr-rl integrates logical-probabilistic krr and model-based rl, and is illustrated in figure 1 . the krr component includes both declarative qualitative knowledge from humans and the probabilistic knowledge from model-based rl. when the robot is free, the robot arbitrarily selects goals (different navigation goals in our case) to work on, and learns the world dynamics, e.g., success rates and costs of navigation actions. when a task becomes available, the krr component dynamically constructs a partial world model (excluding unrelated factors), on which a task-oriented controller is computed using planning algorithms. human knowledge concerns environment variables and their dependencies, i.e., what variables are related to each action. for instance, the human provides knowledge that navigation actions' success rates depend on current time and area (say elevator areas are busy in the mornings), while the robot must learn specific probabilities by interacting with the environment. why is krr-rl needed? consider an indoor robot navigation domain, where a robot wants to maximize the success rate of moving to goal positions through navigation actions. shall we include factors, such as time, weather, positions of human walkers, etc, into the state space? on the one hand, to ensure model completeness, the answer should be "yes". human walkers and sunlight (that blinds robot's lidar sensors) reduce the success rates of the robot's navigation actions, and both can cause the robot irrecoverably lost. on the other hand, to ensure computational feasibility, the answer is "no". modeling whether one specific grid cell being occupied by humans or not introduces one extra dimension in the state space, and doubles the state space size. if we consider (only) ten such grid cells, the state space becomes 2 10 ≈ 1000 times bigger. as a result, rl practitioners frequently have to make a trade-off between model completeness and computational feasibility. in this work, we aim at a framework that retains both model scalability and computational feasibility, i.e., the agent is able to learn within relatively little memory while computing action policies accounting for a large number of domain variables. ## experiments in this section, the goal is to evaluate our hypothesis that our krr-rl framework enables a robot to learn from model-based rl, reason with both the learned knowledge and human knowledge, and dynamically construct task-oriented controllers. specifically, our robot learns from navigation tasks, and applied the learned knowledge (through krr) to navigation, dialog, and delivery tasks. we also evaluated whether the learned knowledge can be represented and applied to tasks under different world settings. in addition to simulation experiments, we have used a real robot to demonstrate how our robot learns from navigation to perform better in dialog. figure 3 shows the map of the working environment (generated using a real robot) used in both simulation and real-robot experiments. human walkers in the blocking areas ("ba") can probabilistically impede the robot, resulting in different success rates in navigation tasks. we have implemented our krr-rl framework on a mobile robot in an office environment. as shown in figure 3 , the robot is equipped with two lidar sensors for localization and obstacle avoidance in navigation, and a kinect rgb-d camera for human-robot interaction. we use the speech application programming interface (sapi) package ( http://www.iflytek.com/en ) for speech recognition. the robot software runs in the robot operating system (ros) @xcite . an illustrative trial on a robot: figure 4 shows the screenshots of milestones of a demo video, which will be made available given its acceptance. after hearing "a coke for bob to office2", the three sub-beliefs are updated (turn1). since the robot is aware of its unreliable speech recognition, it asked about the item, "which item is it?" after hearing "a coke", the belief is updated (turn2), and the robot further confirmed on the item by asking "should i deliver a coke?" it received a positive response (turn3), and decided to move on to ask about the delivery room: "should i deliver to office 2?" after this question, the robot did not further confirm the delivery room, because it learned through model-based rl that navigating to office2 is relatively easy and it decided that it is more worth risking an error and having to replan than it is to ask the person another question. the robot became confident in three dimensions of the service request (<coke,bob,@xmath0, because of the prior knowledge (encoded in p-log) about bob's office. figure 5 shows the belief changes (in the di-mensions of item, person, and room) as the robot interacts with a human user. the robot started with a uniform distribution in all three categories. it should be noted that, although the marginal distributions are uniform, the joint belief distribution is not, as the robot has prior knowledge such as bob's office is office2 and people prefer deliveries to their own offices. demo video is not included to respect the anonymous review process. learning to navigate from navigation tasks in this experiment, the robot learns in the shop-room1 navigation task, and extracts the learned partial world model to the shop-room2 task. it should be noted that navigation from shop to room2 requires traveling in areas that are unnecessary in the shop-room1 task. figure 6 presents the results, where each data points corresponds to an average of 1000 trials. each episode allows at most 200 (300) steps in small (large) domain. the curves are smoothed using a window of 10 episodes. the results suggest that with knowledge extraction (the dashed line) the robot learns faster than without extraction, and this performance improvement is more significant in a larger domain (the right subfigure). learning to dialog and navigate from navigation tasks robot delivering objects requires both tasks: dialog management for specifying service request (under unreliable speech recognition) and navigation for physically delivering objects (under unforeseen obstacles). our office domain includes five rooms, two persons, and three items, resulting in 30 possible service requests. in the dialog manager, the reward function gives delivery actions a big bonus (80) if a request is fulfilled, and a big penalty (-80) otherwise. general questions and confirming questions cost 2.0 and 1.5 respectively. in case a dialog does not end after 20 turns, the robot is forced to work on the most likely delivery. the cost/bonus/penalty values are heuristically set in this work, following guidelines based on studies from the literature on dialog agent behaviors @xcite . 0 1 2 3 4 5 6 7 episodes (x10) -200 -150 -100 -50 0 reward 30x30 grid w/o extraction w/ extraction 0 1 2 3 4 5 6 7 8 9 10 11 12 episodes (x10) -350 -300 -250 -200 -150 -100 -50 0 50 reward 50x50 grid w/o extraction w/ extraction figure 6: navigation tasks in small (left: 30 × 30 grid) and large (right: 50 × 50 grid) domains. with extraction (krr-rl in dashed line), the robot learns faster in the target navigation task. ## conclusions and future work we develop a krr-rl framework that integrates computational paradigms of logical-probabilistic knowledge representation and reasoning (krr), and model-based reinforcement learning (rl). our krr-rl agent learns world dynamics via modelbased rl, and then incorporates the learned dynamics into the logical-probabilistic reasoning module, which is used for dynamic construction of efficient run-time task-specific planning models. experiments were conducted using a mobile robot (simulated and physical) working on delivery tasks that involve both navigation and dialog. results suggested that the learned knowledge from rl can be represented and used for reasoning by the krr component, enabling the robot to dynamically generate task-oriented action policies. the integration of a krr paradigm and modelbased rl paves the way for at least the following research directions. we plan to study how to sequence source tasks to help the robot perform the best in the target task (i.e., a curriculum learning problem within the rl context @xcite ). balancing the efficiencies between service task completion and rl is another topic for further study -currently the robot optimizes for task completions (without considering the potential knowledge learned in this process) once a task becomes available. fundamentally, all domain variables are endogenous, because one can hardly find variables whose values are completely independent from robot actions. however, for practical reasons (such as limited computational resources), people have to limit the number of endogenous. it remains an open question of how to decide what variables should be considered as being endogenous.