Interactive Reinforcement Learning for Table Balancing Robot (2021)

Abstract: With the development of robotics, the use of robots in daily life is increasing, which has led to the need for anyone to easily train robots to improve robot use. Interactive reinforcement learning(IARL) is a method for robot training based on human–robot interaction; prior studies on IARL provide only limited types of feedback or require appropriately designed shaping rewards, which is known to be difficult and time-consuming. Therefore, in this study, we propose interactive deep reinforcement learning models based on voice feedback. In the proposed system, a robot learns the task of cooperative table balancing through deep Q-network using voice feedback provided by humans in real-time, with automatic speech recognition(ASR) and sentiment analysis to understand human voice feedback. As a result, an optimal policy convergence rate of up to 96% was realized, and performance was improved in all voice feedback-based models

## introduction service robots equipped with artificial intelligence technology are increasing in daily life. examples include museum exhibition guide robot @xcite , café-serving robot @xcite , and object carrying robot @xcite . robots increasingly perform tasks instead of or together with humans in various environments in daily life, and there has been an active research on robots that cooperate with humans @xcite . reinforcement learning (rl) --a robot learning technique-is a method in which an agent robot learns the action of obtaining maximum rewards through trial and error. in rl, rewards are generally given by agent action in a state, and if rewards are given through real-time human-agent interaction, it is called interactive reinforcement learning(iarl). reward shaping(rs) @xcite --an iarl method--is a technique in which a human trainer modifies reward functions by providing positive or negative feedback on the action of rl agents. in previous studies on iarl using natural language, the type of feedback is very limited using fewer than 10 feedbacks @xcite .to facilitate the use of robots, the need for a training system through various feedbacks is raised so that robot training can be naturally performed using various voice feedbacks. therefore, in this study, we propose an interactive deep rl model based on voice feedback to facilitate robot use. in the proposed system, a robot uses deep q-networks(dqns) @xcite to perform table balancing @xcite tasks that require cooperation with humans and learns the rl policy through rs by human voice feedback. using rs, a human trainer who collaborates table balancing task with robot and knows how to perform a task provides positive or negative feedback in real time about a robot's action via speech. therefore, the agent provided with voice feedback learns the optimal policy--a policy that always leads to the balanced table state--faster and more naturally than when feedback is not used. the rest of the paper is organized as follows. section 2 explores the flow and limitations of prior iarl studies through related work, and section 3 describes the proposed interactive deep rl system based on voice feedback. in section 4, we describe the results of table balancing task training based on the proposed system, and compare the difference in learning performance against conventional dqn as a baseline and between voice feedback provision types. finally, section 5 concludes this study and suggests future research directions. ## related work one of the strategies to improve learning performance in rl is that humans guide agents as external trainers. representative examples include learning by imitation @xcite , demonstration @xcite , and by feedback. among them, focusing on feedbackproviding learning, we examine: (1) the design of iarl platforms that provide feedback through mouse or remote controls @xcite , (2) design of iarl algorithms @xcite @xcite and ( 3 ) studies of iarl through voice feedback @xcite . what these studies have in common is that rs reduces training time and fosters the robot or computer to learn the target action. regarding methods that adopt hardware input devices, some approaches use a mouse or remote control to design an iarl platform @xcite . @xcite revealed that iarl can improve robot's learning efficiency in an interactive qlearning platform for cooking simulation robots, where humans can use mouse scrolls to provide feedback for robot actions by giving a number between -1 and +1. in the study of @xcite , aibo robots learned action sequences such as singing after hearing a command from a human feedback given by remote control. however, in these prior studies on the design of such an iarl platform, input hardware, such as a mouse and remote control, is required to provide human feedback, which is difficult to see as a natural interaction with human. studies on developing iarl algorithms using human feedback include tamer (knox and stone, 2009), advise @xcite and repair algorithm @xcite . in tamer--an interactive reinforcement learning algorithm proposed by @xcite --an agent learns a human feedback function by receiving two evaluation signals of positive and negative from the human on their keyboards; it was tested in tetris game and mountain car problem. in advise proposed by @xcite , a human modifies an agent's action choice probability, i.e., the policy, by giving the agent binary feedback--positive or negative. as a result, advise outperformed conventional rl algorithms on game tasks such as pac-man. @xcite proposed the re-pair algorithm, which estimates the correctness of human feedback over time; virtual and physical robots performed tasks, such as putting a ball into the box in a simulation environment and grasping cup in the real world. they proved that the repair algorithm matched or improved the performance of conventional q-learning algorithms. however, these approachs that focused on feedback learning algorithms for iarl required the design of an appropriate shaping function, and additional time to calculate rewards or policies. moreover, in the framework proposed in this study, natural language voice feedback is directly integrated into a reward so that the amount of additional computation required for dqn learning is relatively small. studies that investigated iarl using natural language speech voice feedback itself include dynamic rs @xcite and iarl through speech guidance @xcite . @xcite showed that robots can use human voice feedback in rl to learn navigation tasks by assigning specific scalar rewards to feedback vocabulary, such as +100 to "excellent" and -10 to "bad" in simulation environments. @xcite used voice commands and automatic speech recognition(asr) to transcribe input voice commands, and then compared the input sentence and predefined lists using levenshtein distance for cleaning tasks of robot arm agents. however, in these approaches using voice feedback, the rs function was designed by assigning a static reward value to a list of very limited words and sentences defined in advance. therefore, when a feedback vocabulary that has not been defined in the list is input, the agent may have difficulty in learning. moreover, the framework proposed in this study analyzes the positive and negative degrees of input voice feedback using a pretrained sentiment analysis module and converts it into a reward value. therefore, no matter what feedback phrase is input, the sentiment polarity of voice feedback can be analyzed and used for dqn rl. through the examination of prior studies, we can summarize that iarl ordinarily improves learning performance. however, most studies did not adopt a natural interaction method with humans by requiring hardware input devices such as a keyboard or mouse. further, studies using voice feedback used a small number of feedbacks. in this current study, we designed an iarl system for natural robot learning using voice feedback with asr and sentiment analysis techniques to resolve these limitations. ## proposed method in this section, we describe the proposed deep rl framework for table balancing robots based on voice feedback. the task that the robot aims to learn is to maintain balance when lifting a table cooperatively with a human. figure 1 shows the overall work diagram of the proposed system. first, the robot takes a table state image with a camera and forwards it to the dqn. next,the robot drives the balancing action predicted by dqn through image analysis. then, the robot receives evaluative feedback from humans on the executed action; the voice feedback is input via the robot's microphone, converted to numerical values by voice feedback recognition and conversion module, and then incorporated into the environmental rewards of the dqn algorithm. through repetition of the above process, the robot learns a policy in which the sum of environmental rewards and human voice feedback are maximized, and because of the learning, the robot can perform a cooperative table balancing task. in this work, the robot that will learn the table balancing task is softbank's nao robot, and the table is a rectangular box with width, length, and height of 31, 23, 6cm respectively. in addition, the table states to be used for learning were imaged using the lower camera mounted on the nao robot. algorithm 1 interactive deep q-network based on voice feedback initialize action-value function with random weights θ initialize target action-value function q with random weights θ -= θ for @xmath0, 20000 do initialize sequence for @xmath1, t do get table state image @xmath2, at) execute action at and observe reward rt and image xt+1 if human trainer provides voice feedback ft on state st then let rt←rt + ft end if @xmath3, a ; θ -)) otherwise (1) perform a gradient descent step on l(θ) = e[(yt -q(st, at; θt)) 2 ] with respect to the network parameters θ every 5 steps reset θ -= θ end for end for 3.1 deep reinforcement learning process based on voice feedback the robot in the proposed system uses the dqn to recognize the table state image and output the table balancing action based on human voice feedback. a dqn combines q-learning with a deep convolutional neural network to estimate a state-action value function (q function) given an input image and action. depending on the degree of raising and balancing state of the table, the human action states are divided into five in our system: up (s up ), keep (s 0 ), down (s down ), up a lot(s upup ) and down a lot (s downdown ). the subscripts of s represent human actions. the robot executes the table balancing action a by adjusting the knee joint drive value. five robot actions are defined depending on the direction and degree of table movement: a up , a up , a 0 , a down , and a down . algorithm 1 represents the training process of an interactive dqn based on voice feedback. this training process is identical to the dqn training process;an interactive voice feedback-based process is added after the robot action operation. the input state s is a table image(x t ),which is an rgb image of 128 × 170 size representing the balance status of the table imaged by the robot camera. the environment selects a table state image from the training dataset and feeds it to the robot, which is a dqn agent. the robot determines the action in the current time step according to the -greedy policy, which selects a random action with a probability of for exploration. if no random action is selected, the agent chooses the action that maximizes the value of the q function. the q function that dqn aims to predict is as follows: after executing an action, the agent receives evaluative voice feedback from human and environmental rewards. interactive voice feedback is a human speech evaluation of the robot's action. after checking the balance state of the table that has changed by the robot's action, the human provides positive voice feedback when the robot reaches the target state, and negative voice feedback otherwise. the provided voice feedback is converted into a numerical value through the voice feedback recognition and conversion module, and then added to the rl environment rewards. when the human provides voice feedback, the robot uses both feedback and envi-ronmental reward; and without feedback, the robot uses only environmental reward for learning. in subsection 3.2, the voice feedback recognition and conversion module is described in depth. in algorithm 1, θ stands for the parameters of neural networks. dqn considers y t as a target and proceeds learning in a direction that reduces the error of y t and estimated q(s t , a t ) by neural networks. therefore, the dqn model is updated in every episode via the loss function l(θ), which computes the mean squared error. with a repetitive update of θ in the direction of minimizing l(θ), the q function gets closer to the optimal state-action value function, and the agent learns the optimal action in the given state. through this process, the robot can train dqn for table balancing with human voice feedback. to incorporate voice feedback in the dqn framework, we implemented voice feedback recognition and conversion module. ## experimental results in this section, we discuss the construction of a feedback dataset for the experiment, evaluation of the voice feedback recognition and conversion module, and verification of the proposed interactive deep rl model through experiments. ## conclusion in this study, we proposed an interactive deep rl model based on voice feedback for table balancing robot. the proposed system suggests dqn incorporating human voice feedback using asr and sentiment analysis, where feedback given by humans are incorporated into the reward function. experiment results show that the consec-vf model, which pro-vides consec-vf early in learning, achieves an optimal policy convergence rate higher than the baseline model in all optimizer settings. there are several areas of extensions of our approach. future direction for our work includes incorporating multimodal feedback to dqn using various robot sensors. we could also focus on deepening model optimization technique that improves learning performance of interactive rl model in varying settings. robot could also learn when to use feedback and when to discard it or incorporate text semantics such as guiding robot behavior. mans ullerstam and makoto mizukawa. 2004. teaching robots behavior patterns by using reinforcement learning: how to raise pet robots with a remote control. in sice 2004 annual conference, volume 1, pages 143-146. ieee. kazuhiko yokoyama, hiroyuki handa, takakatsu isozumi, yutaro fukase, kenji kaneko, fumio kanehiro, yoshihiro kawai, fumiaki tomita, and hirohisa hirukawa. 2003. cooperative works by a human and a humanoid robot. in 2003 ieee international conference on robotics and automation (cat. no. 03ch37422), volume 3, pages 2985-2991. ieee. zuyuan zhu and huosheng hu. 2018. robot learning from demonstration in robotic assembly: a survey. robotics, 7(2):17.