Automatic Verbal Depiction of a Brick Assembly for a Robot Instructing Humans (2022)

Abstract: Verbal and nonverbal communication skills are essential for human-robot interaction, in particular when the agents are involved in a shared task. We address the specific situation when the robot is the only agent knowing about the plan and the goal of the task and has to instruct the human partner. The case study is a brick assembly. We here describe a multi-layered verbal depictor whose semantic, syntactic and lexical settings have been collected and evaluated via crowdsourcing. One crowdsourced experiment involves a robot instructed pick-and-place task. We show that implicitly referring to achieved subgoals (stairs, pillows, etc) increases performance of human partners.

## introduction task-oriented interactions between systems and humans, in order to achieve a common goal, are present in many applications. for instance, receiving directions via gps is a task-oriented communication with the instructions being delivered visually and verbally @xcite . similarly, robots have been used to give directions @xcite , describe its route experience @xcite or instruct students in a tutorial class @xcite . in the later work (see figure 1 ), a robot helps two participants to perform a jigsaw assembly task using verbal and non-verbal communication. figure 1 : face-to-face interaction on a jigsaw reassembly task with an icub robot (right) acting as the instructor for two students (left). from @xcite . the rise of social robots endowed with verbal, co-verbal and non-verbal communication capabilities, now raises the question from the robotic point of view. how a robot and a human can communicate to achieve a common goal and share plans, involving manipulating objects in their common working space? in this paper, we study this problem by focusing on verbal communication. indeed, a verbal description of how the task is to be done is a more effective way of communicating objectives than non-verbal descriptions: not only does it improve task performance but also gives rise to more compliance and better mutual adaptation (see @xcite . more precisely, we propose to explore the impact of the verbalization strategy in an extreme case where the robot is the only agent that knows the plan and the goal, and human coworkers are awaiting instructions planned by the robot to achieve the goal. note that we limit here the number of human partners to one: the opportunistic allocation of tasks between available coworkers will be addressed in a following paper. when using verbalisation as the main means of communication, an important question is to see how the style, i.e. saying the same thing in different ways, quoted as the verbalization space by @xcite , in which the instructions are being delivered, can affect the execution of the task, especially when communicating complex tasks. in this context, our contribution is threefold: styles: we test four different styles on an assembly task (see figure 2 ) that offers a large verbalization space, i.e. many stylistic dimensions including choice of geometric relations between bricks, of syntactic and lexical descriptions, etc. one primary style parameter is the use of context. by context, we mean implicit referencing to elements of the environment that go beyond the previous and current actions. we compare two ai-generated styles (inclusion vs. dismissal of the use of context) with the lowest vs. highest human instructions in terms of: time-to-complete, comprehension/complexity of the instruction, and efficiency/effectiveness of the task completion. architecture: we propose a robotic control architecture and its sensorimotor capabilities for task-oriented human interaction. we focus on two key components: (a) the planner and (b) the verbalizer. while the former takes decisions on what to do next, the verbalizer puts each elementary instruction into words for a text-to-speech synthesizer. evaluation: we propose an evaluation framework based on a series of three crowdsourced experiments for: (a) collecting and (b) scoring human verbalizations for parameterizing our flexible verbalizer as well as (c) assessing and evaluating the performance of human vs. automatic verbalizations on actual task assembly. this paper is organized as follows: section 2 contains related work for task-oriented communication and plan description; section 3 describes the overall architecture of our control model, with a closer look on the planner and the verbalizer with its different layers (fig. 6 ); section (4) introduces the three experiments used to parametrize the verbalizer; section 4.1 presents the results of the first two web-based experiments used for data collection and assessment of human verbalizations; finally, section 4.2 presents the setup and results of the last web-based experiment used to validate the efficiency of the set of rules in our automatic verbalizer and to compare it with human verbalization. ## related work verbal communication of plans has been used in a large variety of human-robot interaction (hri) scenarios. they mainly vary along four main cate-gories: (1) task type (e.g. commentating, instructing, navigation). (2) perception capabilities (auditive/visual sensors). (3) style and its use in sentence generation (e.g. information tagging). ( 4 ) role (e.g. receptionist, instructor, navigator). most hri tasks require some form of communication. it could be used to describe what happens in the scene: @xcite presented rocco, a fully automated robocup @xcite commentator, aiming at generating real-time summaries of the actions in the games @xcite . for navigation tasks, @xcite ; perera et al. ( 2016 ) presented algorithms for generating routing narratives with varying parametrized styles. @xcite presented a real-time spoken language navigation system able to respond to natural conversational queries. the queries were mainly regarding details of a step in the route. however, responses were generated using simple pre-written "holly sentences" filled with variables extracted from the plan. for assembly tasks, the 'shrdlu' system @xcite is quite inspiring: the task focused on manipulating blocks with a robot arm on the basis of the user's textual input. the system translates the user's input into procedures to move the blocks and question the scene. our work exchanges the roles of the agents: our robot instructs human agents verbally. @xcite also shares some similarities with our work, i.e. verbalising the actions in the plan for the user as well as explaining which actions should be executed and in what order. however, the task they chose does not require the same level of precision -and their focus was not on verbalization. finally, the robert system @xcite , installed on bosch equipment, provides its user with a step-by-step instruction (on a screen using text, images, voice and videos) detailing how to complete a given diy project successfully. similarly to us, the sequence of instructions (plan) is obtained using htn planning. they also added a new feature to perceive the scene using connected tools (sensors), enabling the system to check whether the user is performing the project's steps correctly and to provide help in the case of failure. @xcite proposed a verbalization system able to generate explanations for navigation as well as grasping and manipulation tasks (pick-andplace kitchen scenario). they used pre-written sentence templates. @xcite proposed plan-verb, a domain and planner-independent method for the verbalization of task plans based on semantic information tagging of the actions and predicates in the domain (for both pddl and rddl). several works showed the importance of having different styles @xcite @xcite . in line with @xcite ; @xcite , our verbal generation framework relies on crowdsourcing experiments, for both defining the different styles of the instructions and assessing their efficiency. verbal communication has been used in a large variety of situations. @xcite proposed a robot receptionist with pre-written storylines. their focus was on long-term interactions with a robot that exhibits personality and character. for their robot bartender, petrick and foster (2013) construct plans with tasks, dialogue, and social actions. they advocate for a stronger link between planning and language. ## the architecture figure 3 shows the overall architecture of our hri system monitoring the interaction between the agents (humans and robot) and the working environment. while no single architecture has proven to be best for all applications, layered architectures have proven to be increasingly popular, due to their flexibility and ability to operate at multiple levels of abstraction simultaneously @xcite . similarly to what can be found in @xcite , our robot's architecture can be divided into three levels: perception, decision making and action. with different robots, capabilities change and so do their perception/action modalities. this architecture allows us to add/remove figure 4 : an example where the task is to build a lego ™ arch -hierarchical decomposition in fig. 5 modalities, in order to cope with both industrial (e.g. no gaze/head) and humanoid robots (e.g. no grippers). the perception level is in charge of capturing the current state of the environment as well as the agents acting on it, e.g. analyzing verbal requests coming from the human agent and all changes happening in the working environment. the action level takes charge of all actions towards the environment (e.g robot moving around to pick an object) and agents (e.g. coordinated gaze, speech and pointing to attract partners' attention). the controller is responsible for orchestrating action/perception loops according to the current objective given on request by the planner. in particular, the controller is in charge of monitoring the addressee's activity when processing the robot's instruction, such as on-line attention, task comprehension and correct execution. this includes the chunking or repetition of the instruction if necessary. it all starts with the controller that receives a "go" signal and requests the first action from the planner. provided the requested information, the controller (via the action modules) either applies the action or instructs the human agent to apply it. the environment is modified, the controller perceives the updated current state, and the loop continues until the planner deems this task as completed. the following subsections detail the key modules for the generation of verbal instructions: the planner and the verbalizer. ## experiments the purpose of the first two experiments is to provide us with a ground-truth corpus of verbal descriptions and obtain the highest and lowest ranked human exemplars for giving an instruction. the third experiment introduces an assembly task in order to test the efficiency of ai-generated instructions with reference to the natural ones. following the spatial representation -using metric, directional and topological operators as well as using multiple types of object references (point/corner, line/axis . . . ) @xcite ) -we chose a set of elementary actions which (1) spans most of these operators and (2) allows the implicit use of the context of that action. thus, the instructions studied in the following describe the placement of the first brick of a new structure, i.e. giving the semantic generator the possibility to refer (or not) to the just finished one. we describe below how we use crowdsourcing to gather human descriptions of these placements (mainly to parametrize the semantic and syntactic generator) and compare the efficiency of human vs. automatic descriptions. for this, we asked subjects to actually perform the actions. we expect conditions (human vs. ai-generated utterances, effective vs. no use of context) will impact placement error or time-to-complete. all experiments are performed in french. ## conclusions and future work we evaluate the impact of using hierarchical context when giving instructions in an assembly task. we gathered and ranked crowdsourced human instructions. we set up a multi-layer verbalizer that computes ai-generated instructions. we then compared the performance of these verbalization policies on a web-based assembly task. the differences between users' preferences and actual performances claim for an evaluation method in two steps: first, selecting candidate policies by subjective preference but then assess their efficiency by objective performance. we see that referring to hierarchical context improves human performance, compared to refraining from using it, in particular when context is unambiguous. we also show that our ai-generated instructions often outperform the least popular human instructions, validating the efficiency of our verbalizer. while verbalizing, pointing towards the intended object would improve the understanding of the robot's intention, and reduce the effort in the verbal explanation to ensure task completion. therefore, future work will include this modality in the action layer of our architecture along with speechhand-gaze coordination. incremental monitoring of actions by perception, in particular for on-line comprehension and attention, is a key issue for hri.