Teaching Arm and Head Gestures to a Humanoid Robot through Interactive Demonstration and Spoken Instruction (2021)

Abstract: We describe work in progress for training a humanoid robot to produce iconic arm and head gestures as part of task-oriented dialogue interaction. This involves the development and use of a multimodal dialog manager for non-experts to quickly ‘program’ the robot through speech and vision. Using this dialog manager, videos of gesture demonstrations are collected. Motor positions are extracted from these videos to specify motor trajectories where collections of motor trajectories are used to produce robot gestures following a Gaussian mixtures approach. Concluding discussion considers how learned representations may be used for gesture recognition by the robot, and how the framework may mature into a system to address language grounding and semantic representation.

## introduction a conventional way of programming robots to make iconic gestural movements is to animate movements as sequences of static motor positions. this method is slow and tedious and an easier method is sought. ideally, people should be able to teach a robot how to make new gestures through visual demonstration and verbal instruction, as they might teach another person how to make a new arm and head gesture. such a multimodal interactive approach is one of today's current challenges in robotics. perhaps one reason that multimodal interaction with robots is problematic relates to the compartmentalization of research specialties. speech engineers are generally not experts at computer vision and motor control. likewise, robotics engineers and computer vision engineers tend to treat speech and language as a 'black box' problem best left to speech and language technologists. the result is that language, vision, and motor control tend to be segregated during software planning and im-plementation. it is left to the robot interaction engineer to cobble these segregated modalities together into a cohesive software framework. the broad aim of our project is to pragmatically address this challenge by developing a processing architecture where communicative information across modalities can be more integrated. teaching a robot how to produce gestures through visual demonstration and spoken dialogue is a task that is well suited for addressing the challenge. robot learning from demonstration (lfd), sometimes also referred to as "robot programming by demonstration," "teaching by example," or "imitation learning" is an established approach for training robots through vision. as alluded to above however, one issue with lfd is that lfd practitioners generally fail to incorporate the power of verbal instruction, see @xcite . we posit that with the relatively recent advent of deep learning and related breakthroughs in computer vision, artificial speech recognition, and related technologies, the time is ripe to integrate natural verbal instruction with lfd. lfd and training by example has a rich history and is a popular research area in modern robotics, for example see: @xcite @xcite @xcite @xcite . lfd sidesteps more traditional and tedious methods of manually specifying motor control or where math and computer programing expertise is required. the essence of lfd is that robot movements may be acquired by having a person act out the movements to be learned (either through telepresence, kinetically, or visually), and transposing those movements into representations that a robot may use in combination with the robot's knowledge and internal processing to then produce the movement. it is important to note that lfd is not merely a 'record and replay' technique. generalization is required so that, for example, starting and ending positions of the movements are not pre-determined. exact trajectories as well as amplitudes of movements may vary insofar as the task demands, and resulting movements should be robust in the face of changing environmental conditions and actuator imprecisions. for our present purpose, the idea is also to avoid exact monotonous repetitions, and to develop robust representations that may also be used for perceiving learned gestures. interacting with robots through natural language is another popular area of research. e.g. see: @xcite @xcite @xcite . perhaps the most popular domain for linguistic information transfer between people and robots is in giving travel or route instructions, such as in the spoken guidance of robotic wheelchairs, for a review see: @xcite . it is important to note that speech communication also contains non-linguistic cues, both vocal (e.g. laughter, affect, tone) and non-vocal (e.g. gestures, eye gaze, face expressions, environmental context). for related review, see: @xcite . in addition to the linguistic signal, these and related cues should be readily available for incorporation into interaction designs. ## method the robot this work uses is "voxhead," a 3d printed humanoid robot @xcite . figure 1 displays the robot. the robot serves as a life-sized and relatively low cost platform for interactive social robotics research. the robot has motors for mouth, eye cameras, and facial expressions. for the present work we do not concern ourselves with facial motors. instead, focus is on general head, neck, and arm movements. in total there are sixteen degrees of freedom in the head, neck, and arms that we work with. specifically we use: head tilt, head turn, neck tilt, neck turn, and for each arm: arm raise-lower, arm left-right, arm rotate, elbow bend, wrist rotate, and wrist bend. hands with individual fingers or grippers are also not used here. ## discussion we have introduced the infrastructure of an interactive speech-vision-motor system for training a lifesized humanoid robot to produce desired arm and head gestures. the system interfaces a rudimentary cognitive architecture with an interaction manager for robot control. we use an lfd technique combined with spoken instructions and dialogue for training a robot to produce gestures. we lastly turn to consider the relationship between perception and action, the language grounding problem, and semantic representation. there is an intimate relationship between perception and action. the research industry surrounding the mirror neuron hypothesis reifies this @xcite in light of this, our current work also includes the development of a gesture recognition algorithm that depends on production learning. the time-normalized motor trajectories of a class from section 2.2 define a centroid motor trajectory for the class. we call this centroid a gesture prototype. in short, a motor trajectory to be categorized is template-matched against the stored inventory of gesture prototypes using a multidimensional dynamic time warping algorithm @xcite . the best match is taken as the gesture's category. plans are to develop our system to addresses the symbol grounding problem @xcite @xcite . establishing a socially situated and embodied system for interactive gesture learning was but a first step. semantic meaning must be grounded in experience, where different modalities (speech, vision, motor feedback) are integrated. interactive audio-visualmotor recordings from our system may be used for machine learning approaches, e.g. @xcite to train multi-modal speech recognizers. in order for meaning to emerge, the robot must 'understand' its own output. by pursuing a paradigm where gesture recognition is based on the robot's representations for gesture production, our hope is to depict representations to be one and the same for perception and production. in viewing speech as a problem of motor control, speech cognition becomes grounded in the robot's experience.