The Learning-from-Demonstration (LfD ) framework aims to map human demonstrations to a robot to reduce programming effort. To this end, an LfD system encodes a human demonstration into a series of execution units for a robot,  referred to as task models. Although previous research has proposed successful task-model encoders that analyze images and human body movements, the encoders have been designed in environments without noise. Therefore, there has been little discussion on how to guide a task-model encoder in a scene with spatio-temporal noises  such as cluttered objects or unrelated human body movements. In human-to-human demonstrations, verbal instructions play a role in guiding an observer's visual attention . Inspired by the function of verbal instructions , we propose a verbal focus-of-attention (FoA) system (i.e., spatio-temporal filters) to guide a task-model encoder. For object manipulation, the encoder first recognizes a target-object name and its attributes from verbal instructions. The information serves as a where-to-look FoA filter to confine the areas where the target object existed in the demonstration. The encoder next detects the timings of grasp and release tasks that occur in the filtered area. The timings serve as a when-to-look FoA filter to confine the period when the demonstrator manipulated the object . Finally, the task-model encoder recognizes  task models by employing the FoA filters.  The contributions of this paper are : (1) to propose  verbal FoA for LfD; (2) to design an algorithm to calculate FoA filters from verbal input ; (3) to demonstrate the effectiveness of a verbal-driven FoA by testing an implemented LfD systemin noisy environments .
The learning-from-observation (LfO ) framework aims to map human demonstrations to a robot to reduce programming effort. To this end, an LfO system encodes a human demonstration into a series of execution units for a robot, which are referred to as task models. Although previous research has proposed successful task-model encoders , there has been little discussion on how to guide a task-model encoder in a scene with spatio-temporal noises , such as cluttered objects or unrelated human body movements. Inspired by the function of verbal instructions guiding an observer's visual attention , we propose a verbal focus-of-attention (FoA) system (i.e., spatio-temporal filters) to guide a task-model encoder. For object manipulation, the system first recognizes the name of a target object and its attributes from verbal instructions. The information serves as a where-to-look FoA filter to confine the areas in which the target object existed in the demonstration. The system then detects the timing of grasp and release that occurs in the filtered area. The timings serve as a when-to-look FoA filter to confine the period of object manipulation . Finally, a task-model encoder recognizes the task models by employing FoA filters. We demonstrate the robustness of the verbal FoA in attenuating spatio-temporal noises by comparing it with an existing action localization network. The contributions of this study are as follows : (1) to propose a verbal FoA for LfO, (2) to design an algorithm to calculate FoA filters from verbal input , and (3) to demonstrate the effectiveness of a verbal FoA in localizing an action by comparing it with a state-of-the-art vision system .
R
Learning-from-Demonstration (LfD
learning-from-observation (LfO
R
LfD
LfO
A
which are
R
that analyze images and human body movements, the encoders have been designed in environments without noise. Therefore,
,
A
,
R
In human-to-human demonstrations, verbal instructions play a role in
Inspired by the function of verbal instructions
D
. Inspired by the function of verbal instructions
R
encoder first recognizes a target-object name
system first recognizes the name of a target object
R
where
in which
R
encoder next detects the timings
system then detects the timing
R
tasks that occur
that occurs
R
when the demonstrator manipulated the object
of object manipulation
R
the
a
A
the
D
the
A
We demonstrate the robustness of the verbal FoA in attenuating spatio-temporal noises by comparing it with an existing action localization network.
R
paper are
study are as follows
A
a
R
LfD;
LfO,
R
;
, and
R
verbal-driven FoA by testing an implemented LfD systemin noisy environments
verbal FoA in localizing an action by comparing it with a state-of-the-art vision system