  The paradigm of learning-from-observation (LfO) enables a robot to learn how
to perform actions by observing human-demonstrated actions. Previous research
in LfO have mainly focused on the industrial domain which only consist of the
observable physical constraints between a manipulating tool and the robot's
working environment. In order to extend this paradigm to the household domain
which consists non-observable constraints derived from a human's common sense;
we introduce the idea of semantic constraints. The semantic constraints are
represented similar to the physical constraints by defining a contact with an
imaginary semantic environment. We thoroughly investigate the necessary and
sufficient set of contact state and state transitions to understand the
different types of physical and semantic constraints. We then apply our
constraint representation to analyze various actions in top hit household
YouTube videos and real home cooking recordings. We further categorize the
frequently appearing constraint patterns into physical, semantic, and
multistage task groups and verify that these groups are not only necessary but
a sufficient set for covering standard household actions. Finally, we conduct a
preliminary experiment using textual input to explore the possibilities of
combining verbal and visual input for recognizing the task groups. Our results
provide promising directions for incorporating common sense in the literature
of robot teaching.
