  Imitation Learning (IL) is an effective framework to learn visuomotor skills
from offline demonstration data. However, IL methods often fail to generalize
to new scene configurations not covered by training data. On the other hand,
humans can manipulate objects in varying conditions. Key to such capability is
hand-eye coordination, a cognitive ability that enables humans to adaptively
direct their movements at task-relevant objects and be invariant to the
objects' absolute spatial location. In this work, we present a learnable action
space, Hand-eye Action Networks (HAN), that can approximate human's hand-eye
coordination behaviors by learning from human teleoperated demonstrations.
Through a set of challenging multi-stage manipulation tasks, we show that a
visuomotor policy equipped with HAN is able to inherit the key spatial
invariance property of hand-eye coordination and achieve zero-shot
generalization to new scene configurations. Additional materials available at
https://sites.google.com/stanford.edu/han
