  Robots that must operate in novel environments and collaborate with humans
must be capable of acquiring new knowledge from human experts during operation.
We propose teaching a robot novel objects it has not encountered before by
pointing a hand at the new object of interest. An end-to-end neural network is
used to attend to the novel object of interest indicated by the pointing hand
and then to localize the object in new scenes. In order to attend to the novel
object indicated by the pointing hand, we propose a spatial attention
modulation mechanism that learns to focus on the highlighted object while
ignoring the other objects in the scene. We show that a robot arm can
manipulate novel objects that are highlighted by pointing a hand at them. We
also evaluate the performance of the proposed architecture on a synthetic
dataset constructed using emojis and on a real-world dataset of common objects.
