  One of the fundamental goals of visual perception is to allow agents to
meaningfully interact with their environment. In this paper, we take a step
towards that long-term goal -- we extract highly localized actionable
information related to elementary actions such as pushing or pulling for
articulated objects with movable parts. For example, given a drawer, our
network predicts that applying a pulling force on the handle opens the drawer.
We propose, discuss, and evaluate novel network architectures that given image
and depth data, predict the set of actions possible at each pixel, and the
regions over articulated parts that are likely to move under the force. We
propose a learning-from-interaction framework with an online data sampling
strategy that allows us to train the network in simulation (SAPIEN) and
generalizes across categories. But more importantly, our learned models even
transfer to real-world data. Check the project website for the code and data
release.
