  Robots in many real-world settings have access to force/torque sensors in
their gripper and tactile sensing is often necessary in tasks that involve
contact-rich motion. In this work, we leverage surprise from mismatches in
touch feedback to guide exploration in hard sparse-reward reinforcement
learning tasks. Our approach, Touch-based Curiosity (ToC), learns what visible
objects interactions are supposed to "feel" like. We encourage exploration by
rewarding interactions where the expectation and the experience don't match. In
our proposed method, an initial task-independent exploration phase is followed
by an on-task learning phase, in which the original interactions are relabeled
with on-task rewards. We test our approach on a range of touch-intensive robot
arm tasks (e.g. pushing objects, opening doors), which we also release as part
of this work. Across multiple experiments in a simulated setting, we
demonstrate that our method is able to learn these difficult tasks through
sparse reward and curiosity alone. We compare our cross-modal approach to
single-modality (touch- or vision-only) approaches as well as other
curiosity-based methods and find that our method performs better and is more
sample-efficient.
