  In this paper we address the challenge of exploration in deep reinforcement
learning for robotic manipulation tasks. In sparse goal settings, an agent does
not receive any positive feedback until randomly achieving the goal, which
becomes infeasible for longer control sequences. Inspired by touch-based
exploration observed in children, we formulate an intrinsic reward based on the
sum of forces between a robot's force sensors and manipulation objects that
encourages physical interaction. Furthermore, we introduce contact-prioritized
experience replay, a sampling scheme that prioritizes contact rich episodes and
transitions. We show that our solution accelerates the exploration and
outperforms state-of-the-art methods on three fundamental robot manipulation
benchmarks.
