  In the absence of external rewards, agents can still learn useful behaviors
by identifying and mastering a set of diverse skills within their environment.
Existing skill learning methods use mutual information objectives to
incentivize each skill to be diverse and distinguishable from the rest.
However, if care is not taken to constrain the ways in which the skills are
diverse, trivially diverse skill sets can arise. To ensure useful skill
diversity, we propose a novel skill learning objective, Relative Variational
Intrinsic Control (RVIC), which incentivizes learning skills that are
distinguishable in how they change the agent's relationship to its environment.
The resulting set of skills tiles the space of affordances available to the
agent. We qualitatively analyze skill behaviors on multiple environments and
show how RVIC skills are more useful than skills discovered by existing methods
when used in hierarchical reinforcement learning.
