  Building agents capable of understanding language instructions is critical to
effective and robust human-AI collaboration. Recent work focuses on training
these instruction following agents via reinforcement learning in environments
with synthetic language; however, these instructions often define long-horizon,
sparse-reward tasks, and learning policies requires many episodes of
experience. To this end, we introduce ELLA: Exploration through Learned
Language Abstraction, a reward shaping approach that correlates high-level
instructions with simpler low-level instructions to enrich the sparse rewards
afforded by the environment. ELLA has two key elements: 1) A termination
classifier that identifies when agents complete low-level instructions, and 2)
A relevance classifier that correlates low-level instructions with success on
high-level tasks. We learn the termination classifier offline from pairs of
instructions and terminal states. Notably, in departure from prior work in
language and abstraction, we learn the relevance classifier online, without
relying on an explicit decomposition of high-level instructions to low-level
instructions. On a suite of complex grid world environments with varying
instruction complexities and reward sparsity, ELLA shows a significant gain in
sample efficiency across several environments compared to competitive
language-based reward shaping and no-shaping methods.
