  When intelligent agents communicate to accomplish shared goals, how do these
goals shape the agents' language? We study the dynamics of learning in latent
language policies (LLPs), in which instructor agents generate natural-language
subgoal descriptions and executor agents map these descriptions to low-level
actions. LLPs can solve challenging long-horizon reinforcement learning
problems and provide a rich model for studying task-oriented language use. But
previous work has found that LLP training is prone to semantic drift (use of
messages in ways inconsistent with their original natural language meanings).
Here, we demonstrate theoretically and empirically that multitask training is
an effective counter to this problem: we prove that multitask training
eliminates semantic drift in a well-studied family of signaling games, and show
that multitask training of neural LLPs in a complex strategy game reduces drift
and while improving sample efficiency.
