Multitasking Inhibits Semantic Drift (2021)

Abstract: When intelligent agents communicate to accomplish shared goals, how do these goals shape the agents’ language? We study the dynamics of learning in latent language policies (LLPs), in which instructor agents generate natural-language subgoal descriptions and executor agents map these descriptions to low-level actions. LLPs can solve challenging long-horizon reinforcement learning problems and provide a rich model for studying task-oriented language use. But previous work has found that LLP training is prone to semantic drift (use of messages in ways inconsistent with their original natural language meanings). Here, we demonstrate theoretically and empirically that multitask training is an effective counter to this problem: we prove that multitask training eliminates semantic drift in a well-studied family of signaling games, and show that multitask training of neural LLPs in a complex strategy game reduces drift and while improving sample efficiency.

## introduction a major goal in the study of artificial and natural intelligence is to understand how language can scaffold more general problem-solving skills (e.g. @xcite , and how these skills in turn shape language itself (e.g. @xcite . in nlp and machine learning, latent language policies (llps; @xcite provide a standard framework for studying these questions. an llp consists of instructor and executor subpolicies: the instructor generates natural language messages (e.g. high-level commands or subgoals), and the executor maps these messages to sequences of low-level actions (fig. 1 ). llps have been used to construct interactive agents capable of complex reasoning (e.g. programming by demonstration) and planning over long horizons (e.g. in strategy games; @xcite . they promise an effective and interpretable interface between planning and control. instructor for task 2 (build spearmen to win) build a dragon attack with spearman multitask executor attack with dragon send a peasant to explore instructor for task 1 (build dragons to win) build a spearman single-task executor instructor for task 1 (build dragons to win) executor attacks with spearman executor explores with peasant executor builds a dragon standard training multitask training (a) (b) (c) (d) figure 1 : in latent language policies, instructor agents (a) send natural-language commands (b) to executor agents (c), which execute them in an interactive environment (d). jointly trained instructor-executor pairs learn to use messages in ways inconsistent with their natural language meanings (top shows a real messageaction pair from a model described in section 4). we show that multitask training with a population of taskspecific instructors stabilizes message semantics and in some cases improves model performance. however, they present a number of challenges for training. as llps employ a human-specified space of high-level commands, they must be initialized with human supervision, typically obtained by pretraining the executor. on its own, this training paradigm restricts the quality of the learned executor policy to that exhibited in (possibly suboptimal) human supervision. for tasks like the real-time strategy game depicted in fig. 1 , we would like to study llps trained via reinforcement learning (rl), jointly learning from a downstream reward signal, and optimizing both instructors and executors for task success rather than fidelity to human teachers. training llps via rl has proven difficult. past work has identified two main challenges: primarily, the llp-specific problem of semantic drift, in which agents come to deploy messages in ways inconsistent with their original (natural language) meanings @xcite ; secondarily, the general problem of sample inefficiency in rl algorithms @xcite . model-free deep rl is particularly notorious for requiring enormous amounts of interaction with the environment @xcite . for llps to meet their promise as flexible, controllable, and understandable tools for deep learning, better approaches are needed to limit semantic drift and perhaps improve sample efficiency. while semantic change is a constant and welldocumented feature of human languages @xcite , (human) word meanings are on the whole remarkably stable relative to the rate of change in the tasks for which words are deployed @xcite . in particular, disappearance of lexical items is mitigated by increased population size @xcite and increased frequency of use @xcite . drawing on these facts about stabilizing factors in human language, we hypothesize that training of machine learning models with latent language variables can be made more robust by incorporating a population of instructors with diverse communicative needs that exercise different parts of the lexicon. we describe a multitask llp training scheme in which task-specific instructors communicate with a shared executor. we show that complex longhorizon llps can be effectively tuned via joint reinforcement learning of instructors and executors using multitask training: together, these results show that diverse shared goals and communicative needs can facilitate (and specifically stabilize) learning of communication strategies. ## background and related work deep reinforcement learning (drl) has recently made impressive progress on many challenging domains such as games @xcite , locomotion @xcite and dexterous manipulation tasks @xcite . however, even state-ofthe-art approaches to reinforcement struggle with tasks involving complex goals, sparse rewards, and long time horizons. a variety of models and algorithms for hierarchical reinforcement learning have been proposed to address this challenge @xcite @xcite via supervised or unsupervised training of a fixed, discrete set of sub-policies. language can express arbitrary goals, and has compositional structure that allows generalization across commands. building on this intuition, several recent papers have explored hierarchical rl in which natural language is used to parameterize the space of high-level actions @xcite @xcite @xcite . while there are minor implementation differences between all these approaches, we will refer to them collectively as latent language policies (llps). like other hierarchical agents, an llp consists of a pair of subpolicies: an instructor i(m | o) and an executor e(a | m, o). an llp takes actions by first sampling a stringvalued message m ∼ i from the instructor, and then an action a ∼ e from the executor. for these messages to correspond to natural language, rather than arbitrary strings, policies need some source of information about what human language users think they mean. this is typically accomplished by pretraining executors via human demonstrations or reinforcement learning; here we focus on the ingredients of effective joint rl of instructors and executors. reinforcement learning has been widely used to improve supervised language generation policies, particularly for dialogue @xcite , translation @xcite and summarization @xcite . here, we instead focus on models where language is a latent variable as part of a hierarchical policy for a non-linguistic task. as noted in section 1, an observed shortcoming of reinforcement learning in all these settings is its susceptibility to semantic drift. in the literature on human language change @xcite , semantic drift refers to a variety of phenomena, including specific terms becoming more general, general terms becoming specific, and parts coming to refer to wholes. in machine learning, it refers broadly to the use of messages inconsistent with their natural language meanings in language-generation policies @xcite . @xcite mitigate semantic drift in pivotbased machine translation by using visual grounding, whereas @xcite periodically update a student model on data generated by an rl teacher. work in emergent communication has found that reinforcement learning tends not to learn policies with natural language-like properties @xcite , although population-based training has been found to be helpful @xcite . most relatedly to our work, @xcite train speaker-listener agents jointly in a visual referential communication task and introduce auxiliary loss functions for stabilizing training. our work focuses on a more general setting where the interactions are temporally extended, have large action spaces and is partially observable. @xcite use populations of agents to reduce semantic drift in visual dialogue. we view the current paper's analysis of multitask learning as complementary to these approaches from the emergent communication literature; future work might consider ways of combining the two. a great deal of recent work in both rl (e.g. @xcite and language processing (e.g. @xcite has observed that carefully designed training objectives can serve as a source of model-agnostic inductive bias. our results bring these two lines of work together: multitask train-ing improves the faithfulness and adaptability of learned language understanding models, even when optimizing for a downstream reward. ## multitask communication in theory: lewis signaling games we begin our analysis with the simple signaling game depicted in fig. 2 . in this game, one agent receives an observation, then sends a message to another agent, which then performs an action. signaling games like this one are widely studied in nlp as models of reference resolution and language generation @xcite . the instructor-executor pair may together be viewed as the simplest llp of the kind described in section 2. formally, a (2-observation, 2-message) lewis signalling game is defined by: the game is played between two agents: a instructor (with parameters θ i ), which receives an observation and samples an observation-specific message from a distribution i(m | o; θ i ); and a executor (with parameters θ e ), which receives the instructor's message and uses it to sample an action from a distribution e(a | m; θ e ). the agents then receive a reward r(o, a) that depends on the observation and action but not on the message sent. this policy's expected reward is given by: gradient ascent on eq. ( 1 ) with respect to θ i and θ e (e.g. using a policy gradient algorithm; williams, 1992) can be used to improve the expected reward obtained by an llp. as an example, the right portion of fig. 2 shows two reward functions r and r . in both, each observation is paired with a single action, and the executor must take the action corresponding to the observation to receive a positive reward. for r, two strategies obtain the optimal expected reward of 1: one in which i(m semantic drift suppose, as shown in fig. 2 , the messages m 1 and m 2 are not arbitrary symbols, but correspond to the natural language expressions m 1 = push red and m 2 = push blue. in this case, only of the policies described above corresponds to the semantics of natural language-namely, the one in which e(a what is needed to ensure that a pair of agents playing converge to the natural language strategy? in the analysis that follows, we will consider instructor and executor agents (each with a single scalar @xmath0, 1]): for the game depicted in fig. 2 , we would like to avoid any outcome in which, after training, more generally, let us assume that we have an initial set of executor parameters that are possibly suboptimal but correspond to natural language semantics in the sense that e(a to analyze semantic drift in this game, we consider the final values of the parameters (θ i , θ e ) when optimized from an initialization (θ for the reward function r depicted in fig. 2 , we can perform gradient ascent on eq. ( 1 ) with respect to θ i and θ e in this model by observing that: by considering the limiting behavior of gradient ascent as step size goes to zero (a gradient flow; see appendix a), it is possible to give a closedform expression for the value of these parameters as a function of time: proof is given in appendix a. note in particular that semantic drift will occur whenever θ (0) e , which can occur even assuming a well-initialized executor with θ @xmath1. fig. 5 in the appendix provides a visualization of learning dynamics and these drift-susceptible initializations. however, we will next show that this drift can be eliminated via multitask training. we assume that the two instructors share the same initialization, with θ (0) ## multitask communication in practice: we next verify whether this result extends to the complex llp-learning tasks discussed in section 2. our focus in this section is the minirts environment of @xcite (depicted in fig. 1 ), in which agents must build and control an army of units like archers, spearmen, swordman, cavalry, and dragons, each with specialized abilities, with the goal of destroying the opponent's town center. using this game, @xcite crowdsourced a dataset of high-level instructions (like attack with dragon and send idle peasant to mine) paired with low-level action sequences (fig. 1 ). they showed that an llp trained on this supervised data via behavior cloning significantly outperformed a flat policy trained with imitation learning directly on low-level action sequences. here we investigate (1) whether these policycloned llp can be further improved via reinforcement learning directly on a sparse win-loss signal from the game, (2) whether we can improve sample efficiency during reinforcement learning by jointly training executor models on multiple game variants simultaneously through multitask learning, and (3) whether semantic drift can be avoided during multi-task training. below, section 4.1, section 4.2 and section 4.3 provide more detail about the task, model, and training procedure. section 4.4 reports experimental results. ## conclusions we have presented a theoretical and empirical analysis of semantic drift and sample efficiency in multitask reinforcement learning of latent language policies (llps). in a lewis signaling game, we proved that multitask training can completely eliminate semantic drift. in a two-player real-time strategy game, we showed that multitask training is effective at mitigating semantic drift, improves the quality of learned policies and is sample efficient. future work might integrate these results with other forms of population-based training (like those proposed by @xcite for reference games) and explore other environmental factors affecting dynamics of language change in populations of learned agents.