Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior (2021)

Abstract: We study continual learning for natural language instruction generation, by observing human users’ instruction execution. We focus on a collaborative scenario, where the system both acts and delegates tasks to human users using natural language. We compare user execution of generated instructions to the original system intent as an indication to the system’s success communicating its intent. We show how to use this signal to improve the system’s ability to generate instructions via contextual bandit learning. In interaction with real users, our system demonstrates dramatic improvements in its ability to generate language over time.

## introduction natural language provides an expressive and accessible avenue to instruct non-expert users. the ability to generate instructions is critical for systems that collaborate with users, for example, to delegate tasks. in such scenarios, the system generates language to communicate to the user a latent intent. when users are cooperative and proficient in the language, whether they accomplish the system's intent provides an informative, albeit noisy signal of the quality of instruction generation. this implicit signal is fundamentally different from supervised data, including via active learning, in that it does not label the system's intent with a written instruction, but only provides evidence to the quality of a given instruction in relaying this intent. as a natural byproduct of interaction with users, it also differs from explicit user feedback in not requiring user action beyond what they already do as part of the interaction. despite its potential and prevalence, this signal is understudied for learning to generate natural language in this paper, we study this learning signal. we formalize continually improving instruction generation by observing human users executing gen-erated instructions. we learn by comparing instruction execution to the system intent, and demonstrate how this results in a system that continually improves its natural language generation ability through interaction with users. figure 1 illustrates our learning process. we design a task-oriented collaborative scenario using the cerealbar game environment @xcite . in cerealbar, two agents, a leader and a follower, work together to complete tasks. the leader plans the tasks to complete, and communicates goals to the follower using natural language. cerealbar was originally introduced for studying follower instruction execution. we modify it to focus on generation of leader instructions, which are then executed by human followers. the collaborative, embodied setup effectively engages users, and aligns their incentives with executing the system's instructions to the best of their abilities. a major challenge is inferring a learning signal from observed user behavior. given the user execution, we create positive and negative examples, depending on how the user execution aligns with the system's plan and the user's perceived correctness of their own execution. for example, consider an execution that does not align well with the system's plan, but that the user considers correct given the instruction. because of the misalignment, we cannot consider the instruction as a successful example given the system's plan. however, given the user's perceived correctness, we can generate a positive example treating the user's execution as a plan paired with the instruction. in contrast to supervised learning with gold-standard per-token labels @xcite , such utterance-level binary labels form a challenging signal for learning, because they do not distinguish between correct and incorrect tokens. we do not make the typical distinction between training and deployment; as human users follow generated instructions, we continually collect new data, periodically train using this data, and evaluate the system through the interaction itself. we formalize learning as an off-policy contextual bandit learning problem. we show that positive examples can be treated in a manner that reduces to supervised learning, allowing for simple effective use of the data. however, using negative examples is more challenging, because simply minimizing their likelihood gives an unbounded negative loss. we weigh negative examples using an inverse propensity score @xcite @xcite to address this issue. we experiment with our approach through interaction with human users, tracking both task performance and how the generated language changes. we observe dramatic improvements in the quality of instructions generated as reflected in users' execution: task completion in accordance to the system intent increases from 44.7% to 79.3%. this is accompanied by significant language change: the occurrence of erroneous phrases decreases as desired, but the effective system vocabulary gradually shrinks. although using user feedback for improving language generation has been studied, as we discuss in section 8, to the best of our knowledge, this study is the first to show effective instruction generation learning by observing user execution. our experiments demonstrate the effectiveness of our process, but also illustrate limitations and important directions for future work. code and data are available at https://lil.nlp.cornell .edu/cerealbar/ . ## technical overview and notation our goal is to continually improve a natural language instruction generation model, by observing human executions of generated instructions. interaction scenario we focus on a collaborative scenario, where two agents, a leader and a follower, complete tasks in an environment. the system is the leader, and the human user is the follower. the leader plans tasks to accomplish, acts in the world, and instructs the follower using natural language. we use a deterministic procedure for planning and executing leader actions, and focus on learning the leader instruction generation model. the human follower acts in the world following the system instructions. we instantiate this scenario using cerealbar (section 3), a collaborative game, where two agents collect sets of cards together by moving in a 3d environment. task a world state s describes the current environment; in cerealbar, this includes the location of landmarks, cards, and both agents. a plan p is a sequence of poses p 1 , . . . , p | p| the system intends for the human user to take starting from a start state s 1 . in cerealbar, a plan includes moving in the environment with the intent of collecting cards; each pose p j is a tuple (h j , w j , α j ), where h j and w j are height and width coordinates, and α j is a discrete orientation angle. an instruction x is a sequence of tokens x 1 , . . . , x |x| . an instruction execution ē is the sequence of poses p 1 , . . . , p |ē| a user takes executing x, starting in a start state s 1 . the generation distribution p (x | s 1 , p; θ) is parameterized by θ. the goal of instruction generation is that given a generated instruction x ∼ p (• | s 1 , p; θ), the user execution ē from s 1 will follow the plan p. the user does not have access to p, but only to its description x. ## model we design a relatively simple encoder-decoder architecture to model the generation distribution p (• | s 1 , p; θ), leaving more complex model development for future work. the inputs are a start state s 1 and a plan p. the model parameters are θ. our design considers the environment and plan to generate relevant, grounded instructions. figure 3 illustrates the model. inputs similar to @xcite , we represent the world state s 1 ∈ {0, 1} p ×h×w as a binary 3d tensor, where p is the number of position properties, and h and w are the environment's height and width. each of the w × h positions is represented as a binary properties vector of length p (encoding the type of object in the position, its color, etc.). the system plan @xmath0, . . . , p | p| is a sequence of follower poses along the intended execution. each pose p j is a tuple (h j , w j , α j ) of height h j and width w j coordinates, and a discrete orientation angle α j . encoder the encoder computes a set of hidden states, which the decoder attends to during generation. we use a learned embedding function φ s to map each position vector to a dense embedding of size n s by summing the embeddings of each of the position's properties. we combine the embeddings into a tensor @xmath1, and compute: s ′ = cnn 1 (s), where cnn 1 is a learned convolution and s ′ ∈ r n s ′ ×h×w . because the cerealbar environment is a grid of hexagons, we use hexaconv @xcite . we encode the plan positions into a sequence of vectors p s ′ 1 , . . . , p s ′ | p| by cropping a n s ′ × n p × n p -sized tensors from s ′ centered around each (h j , w j ) and rotated by α j . these tensors represent the pose of the follower and its surroundings during execution. each p s ′ j is en-coded to p @xmath2, while retaining the dimensionality of p s ′ j . we concatenate an orientation embedding φ α (α j ) to each p j , and process we construct the set of hidden states p the decoder attends to by concatenating each h j with the n p × n p position vectors encoded in each p j : ## learning we continually improve our model by observing users following generated instructions and re-estimating the model parameters. we initialize the model parameters θ 1 using an existing language model and training on a static dataset of instructions d 0 (section 5.1). we then perform a series of rounds, each round r includes deploying the model with human users and training on the collected interactions (section 5.2). in round r, we collect interactions between our model parameterized by θ r and human followers, to create a dataset d @xmath3 ## results and analysis we conduct a long-term experiment with 14 rounds using our approach, and separate sevenround experiments to compare system variants. in both experiments, we collect roughly 100 interactions for each system per round. in the seven-round experiments, we deploy methods simultaneously to ensure that our observations are not sensitive to changes in user behavior, for example, because of adaptation and increased expertise. we do not inform workers about the model they are interacting with. we train each system only on data collected by the same method in previous rounds. ## related work learning for instruction generation has been studied using supervised methods, with examples of task specifications (i.e., contexts) paired with human-written instructions (e.g., @xcite , including to improve instruction following @xcite . we focus on continually learning by observing users executing generated instructions. this reduces annotation needs, and delegates much of the learning to interaction with users during system deployment. language generation in context was also studied in scenarios that are not explicitly instructional, but aim to elicit specific behavior, such as negotiation games (e.g., @xcite and referring expression generation (e.g., @xcite . @xcite survey existing work on language generation, including using rulebased methods. similar to our approach, some rule-based methods were evaluated with human followers in situated environments using task success (e.g., @xcite . such methods are accurate and reliable, but are limited to pre-specified rules and remain static following development. our focus is on studying the potential for learning by observing human behavior. the two approaches can be combined, for example by using rule-based methods to generate initialization data for our approach. bandit learning has been studied with simulated user ratings for machine translation @xcite @xcite and semantic parsing @xcite . we learn from real users, similar to recent studies in machine translation @xcite . in general, such learning assumes users can judge the system output, for example via proficiency in the language they wish to translate to. our learning signal does not require such expertise, and is available naturally from the interaction. explicit human feedback has also been incorporated into reinforcement learning methods @xcite @xcite @xcite @xcite , including in the context of dialogue system learning @xcite . @xcite study forming a reward from implicit feedback for non-task-oriented dialogue language generation, by training multiple models to detect linguistic signals, such as sentiment and lexical overlap, that correlate with explicit user feedback. learning from users has also been studied by asking users to rank system outputs (e.g., @xcite , including for instruction following @xcite and summarization @xcite . unlike our approach, such ranking requires knowing the true system intent, and is not part of the system's normal operation (i.e., instructing users in our case). incorporating human users into learning is related to active learning @xcite , where a policy selects examples for an oracle to label during learning. unlike common active learning scenarios we do not select examples from a static underlying distribution (i.e., a training set) for annotation, but generate examples with the learned model. this is similar to query synthesis active learning @xcite , where examples are generated for annotation, rather than being selected from a set of unannotated examples. a more significant difference is that active learning methods solicit model output annotations by presenting an oracle with model inputs. in contrast, our approach exposes users to model outputs (i.e., generated instructions). it does not solicit written instructions, as would be expected if requesting labels. we also do not show model inputs (i.e., plans) to users. finally, our model interacts with users during system operation, while completing its task. it does not require oracle annotators. language learning from behavioral signals has been studied in the cognitive science and psychol-ogy literature. 11 @xcite study two types of feedback in human studies: concurrent linguistic feedback and behavioral intent confirmation, and show how both influence linguistic adaptation in an interaction over time. studies of reference games reproduced the effect of confirmation feedback, showing that successful intent communication reinforces convention formation in the form of shorter references @xcite . our learning signal is a type of confirmation feedback. however, our interaction procures and makes use of more complex learning signals than a simple binary intent communication success, by using the path the listener takes in response to the generated instruction as an alternative intent when constructing data for learning (section 5.2). 12 ## discussion we propose a methodology to continually improve an instruction generation model by observing human users executing natural language instructions, and demonstrate its efficacy within a collaborative instruction following scenario. our study shows that observation of user behavior is an informative signal for generating language to relay instructional intent. to the best of our knowledge, this type of learning signal has not been studied before. this learning setting facilitates continual learning through interaction with users, and is particularly compelling for interactions with collaborative agents, including robots and software agents. such agents are likely to operate in constantly changing environments (e.g., robots in homes), where continual learning is necessary to adjust to changes. our continual learning approach also provides systems the flexibility to co-adapt to human users, who are likely to change preferences and behaviors in response to system behavior. our experiments demonstrate the learning process is robust to various learning and process design choices. however, they also show it is accompanied by a reduction of language complexity, including reducing the effective vocabulary and sentence length. while much of the decrease in the effective vocabulary size throughout the system lifetime relates to generating fewer erroneous phrases, it also reduces the language diversity and descriptiveness. our experiments show that this trend can be slowed down by using negative examples, and appears to be less pronounced when using fine-tuning. the combination of this decrease with the preference for shorter instructions makes it difficult for the system to describe longer, complex trajectories. qualitatively, we observe this open problem is responsible for a significant portion of the remaining errors. an important direction for future work is experimenting with directly encouraging more diverse language. this can be combined with approaches that allow for introducing new word types, which is unlikely in our approach, even though it uses sub-word tokenization. a potential direction in this vein is combining active learning to solicit human-written oracle instructions for plans the system fails to communicate. our work highlights several other directions for future work. there is a strong need for a reliable automated metric to evaluate instruction generation. in absence of such a metric, we use a simple, but likely sub-optimal stopping criteria for learning. beyond the learning signal we explored in our experiments, there are additional potential cues available during interaction. for example, using continuous-valued similarity between system intent and user execution, modeling follower quality to discount the learning signal from interactions with bad followers, or weighing the feedback questions differently for more nuanced reward. finally, the decrease in utterance length and vocabulary size mirrors similar trends observed in studies of human communication @xcite . this illustrates the potential of continual learning systems to reflect the dynamics of language change human participants expect in natural language interactions. observations of human learning also indicate the potential of integrating our approach with conversational self-repair @xcite and partner reformulation @xcite , both important components of child language acquisition that likely provide better credit assignment for learning compared to our binary bandit signal.