Supervised Seeded Iterated Learning for Interactive Language Learning (2020)

Abstract: Language drift has been one of the major obstacles to train language models through interaction. When word-based conversational agents are trained towards completing a task, they tend to invent their language rather than leveraging natural language. In recent literature, two general methods partially counter this phenomenon: Supervised Selfplay (S2P) and Seeded Iterated Learning (SIL). While S2P jointly trains interactive and supervised losses to counter the drift, SIL changes the training dynamics to prevent language drift from occurring. In this paper, we first highlight their respective weaknesses, i.e., late-stage training collapses and higher negative likelihood when evaluated on human corpus. Given these observations, we introduce Supervised Seeded Iterated Learning (SSIL) to combine both methods to minimize their respective weaknesses. We then show the effectiveness of in the language-drift translation game.

## introduction since the early days of nlp @xcite , conversational agents have been designed to interact with humans through language to solve diverse tasks, e.g., remote instructions @xcite or booking assistants @xcite . in this goal-oriented dialogue setting, the conversational agents are often designed to compose with predefined language utterances @xcite @xcite . even if such approaches are efficient, they also tend to narrow down the agent's language diversity. to remove this restriction, recent work has been exploring interactive word-based training. in this setting, the agents are generally trained through a two-stage process @xcite @xcite @xcite : firstly, the agent is pretrained on a human-labeled corpus through supervised learning to generate grammatically reasonable sentences. secondly, the agent is finetuned to maximize the task-completion score by interacting with a user. due to samplecomplexity and reproducibility issues, the user is generally replaced by a game simulator that may evolve with the conversational agent. unfortunately, this pairing may lead to the language drift phenomenon, where the conversational agents gradually co-adapt, and drift away from the pretrained natural language. the model thus becomes unfit to interact with humans @xcite @xcite . while domain-specific methods exist to counter language drift @xcite , a simple task-agnostic method consists of combining interactive and supervised training losses on a pretraining corpus @xcite , which was later formalized as supervised selfplay (s2p) @xcite . inspired by language evolution and cultural transmission @xcite , recent work proposes seeded iterated learning (sil) @xcite as another task-agnostic method to counter language drift. sil modifies the training dynamics by iteratively refining a pretrained student agent by imitating interactive agents, as illustrated in figure 1 . at each iteration, a teacher agent is created by duplicating the student agent, which is then finetuned towards task completion. a new dataset is then generated by greedily sampling the teacher, and those samples are used to refine the student through supervised learning. the authors empirically show that this iterated learning procedure induces an inductive learning bias that successfully maintains the language grounding while improving task-completion. as a first contribution, we further examine the performance of these two methods in the setting of a translation game @xcite . we show that pretrained agent data initialize teacher teacher interaction data generation imitation duplicate student t duplicate student t+1 figure 1 : sil @xcite . a student agent is iteratively refined using newly generated data from a teacher agent. at each iteration, a teacher agent is created on top of the student before being finetuned by interaction, e.g. maximizing a task completion-score. teacher generates a dataset with greedy sampling and students imitate those samples. the interaction step involves interaction with another language agent. s2p is unable to maintain a high grounding score and experiences a late-stage collapse, while sil has a higher negative likelihood when evaluated on human corpus. we propose to combine sil with s2p by applying an s2p loss in the interactive stage of sil. we show that the resulting supervised seeded iterated learning (ssil) algorithm manages to get the best of both algorithms in the translation game. finally, we observe that the late-stage collapse of s2p is correlated with conflicting gradients before showing that ssil empirically reduces this gradient discrepancy. ## preventing language drift we describe here our interactive training setup before introducing different approaches to prevent language drift. in this setting, we have a set of collaborative agents that interact through language to solve a task. to begin, we train the agents to generate natural language in a word-by-word fashion. then we finetune the agents to optimize a task completion score through interaction, i.e., learning to perform the task better. our goal is to prevent the language drift in this second stage.