Analysis of Language Change in Collaborative Instruction Following (2021)

Abstract: We analyze language change over time in a collaborative, goal-oriented instructional task, where utility-maximizing participants form conventions and increase their expertise. Prior work studied such scenarios mostly in the context of reference games, and consistently found that language complexity is reduced along multiple dimensions, such as utterance length, as conventions are formed. In contrast, we find that, given the ability to increase instruction utility, instructors increase language complexity along these previously studied dimensions to better collaborate with increasingly skilled instruction followers.

## introduction community language change in situated collaborative task-oriented scenarios has been studied with focus on reference games (krauss and weinheimer, 1964; clark and wilkes-gibbs, 1986; hawkins et al., 2017, 2020a,b), where two participants coordinate using language to select to a single item from a set of available items. these studies found that utility-maximizing participants trade surfaceform linguistic complexity with established norms, as the familiarity and expertise of the interaction partners increase. in practice, this emerges as a reduction in utterance length and vocabulary size. we study the generality of these observations by analyzing language change in a collaborative instructional task, where instructors can specify multiple goals within a single instruction to increase their utility. this option, not present in reference games, creates competing incentives: increasing utility by issuing more goals in a single instruction versus decreasing language effort by utilizing established norms (e.g., by shortening instructions). we use the cerealbar game environment and its accompanying dataset (figure 1 ; suhr et al., ## scenario and data overview we use the cerealbar game and accompanying dataset (suhr et al., 2019) in our analysis. cere-albar is a collaborative, two-player game, where a leader and a follower collect matching sets of cards by moving in an environment. the game is turn-based, and each player has a limited number of steps per turn. the leader both collects cards and instructs the follower using natural language. @xcite the follower executes leader instructions. the players' abilities differ: the leader observes the complete environment and plans sets to collect; the follower only observes what is ahead, but has more steps per turn. for each set made, players receive one point and additional turns, allowing them to complete more sets. success requires the players to collaborate via natural language: the leader must write informative instructions to the follower, and the follower must efficiently follow these instructions. figure 1 shows a snapshot of the game. the cerealbar dataset contains 1,202 humanhuman game interactions collected over the course of four months. workers were randomly assigned as leader or follower for each interaction. the collection process created a wizard-of-oz setup: the system user, as the leader, provides instructions and acts in the world, and the human follower is a wizard, executing instructions to emulate the desired system behavior. we only use interactions from the training split for our analysis. we prune interactions by inexperienced workers, as classified when the data was collected, to focus on the impact of experience. @xcite in total, we consider 795 interactions. ## data analysis to analyze trends over the data collection period, we split the data chronologically into 10 deciles of roughly equal size (79 or 80 interactions). an average of 40 workers participated in each decile (figure 2 , left). the community stabilized after decile 4, as worker recruitment slowed and the community was split by expertise. @xcite interaction goals are increasingly achieved over time. mean score per game increases from 3.8 to 12.3 (@xmath0.0001) (figure 2 , right). @xcite execution efficiency and game expertise also improve. @xcite our focus is how leader language -the sole communication conduit -changes to enable these gains. we design our analysis to be as similar as possible to existing work on reference games (hawkins et al., 2020a), which shows that certain language aspects are simplified as community conventions form. cerealbar allows for a different realization of common ground development than previously studied reference games, and we observe trends that are in contrast to this line of prior work. instruction length and vocabulary mean 6 instruction length increases from 11.9 to 14.1 tokens 7 (@xmath1.0001) over time, while vocabulary size in-creases from 752 to 1,070 unique tokens (figure 3 ). this contrasts with reference games, where utterance length and vocabulary size reduce (clark and wilkes-gibbs, 1986; hawkins et al., 2017). some of the words added more specifically describe props or movements. however, the overall trend is relatively complex, and identifying clear patterns likely requires a more targeted scenario design. ## discussion and related work the cerealbar scenario is related to reference games (krauss and weinheimer, 1964; clark and wilkes-gibbs, 1986; hawkins et al., 2017; monroe et al., 2017; he et al., 2017; udagawa and aizawa, 2019; haber et al., 2019), which require two players to agree on a single referent from a set via dialogue. cerealbar differs in several ways. it allows only unidirectional language communication, and utterances in cerealbar are instructions specifying desired follower behavior with any number of tasks to complete (i.e., with flexible utility), not a description of a single target referent. these differences lead to different language dynamics. in reference games, hawkins et al. (2020a) observed the development of specialized reference phrases for ambiguous shapes, which allows players to reduce their utterances' length and syntactic complexity. given that cerealbar objects are generally unambiguous and familiar, players do not begin with overly verbose references, and have less potential for reduction to more concise references. in contrast, we observe increased instruction length and complexity. leaders issue an increasing number of tasks to the follower per instruction, utilizing the flexibility afforded by cerealbar's design. this less constrained scenario better reflects real-life collaborations, where participants complete many tasks to achieve complex goals. our observations show the competing effects of cost-minimization and utility-maximization. the formation of common ground and expectations on partners' behavior enables leaders to use language differently to convey more information-dense instructions to optimize game performance. this is aligned with the expectation of better communication grounding between community members in clark and marshall (1981), and with how grounding in clark and wilkes-gibbs (1986) manifests as reduced complexity when utterance utility is fixed. because there are conflicting forces at work in ce-realbar, common ground is realized differently. the most related setup to cerealbar is the cards task (djalali et al., 2012; potts, 2012), where two players collect a single set of cards. it uses four static environments and studies dialogue, not instructions. djalali et al. (2011) showed cards players increase the interaction complexity by developing a rich common ground, including terms for the fixed board locations. this is less likely with the randomly generated cerealbar envi-ronments. utterances in cards also become shorter, potentially due to the predefined number of goals. language complexity also increases in communities where users jointly build a natural-languagelike programming language (wang et al., 2017; gavran et al., 2018). this scenario differs from ours in lacking explicit collaboration on tasks, focusing on a learned programming language rather than natural language, and training a single model, differently from our many-listeners community. the language dynamics observed in cereal-bar contrast with those previously observed in reference games, providing evidence that gradual formation of common ground among interaction participants does not necessarily result in reduced complexity of sentences, and may even result in increased complexity. our conclusions do not void nor mutually exclude previous work, but illustrate the complexity of language change over time in a community. an important direction for future work is controlled studies to observe the effects of scenario design on the interaction between the development of common ground and language change. takuma udagawa and akiko aizawa. 2019. a natural language corpus of common grounding under continuous and partially-observable context. in proceedings of the conference on artificial intelligence. sida i. wang, samuel ginn, percy liang, and christopher d. manning. 2017. naturalizing a programming language via interactive learning. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 929-938, vancouver, canada. association for computational linguistics. yang xu and david reitter. 2016. convergence of syntactic complexity in conversation. in proceedings of the annual meeting of the association for computational linguistics.