Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Game (2024)

Abstract: In this work, we evaluate the adaptability of neural agents towards assumed partner behaviors in a collaborative reference game. In this game, success is achieved when a knowledgeable guide can verbally lead a follower to the selection of a specific puzzle piece among several distractors. We frame this language grounding and coordination task as a reinforcement learning problem and measure to which extent a common reinforcement training algorithm (PPO) is able to produce neural agents (the guides) that perform well with various heuristic follower behaviors that vary along the dimensions of confidence and autonomy. We experiment with a learning signal that in addition to the goal condition also respects an assumed communicative effort. Our results indicate that this novel ingredient leads to communicative strategies that are less verbose (staying silent in some of the steps) and that with respect to that the guide’s strategies indeed adapt to the partner’s level of confidence and autonomy.

## introduction sometimes we feel like we could continue another person's sentence. this happens in particular with people we know well or we often interact with. a common phrase coined to this phenomenon is that "people are on the same wavelength". indeed @xcite found that brain activities somewhat synchronize between teachers and students during lessons. even more surprising, synchronicity becomes a good predictor of the learning success of the students. a psycho-linguistic study by @xcite observed the language use of collaborative partners during an ongoing goal-oriented interaction: they (implicitly) agree on newly introduced noun phrases and a common strategy to achieve the goal together. interestingly, the number of used words drastically decreases during the collaboration. the participants strive towards reduced individual efforts while the number of successful outcomes stays high. we see that human-human interaction is characterized by synchronicity (adaption) and the reduction of individual efforts. still, the modelling of changing behaviors (or different others) remains an open problem "due to the essentially unconstrained nature of what other agents may do" @xcite . are neural agents capable of adapting to their interactants and converge to useful strategies when the partner's behavior becomes apparent only during an ongoing interaction itself? in this work, we frame a collaborative language coordination and grounding task (see figure 1 ) as a reinforcement learning problem @xcite and evaluate, if and to which extent a common training algorithm proximal policy optimization (ppo) @xcite is able to produce neural agents that perform well with a variety of partner behaviors. to study how learning agents potentially adapt to an assumed partner's behavior, we propose a challenging vision and language grounding task where two players have to coordinate on the selection of a puzzle piece (a pentomino, a shape of five adjacent squares; @xcite ) among several distractors while (i) the actual target piece is only known to one of them (the guide), and (ii) only the other can perform the selection (the follower). the main idea is that we assume an ongoing interaction in which the follower's behavior changes. after some time the follower should become more autonomous and more confident in choosing actions and executing its own plan (as pointed out by @xcite ). but instead of treating this as a multi-agent setting directly, we follow @xcite with the notion of assigning different agents to different sub-tasks and learn a policy for each of the controllable follower behaviors (the sub-tasks) separately. the resulting policies represent a guide's communicative strategy at certain points in time of the assumed ongoing interaction. our expectations on the learned communicative strategies of the guide are that in the beginning (with a less autonomous, less confident follower) more is to be said. later on, with a more autonomous and confident follower, the guide learns that it "does not need to say anything" to be successful and consequently reduces its effort. our contributions are as follows 1 : natural language goals in rl. using natural language to describe the goal state in an rl problem has become a common theme @xcite @xcite @xcite . this research direction is interesting because it could allow humans to interact more easily with learned agents. there is work that shows that intermediate language inputs are a valuable signal in task-oriented visual environments @xcite . indeed @xcite found that natural language can "provide a gradient" towards the goal state. but they also point out the "brittleness" of these signals because the language input might align badly with sub-trajectories. a key challenge here is the variability of expressions in language that can be produced and understood in the defined action space. even in relatively simple environments, there might arise an overwhelming amount of situations for an agent to handle @xcite . we weaken the action space exploration problem by using ideas from natural language understanding @xcite and let the guide produce language actions in a well-defined reduced "intent space". these intents are then verbalized (using templates; which could be a conditioned pretrained language model) and given to the follower. interactive sub-goal generation in rl. @xcite use a pre-trained large language model to generate possible plans (in the form of source code) for the completion of a task. the learning process is extended with a mechanism that allows then, the follower receives the utterance l t , the gripper coordinate g t and a symbolic representation of a partial view of the scene p t . the hand-crafted policy updates the plan accordingly based on its given representation of the world. finally, the follower's next planned action (or wait) is performed with a certain chance defined by the attached confidence. the process repeats until a piece is taken or time runs out. the model to learn the refinement of single actions or an entire plan respectively. indeed neural agents perform better when they self-predict sub-goals to be achieved (with an intrinsic reward) instead of reaching for the final goal immediately @xcite @xcite . for example, lee and kim (2023) study the task of finding the best route in a simple visual domain by training a subgoal system that predicts intermediate coordinates. in contrast to them, our guiding agent has to produce utterances to describe a sub-goal (and we use referring expressions or directions). @xcite also address the question of "when to provide sub-goals", which is necessary in our task. nevertheless, in distinction to these works, we treat the sub-goal generation not just as additional information for the follower's success but are interested in the learned communicative strategies themselves. we treat the sub-goal providing guide as an individual participant in the environment similar to a multi-agent setting. skill learning in cooperative multi-agent rl. we treat both guide and follower as agents in a cooperative setting and follow work that uses handcrafted policies @xcite @xcite . in this sense, our approach is similar to heterogeneous skill learning @xcite @xcite where a single agent is trained to acquire a variety of skills (in our case communication strategies). this is, in particular, helpful due to the differences in the action spaces of the guide (language acts) and the follower (movements). in addition, this method of having a hand-crafted follower policy allows us to avoid the problem of emergent communication where agents agree on a language that becomes inaccessible to humans @xcite . ## the collaborative reference game we use a collaborative game of referential and interactive language with pentomino pieces @xcite and extend it for guidance learning. a guide has to instruct a follower to select a specific target piece with a gripper. in this setting, both players are constrained as follows: the guide can provide utterances but cannot move the gripper. the follower can move the gripper but is not allowed to provide an utterance. this asymmetry in knowledge and skill forces them to work together and coordinate. @xcite found that such a reference game leads to diverse language use on the guide's side. ## the follower behaviors for the follower, we take inspiration from @xcite who suggest a plan-based approach towards solving text-based tasks with language models: given a task's natural language instruction their model initially produces a plan, which is then executed and repeatedly refined or revised. we implement a policy that keeps track of a plan that contains up to 10 actions (the plan horizon; which is exactly the number of actions needed to reach the diagonal corner of the partial view). our follower's behavior of following the plan is adjustable along two dimensions: confidence and autonomy. confidence. the actions in the plan are associated with a decreasing probability of being executed (the "confidence triangle" in figure 2 ) so that given a discount factor ϕ ∈ [0, 1] and a lower threshold @xmath0, 1] we calculate: which introduces a notion of confidence: either the planned action is executed or a wait action occurs (hesitation). furthermore, this conceptualizes that a follower becomes increasingly unsure about the continuation of the plan without receiving feedback from the guide. autonomy. the revision process for our follower policy is conceptually divided into five subprograms that run after the guide's utterance is received, parsed and the assumed intent type is determined, as follows: then, the next action is chosen as described under on_silence. now, the autonomy defines which procedures the follower undertakes, when intermediate feedback is missing (the guide stays silent). the cautious follower is performing solely the previously defined procedures: when the plan is exhausted, then it waits until a new directive or reference is given. if this follower is over an assumed target piece, then it waits until the "take" directive is given by the guide. in contrast, the eager follower aims to actually take an assumed target piece when approaching it in the current field of view. furthermore, the eager follower autonomously looks for target candidates at each step (as described in the on_reference procedure) and potentially revises the plan (also when the guide stays silent). ## conclusions in this work, we examined an interesting intersection between psycho-linguistic studies and deep learning with reinforcement learning. we considered neural agents as possible interaction partners (for humans) in a challenging reference game where a guide has to learn when, what, and how information (actionable intents) is to be provided to a follower. as a proxy for different follower behaviors, we implemented a hand-crafted policy that is controllable along two dimensions: autonomy in exploration and confidence in executing an action. we experimented with a learning signal that in addition to the goal condition also respects an assumed communicative effort. our results indicate that this formulation of the learning signal leads to communicative strategies that are less verbose (stay silent more often) and that the resulting guide behaviors are adapted (in terms of intent selection distributions) to the follower's autonomy and confidence levels. we think this work presents a useful case study of neural agents that have to learn adapted communication strategies in an interactive setting (possibly with humans). in future work, we want to investigate other reward formulations for the reference game and evaluate the learning of communication policies where the utterance production process spans multiple time steps (one word at a time) and the production must be possibly interrupted and revised during the interaction. figure 5 : the distribution of the preference order choices for the reference action (from figure 4 ). the preferences over position (p), shape (s) and color (c) are given to the ia for reference production. figure 6 : the mean number of silent turns performed by the learnt policies (incl. all seeds) during the test episodes. we fitted a linear regression with a confidence interval of 99% through the data points separately for the followers with ϕ = {75, 85} and ϕ = {90, 95, 97, 99}. the latter shows a trend towards more silence turns when the guide is paired with more confident followers. the ia on symbolic properties as based on the formulation by van deemter (2016) require: a set of distractors m , a set of property values p of a referent r and a linear preference order o over the property values p 1: d ← ∅ 2: for p in o(p) do the algorithm has a meta-parameter o, indicating the preference order, which determines the order in which the properties of the referent are tested against the distractors. in our domain, for example, when color is the most preferred property, the algorithm might return blue, if this property already excludes all distractors. when shape is the preferred property and all distractors do not share the shape t with the referent, t would be returned. hence even when the referent and distractor pieces are the same, different preference orders might lead to different expressions. there are 3 expression templates that are used when only a single property value of the target piece is returned by the incremental algorithm (ia): the maximal sentence length is 12.