A Dialogue Game for Eliciting Balanced Collaboration (2024)

Abstract: Collaboration is an integral part of human dialogue. Typical task-oriented dialogue games assign asymmetric roles to the participants, which limits their ability to elicit naturalistic role-taking in collaboration and its negotiation. We present a novel and simple online setup that favors balanced collaboration: a two-player 2D object placement game in which the players must negotiate the goal state themselves. We show empirically that human players exhibit a variety of role distributions, and that balanced collaboration improves task performance. We also present an LLM-based baseline agent which demonstrates that automatic playing of our game is an interesting challenge for artificial systems.

## introduction language use is a highly collaborative process that involves constant negotiation and cooperation between interlocutors, with the ultimate goal of facilitating mutual understanding @xcite @xcite ). an improved understanding of these negotiation processes would benefit the development of future systems for effective human-ai cooperation and go beyond today's rigid division of roles between dialogue systems and users @xcite . collaborative dialogue is often studied through dialogue games involving reconstruction, where one player has information about a target configuration and guides the other player towards it @xcite @xcite . this approach places the players in fixed roles (instruction giver/follower), which is in contrast to the fluid and implicit negotiation of these roles in naturally occurring collaborative dialogue, in turn limiting the ability of such games to elicit negotiation about collaborative roles. in this paper, we directly address this issue by introducing a collaborative game designed to elicit dialogues with more flexible role-taking -a 2d object placement game in which the target configuration is not predetermined, but must be negotiated by the players. the players use online chat to jointly decide how to arrange movable objects, without seeing each other's boards. the initially symmetric roles ensure a level playing field between players in terms of environment knowledge and the goal state. by describing the target state as only "an identical placement", we transfer the task of goal state selection onto the players, which, in turn, enables us to study the task-solving approach that they choose to take. we observe that players indeed exhibit a variety of collaboration strategies in this dialogue game, further illustrated by a metric we define, the dominance score, representing the degree to which one player controls the gameplay. only a minority of player dyads choose an asymmetric strategy in which one player always dominates; this strategy is also associated with systematically lower scores than more balanced strategies. finally, we describe a baseline computational agent for this game. it achieves a significantly lower average score than a human player using a limited collaboration strategy, indicating that natural and effective collaboration in balanced games like ours is an interesting avenue for future research. foot_0 2 background collaboration in dialogue. in situated dialogue, common ground and shared context are paramount to avoiding misunderstandings @xcite . interlocutors commonly engage in a collaborative effort, i.e., negotiation, to establish these commonalities, and, ultimately, a joint purpose @xcite , frequently through a coordinated referencing approach @xcite . in order player 1 task view let's place the pink pillow "inside" the oven to the left or the right side of the tap? done; blue hat on toaster, covering the toaster itself yep covering the toaster once it's done you can only see the top left side of the toaster done, now cowboy hat, lets place it on the counter next to the tap left side so not on top, but covering the toaster? done chat box (shared information) player 2 task view to arrive at a unified goal, the interlocutors must work together and coordinate their actions over time @xcite . this ongoing coordination process can lead to the acquisition of new knowledge, including how to coordinate better @xcite . collaborative games. there have been many reconstruction game environments developed for the purpose of studying collaboration and negotiation (e.g. 2024) ). however, all previously cited environments assume a predetermined target state to which one player must guide the other. this inherently places the participants on different levels dependent on the role they are assigned (instructor vs. follower), determined by the information they are given. we go beyond this by removing these constraints and allowing for more balanced task-solving approaches, which are necessary for a holistic study of collaboration @xcite . human-computer collaboration. here we refer to all collaborative situations in which "agents may be able to achieve joint gains or avoid joint losses" @xcite . in the field of humancomputer dialogue systems, the most frequent such agents are instruction-giving @xcite @xcite @xcite , or instruction-following @xcite . while they do involve a level of first-hand human-computer interaction and dialogue necessary for completing a given task, both cases are characterized by a built-in asymmetry, analogous to the aforementioned reconstruction games. in order to ensure successful and robust human-computer cooperation, and facilitate trust, it is integral for inherently collaborative systems (e.g., assistants) to be able to handle balanced collaboration, as well @xcite . ## collaborative object-placement game we developed a collaborative, 2d object placement game that can be played by two players over the internet. in each round, the two players see an identical, static background, upon which movable objects have been placed in random positions that are different for the two players (see figure 1 ). the goal of the game is for the players to place each object in the same position by dragging it with the mouse. players cannot see each other's scene; they can only communicate through a chat window. each pair of players played two rounds of the game together, with a kitchen background in the first round and a living room in the second (see appendix a.1 for more images). this allowed us to study how their collaboration strategies evolved as they became more familiar with each other. we make the game available online by integrating it into slurk @xcite , which is a dialogue collection platform built to deal with server-side client events and api calls, ensuring participants could play the game online; additionally, it provides a straightforward and customizable logging system, as well as an off-the-shelf front-end interface with a built-in chat box feature. all the images are "cartoonish" illustrations of real rooms and objects, in order to facilitate natural-language communication while creating a "game" feeling. there were a total of five movable objects: a pillow, pair of pants, trash bag, flat cap, and cowboy hat. we found five items to strike a good balance between rich interactions and efficient gameplay. our game implementation prevented placing objects on top of each other in order to enforce nontrivial reference to locations through background landmarks. the players were scored jointly, based on the mean manhattan distance between identical objects. the closer the two common objects were placed on the grid, i.e., the smaller the distance between them was, the higher the score the pair received. the score was normalized on a scale from 0 to 100, contributing to the typical game "feel". participants with very high scores (>99) got awarded a bonus. ## game playing strategies we gathered a dataset of 71 games by crowdsourcing participants via prolific. we used this data to analyze human dialogue behavior in a collaborative environment. ## baseline llm agent our game is intended as a testbed for computational agents that collaborate effectively with humans. to gauge how challenging it is for such agents, we evaluated a simple baseline agent based on llms. the agent enforces a leader strategy, with the human player as the leader, by asking the human player for instructions in the first message and remaining passive and reactive otherwise. it uses an llm to perform simple semantic parsing of the human's instruction into triples of the form (object to move, landmark in the scene, spatial relation) and then uses simple handwritten rules to map such triples into (x, y) positions in the scene. for in-stance, if the centerpoint of the fridge is at position (x, y), the description "above the fridge" will be resolved to (x, y -10). we use few-shot instruction giving with gpt 3.5 turbo instruct (openai, 2023); see appendix a.2 for details. in an online evaluation with ten human participants (referred to as "the llm batch"), the agent obtained a mean score of 84.2 (left plot of fig. 2 ). this shows that the task is within reach of llmbased agents; at the same time, the agent considerably lags behind even the human-human leader strategy, suggesting that effective collaboration remains a challenge. moreover, we calculate the mean dominance score difference and report a score of 2.02 (see ## conclusion and future work we have presented a 2d object placement game which is suitable for eliciting dialogues with varied collaboration strategies. this is in contrast to earlier dialogue games, in which one player typically takes the lead. the key innovation of our game is that players must negotiate their joint goal state. a baseline computational agent achieves a task performance that is within reach of, but still considerably below human performance, indicating that variants of our game would be an interesting and challenging platform for investigating humancomputer collaboration. in the future, it would be interesting to explore even more balanced versions of the game, e.g. by adding rules that increase the cost of failed collaboration. another avenue of future research is to investigate the interplay of collaboration strategy and mutual adaptation of the player's lexica. additionally, it would be insightful to empirically verify the dominance distribution of the other aforementioned dialogue games' outputs, as well as to further investigate the contributing factors to the occurrence of mixed-leader dialogues beyond the symmetric roles and lack of predefined goal state. moreover, the dominance score is a useful operationalization of the collaborative imbalance between players, but it is an approximation that does not actually take the content of the players' chat messages into account. it might be interesting to refine this measure in the future, e.g. by having the messages evaluated by an llm. nevertheless, the post-hoc manual analysis of the games in the four strategies indicates that the dominance score captures differences in collaboration strategy well. lastly, the llm agent presented in section 5 is a relatively simple baseline. it is conceivable that a more intricate llm model would close the gap to human performance, at least to the leader strategy. we leave the exploration of such models, and of more intricate versions of our game that would remain challenging for them, for future research.