CPS - T ask F orge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks (2024)

Abstract: Teams can outperform individuals; could adding AI teammates further bolster performance of teams solving problems collaboratively? Collaborative problem solving (CPS) research commonly studies teams with two agents (human-human or human-AI), but team research literature finds that, for complex tasks, larger teams are more effective. Progress in studying collaboration with more than two agents, through textual records of team interactions, is hindered by a major data challenge: available CPS corpora are predominantly dyadic, and adapting pre-existing CPS tasks to more agents is non-trivial. We address this data challenge by developing a CPS task generator, CPS-TaskForge, that can produce environments for studying CPS under a wide array of conditions, and releasing a CPS task design checklist grounded in the theoretical PISA 2015 CPS framework to help facilitate the development of CPS corpora with more agents. CPS-TaskForge takes the form of a resource management (tower defense) game, and different CPS tasks can be studied by manipulating game design parameters. We conduct a case study with groups of 3–4 humans to validate production of diverse natural language CPS communication in a game instance produced by CPS-TaskForge. We discuss opportunities for advancing research in CPS (both with human-only and human-AI teams) using different task configurations. We release all data and code.

## introduction modern life requires teamwork to solve problems @xcite , but what makes a team work well together? this area of study, known as collaborative problem solving (cps), is active across many disciplines, e.g., psychologists study the construction of team mental models in team discussions @xcite , business management sciences investigate how communication style affects 1 https://github.com/nhaduong/cps-taskforge performance evaluation @xcite , and educators develop tools to teach team communication strategies @xcite , emphasizing the research direction of discovering how team members talk to one another. conducting empirical work in cps faces many challenges, in large part because of a large cps task design space (e.g., what is the problem, who makes up the team, and who knows what information when). as a result, despite extensive interdisciplinary work in cps, task designs in empirical studies have often focused on teams of two collaborating to solve problems such as selecting a designated object, modeling search and rescue, and making decisions. ai agents have the potential to increase team effectiveness, and developing ways to integrate ai into teams is an active area of research in communities such as hci @xcite , nlp @xcite , and ai fairness @xcite . example integrations include aiassisted decision making with one human and one ai (e.g., cancer diagnosis, @xcite and ai-assisted creative tooling (e.g., @xcite . developing these collaborative tools is made possible through open datasets. for example, various amazon reviews datasets (e.g., @xcite @xcite have been used to develop sentiment classifiers and deception detectors that can be used as ai-assisted decision makers, and the reddit writingprompts dataset @xcite has been valuable in developing co-writing ai systems. unfortunately, a paucity of open datasets with more than two parties leads to challenges in integrating ai with larger human teams, as we lack understanding of team dynamics when an ai communicates to a team, rather than an individual. to support cps study across different designs (e.g., adding a third ai teammate to a two-human team or using voice instead of text communication), we introduce a cps task environment genera-tor, cps-taskforge. cps-taskforge instantiates a resource management activity through a tower defense game and supports adjusting a range of cps design parameters such as team composition, communication method, and how stressful the task is. in a tower defense game, players must defend their base by using limited resources to construct towers that can defeat enemies before the enemies destroy the base. we provide a cps task design checklist, cps-✓, adapted from the pisa 2015 theoretical cps framework (pisa2015) developed by the oecd @xcite , to support generating the desired task environment with cps-taskforge. we illustrate cps-taskforge capabilities by presenting several cps task designs and conducting a case study that can collect human communication data exhibiting a range of cps skills, including social skills such as maintaining group communication and cognitive skills such as developing strategic plans. our study has small groups of 3-4 participants complete a task multiple times with increasing difficulty. we observe many different successful strategies and a wide range in cps skill usage across teams, demonstrating the versatility of collecting data through cps-taskforge. to summarize our contributions: ## collaboration and problem solving collaborative problem solving (cps) processes are well-studied for human teams, but when human-ai teams are considered, downstream task performance has been prioritized, leaving human-ai cps processes understudied. for example, @xcite found human team communication more effective when the appropriate style was used in conjunction with the delivery of relevant information. humans have different expectations towards ai teammates @xcite @xcite , so human-ai teams may value communication style differently. studying human-ai cps processes requires developing the appropriate datasets, but resources for creating such data is deficient. understanding how effective and efficient communication can predict successful teamwork requires collecting data in a variety of cps settings. the tasks used to elicit relevant data often model real-world activities, e.g., rescuing humans from a burning building (asist; @xcite , instruction following through selecting designated objects (e.g., pentoref, @xcite @xcite @xcite , and navigating environments (e.g., hcrc map task, @xcite , and use human participants. the resulting datasets have been used to study a wide variety of communication and linguistic phenomena, including language entrainment (i.e., when communicative behavior becomes similar among interlocutors, including lexical choice and rhythm) and common ground building (i.e., when interlocutors develop their own code). to the best of our knowledge, analogous settings incorporating an ai team member in a cps task have not explored similar communication and linguistic phenomena because only recently has ai-generated natural language become indistinguishable from humans @xcite , enabling exploration of ai teammates as peers. unfortunately, expanding pre-existing datasets to other cps settings, such as involving an ai agent or a third human team member, is challenging because the tasks were designed to study a specific team composition; for example, what role would a third participant play in a navigation task originally designed for one human to tell another human where to go? despite the extensive body of literature studying cps, publicly available resources remain scarce, particularly when more than two agents are involved. we summarize a sample of cps task ac-task type ## cps-taskforge and tower defense to advance cps research, we need ways to systematically study cps when varying factors, allowing comparison of cps results across settings. we therefore develop a cps task environment generator, cps-taskforge, which can generate cps environments with different design factors. we also release a cps task design checklist, cps-✓, that describes how varying design factors produces different environments. we defer discussion of cps-✓to section 4; here we give a concrete description of the task environments our work targets. we start with several requirements: (r1) cps-taskforge should be built on an activity that can support the different values in cps-✓; (r2) the activity should be fun, to motivate participant signups, because cps studies require multiple participants, making scheduling a logistical barrier to conducting cps research; (r3) the activity should be easy to learn for both participants and researchers, in order to minimize time spent in tutorials and allow researchers to quickly design different cps studies; and (r4) the activity should easily scale in difficulty to enable cps research studying effects of expertise on collaboration. we meet our design requirements by using the tower defense (td) game genre as our cps-taskforge activity. the premise of a td game is to defend a base from enemies by placing towers on the map, which can destroy the enemies. td games require strategy and resource management-a vital aspect of cps tasks @xcite -and games have been successfully used by the research community to study communication (e.g., codenames; @xcite ) and collect data (e.g., verbosity; @xcite , duolingo (von ahn, 2013), searchwar @xcite , and matchin @xcite . td games are known for having a gentle learning curve, short levels (r3), and ease in scaling difficulty through simple designs (r1, r4; @xcite . the 2021 mobile market value for td games was estimated at 940 million usd (analytica, 2022); this popularity suggests the potential for participants to play the game of their own volition (r2). it is also known to support 1-4 players in cooperative play, foot_0 natively supporting studying human-ai teams involving as few as one human. we briefly describe what a td game involves, referencing an in-game screenshot (figure 1 ) of an environment produced by cps-taskforge. in a td game, the player needs to defend their base (7) from enemies by placing towers on the map whose inhabitants can attack the oncoming enemies. the enemies will appear at designated spawn points (1) and traverse the map along specific paths known to the player, allowing the player to strategize where to place towers effectively. players must manage their resources (3) (e.g., gold and map real estate) when developing their defense strategy. levels differ in the enemy spawning behavior (e.g., enemies can spawn without a break, or there is time in between groups of enemies), enemy variants (e.g., a faster or slower enemy), map terrain (e.g., obstacles can prevent tower placement), and player resources (e.g., types of towers, amount of starting gold). the standard td game has two phases: planning, a static phase where players can place towers on the map, and attack, a dynamic phase during which enemies spawn, and players can react to the changing situation by adjusting their towers. cps-taskforge is built on the open-source godot foot_1 game engine, and further details of implementation and the tower defense games it produces are available in appendix a and the documentation of our open-source release. ## cps-✓ : a cps task design checklist the pisa 2015 cps framework (pisa2015) (oecd, 2017) describes cps tasks through a set of 15 design factors, showing how different cps settings can be studied by manipulating different combinations of factors (e.g., team size and composition). to operationalize cps research goals as design parameters that cps-taskforge can use to generate the environment, we define cps-✓, a design checklist adapted from pisa2015 ( ## case study: communication of small groups as task difficulty increases to validate its flexibility, we want to explore whether cps-taskforge is capable of producing an environment that elicits diverse collaborative problem solving behavior. prior work in cps primarily used tasks with dyads or task reptitions at the same difficulty level, so we design a cps task where teams of 3-4 people complete a task, aiming to minimize expenditure of gold, at multiple difficulty levels. we design our cps-taskforge environment as follows, referencing the questions from cps-✓. task success is evaluated by the amount of money left unused, enemies destroyed, and health of the base (q1). a single level takes 5-8 minutes to complete, depending on level difficulty, and we design 3 levels with increasing difficulty (appendix figure 4a; q2-3). all players are human (q4), and each player is given 2-4 unique towers from a pool of 12 towers with different properties (subsection a.2) so that players have different roles, encouraging all players to engage and suggest usage of their own towers (q5-6). players are provided a surplus of gold, and costs are balanced to slightly favor upgrading over placing more towers, giving teams the opportunity to find many successful strategies (q7). all new information is distributed to players simultaneously (e.g., how much damage an enemy receives from a tower) (q8). players are under moderate time stress because each level is calibrated to give ample but limited time (5-6 minutes) to discuss strategy and place towers, and we disabled interaction during the attack phase (q9). players could end the planning phase early. we designated level-specific planning time to ensure the study is completed in a reasonable amount of time. players can only communicate through text chat (q10). these design decisions showcase the simplicity with which the td genre affords the ability to create different cps task environments. ## conclusion human-ai collaborative problem solving tools are rapidly being integrated in real-world work environments. the modern workforce uses teams with more than two parties, but empirical research with larger teams lags behind. the task design space for conducting cps research is large, and the tooling to systematically explore cps designs is lacking. our cps task environment generator, cps-taskforge, enables diverse, systematic cps research through a tower defense game environment that appeals to human subjects and is grounded in theory. it enables the study of larger team cps (multiple people and/or multiple ai agents) grounded in an environment and task that is accessible yet still carries real-world resemblance. the data generated in our case study reveals different collaborative tasks required to succeed in the overall tower defense task, such as decision making and ensuring teammates have the same understanding of the task. we will release all code for cps-taskforge and communication data collected in our case study to encourage studying multi-human and multi-ai collaborative problem solving.