LLM Agents for Coordinating Multi-User Information Gathering (2025)

Abstract: This paper introduces PeopleJoin, a benchmark for evaluating LM-mediated collaborative problem solving. Given a user request, PeopleJoin agents must identify teammates who might be able to assist, converse with these teammates to gather information, and finally compile a useful answer or summary for the original user. PeopleJoin comprises two evaluation domains: PeopleJoin-QA, focused on questions about tabular data, and PeopleJoin-DocCreation, focused on document creation tasks. The two domains are adapted from existing NLP benchmarks for database question answering and multi-document summarization; here, however, the information needed to complete these tasks is distributed across synthetic “organizations” of 2–20 users, simulating natural multi-user collaboration scenarios. We implemented several popular LM agent architectures, evaluating their accuracy and efficiency at completing tasks, and highlight new research questions that can be studied using PeopleJoin.

## introduction in today's fast-paced and interconnected world, effective collaboration is essential for achieving complex tasks and making informed decisions @xcite . many decision-making, content creation, and information-gathering tasks require collecting information from multiple people. for example, preparing a list of interns across teams in an organization by reaching out to the leader of each team; preparing a newsletter for project updates might necessitate coordinating with multiple contributors; identifying a suitable time to meet might require several rounds of negotiations @xcite . identifying what information is available, judiciously determining who to contact, asking precise questions, and compiling research results can be a challenging and time-consuming processespecially when real-time interaction between team members is difficult to coordinate. at the same time, recent large language models (llms), such as gpt-4 (openai, 2023), phi-3 @xcite , llama @xcite , and gemini @xcite , are becoming a crucial building block in developing automated agents that can assist human users with complex tasks @xcite @xcite . these tasks include chat applications for assisting individual users with searching and summarizing information (such as in microsoft copilot chat foot_1 ), and even supporting these users in workplace decision-making @xcite . could these agents be extended to improve collaboration among multiple users? in this paper, we introduce peoplejoin, an evaluation framework for studying effectiveness of llm-powered agents to assist with multi-user collaboration tasks. each peoplejoin task takes place within a fictitious organization with 2-20 employees, some of whom possess a collection of documents necessary to solve some task. one of the users (the initiating user) communicates the task to an agent (fig. 1 ). agents have direct access to the initiating user's documents, and can engage in conversations with other users to gather relevant information. they must rely on limited descriptions of other users, and potentially previous interactions, to determine who to contact for a given task. peoplejoin comprises two families of tasks: peoplejoin-qa and peoplejoin-doccreation, derived from the spider @xcite and multinews @xcite datasets respectively. it evaluates agents' ability to answer questions involving complex relational reasoning and document summarization. our initial benchmark release also includes reference agent implementations based on popular prompting and orchestration strategies, and a suite of evaluation metrics. we report evaluation results using phi-3-medium @xcite , gpt-4turbo and gpt-4o (openai, 2023) language models to implement these agents. our results indicate that lm-powered agents can struggle to coordinate with multiple users to correctly address information seeking and document authoring requests. major research questions remain around how to optimally determine which people to contact and in what order, how to ask high-quality questions, and how to learn and adapt to the structure of an organization. peoplejoin thus provide a test-bed for building ai-driven systems that can enhance human collaboration, and will also enable future work on learning from interaction, distributing tasks equitably, and maintaining user privacy in such agentic systems. ## data each peoplejoin domain comprises a set of organizations. each organization contains a set of collaborators, and each collaborator has privileged access to a set of documents. the benchmark provides llm-based simulators for each collaborator, a search interface that can be used to find collaborators, and a messaging interface that can be used to ask collaborators about their documents. then an agent must take as input a query from one collaborator, use the search and messaging interfaces to interact with other collaborators, and finally return an answer to the originator. drawing analogies between multi-user collaboration tasks and existing multi-datasource tasks commonly studied in nlp, we develop peoplejoin by re-purposing existing high-quality resources for database question answering (to produce peoplejoin-qa) and multi-document summarization (to produce peoplejoin-doccreation). ## baseline agent architectures to demonstrate the usefulness of peoplejoin as a research platform, we develop and evaluate a reference lm-powered agent implementation to perform tasks by coordinating interactions, retrieving relevant information, and posing targeted queries to other organization members. we consider an event-based reactive agent, which is triggered by user actions: upon getting a message from any organization member, the agent follows react-style prompting loop @xcite , taking actions, making observations, and performing reflection, until it decides to pause and wait for a next event, or terminate the session. ## evaluation peoplejoin provides metrics for evaluating the efficiency and correctness of user interactions. ## conclusions and future directions peoplejoin is a new benchmark designed to evaluate the role of large language model (llm) agents in facilitating collaborative information gathering within multi-user environments. it comprises two domains, peoplejoin-qa and peoplejoin-doccreation, which challenge llm agents to handle tasks related to question-answering and document creation. experiments with popular llm agent architectures revealed both their potential and limitations in accurately and efficiently completing complex collaborative tasks. future work could consider ai agents that learn over time from interactions for improving their performance over time @xcite . by analyzing past conversations, they can improve information source selection and communication strategies, making future interactions more efficient. secondly, privacy risks emerge when agents access personal documents, as large language models may not fully adhere to privacy guidelines @xcite . future work could focus on privacycentric evaluations and explore new information access models to mitigate such risks. finally, an excessive number of ai-initiated requests can overwhelm users, hindering productivity. building agents that can minimize human effort and prioritize urgent requests remains a challenge.