MEETING DELEGATE : Benchmarking LLM s on Attending Meetings on Our Behalf (2025)

Abstract: In contemporary workplaces, meetings are essential for exchanging ideas and ensuring team alignment but often face challenges such as time consumption, scheduling conflicts, and inefficient participation. Recent advancements in Large Language Models (LLMs) have demonstrated their strong capabilities in natural language generation and reasoning, prompting the question- can LLMs effectively delegate participants in meetings? To explore this, we develop a prototype LLM-powered meeting delegate system and create a comprehensive benchmark using real meeting transcripts. Our evaluation shows GPT-4/4o balance active and cautious engagement, Gemini 1.5 Pro leans cautious, and Gemini 1.5 Flash and Llama3-8B/70B are more active. About 60% of responses capture at least one key point from the ground truth. Challenges remain in reducing irrelevant or repetitive content and handling transcription errors in real-world settings. We further validate the system through practical deployment and collect feedback. Our results highlight both the promise and limitations of LLMs as meeting delegates, providing insights for their real-world application in reducing meeting burden

## introduction nowadays, the nature of work has increasingly become more collaborative @xcite , with meetings becoming an essential component @xcite to facilitate the exchange of ideas and information, fostering innovation and ensuring alignment among team members. attending meetings, however, poses notable difficulties. firstly, the rapid increase in the number of meetings can consume a substantial amount of time, diverting attention from core tasks and reducing overall productivity @xcite . secondly, scheduling conflicts often arise when multiple meetings are double-booked, forcing participants to prioritize or miss valuable discus-sions altogether. thirdly, not all meetings require full attendance; participants may only need to contribute to specific topics, leading to inefficiencies when attendees are required for entire duration. in this study, we investigate the feasibility of developing a meeting delegate system to represent individuals in meetings. this concept is becoming increasingly viable with the advancement of large language models (llms). these llms, renowned for their remarkable capabilities in natural language understanding and generation @xcite @xcite , demonstrate potential to comprehend meeting context, participate in dynamic conversations, and provide informed responses. developing llm-powered meeting delegate systems faces several challenges. firstly, such systems must navigate complex, context-rich conversations involving multiple participants, requiring them to discern opportune moments for engagement and restraint. secondly, human conversations often contain ambiguities and uncertainties, such as queries directed ambiguously or pronunciation-related ambiguities, which challenge the system's ability to respond effectively. thirdly, ensuring user privacy is crucial to prevent over-sharing of information and safeguard the user's personal image. finally, these systems must operate in real-time, necessitating low-latency responsiveness. this study evaluates llms in the meeting delegate role, initially addressing the first two challenges while leaving privacy and latency considerations for future work. unlike prior research that examines the facilitator role in meetings @xcite , our work addresses the more prevalent participant role. specifically, we focus on how llms manage the nuances of meeting discussions, including their ability to discern when to intervene, respond to ambiguities, and maintain an accurate understanding of evolving contexts. in the absence of established benchmarks, we constructed an evaluation dataset from real meeting transcripts. our evaluation reveals that gpt-4/4o maintain balanced performance between active and cautious engagement strategies, while gemini 1.5 pro is more cautious, and gemini 1.5 flash and llama3-8b/70b are more active. overall, 60% of responses address at least one main point from the ground-truth. while the results demonstrate the potential of llms in meeting scenarios, improvements are needed to reduce irrelevant or repetitive content and enhance tolerance for transcription errors. to extend this evaluation beyond datasets, we also implemented and tested a prototype system in real-world settings. our contributions are summarized as follows: facilitator in multi-participant chat. muca @xcite presents a framework that leverages llms to facilitate group chats by simulating users, demonstrating notable effectiveness in goaloriented conversations. similarly, approaches like gpt-4o demo for meetings (openai, 2024a) are designed to serve as facilitators in group discussions. while these studies underscore llms' capabilities in managing group chats, they primarily focus on llms guiding the meeting process rather than representing individuals with different roles. role-playing with llms: characters and digital twins. role-play prompting @xcite has proven effective in triggering chain-ofthought reasoning in llms. research on simulating famous personalities @xcite has also explored character consistency and social interactions in agent-based group chats. while reid hoffman's @xcite gpt-4powered digital twin showcased the potential of aidriven representations, it was limited to one-on-one interactions, leaving group discussions largely unexplored. unlike prior work, we focus on llms as meeting delegates, providing targeted engagement for multi-participant, goal-specific meetings. our evaluation and real-world deployment show their potential to reduce individual meeting burdens and advance llm applications in professional settings. given the absence of a standardized meeting delegate architecture, this study adopts the design depicted in figure 1 . the system first employs an information gathering module to collect meetingrelated information, facilitating llm-driven participation. users can manually provide topics of interest, background knowledge, and shareable materials prior to the meeting. alternatively, if the user has a personal knowledge base or an intelligent personal assistant/agent, the system can query them in real-time, provided latency is manageable. ## llm-based meeting delegate system once in the meeting, the system monitors proceedings and employs llms to determine appropriate engagement timing and content. while various contextual data sources (e.g., transcripts, screen sharing, audio) are available, this work focuses on transcripts obtained from meeting software or speech-to-text tools. figure 1 outlines three response types: leading discussions, responding to others, and chiming in. this study prioritizes the latter two, emphasizing the participant role. the generated text-based response can then be converted into speech using tts technology, potentially mimicking the user's voice. an example of this workflow is depicted in figure 2 , drawn from a real prototype implementation (detailed in section 6.2). in this example, bob uses his meeting delegate to participate in a meeting with alice and others. before the meeting, bob provides topics of interest and relevant shareable information to the meeting delegate 1 . this information, along with instructions, forms the prompt for the meeting delegate 2 . the delegate then joins the meeting 3 and determines, based on the ongoing meeting transcript, whether to engage 4 . during the meeting, alice discusses updates on the voice function, which aligns with bob's goal to learn about its progress. the meeting delegate then chimes in 5 , generating a text-based response (converted to speech 6 ), asking for more details, thus achieving bob's objectives and engaging in the conversation. ## benchmark dataset while the preceding example illustrates the meeting delegate in action, a more systematic evaluation is needed to gauge how effectively it times its interventions and generates relevant responses. no existing benchmark datasets meet these objectives, prompting us to create one. ## conclusion we introduce an llm-powered meeting delegate system addressing challenges in collaborative work. focusing on participant roles, our prototype and benchmark demonstrate llms' potential to improve meeting efficiency. real-world evaluation reveals varying performance, with strengths and areas for improvement, particularly in handling transcription errors and reducing irrelevant or repetitive responses. future work should enhance real-time responsiveness and privacy safeguards to fully realize llms' role in collaborative environments.