CRMA rena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments (2025)

Abstract: Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM systems can automate routine processes and enhance personalized service. However, deploying and evaluating these agents is challenging due to the lack of realistic benchmarks that reflect the complexity of real-world CRM tasks. To address this issue, we introduce CRMArena, a novel benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments. Following guidance from CRM experts and industry best practices, we designed CRMArena with nine customer service tasks distributed across three personas: service agent, analyst, and manager. The benchmark includes 16 commonly used industrial objects (e.g., account, order, knowledge article, case) with high interconnectivity, along with latent variables (e.g., complaint habits, policy violations) to simulate realistic data distributions. Experimental results reveal that state-of-the-art LLM agents succeed in less than 58% of the tasks with ReAct prompting, and less than 65% even with function-calling abilities. Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments.

## introduction customer relationship management (crm) systems are pivotal in modern enterprises, serving as the backbone for managing interactions with current and potential customers @xcite . the integration of intelligent agents based on large language models (llms) into crm systems promises to automate routine tasks, enhance operational efficiency, and revolutionize customer experiences. however, evaluating llm agents in real-world professional environments remains a challenge, due to the absence of robust benchmarks that faithfully capture the complexity of tasks encountered in real-world crm environments, largely due to data privacy concerns within enterprises. prior benchmarks on evaluating llm agents on work-related tasks, such as workarena @xcite , workbench @xcite , and tau @xcite tend to focus on basic functionality, and fall short in two key areas. first, the complexity of the objects (e.g., tables in databases) and dependencies (e.g., foreign keys) between these objects is often overly simple, lacking the complexity of real-world scenarios. second, the tasks included in the benchmarks, such as navigating web pages and filtering lists, are typically too straightforward and do not represent real-world work tasks. to address these limitations, we introduce crmarena, a comprehensive benchmark tailored to evaluate llm agents on performing realistic crm tasks in real-world work environments. crmarena features a realistic sandbox environment modeled after salesforce's schema, developed using an extensible data generation pipeline powered by llms (top left of figure 1 ). specifically, the pipeline tackles two key challenges: (1) object connectivity: reflecting the complex relationships between data objects (e.g., account associated with case and order) by mirroring salesforce's service cloud schema 2 . (2) introducing latent variables to better simulate realistic data dynamics, such as influenc-figure 1 : an overview of the contribution of this work. we begin by generating realistic crm data based on the salesforce service cloud schema, ensuring both quality and diversity through rigorous verification processes. this verified data is then stored locally and uploaded to a salesforce organization (org). an expert study, conducted with domain experts, validated the data's realism. using this org as a sandbox environment, we create query instances and benchmark various llms across different agentic frameworks. crmarena seamlessly integrates with salesforce, 3 enabling interaction via both the user interface and api access (see bottom of figure 1 ). this integration facilitated an expert study with crm professionals to assess the quality of our synthesized organization ( §2.5). study findings revealed that 90% of domain experts found the test environment to be realistic or better, underscoring the benchmark's fidelity to real-world crm scenarios. upon verifying the realism of crmarena, we then assess various agentic systems through 3 https://www.salesforce.com/crm/ api access. we develop two sets of tools generalpurpose vs. task-specific tools, combine them with three agentic frameworks and various llms. findings indicate that all llm agents struggle to reliably complete tasks when using general-purpose tools, with top performing systems completing less than 40% of the tasks. incorporating manually designed tools can enhance performance, with top llm agents solving up to 55% of the tasks. however, we discover that weaker llms often do not benefit from manually-crafted tools due to their weaker function calling capabilities. in summary, our main contributions are: motivated by tasks commonly addressed by crm personas: service manager, service agent, and service analyst, crmarena comprises nine tasks that reflect real-world crm scenarios. verified by domain experts as common occurrences in crm, an overview of these tasks is presented in figure 2 . below, we provide detailed illustrations of each task. ## related work agent benchmark several benchmarks have been developed to evaluate llm-based agents @xcite @xcite . recently, major efforts have focused specifically on web agents, which challenges llms to navigate and perform actions on websites. these websites are often about everyday scenarios, such as ecommerce, and social discussion form @xcite @xcite @xcite . another line of work focus on evaluating the safety of deploying agents ( @xcite @xcite . work-oriented datasets a few studies have developed datasets specifically for work-oriented tasks. the crm benchmark @xcite aims to assess llms' text generation and summarization abilities in business applications. work-bench @xcite consists of five databases designed to evaluate llm agents' performance in simple work tasks, such as sending emails, creating calendar invites, and counting traffic sources for a website. τ -bench @xcite creates tasks that require interactions with users to obtain relevant information and authorization, achieved by using llms to simulate users. workarena @xcite builds a webbased work environment that allows for testing agents with visual capabilities. ## conclusion this work introduces crmarena, a novel benchmark for evaluating llm agents in performing realistic crm tasks within professional work environments. by incorporating expert-validated tasks and modeling intricate data interconnections typical of crm systems, crmarena offers a comprehensive and realistic challenge for llm agents. our experiments demonstrate that even state-of-the-art llms struggle with these realistic tasks, achieving limited success rates even with function-calling capabilities. these findings highlight the gap between current llm capabilities and the requirements of real-world crm scenarios. crmarena serves as a foundational step towards more sophisticated evaluations of llm agents in realistic work environments.