Automated test generation to evaluate tool-augmented LLM s as conversational AI agents (2024)

Abstract: Tool-augmented LLMs are a promising approach to create AI agents that can have realistic conversations, follow procedures, and call appropriate functions. However, evaluating them is challenging due to the diversity of possible conversations, and existing datasets focus only on single interactions and function-calling. We present a test generation pipeline to evaluate LLMs as conversational AI agents. Our framework uses LLMs to generate diverse tests grounded on user-defined procedures. For that, we use intermediate graphs to limit the LLM test generator’s tendency to hallucinate content that is not grounded on input procedures, and enforces high coverage of the possible conversations. Additionally, we put forward ALMITA, a manually curated dataset for evaluating AI agents in customer support, and use it to evaluate existing LLMs. Our results show that while tool-augmented LLMs perform well in single interactions, they often struggle to handle complete conversations. While our focus is on customer support, our test generation pipeline is general enough to evaluate different AI agents.

## introduction large language models (llms) are revolutionizing ai agents and have demonstrated remarkable generalization capabilities across various domains @xcite @xcite . in particular, llms have made a profound impact as chatbots and as ai agents in customer support systems @xcite . nevertheless, carelessly deploying an llm as an ai agent, and allowing them to interact with real users and apis, can lead to misinformation, reputational damage and costs to the company. thus, it is critical to evaluate ai agents beforehand. despite this need, evaluating the performance of llms in real-world scenarios remains a significant challenge. this is specially true in a conversational context, which is more complex than answering singleinteraction requests. most current approaches to evaluate llms focus primarily on specific tasks such as multi-qa @xcite or code generation @xcite , which do not fully evaluate the broader set of capabilities that llms are expected to possess to truly function as an effective conversational ai agents. focusing on customer support, an effective ai agent is should be capable of interacting with tools and the customer in order to resolve customer issues, while stricly adhering to procedures described by customer support admins. in order to assess the ai agent's performance, it is crucial to measure its ability to follow a given set of procedures and their resilience against potential customer manipulations. for that, it is key to have a comprehensive evaluation dataset, which can lead to valuable insights into the agent's abilities and limitations. we propose a method to generate evaluation datasets for tool-augmented llms as conversational ai agents. our method automates dataset generation using an llm to create conversations based on procedures, which are then transformed into tests. we use intermediate graph structures to improve the quality of the generated dataset (i.e., tests follow user-defined procedures) and make it more comprehensive (i.e., tests cover most relevant cases). to assess the ai agent's ability to handle attacks, we incorporate red teaming in our examples. our generation pipeline, illustrated in figure 1 , builds diverse datasets autonomously by using synthetically generated intents as seeds for procedures. additionally, our pipeline also allows for the inclusion of real data where available, such as actual procedures or apis used by a company to generate synthetic conversations. while datasets can be created fully automatically, we also put forward almita (automated benchmark of language models for intelligent tool-augmented agents), a manually curated dataset. we use this high-quality dataset to benchmark llms as conversational toolaugmented ai agents. intent "cancel order" procedure generator 1 procedure api extractor apis 2 flowgraph generator 3 flowgraph conversation graph generator 4 conversation graph noise generator 5 path sampler 6 paths cancel my order need the id 123 cancel(123) success done conversation generator conversation 7 test extractor 8 cancel my order need the id 123 cancel(123) success done cancel my order need the id cancel my order need the id 123 cancel(123) success test #1 figure 1 : automated test generation pipeline. for a given intent (e.g., cancel order) (1) we use an llm to generate a corresponding procedure. then, (2) an llm extracts relevant apis from the procedure, and (3) generates a flowgraph from the procedure and its apis. next, (4) an llm generates a conversation graph from the flowgraph and ( 5 ) adds noise to the graph (e.g., users going out of the expected procedure), to make the graph more realistic. to obtain conversations from the graph, (6) we sample paths from it, which correspond to different interactions. finally, (7) an llm generates conversations from the paths and ( 8 ) we extract tests from the sampled conversations. our main contributions are: we also note that, while our evaluation focuses on customer support, the same method could be applied, with some changes, to other domains. 1 almita, along with all other datasets generated using our pipeline and referenced in the paper, are available in https: //github.com/zendesk/almita-dataset . ## related work with the increasing use of llms as ai agents, significant efforts have been made to develop benchmarks to evaluate their ability to correctly answer customer requests in conversational settings. gaia proposes 466 human-annotated questions covering tasks like general knowledge, daily tasks, and data analysis @xcite . recently, agentinstruct introduced a framework for generating synthetic data from diverse sources, such as code, web articles, and textbook chapters, to help agents generate and refine instruction sets @xcite . unlike our work, these datasets do not assess toolaugmented ai agents. datasets to evaluate tool-augmented llms have been proposed. @xcite propose agent-tuning and compile multiple agent datasets to create sequences of api calls. agentbench features multi-step interactions between an agent and the environment, using various tools to solve user requests @xcite . @xcite and @xcite build datasets of apis from sources like torchhub, tensorhub, and rapidai, prompting an llm to generate instructions solvable by these apis. @xcite combine multiple datasets to convert user instructions into api calls. apigen introduced an automatic method to generate synthetic datasets for tool function calling @xcite . unlike our work, these datasets are not conversational and just focus on mapping utterances to api calls, and they do not use intermediate structures (i.e., graphs) to ensure coverage and reduce hallucinations in generated tests. other relevant work focuses on graph learning and on using different intermediate structures to reducing hallucinations. @xcite propose instructglm, which uses natural language to describe node features used to tune an llm for inference on graphs. @xcite introduce nl-graph, a benchmark for graph-based problems written in natural language, demonstrating that llms can perform structured operations on textual descriptions of graphs. additionally, @xcite propose using question-answer blueprints as intermediate representations to reduce hallucinations. these works do not fully encompass our problem setting of generating conversations in dialog format, calling apis, and extracting tests. ## method our automated test generation pipeline, illustrated in figure 1 , begins by generating textual procedures from input intents. while one could use an llm to directly generate conversations from procedures, our approach converts the procedures into a flowgraph and then into a conversation graph. our assumption is that using these intermediate structured representations makes the task of creating the conversations grounded on the procedures more accurate; see section 4.2 for supporting evidence. additionally, the graphs allow us to introduce noise into the conversations, making conversations more realistic and challenging, and enable us to sample paths, ensuring path coverage and conversation diversity. we then generate conversations from the sampled paths. finally, we extract tests from these conversations by breaking down the conversation at each user message, storing the context, and recording the generated response as the correct reply. ## results in section 4.1 we detail the creation of almita, a manually curated dataset for evaluating llms as ai customer support agents. two annotators independently review each datapoint to identify incorrect instances, followed by a discussion to align their assessments and minimize disagreements. any datapoint deemed incorrect by at least one annotator is then removed. gpt-4 is used for all generation steps (see figure 1 ). to assess the benefits of the graph intermediate structures, we conduct an ablation study comparing conversations generated directly from procedures to those using the intermediate structures, with manual curation for quality assessment (section 4.2). in section 4.3, we evaluate various ai agents on almita. finally, in section 4.4, we assess the effectiveness of our pipeline in generating high-quality test sets automatically. we do this by comparing the ai agents' performance on almita with those on its fully automated counterpart, auto-almita. ## conclusions llms are being used as customer support ai agents. however, existing evaluation datasets are limited in their scope. we propose an automated test generation pipeline to evaluate tool-augmented conversational ai agents. our proposed method uses llms to generate intermediate graph structures that help limit hallucinations and improve diversity in the generated tests. we evaluate different llms to analyze the current capabilities of llms implemented as ai agents. to facilitate this, we developed the almita dataset, which we used to thoroughly evaluate these ai agents and identify their limitations. almita allows for a multifaceted evaluation across several key dimensions, such as reply accuracy, api call correctness, and overall conversation integrity. our findings highlighted significant limitations in current llms, particularly in maintaining correct conversations throughout a user interaction. importantly, the almita dataset can be used by other researchers to evaluate ai agents, providing a comprehensive benchmark for assessing various aspects of their performance, possibly in other target domains. additionally, since our test generation pipeline is fully automated, we have the capability to create new, more challenging versions of the dataset. this adaptability ensures that our framework can be continually updated to reflect more complex and realistic scenarios, further enhancing its utility for ongoing research and development of ai agents in customer support and beyond.