Evaluating Compound AI Systems through Behaviors, Not Benchmarks (2025)

Abstract: Compound AI (CAI) systems, also referred to as LLM Agents, combine LLMs with retrievers and tools to enable information-seeking applications in the real-world. Thus, ensuring these systems perform reliably is critical. However, traditional evaluation using benchmark datasets and aggregate metrics often fails to capture their true operational performance. This is because understanding the operational efficacy of these information-seeking systems requires the ability to probe their behavior across a spectrum of simulated scenarios to identify potential failure modes. Thus, we present a behavior-driven evaluation framework that generates test specifications - explicit descriptions of expected system behaviors in specific scenarios - aligned with real usage contexts. These test specifications serve as formal declarations of system requirements that are then automatically transformed into concrete test cases. Specifically, our framework operates in two phases: (1) generating diverse test specifications via submodular optimization over semantic diversity and document coverage of the tests, and (2) implementing these specifications through graph-based pipelines supporting both tabular and textual sources. Evaluations on QuAC & HybriDialogue datasets, across SoTA LLMs, reveal that our framework identifies failure modes missed by traditional metrics, demonstrating failure rates twice as high as human-curated datasets.

## introduction recent advances in large language models (llms) have transformed user-facing applications, enabling natural conversations from simple inquiries @xcite to complex tasks like travel booking @xcite . this has given rise to compound ai (cai) systems @xcite (often referred to as agents @xcite ) that integrate llms with * equal contribution retrievers and specialized tools for multi-step reasoning and information synthesis. these systems have gained widespread use in information-seeking applications @xcite @xcite , yet their transition to realworld deployments raises a critical challenge: how do we evaluate their operational performance? one solution is collecting real user interactions with the cai system. while such interactions would authentically assess system capabilities -particularly in identifying failure modes, direct user evaluation faces scalability challenges. another solution, one that is adopted frequently, is a data-centric approach of evaluating cai systems through benchmarks, which can be broadly categorised into: (1) assessing individual components (llms, retrievers) with benchmarks @xcite @xcite , which overlook system-level behavior; or (2) holistic evaluation of cai system using crowdsourced or synthetic qa pairs. the latter has three limitations: (i) aggregate metrics in existing benchmarks @xcite @xcite mask detailed performance, (ii) synthetic methods @xcite are limited to text documents, and (iii) generated questions often reflect document structure @xcite rather than real-world use, as annotators (or llms) annotate with an objective i.e. independent of cai system. thus, instead of relying solely on costly realuser evaluations (solution 1) or coarse, documentcentric benchmarks (solution 2), we advocate for a middle path, taking inspiration from software engineering -scenario-driven behavioral testing of cai systems against expected usage scenarios. concretely, we draw inspiration from behavior-driven development (bdd) @xcite in software engineering, which focuses on application centric behaviors -e.g., "given a user is on the login page, when they enter valid credentials, then they should be redirected to the homepage". bdd focuses on this instead of testing code modules in isolation. to instantiate this concept in our context, we propose a novel evaluation framework that expresses expected cai system behaviors through behavior-driven test specifications (for eg. see as can be seen from the above example, our framework generates interpretable tests by providing explicit scenarios and acceptable behaviors. as shown in figure 1 (b), this contrasts with traditional benchmark evaluation that capture implicit scenarios, which suffer from clustering effects due to their data-centric approach and the homogeneous structure of documents -depicted in figure 1 (c ). however, please note that our framework is meant to complement existing benchmarks by enabling generation of targeted test data-data that is otherwise hard to acquire without real-user interactions. our evaluation framework operates in two phases. first, we leverage llms' ability to simulate human behavior @xcite to generate diverse set of initial test specifications. from these, we employ submodular optimization to select a subset that maximizes both specification diversity (measured by log determinant distance between test embeddings) and document coverage. although this subset selection is combinatorially hard, the submodularity of our combined metric ensures a greedy solution provides near-optimal results @xcite . in the second phase, these selected specifications are transformed into document-grounded conversational questionanswers (qas) using graph-based procedures, supporting both tabular and textual data sources. it is worth noting that our notion of test specifications differs from "goals" in goal-oriented dialogue datasets. while "goals" are typically derived from document structure and hierarchical headers @xcite , our specifications aim to capture the intended purpose of cai system interactions. below we list our contributions: ## preliminaries information-seeking compound ai system: let s := (r, l, t ools, d, τ ) be an informationseeking compound ai system, where r is a set of retrievers, l is a set of llms, t ools is a set of functions that extend llm abilities (e.g., code executor, internet search), d is a document collection, and τ is the topology of the network connecting r, l, t ools and d to solve tasks. in this work, we assume black-box access to s, described by s desc , which outlines its purpose, capabilities, and out-of-scope topics. document types and structure: here, we consider text documents d @xmath0, passage i ), table documents d @xmath1, headers i , items i ) and policy documents (same definition as text documents). we define policy documents to be separate from text documents as policies provide rules and guidance with regard to the cai system, whereas a text document (w.r.t. our definition) contains explicit facts. bdd test specifications @xcite ): let t be the space of bdd test specifications, where each @xmath2: {"scenario", "given", "when", "then"} → {s, g, w, th}). here, 'scenario' defines the situation being tested, 'given' sets the context, 'when' specifies user need, and 'then' defines the expected system behavior. for instance, in a website development setting, a test specification might describe website behavior during user login. -given a user is on the login page, when they enter valid credentials, then they should be taken to the homepage. problem statement: given an informationseeking compound ai system s with description s desc and document collection d, our objective is to identify a relevant subset of test specifications t s ⊆ t and generate a behaviordriven evaluation dataset e bdd of test-aligned conversational question-answer pairs. for each test specification t @xmath3, we select a relevant document subset d t i ⊆ d semantically related to t i , and construct the evaluation dataset as - ## methodology our evaluation framework operates in two complementary phases: (1) test specification generation (section 3.1), where we use llms to generate and optimize test specifications that simulate realistic human interactions for diversity and document coverage, and (2) test specification implementation (section 3.2), where we transform these specifications into document-grounded conversational qa pairs across both tabular and textual data sources. ## results and analysis we evaluate our behavior-driven framework through following research questions (rqs): rq1: effectiveness in identifying system weaknesses compared to traditional benchmarks. rq2: quality of generated test specifications in terms of diversity and document coverage. rq3: reliability of llms for evaluation. rq4: quality of the generated conversations prior to human verification. rq5: difficulty of generated conversations compared to existing benchmarks. ## conclusion we have presented a behavior-driven framework for evaluating cai systems that combines diverse test specifications with graph-based document traversal across tabular and textual data. experiments on popular datasets show our framework's effectiveness in identifying failure modes missed by traditional metrics.