Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLM s (2023)

Abstract: Current model testing work has mostly focused on creating test cases. Identifying what to test is a step that is largely ignored and poorly supported. We propose Weaver, an interactive tool that supports requirements elicitation for guiding model testing. Weaver uses large language models to generate knowledge bases and recommends concepts from them interactively, allowing testers to elicit requirements for further testing. Weaver provides rich external knowledge to testers and encourages testers to systematically explore diverse concepts beyond their own biases. In a user study, we show that both NLP experts and non-experts identified more, as well as more diverse concepts worth testing when using Weaver. Collectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT. Our case studies further show that Weaver can help practitioners test models in real-world settings, where developers define more nuanced application scenarios (e.g., code understanding and transcript summarization) using LLMs.

## introduction despite being increasingly deployed in real-world products, ml models still suffer from falsehoods @xcite , biases @xcite , and shortcuts @xcite , leading to usability, fairness, and safety issues in those products @xcite . for example, toxicity detection models are used by social media platforms to flag or remove harmful content, but their biases amplify harm against minority groups @xcite . as standard benchmarks are often too coarse to expose these issues, recent work has proposed to test nuanced behaviors of ml models @xcite @xcite . figure 1 : the v-model @xcite , a widely used development process in many fields of engineering, adapted for testing models within a system. this layering and planning is also compatible with more agile development approaches, where the different activities may be iterated in different ways but are still linked. such model testing of nuanced behaviors usually requires translating behavior expectations into test cases (input-output pairs). to enable such test case creation, prior work has taken inspiration from the long-established software testing approaches: for example, in their checklist framework, @xcite used templates to form minimal functionality test cases, which was inspired by unit tests. @xcite 's work on editing inputs (e.g., synonym swap) for testing model invariances is akin to metamorphic testing. to enable testing generative models, @xcite have also explored specifying properties that any correct output should follow, similar to property-based testing. however, prior work has focused on how to write test cases, not what tests to write. in software engineering, tests are fundamentally grounded in requirements and design work, as commonly expressed in the v-model @xcite 1 ). ideally, each test can be traced back to a requirement and all requirements are tested. software engineering research has long established the importance of requirements for development and testing, and studied many approaches for requirements elicitation @xcite . in comparison, little work has explicitly supported identifying what to test in model testing. researchers and practitioners seem to rely mostly on intuition and generic domain knowledge @xcite , or debug a small set of issues initially identified through error analysis @xcite . such approaches are often shaped heavily by individual knowledge and biases @xcite , driving practitioners to focus on local areas of a few related concepts where they find problems, while neglecting the larger space @xcite , as exemplified in figure 2 . for example, to test toxicity detection models, practitioners may identify and test (1) racism with handcrafted test cases and (2) robustness to paraphrasing with checklist. however, they are likely to miss many concepts in the space (e.g., spreading misinformation as a way to spread toxicity) if they have never seen or worked on similar aspects, as often observed in the fairness literature @xcite . in this work, we contribute the concept and method of requirements engineering ("what to test") for improving model testing. we connect testing to requirements with weaver, an interactive tool that supports requirements elicitation for guiding model testing. our goal is to provide comprehensive external knowledge, encouraging testers to systematically explore diverse concepts beyond their biases (figure 2 ), while balancing the completeness of requirements collection and the effort of practitioners exploring these requirements. weaver systematically generates knowledge bases (kb) by querying large language models (llms) with well-defined relations from con-ceptnet, allowing testers to elicit requirements for any use cases (as opposed to being limited by prebuilt kbs). for example, in figure 3 , from a seed concept "online toxicity", the llm generates a kb showing "spreading misinformation" with relation "done via." to not overwhelm users and encourage iterative exploration, weaver shows only a subset of the concepts from the kbs, balancing relevance and diversity, while allowing users to still explore more concepts on demand. we demonstrate the usefulness and generality of weaver through a user study and case studies. in the user study ( §4), users identified more, as well as more diverse concepts worth testing when using weaver. collectively, they found more than 200 failing test cases for stance detection with zero-shot chatgpt (gpt-turbo-3.5). our case studies ( §5) further show that weaver can help practitioners test models in real-world settings, from transcript summarization to code understanding. ## weaver our key idea is to provide a knowledge base to support systematic exploration of domain concepts to guide model testing. this allows testers to consider requirements broadly, mitigating their biases as testers otherwise tend to opportunistically explore concepts in local areas (figure 2 ). our tool, weaver, has three primary building blocks: (1) an llm-generated knowledge base for a given testing task, for encouraging more systematic and diverse requirement elicitation beyond individual biases; (2) a graph-inspired recommendation paradigm that prioritizes diverse yet relevant concepts for user inspection; and (3) an intuitive interface that can be easily paired with any test case creation methods, for supporting users to interactively navigate the knowledge base(s) for their own purposes. below, we walk through the design choices and corresponding rationales. ## intrinsic evaluation as the primary goal of weaver is to provide external knowledge to guide testing, it is important that the knowledge provided is comprehensive in the first place. here, we quantitatively evaluate: q.1 how comprehensive are the knowledge bases generated by weaver? tasks, data, and metrics. we select four tasks for the evaluation: hateful meme detection, pedestrian detection, stance detection for feminism, and stance detection for climate change (task descriptions in appendix c). these tasks cover diverse domains and modalities, and importantly, provide us with gold concepts that can be used to evaluate our llm-generated kb. the first two tasks have been studied in prior work, and we directly use their ground-truth concepts collected from existing documents @xcite and user studies @xcite . for the last two tasks, we aggregate all concepts identified by 20 participants without using weaver as part of our user study (discussed later in §4), which we consider as our ground truth. intuitively, such aggregation should help represent what concepts are generally deemed important. as shown in we evaluate the comprehensiveness of the generated knowledge using recall, i.e., the fraction of existing concepts that also appear in the kb. since there are many phrasing variations of the same concept, we decide that a concept is in the kb if it appears exactly the same in the kb, or our manual check decides that it matches one of the 10 most similar concepts from the kb, as measured by the cosine distance (cf. §2.1). we established that the manual process is reliable by evaluating inter-rater reliability where two authors independently labeled a random sample of 50 concepts, finding substantial agreement (κ = 69.4%). we also evaluate the validity of the generated knowledge using precision, i.e., the fraction of kb edges that are valid. note that because our ground truths are incomplete by nature (collected from dataset analysis and user study), kb edges that are not in the ground truths can still be valid. following prior work @xcite , we performed manual validation on sampled kb edges. we sampled 50 edges from each of the four generated kbs. ## case studies using two case studies, we demonstrate that weaver can help practitioners test their own models and find various bugs in real-world settings, and has the potential to provide support beyond post-hoc model testing. in the studies, we provided sufficient supports to the practitioners, including integrating weaver into their natural evaluation environment (jupyter notebook), joining each user to explore their models for approximately three hours, and offering feedback and discussions whenever necessary (e.g., as practitioners brainstormed their seed concepts). case selection. we approached researchers in our contact network to identify projects that were actively developing and improving models to be integrated into software products, such that (1) model testing has real stakes for them, and (2) the model needs to meet real-world requirements of a product, not just succeed on a standard benchmark. we ended up recruiting practitioners working on two projects that matched our scope to try weaver. the first practitioner (c1) is building a pipeline for knowledge transfer, where they prompt an llm to summarize content from transcripts into instructions. the second practitioner (c2) is building an ide plugin to help developers understand unfamiliar code, developing llm prompts to generate text summary for code segments. while these are two distinct scenarios, their shared challenge is that for practitioners working on novel tasks, it is often non-trivial to perform prompt engineering, especially because they do not have appropriate datasets for evaluating their prompts. the case studies were irb-approved and participants were compensated for their time. ## discussion and conclusion in this work, we propose weaver, a tool that uses knowledge bases to guide model testing, helping testers consider requirements broadly. thorough user studies and case studies, we show that weaver supports users to identify more, as well as more diverse concepts worth testing, can successfully mitigating users' biases, and can support real-world applications. beyond being a useful testing tool, the underlying concept of weaver have interesting implications on ml model testing and development, which we detail below. model testing in the era of llms. throughout our user studies and case studies, we focused on testing "models" achieved by prompting llms. here, we would like to highlight the importance of requirements in such cases. llms are increasingly deployed in different applications, and traditional model evaluations are becoming less indicative. with these models trained on massive web text, it is unclear what should be considered as "in-distribution evaluation data." instead, the evaluation objectives heavily depend on what practitioners need, which should be reflected through well-documented requirements. meanwhile, as most practitioners are not nlp experts, they face challenges articulating how and what they should test about their prompted models @xcite . as their use cases become more nuanced, it is also less likely for them to find pre-existing collections on important concepts. as such, enabling each individual to identify what-to-test is essential. we hope weaver can be used for democratizing rigorous testing, just as llms democratized access to powerful models. still, currently weaver relies purely on practitioners to identify requirements worth testing, which may result in mis-matched requirement granularity (cf. §3). future work can explore more complex structures that can represent knowledge (e.g., from kbs to knowledge graphs), and advanced recommendation mechanisms for practitioners to find the best requirements to explore first.