Beyond Accuracy: Behavioral Testing of NLP Models with C heck L ist (2020)

Abstract: Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

## introduction one of the primary goals of training nlp models is generalization. since testing "in the wild" is expensive and does not allow for fast iterations, the standard paradigm for evaluation is using trainvalidation-test splits to estimate the accuracy of the model, including the use of leader boards to track progress on a task @xcite . while performance on held-out data is a useful indicator, held-out datasets are often not comprehensive, and contain the same biases as the training data @xcite , such that real-world performance may be overestimated @xcite . further, by summarizing the performance as a single aggregate statistic, it becomes difficult to figure out where the model is failing, and how to fix it @xcite . a number of additional evaluation approaches have been proposed, such as evaluating robustness to noise @xcite or adversarial changes @xcite , fairness @xcite , logical consistency @xcite , explanations @xcite , diagnostic datasets @xcite , and interactive error analysis @xcite . however, these approaches focus either on individual tasks such as question answering or natural language inference, or on a few capabilities (e.g. robustness), and thus do not provide comprehensive guidance on how to evaluate models. software engineering research, on the other hand, has proposed a variety of paradigms and tools for testing complex software systems. in particular, "behavioral testing" (also known as black-box testing) is concerned with testing different capabilities of a system by validating the input-output behavior, without any knowledge of the internal structure @xcite . while there are clear similarities, many insights from software engineering are yet to be applied to nlp models. in this work, we propose checklist, a new evaluation methodology and accompanying tool foot_0 for comprehensive behavioral testing of nlp models. checklist guides users in what to test, by providing a list of linguistic capabilities, which are applicable to most tasks. to break down potential capability failures into specific behaviors, checklist introduces different test types, such as prediction invariance in the presence of certain perturbations, or performance on a set of "sanity checks." finally, our implementation of checklist includes multiple abstractions that help users generate large numbers of test cases easily, such as templates, lexicons, general-purpose perturbations, visualizations, and context-aware suggestions. test case expected predicted pass? testing negation with mft template: i {negation} {pos_verb} the {thing}. i can't say i recommend the food. neg pos ︎ x i didn't love the flight. neg neutral ︎ x failure @xmath0.4% testing ner with inv @americanair thank you we got on a different flight to [ @xmath1. inv ︎ x @virginamerica i can't lose my luggage, moving to [ @xmath2, ugh. inv ︎ x failure @xmath3.8% testing vocabulary with dir @americanair service wasn't great. you are lame. ↓ ︎ x @jetblue why won't you help them?! ugh. i dread you. ↓ ︎ x failure @xmath4.6% capability min func test invariance directional vocabulary fail. @xmath5.0% 16.2% 34.6% ner 0.0% 20.8% % n/a negation 76.4% n/a n/a b a c a b c pos neutral same pred. (inv) after removals / additions labels: negative, positive, neutral sentiment monotonic decreasing (↓) … … … … neutral neg neg neutral neg neutral as an example, we checklist a commercial sentiment analysis model in figure 1 . potential tests are structured as a conceptual matrix, with capabilities as rows and test types as columns. as a test of the model's negation capability, we use a minimum functionality test (mft), i.e. simple test cases designed to target a specific behavior (figure 1a ). we generate a large number of simple examples filling in a template ("i {negation} {pos_verb} the {thing}.") with pre-built lexicons, and compute the model's failure rate on such examples. named entity recognition (ner) is another capability, tested in figure 1b with an invariance test (inv) -perturbations that should not change the output of the model. in this case, changing location names should not change sentiment. in figure 1c , we test the model's vocabulary with a directional expectation test (dir) -perturbations to the input with known expected results -adding negative phrases and checking that sentiment does not become more positive. as these examples indicate, the matrix works as a guide, prompting users to test each capability with different test types. we demonstrate the usefulness and generality of checklist via instantiation on three nlp tasks: sentiment analysis (sentiment), duplicate question detection (qqp; @xcite , and machine comprehension (mc; @xcite . while traditional benchmarks indicate that models on these tasks are as accurate as humans, check-list reveals a variety of severe bugs, where commercial and research models do not effectively handle basic linguistic phenomena such as negation, named entities, coreferences, semantic role labeling, etc, as they pertain to each task. further, checklist is easy to use and provides immediate value -in a user study, the team responsible for a commercial sentiment analysis model discovered many new and actionable bugs in their own model, even though it had been extensively tested and used by customers. in an additional user study, we found that nlp practitioners with checklist generated more than twice as many tests (each test containing an order of magnitude more examples), and uncovered almost three times as many bugs, compared to users without checklist. ## checklist conceptually, users "checklist" a model by filling out cells in a matrix (figure 1 ), each cell potentially containing multiple tests. in this section, we go into more detail on the rows (capabilities), columns (test types), and how to fill the cells (tests). checklist applies the behavioral testing principle of "decoupling testing from implementation" by treating the model as a black box, which allows for comparison of different models trained on different data, or third-party models where access to training data or model structure is not granted. ## user evaluation the failures discovered in the previous section demonstrate the usefulness and flexibility of check-list. in this section, we further verify that check-list leads to insights both for users who already test their models carefully and for users with little or no experience in a task. ## related work one approach to evaluate specific linguistic capabilities is to create challenge datasets. @xcite note benefits of this approach, such as systematic control over data, as well as drawbacks, such as small scale and lack of resemblance to "real" data. further, they note that the majority of challenge sets are for natural language inference. we do not aim for checklist to replace challenge or benchmark datasets, but to complement them. we believe checklist maintains many of the benefits of challenge sets while mitigating their drawbacks: authoring examples from scratch with templates provides systematic control, while perturbation-based inv and dir tests allow for testing behavior in unlabeled, naturally-occurring data. while many challenge sets focus on extreme or difficult cases @xcite , mfts also focus on what should be easy cases given a capability, uncovering severe bugs. finally, the user study demonstrates that checklist can be used effectively for a variety of tasks with low effort: users created a complete test suite for sentiment analysis in a day, and mfts for qqp in two hours, both revealing previously unknown, severe bugs. with the increase in popularity of end-toend deep models, the community has turned to "probes", where a probing model for linguistic phenomena of interest (e.g. ner) is trained on intermediate representations of the encoder @xcite . along similar lines, previous work on word embeddings looked for correlations between properties of the embeddings and downstream task performance @xcite . while interesting as analysis methods, these do not give users an understanding of how a fine-tuned (or end-to-end) model can handle linguistic phenomena for the end-task. for example, while @xcite found that very accurate ner models can be trained using bert (96.7%), we show bert finetuned on qqp or sst-2 displays severe ner issues. there are existing perturbation techniques meant to evaluate specific behavioral capabilities of nlp models such as logical consistency @xcite and robustness to noise @xcite , name changes @xcite , or adversaries @xcite . checklist provides a framework for such techniques to systematically evaluate these alongside a variety of other capabilities. however, checklist cannot be directly used for non-behavioral issues such as data versioning problems @xcite , labeling errors, annotator biases @xcite , worst-case security issues @xcite , or lack of interpretability @xcite . ## conclusion while useful, accuracy on benchmarks is not sufficient for evaluating nlp models. adopting principles from behavioral testing in software engineering, we propose checklist, a model-agnostic and task-agnostic testing methodology that tests individual capabilities of the model using three different test types. to illustrate its utility, we highlight significant problems at multiple levels in the conceptual nlp pipeline for models that have "solved" existing benchmarks on three different tasks. further, checklist reveals critical bugs in commercial systems developed by large software companies, indicating that it complements current practices well. tests created with checklist can be applied to any model, making it easy to incorporate in current benchmarks or evaluation pipelines. our user studies indicate that checklist is easy to learn and use, and helpful both for expert users who have tested their models at length as well as for practitioners with little experience in a task. the tests presented in this paper are part of check-list's open source release, and can easily be incorporated into existing benchmarks. more importantly, the abstractions and tools in checklist can be used to collectively create more exhaustive test suites for a variety of tasks. since many tests can be applied across tasks as is (e.g. typos) or with minor variations (e.g. changing names), we expect that collaborative test creation will result in evaluation of nlp models that is much more robust and detailed, beyond just accuracy on held-out data. checklist is open source, and available at https://github.com/marcotcr/checklist .