T est A ug: A Framework for Augmenting Capability-based NLP Tests (2022)

Abstract: The recently proposed capability-based NLP testing allows model developers to test the functional capabilities of NLP models, revealing functional failures for models with good held-out evaluation scores. However, existing work on capability-based testing requires the developer to compose each individual test template from scratch. Such approach thus requires extensive manual efforts and is less scalable. In this paper, we investigate a different approach that requires the developer to only annotate a few test templates, while leveraging the GPT-3 engine to generate the majority of test cases. While our approach saves the manual efforts by design, it guarantees the correctness of the generated suites with a validity checker. Moreover, our experimental results show that the test suites generated by GPT-3 are more diverse than the manually created ones; they can also be used to detect more errors compared to manually created counterparts. Our test suites can be downloaded at https://anonymous-researcher-nlp.github.io/testaug/ .

## introduction in recent years, natural language processing (nlp) has seen major breakthroughs in the model performances. conventional approaches to evaluating nlp models' performance rely on reporting aggregate metrics such as the accuracy and f-1 scores on the held-out dataset. however, the held-out scores do not represent the model's performance on data in the wild @xcite @xcite . moreover, the aggregated scores cannot shed lights on where the model fails and how to fix the failures. for example, recent studies show that even stress-tested commercial nlp apis (e.g., google cloud's natural language api foot_0 and microsoft's text analytics api foot_1 ) often to assist developers in finding behavioral failures in their models, recent work has proposed a framework called checklist for behavioral or capability-based nlp testing @xcite . such tests include input and output pairs to examine the model's performance on each linguistic capability. in existing capability-based testing frameworks @xcite , the test suites are generated from manually created natural language templates (e.g., "she is a [] person" and "she is not a [] person.") and a pre-defined list of words to fill the template (e.g., generous, mean). existing work thus has two disadvantages: even worse, despite the crowdsourcing avail-ability, expert annotations are often required: the templates need to both follow the linguistic rules and capture potential nlp pitfalls. for example, for some capabilities in @xcite , the only variation comes from persons' names (e.g., "if {male name} and {female name} were alone, do you think he would reject her?" and "if {male name} and {female name} were alone, do you think she would reject him?"). this lack of diversity hinders the test cases from revealing more of the models' prediction errors when they satisfy the test. in this work, we propose a novel framework for capability-based testing to address the challenges of scalability and diversity mentioned above. in our framework, testaug, the developer first annotates a few seed test cases, testaug then leverages the gpt-3 engine @xcite to generate test cases similar to the seed. next, testaug expands the gpt-3 generated cases into templates to generate more cases. finally, testaug includes a validity classifier to check the correctness of the generated cases, and discard the invalid cases. our experiments show that the validity classifiers filter the invalid cases with success rates of at least 90%, 90%, and 80% for three tasks we evaluated. furthermore, the valid generated cases are more diverse and can detect more bugs than existing work @xcite . our contributions are three folds: to help gain a more comprehensive understanding of the model performance, researchers proposed a new approach to evaluating nlp models called linguistic capability-based testing @xcite @xcite . that is, instead of testing and reporting the average performance on one dataset, we test and report multiple metrics by assessing the model's capabilities of handling different test scenarios. the taxonomy of the capabilities can be organized by linguistic theory @xcite , logic, domain knowledge @xcite , or the functional requirements defined by the specific application @xcite @xcite . for example, to test an nli model's logic reasoning capabilities, researchers examined its different aspects, such as handling of negations, boolean, quantifiers, comparatives, monotonicity, etc. @xcite . later, ribeiro et al. (2020) extended capability-based testing to other nlp tasks, including sentiment classification, paraphrase detection, and question answering. the capabilities for testing would be listed by software developers or by the subject matter experts who manually identify a taxonomy of errors based on their expertise in data annotation @xcite . the construction method for the test suites can be divided into fully manual @xcite and semi automatic approaches. the manual approaches often suffer from scalability issues @xcite . some existing approaches proposed to scale up the annotation by leveraging non-expert annotators, but had to restrict the capabilities to avoid making the tasks too complicated for the annotators @xcite . to construct a massive scale test suite without large manual annotation efforts, @xcite proposed to recast 13 existing datasets on 7 different tasks (e.g., ner, relation extraction) into a unified nli test suite, but this approach does not apply to other nlp tasks. other works remedy the scalability issue by manually coming up with templates where the blanks can be filled with interchangeable tokens or a cloze-style prediction from language models @xcite , but automatically generating the templates remain a challenging task @xcite . finally, @xcite proposed a formal language for generating templates, although it can be used to generate examples of contradictions in nli. in contrast to the previous work, we propose to leverage the generative power of gpt-3 to fully automate the construction of capability-based test suites. our framework thus overcomes the scalability issue in existing work. prompt learning for gpt-3. our work has employed the gpt-3 engine @xcite for the generation and verification of the test suites, where we have manually engineered and optimized the prompt messages (section 4). prompt learning was found to be helpful for a wide range of tasks @xcite , including major natural language generation tasks (li and liang, 2021). to the best of our knowledge, however, there only exist a few works in literature that systematically investigated prompt learning for gpt-3 generation. mishra et al. (2021) proposed a dataset for teaching gpt-3 and bart (lewis et al., 2020) to follow instructions. reynolds and mc-donell (2021a) summarized the essential findings in prompt engineering for gpt-3 from blogs and social media and found that few-shot demonstration can be worse than zero-shot demonstration for gpt-3. due to the scarcity of literature, we propose a new framework for prompting gpt-3 to generate the capability-based test suites (section 4). ## problem definition software testing refers to the process of identifying the inconsistencies between software's actual and expected execution process @xcite . software testing includes white-box testing and black-box testing. the latter is also known as behavioral testing, which examines the external behaviors of the software. it often requires the developers to collect test cases (i.e., input/output pairs) to constitute a test suite (i.e., a collection of cases for testing specific software behaviors). in recent years, following the success of natural language processing, behavioral testing was introduced to test nlp models @xcite , especially large language models that show stateof-the-art performance. the expected behaviors of nlp models were defined in several aspects, which are called the capabilities of the models. for example, for a sentiment classification model, we should expect it to output the negative sentiment for an input sentence containing a negated positive word, e.g., i don't like the food. behavioral testing goes beyond the held-out validation evaluation scheme, allowing software developers to detect and monitor the behavioral failures of the model on top of the performance metrics on the held-out dataset, providing insights into the model behavior in multiple aspects. the most recent work on nlp behavioral testing is called checklist @xcite . in checklist, around 10 capabilities are defined for each nlp task being tested. for each capability, several tests were created by the developer, and the requirement of each test is described with a natural language description as in ## the testaug framework given a linguistic capability and their specific tests, the previous approach to generating test cases relies on manual templates. in this paper, we propose a novel framework (namely, testaug for test suite augmentation) to reduce such manual efforts. figure 1 shows the control flow of our framework. first, testaug starts with a few seed test cases from the checklist test suite @xcite . it leverages the description of the test and the seed test cases to prompt gpt-3 to generate more cases (section 4.1). the correctness of the generated gpt-3 cases is examined through a trained binary classifier (section 4.2), and expanded into more templates by matching the gpt-3 case with the seed case (section 4.3). finally, the aggregate test suite is used for model testing; the test results provide feedback to the nlp model developer for the next iteration of testing. ## experiments in this section, we evaluate the effectiveness of tes-taug and compare it with existing work @xcite in multiple aspects. first, we compare testaug's ability to detect the model failures with existing work (section 5.2). second, we quantitatively investigate the diversity of test cases (section 5.3). since test cases are graduate students in computer science working on nlprelated research. their agreement rate is reported in ## discussion, conclusions, and future work this paper introduces a novel framework testaug for capability-based nlp testing. addressing current system's heavy dependence on manual creation of templates, our framework can save at least 98.0% of the manual annotation effort with gpt-3; meanwhile, the test suites generated with our framework reveal more bugs than existing systems and show better diversity. our framework guarantees the correctness of the generated cases by removing the invalid output from gpt-3. the main limitation of testaug is that gpt-3 fails to generate highly structured test cases, such as cases for extractive question answering. it also struggles to generate cases that require logic or math reasoning. we leave the exploration of these cases for future work. 12 additional capabilities in nli tasks are below 80% valid threshold. automatic filtering these cases out with trained classifiers are left as future work.