A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist (2021)

Abstract: Despite state-of-the-art performance, NLP systems can be fragile in real-world situations. This is often due to insufficient understanding of the capabilities and limitations of models and the heavy reliance on standard evaluation benchmarks. Research into non-standard evaluation to mitigate this brittleness is gaining increasing attention. Notably, the behavioral testing principle ‘Checklist’, which decouples testing from implementation revealed significant failures in state-of-the-art models for multiple tasks. In this paper, we present a case study of using Checklist in a practical scenario. We conduct experiments for evaluating an offensive content detection system and use a data augmentation technique for improving the model using insights from Checklist. We lay out the challenges and open questions based on our observations of using Checklist for human-in-loop evaluation and improvement of NLP systems. Disclaimer: The paper contains examples of content with offensive language. The examples do not represent the views of the authors or their employers towards any person(s), group(s), practice(s), or entity/entities.

## introduction nlp systems have been known to learn spurious patterns from data to achieve high accuracy on test sets @xcite @xcite @xcite . evaluating models on static benchmarks and on test sets that have a similar distribution to the training data has resulted in an overestimation of model performance @xcite and models becoming increasingly fragile or less useful in real-world settings. this can be due to various factors such as language complexity and variability, the difference between training, testing, and real-world data, and insufficient understanding of the capabilities and limitations of the model itself. when deployed in the wild, such systems tend to break down, resulting in grossly incorrect predictions. this leads to mistrust in the system at two levels -first, on individual predictions and second, on the system's soundness in uncontrolled environments such as usage after deployment @xcite . further, evaluation benchmarks are also becoming increasingly obsolete due to the exponential rise in data and compute-heavy systems that exceed performance expectations, bringing the benchmark's 'toughness' and hence, its reliability into question @xcite . in order to mitigate this limitation of static evaluation, several approaches are used to evaluate other model aspects including, but not limited to, robustness @xcite , fairness @xcite , consistency @xcite , explanations @xcite , and adversarial performance @xcite @xcite . human-in-loop processes can be used to complement the capabilities of automation with human expertise @xcite @xcite . previous studies have shown that using humans to close the loop of the process of evaluation, explanation, or improvement can lead to a much better understanding of the system through higher explainability @xcite , better detection of model failures @xcite , and easier bug-fixing @xcite , resulting in robustness of the model in practical scenarios and increased trust in its predictions. @xcite introduced a behavioral testing strategy that decouples testing from model implementation. using human-generated test sets, they showed that state-of-the-art nlp models for multiple tasks fail to perform well for basic capabilities. we describe the framework in detail in section 2. in this paper, we describe a case study of using the checklist paradigm of evaluation in a practi-cal scenario. specifically, we used checklist to evaluate and debug an offensive content detection system. we found that checklist can lead to effective pinpointing of specific capabilities for which the model, despite impressive performance on a standard benchmark test set, failed. further, these insights can be used to improve the model using targeted data augmentation to debug specific model failures. however, we found that using checklist for evaluation and improvement is not always foolproof. we discuss the challenges and open questions observed during these experiments. the rest of the paper is organized as follows: first, we give a brief overview of the checklist framework. in section 3, we describe our case study, including the capabilities we test, results on the base model, the method applied to use these insights for improvement, and the results thereafter. finally, we present a detailed analysis of some of the most imminent challenges with using checklist for our experiments. ## overview of checklist the checklist framework @xcite introduces a human-in-loop behavioral testing technique for evaluating nlp systems. the authors argue that even though models perform well on static benchmarks, they fail to perform in realworld scenarios for basic capabilities. they release an open-source package foot_0 with functionality to create template sets and run software engineering-like decoupled testing on black-box models. the individual phenomena tested using checklist are known as capabilities. these capabilities are based on model expectations and the language usage that it needs to handle. for example, negation is a capability of a sentiment analysis model -the model should be able to distinguish 'happy' and 'not happy' as two opposite sentiments despite the overlapping word 'happy'. the checklist framework provides three different test types. the minimum functionality tests (mfts) are simple tests, similar to unit tests in software testing, that can test predictions on specific model capabilities. most of the capabilities tested in our case study are mfts. invariance tests (invs) are a test type where small semantic-preserving perturbations are applied to the test cases, and it is expected that the model output should not change. for example -in our case, while testing for the ro-bustness of the model, small typos are introduced. directional expectation tests (dirs) are similar to inv, except that the model output is expected to change in a certain way. in order for humans to generate test cases, checklist uses templates and lexicons. for example: 'i {positive verb} {activity}.' is a template. 'positive verb' and 'activity' in this template are two different keywords in the lexicon, each taking a specific set of values. for example, positive @xmath0, 'love', 'enjoy'] and @xmath1. the template generates 12 examples -the cartesian product of the values of the two lexicon keywords in the template. in the original checklist work, the authors test state-of-the-art and commercial systems across three tasks revealing unprecedented failure rates even for the most basic capabilities. for more detailed information on these results, we refer the reader to the original paper. ## a case study of using checklist we used checklist to evaluate and improve an inhouse offensive content detection system. while we studies a particular system as a use case, the process can be applied to any nlp model. the first step was to create a documentation of expectations. this documentation was important to lay out guidelines for what constitutes positive or negative examples in specific settings. in accordance with standard text classification annotation, for this task, offensive content is the positive class, and non-offensive content is the negative class. the documentation or guidelines contained concrete definitions of various offensive content categories that the model is expected to detect. these are henceforth known as the model capabilities and are described in section 3.1. this is important for checklist evaluation because it relies on building templates that target specific model capabilities. we gave this documentation to the annotators who were expected to use these guidelines to come up with templates of specific capabilities that could be used to find bugs. since offensive classification is a task that inherently has a class imbalance with fewer examples in the positive class in most publicly available datasets @xcite , including ours, we asked the annotators to develop templates that would result in offensive data. an-other reason for this design decision was that while offensive content is more likely to follow a certain pattern -a combination of entities and offensive language, non-offensive content would not follow any discernible patterns making it difficult to convert it into templates. further, from an application perspective in real-world settings, where such a model is employed to flag or filter offensive content, a higher recall is preferable, as false negatives (examples that are offensive but predicted as nonoffensive) need to be avoided even at the cost of a few false positives. ## challenges and open questions our case study was an experiment of using checklist to debug nlp systems. it presents optimistic findings for using human-in-loop for improving model performance. however, using this technique for evaluation and improvement is not straightforward or foolproof. in this section, we discuss some nuances and challenges that we observed while conducting these experiments. ## conclusion state-of-the-art systems have been known to break down when deployed in the wild because of heavy reliance on static evaluation benchmarks that fail to holistically test the system. several non-standard forms of evaluation into specific aspects of the models can lead to insights that might otherwise go unnoticed. human-in-loop processes have been known to aid better explainability, trust, debugging, and improvement of nlp models by combining automation with human-expertise of language use. the checklist framework introduced a behavioral testing approach for finding bugs in nlp models, which showed that state-of-the-art systems fail on the simplest of capabilities. we presented a case study of using checklist to debug an english offensive content detection system. the process we utilized was two-staged: first, we employed a human annotator to generate templates for evaluating specific model capabilities. these results were leveraged to find bugs, or capabilities in which the model is not performing as per expectation. the second step was to augment the data generated from these templates and re-train the model. this led to targeted bug-fixing and better performance not just on the test sets created from the same templates, but more generally, on independently created template sets. using this technique led to not only improved models but also a better understanding of the limitations and capabilities of the model in context of specific requirements. our findings add to the growing optimism of using human expertise and non-standard evaluation to improve performance, better explainability, and increase trust in nlp systems deployed in real-world uncontrolled usage environments. we also discuss various challenges of employing such a human-in-loop strategy. these include resource requirements, different methods to improve the model, determining the quality of templates and template sets, finding the optimal and representative template set, the difficulty for human subjects to create templates from scratch, and extension of the paradigm to languages other than english. this leads to the conclusion that the process, even though beneficial, leaves many open questions that need to be addressed. we hope that our work further increases attention to the checklist paradigm and motivates researchers to evaluate and improve black box nlp models using non-standard and explainable humanin-loop evaluation and investigate its challenges.