White-box Testing of NLP models with Mask Neuron Coverage (2022)

Abstract: Recent literature has seen growing interest in using black-box strategies like for testing the behavior of NLP models. Research on white-box testing has developed a number of methods for evaluatinghow thoroughly the internal behavior of deep models is tested, but they are not applicableto NLP models. We propose a set of white-box testing methods that are customized for transformer-based NLP models. These include MASK NEURON COVERAGE (MNCOVER) that measures how thoroughlythe attention layers in models are exercised during testing. We show that MNCOVER can refine testing suites generated by CheckList by substantiallyreduce them in size, for more than 60% on average, while retaining failing tests – thereby concentrating the faultdetection power of the test suite. Further we show how can be used to guide CheckList input generation,evaluate alternative NLP testing methods, and drive data augmentation to improve accuracy.

## introduction previous nlp methods have used black-box testing to discover errors in nlp models. for instance, checklist @xcite introduces a blackbox testing strategy as a new evaluation methodology for comprehensive behavioral testing of nlp models. checklist introduced different test types, such as prediction invariance in the presence of certain perturbations. black-box testing approaches, like checklist, may produce distinct test inputs that yield very similar internal behavior from an nlp model. requiring that generated tests are distinct both from a black-box and a white-box perspective -that measures test similarity in terms of latent representations -has the potential to reduce the cost of testing without reducing its error-detection effectiveness. researchers have explored a range of white-box coverage techniques that focus on neu-ron activations and demonstrated their benefit on architecturally simple feed-forward networks @xcite @xcite . however, transformer-based nlp models incorporate more complex layer types, such as those computing self-attention, to which prior work is inapplicable. in this paper, we propose a suite of white-box coverage metrics. we first adapt the k-multisection neuron coverage measure from @xcite to transformer architectures. then we design a novel mncover coverage metric, tailored to nlp models. mncover focuses on the neural modules that are important for nlp and designs strategies to ensure that those modules' behavior is thoroughly exercised by a test set. our proposed coverage metric, when used to guide test generation, can cost-effectively achieve high-levels of coverage. figure 1 shows one example of how mncover can work in concert with checklist to produce a small and effective test set. the primary insight is that not all text sentences contain new information that will improve our confidence in a target model's behavior. in this list, multiple sentences were generated with similar syntactic and semantic structure. these sentences cause the activation of sets of attention neurons that have substantial overlap. this represents a form of redundancy in testing an nlp model. coverage-based filtering seeks to identify when an input's activation of attention neurons is subsumed by that of prior test inputs -such inputs are filtered. in the figure the second and third sentences are filtered out because their activation of attention neurons is identical to the first test sentence. as we show in §4 this form of filtering can substantially reduce test suite size while retaining tests that expose failures in modern nlp models, such as bert. the primary contributions of the paper lie in: ## background coverage for testing deep networksthe research of coverage testing focuses on the concept of "adequacy criterion" that defines when "enough" testing has been performed. the white-box coverage testing has been proposed by multiple recent studies to test deep neural networks @xcite @xcite . deepxplore @xcite , a white-box differential testing algorithm, introduced neuron coverage for dnns to guide systematic exploration of dnn's internal logic. let us use d to denote a set of test inputs (normally named as a test suite in behavior testing). the neuron coverage regarding d is defined as the ratio between the number of unique activated neurons (activated by d) and the total number of neurons in that dnn under behavior testing. a neuron is considered to be activated if its output is higher than a threshold value (e.g., 0). another closely related study, deeptest @xcite , proposed a gray-box, neuron coverage-guided test suite generation strategy. then, the study deepgauge @xcite expands the neuron coverage definition by introducing the kmultisection neuron coverage criteria to produce a multi-granular set of dnn coverage metrics. for a given neuron n, the kmultisection neuron coverage measures how thoroughly a given set of test inputs like d covers the range [low n , high n ]. the range [low n , high n ] is divided into k equal bins (i.e., k-multisections), for @xmath0. for d and the target neuron n, its k-multisection neuron coverage is then defined as the ratio of the number of bins covered by d and the total number of bins, i.e., k. for an entire dnn model, the kmultisection neuron coverage is then the ratio of all the activated bins for all its neurons and the total number of bins for all neurons in the dnn. transformer architecturenlp is undergoing a paradigm shift with the rise of large scale transformer models (e.g., bert, dall-e, gpt-3) that are trained on unprecedented data scale and are adaptable to a wide range of downstream tasks @xcite . these models embrace the transformer architecture @xcite and can capture long-range pairwise or higher-order interactions between input elements. they utilize the selfattention mechanism @xcite that enables shorter computation paths and provides parallelizable computation for learning to represent a sequential input data, like text. transformer receives inputs in the general form of word tokens. the sequence of inputs is converted to vector embeddings that get repeatedly re-encoded via the self-attention mechanism. the self-attention can repeat for many layers, with each layer re-encoding and each layer maintaining the same sequence length. at each layer, it corresponds to the following operations to learn encoding of token at position i: here w k is the key weight matrix, w q is the query weight matrix,w v is the value weight matrix, w r and w o are transformation matrices, and b 1 and b 2 are bias vectors. ## method state-of-the-art nlp models are large-scale with millions of neurons, due to large hidden sizes and multiple layers. we propose to simplify and view these foundation models @xcite through two levels of granularity: (1) word level: that includes the position-level embeddings at each layer and (2) pairwise attention level: that includes the pairwise self-attention neurons between two positions at each layer. in the rest of this paper, we denote the vector embeddings at location i for layer l as h l i and name these as the word level neurons at layer l. we also denote the α ij at layer l and head h as α ij lk , and call them as the attention level neurons at layer l. ## experiments our experiments are designed to answer the following questions: datasets and modelswe use pretrained model bert-base @xcite and robertabase @xcite provided by @xcite finetuned on sst-2 dataset and quora question pair (qqp) dataset. for the qqp dataset, we use the model finetuned on the mrpc dataset. we train a word level mask (m w ) and an interaction mask (m a ) for each of these settings. we use a learning rate of 1e -05, β @xmath1.001, β @xmath2.001, and β @xmath3.001 for all models. we have provided the test accuracy of the target models and the models trained with masks in