FIND : H uman-in-the- L oop D ebugging D eep T ext C lassifiers (2020)

Abstract: Since obtaining a perfect training dataset (i.e., a dataset which is considerably large, unbiased, and well-representative of unseen cases) is hardly possible, many real-world text classifiers are trained on the available, yet imperfect, datasets. These classifiers are thus likely to have undesirable properties. For instance, they may have biases against some sub-populations or may not work effectively in the wild due to overfitting. In this paper, we propose FIND – a framework which enables humans to debug deep learning text classifiers by disabling irrelevant hidden features. Experiments show that by using FIND, humans can improve CNN text classifiers which were trained under different types of imperfect datasets (including datasets with biases and datasets with dissimilar train-test distributions).

## introduction deep learning has become the dominant approach to address most natural language processing (nlp) tasks, including text classification. with sufficient and high-quality training data, deep learning models can perform incredibly well @xcite . however, in real-world cases, such ideal datasets are scarce. often times, the available datasets are small, full of regular but irrelevant words, and contain unintended biases @xcite . these can lead to suboptimal models with undesirable properties. for example, the models may have biases against some sub-populations or may not work effectively in the wild as they overfit the imperfect training data. to improve the models, previous work has looked into different techniques beyond standard model fitting. if the weaknesses of the training datasets or the models are anticipated, strategies can be tailored to mitigate such weaknesses. for example, augmenting the training data with genderswapped input texts helps reduce gender bias in the models @xcite . adversarial training can prevent the models from exploiting irrelevant and/or protected features @xcite . with a limited number of training examples, using human rationales or prior knowledge together with training labels can help the models perform better @xcite @xcite . nonetheless, there are side-effects of suboptimal datasets that cannot be predicted and are only found after training thanks to post-hoc error analysis. to rectify such problems, there have been attempts to enable humans to fix the trained models (i.e., to perform model debugging) @xcite . since the models are usually too complex to understand, manually modifying the model parameters is not possible. existing techniques, therefore, allow humans to provide feedback on individual predictions instead. then, additional training examples are created based on the feedback to retrain the models. however, such local improvements for individual predictions could add up to inferior overall performance @xcite . furthermore, these existing techniques allow us to rectify only errors related to examples at hand but provide no way to fix problems kept hidden in the model parameters. in this paper, we propose a framework which allows humans to debug and improve deep text classifiers by disabling hidden features which are irrelevant to the classification task. we name this framework find (feature investigation and disabling). find exploits an explanation method, namely layer-wise relevance propagation (lrp) @xcite , to understand the behavior of a classifier when it predicts each training instance. then it aggregates all the information using word clouds to create a global visual picture of the model. this enables humans to comprehend the features automatically learned by the deep classifier and then decide to disable some features that could undermine the prediction accuracy during testing. the main differences between our work and existing work are: (i) first, find leverages human feedback on the model components, not the individual predictions, to perform debugging; (ii) second, find targets deep text classifiers which are more convoluted than traditional classifiers used in existing work (such as naive bayes classifiers and support vector machines). we conducted three human experiments (one feasibility study and two debugging experiments) to demonstrate the usefulness of find. for all the experiments, we used as classifiers convolutional neural networks (cnns) @xcite , which are a popular, well-performing architecture for many text classification tasks including the tasks we experimented with @xcite @xcite . the overall results show that find with human-in-the-loop can improve the text classifiers and mitigate the said problems in the datasets. after the experiments, we discuss the generalization of the proposed framework to other tasks and models. overall, the main contributions of this paper are: the rest of this paper is organized as follows. section 2 explains related work about analyzing, explaining, and human-debugging text classifiers. section 3 proposes find, our debugging framework. section 4 explains the experimental setup followed by the three human experiments in section 5 to 7. finally, section 8 discusses generalization of the framework and concludes the paper. code and datasets of this paper are available at https://github.com/plkumjorn/find . ## related work analyzing deep nlp models -there has been substantial work in gaining better understanding of complex, deep neural nlp models. by visualizing dense hidden vectors, @xcite found that some dimensions of the final representation learned by recurrent neural networks capture the effect of intensification and negation in the input text. @xcite revealed the existence of interpretable cells in a character-level lstm model for language modelling. for example, they found a cell acting as a line length counter and cells checking if the current letter is inside a parenthesis or a quote. @xcite presented interesting findings about cnns for text classification including the fact that one convolutional filter may detect more than one n-gram pattern and may also suppress negative n-grams. many recent papers studied several types of knowledge in bert @xcite , a deep transformer-based model for language understanding, and found that syntactic information is mostly captured in the middle bert layers while the final bert layers are the most task-specific @xcite . inspired by many findings, we make the assumption that each dimension of the final representation (i.e., the vector before the output layer) captures patterns or qualities in the input which are useful for classification. therefore, understanding the roles of these dimensions (we refer to them as features) is a prerequisite for effective human-in-the-loop model debugging, and we exploit an explanation method to gain such an understanding. explaining predictions from text classifiers -several methods have been devised to generate explanations supporting classifications in many forms, such as natural language texts @xcite , rules @xcite , extracted rationales @xcite , and attribution scores @xcite . some explanation methods, such as lime @xcite and shap (lundberg and lee, 2017), are model-agnostic and do not require access to model parameters. other methods access the model architectures and parameters to generate the explanations, such as deeplift @xcite and lrp (layer-wise relevance propagation) @xcite . in this work, we use lrp to explain not the predictions but the learned features so as to expose the model behavior to humans and enable informed model debugging. debugging text classifiers using human feedback -early work in this area comes from the human-computer interaction community. @xcite studied the types of feedback humans usually give in response to machine-generated predictions and explanations. also, some of the feedback collected (i.e., important words of each category) was used to improve the classifier via a user co-training approach. @xcite presented an explanatory debugging approach in which the system explains to users how it made each prediction, and the users then rectify the model by adding/removing words from the explanation and adjusting important weights. even without explanations shown, an active learning framework proposed by @xcite asks humans to iteratively label some chosen features (i.e., words) and adjusts the model parameters that correspond to the features. however, these early works target simpler machine learning classifiers (e.g., naive bayes classifiers with bag-of-words) and it is not clear how to apply the proposed approaches to deep text classifiers. recently, there have been new attempts to use explanations and human feedback to debug classifiers in general. some of them were tested on traditional text classifiers. for instance, @xcite showed a set of lime explanations for individual svm predictions to humans and asked them to remove irrelevant words from the training data in subsequent training. the process was run for three rounds to iteratively improve the classifiers. @xcite proposed caipi, which is an explanatory interactive learning framework. at each iteration, it selects an unlabelled example to predict and explain to users using lime, and the users respond by removing irrelevant features from the explanation. caipi then uses this feedback to generate augmented data and retrain the model. while these recent works use feedback on lowlevel features (input words) and individual predictions, our framework (find) uses feedback on the learned features with respect to the big picture of the model. this helps us avoid local decision pitfalls which usually occur in interactive machine learning @xcite . overall, what makes our contribution different from existing work is that (i) we collect the feedback on the model, not the individual predictions, and (ii) we target deep text classifiers which are more complex than the models used in previous work. ## experimental setup all datasets and their splits used in the experiments are listed in ## exp 1: feasibility study in this feasibility study, we assessed the effectiveness of word clouds as visual explanations to reveal the behavior of cnn features. we trained cnn models using small training datasets and evaluated the quality of cnn features based on responses from mturk workers to the feature word clouds. then we disabled features based on their average quality scores. the assumption was: if the scores of the disabled features correlated with the drop in the model predictive performance, it meant that humans could understand and accurately assess cnn features using word clouds. we used small training datasets so that the trained cnns had features with different levels of quality. some features detected useful patterns, while others overfitted the training data. ## exp 3: dataset shift dataset shift is a problem where the joint distribution of inputs and outputs differs between training and test stage @xcite . many classifiers perform poorly under dataset shift because some of the learned features are inapplicable (or sometimes even harmful) to classify test documents. we hypothesize that find is useful for investigating the learned features and disabling the overfitting ones to increase the generalizability of the model. ## discussion and conclusions we proposed find, a framework which enables humans to debug deep text classifiers by disabling irrelevant or harmful features. using the proposed framework on cnn text classifiers, we found that (i) word clouds generated by running lrp on the training data accurately revealed the behaviors of cnn features, (ii) some of the learned features might be more useful to the task than the others and (iii) disabling the irrelevant or harmful features could improve the model predictive performance and reduce unintended biases in the model.