Accountable Error Characterization (2021)

Abstract: Customers of machine learning systems demand accountability from the companies employing these algorithms for various prediction tasks. Accountability requires understanding of system limit and condition of erroneous predictions, as customers are often interested in understanding the incorrect predictions, and model developers are absorbed in finding methods that can be used to get incremental improvements to an existing system. Therefore, we propose an accountable error characterization method, AEC, to understand when and where errors occur within the existing black-box models. AEC, as constructed with human-understandable linguistic features, allows the model developers to automatically identify the main sources of errors for a given classification system. It can also be used to sample for the set of most informative input points for a next round of training. We perform error detection for a sentiment analysis task using AEC as a case study. Our results on the sample sentiment task show that AEC is able to characterize erroneous predictions into human understandable categories and also achieves promising results on selecting erroneous samples when compared with the uncertainty-based sampling.

## introduction as machine learning is becoming the method of choice for many analytics functionalities in industry, it becomes crucial to be able to understand the limits and risks of the existing models. in favour of more accurate ai, the availability of computational resources is coupled with increasing dataset sizes that has resulted in more complex models. complex models suffer from lack of transparency, which leads to low trust as well as the inability to fix or improve the models output easily. deep learning algorithms are among the highly accurate and complex models. most users of deep learning models often treat them as a black box because of its incomprehensible functions and unclear working mechanism @xcite . however, customers' retention requires accountability for these systems @xcite . interpreting and understanding what the model has learned, as well as the limits and the risks of the existing model have therefore become a key ingredient of a robust validation @xcite . one line of research on model accountability examines the information learned by the model itself to probe the linguistic aspects of language learnt by the models @xcite @xcite @xcite . other line of research gives machine learning models the ability to explain or to present their behaviours in understandable terms to humans @xcite to make the predictions more transparent, and trustworthy. however, very few studies set the focus on error characterization as well as automatic error detection and mitigation. to address the above-mentioned gaps in characterizing model limits and risks, we seek to improve a model's behavior by categorizing incorrect predictions using explainable linguistic features. to accomplish that, we propose a framework called accountable error characterization (aec) to explain the predictions of a neural network model by constructing an explainable error classifier. the most similar work to ours is by @xcite . they build interpretable decision-tree classifiers for summarizing failure conditions using human and machine generated features. in contrast, our approach builds upon incorrect predictions on a separate set to provide insights into model failure. the aec framework has three key components: a base neural network model, an error characterization model, and a set of interpretable features that serve as the input to the error characterization model. the features used in the error characterization model are based on explainable linguistic and lexical features such as dependency relations, and various lexicons that have been inspired by prior art, which allows the users and model developers to identify when a model fails. the error characterization model also offer rankings of informative features to provide insight into where and why the model fails. by adding the error classification step on top of the base model, aec can also be adopted to identify the highly confident error cases as the most informative samples for the next round of training. although uncertainty based sampling can also be adopted to get the most informative samples @xcite @xcite , as it selects the examples with the least confidence, ghai et al. (2020) show that uncertainty sampling led to an increasing challenge for annotators to provide correct labels. aec avoids such problem by learning from error cases from a validation set. our results show that aec outperforms the uncertainty based sampling in terms of selecting erroneous predictions on a sample sentiment dataset (see we first present the overall framework of aec to construct the error classifier, followed by the experiments and result. finally, we conclude the paper with future directions and work in progress. ## explainable framework figure 1 summarizes our overall method for constructing a human understandable classifier that can be used to explain the erroneous predictions of a deep neural network classifier and thus to improve the model performance. our method consists of the following steps: s1: train a neural network based classifier using labeled dataset i, call it as the base classifier. s2: apply the base classifier on another labeled dataset ii to get correct and incorrect prediction cases, based on which train a second 2class error identification classifier with a set of human understandable features. note here labeled dataset i and ii can be in the same domain or in different domains. s3: rank the features according to their individual predictive power. apply the error identification classifier from step 2, to a set of unlabeled data from the same domain as labeled dataset ii and rank the unlabeled instances according to their prediction probability of being erroneous. these represent the most informative samples that can be further used in an active learning setting. the focus of the current work is to identify and characterize the error cases of a base classifier in an human understandable manner. the following two sections describe the experiments and implementation of the framework using a sentiment prediction task as case study. the integration of these samples into an iterative training set up is a work in progress for future extension. ## evaluation and results in terms of identifying erroneous predictions, in our evaluation, we compare the performance of aec with uncertainty-based sampling, in which the learner computes a probabilistic output for each sample, and select the samples that the base classifier is the most uncertain about based on probability scores. ## conclusion and future work we present an error characterization framework, called aec, which allows the model users and developers to understand when and where a model fails. aec is trained on human understandable linguistic features with erroneous predictions from the base classifier as training input. we used a cross-domain sentiment analysis task as case study to showcase the effectiveness of aec in terms of error detection and characterization. our experiments showed that aec outperformed uncertainty based sampling in terms of selecting the erroneous samples for continuous model improvements (a strong active learning baseline for selecting the most uncertain samples for continuous model improvements) for the task of predicting errors which can act as most informative samples of the base classifier. in addition, errors automatically detected by aec seemed to be more understandable to the model developers. having these explanations lets the end users make a more informed decision, as well as guide the labeling decisions for next round of training. as our initial results on sentiment dataset look promising for both performance and explainability, we are in the process of extending the framework to run the algorithm iteratively on multiple datasets. while applying the error characterization classifier on the unlabeled datasets, not only we will select the top k instances with the highest prediction probability of being correctly predicted and add them back to the original training dataset for retraining purpose, but we will also select top k instances with the highest prediction probability of being incorrectly predicted. we will assign those instances to human annotators for labels and add them back to the original labeled data as well for the next iteration of training process. we will continuously feed these samples to train the base network, and evaluate the actual performance gains for the base classifier.