When and Why a Model Fails? A Human-in-the-loop Error Detection Framework for Sentiment Analysis (2021)

Abstract: Although deep neural networks have been widely employed and proven effective in sentiment analysis tasks, it remains challenging for model developers to assess their models for erroneous predictions that might exist prior to deployment. Once deployed, emergent errors can be hard to identify in prediction run-time and impossible to trace back to their sources. To address such gaps, in this paper we propose an error detection framework for sentiment analysis based on explainable features. We perform global-level feature validation with human-in-the-loop assessment, followed by an integration of global and local-level feature contribution analysis. Experimental results show that, given limited human-in-the-loop intervention, our method is able to identify erroneous model predictions on unseen data with high precision.

## introduction deep learning approaches, especially neural network-based ones, have been widely employed and proven effective in sentiment analysis tasks @xcite . these performance improvements, however, have come at the cost of model transparency and accountability @xcite . many times, deep models are being used as black-box tools by users, even without knowing the model's peculiarities as well as limitations @xcite . in that sense, users can be easily exposed to impropriety or error predictions made in run time, and thus lose trust towards the sentiment classification system. traditional evaluation metrics, such as accuracy and f1-score, can explain the predictive performance of a sentiment model. however, their explanations are from an overall and reactive perspective, as they fail to provide insights into the * yg was affiliated with ibm research -almaden at the time of the work reported in this paper. with this challenge in mind, we introduce in this study an explainable error detection framework with human-in-the-loop for sentiment analysis task. specifically, as shown in figure 1 , given a pre-trained black-box sentiment model, the errordetection framework first analyzes local feature contributions through a data perturbation process. next, the local feature contributions are aggregated for global-level feature contributions. later, humans are brought into the loop to assess the relevance of the top ranked global features to the target sentiment classes, and report errors if any. an erroneous score is calculated based on both global and local features. instances exhibiting erroneous scores above a specific threshold are flagged as problematic predictions. we demonstrate the error detection framework on two sentiment test datasets. from experimental results, we notice high error detection precision of the proposed framework. our contributions are fourfold: first, we present a high precision error detection framework for sentiment analysis task, which can proactively notify users with prediction errors in run-time. second, the proposed framework calculates the likelihood of concerns on model correctness in even unseen cases. third, erroneous predictions are identified based on explainable features which allow the users to easily understand why a prediction fails. fourth, the proposed error detection framework requires little human effort in error detection by labeling on the global feature level, rather than local instance level. ## related work error analysis and detection is important to building accountable ai models, as they allow individuals to understand when and how predictions fail @xcite . the most intuitive error analysis method is to evaluate the algorithms on a broad set of performance metrics, such as, sensitivity and specifity analysis @xcite . but their explanations are from an overall and reactive perspective, as they fail to provide insights on each specific instance level on unseen data. manual error analysis is another common approach, whereas it requires significant human efforts and time, and is not easy to scale. by taking prior knowledge, such as semantic context, into consideration, heuristicbased method brings context-based errors into practice @xcite . comparing to the manual error analysis, heuristic-based method requires no human intervention. however, predefined heuristics applicable to a specific model is difficult to get. uncertainty sampling @xcite based on a model's confidence scores to identify and label potential prediction errors. although no human effort is needed in this approach, its performance is not always reliable. in addition, a series of methods have recently been introduced to identify model errors in a human-machine interac-tive manner @xcite @xcite . by adding humans into the model evaluation step @xcite , the proposed methods allowed the users to improve the model performance iteratively, by identifying model errors, providing new training data rewardingly, and retraining the model. common limitations of these human-in-the-loop based methods are that, they require human labeling on the instance level, which can be labor-intensive, and can not be easily generalized to unseen data. ## methods in this section, we present our framework to detect errors in sentiment predictions with human-in-theloop in detail. given a blackbox sentiment model and a set of unseen test data, the proposed method runs over the following four steps: 1) "local-level feature contributions" module quantifies the feature contributions to the prediction of each individual target instance. 2) "global-level feature contributions" module characterizes the general effect of a feature to the overall prediction across all instances. 3) "human assessment" module brings human into the loop of error detection by allowing them to manually label on an interpretable feature-level, instead of instance-level, to save the labeling efforts. 4) "global-local integration" module quantifies the erroneous probabilities of instance-level predictions made by the model. with the erroneous probabilities, the framework can send users with failure alerts in prediction run-time on unseen sentiment data. ## experiment settings to test our error detection framework, we first create a three-class sentiment classifier, and later treat it as a black-box "pre-trained" model for error detection. we implemented the classifier using a replication of the multichannel cnn model introduced by @xcite , although any algorithm can be applied here as the black-box pre-trained model. the training data contains 2,265,413 positive, 2,704,587 negative, and 2,297,426 neutral cases. they were collected from various sources, ranging from high-quality human-labeled instances to pseudo-labeled instances annotated using emoji or hashtag based indicators @xcite . we evaluated the model on the test dataset of se-meval2016 task 4 subtask a @xcite . our model achieved a f p n 1 of 0.345, and a threeclass prediction accuracy of 0.463, which is comparable to many of the semeval2016 participation systems. we calculated the local and global feature contributions using the training dataset and performed the human-in-the-loop assessment on figure eight 2 . we extracted the top 2,000 non-neutral features with the highest global contribution magnitudes to human assessors to determine their global correctness. we chose only non-neutral features for error assessment as polarity errors, such as predicting positive as negative or vice versa, can greatly impact user's trust towards the model, and we want to focus more on such extreme cases for error detection. 5 unigrams were shown in 1 annotation page and gold questions (easy questions with known answers, e.g. "happy" with the definition "enjoying or showing or marked by joy or pleasure" as "positive" polarity) were embedded on each page for quality check. for each unigram feature, we recruited in total 5 assessors who had to be native english speakers, with the highest level of experience. for all 2,000 unigram features, we collected in total 10,155 judgements within 1 hour. among them only 155 (1.5%) were from untrusted assessors, who have been excluded during the annotation process. this indicated the relative easiness of feature labeling for sentiment task for even non-2 https://www.figure-eight.com/ expert assessors. we obtained the annotations for 1,725 out of the 2,000 assessed unigrams and converted them into binary cases (agree or disagree, 86.25% inter-rater agreement), where at least 3 assessors agree on the same answer. among them 161 were found to be wrongly learned by the pretrained black-box model. global-level features include "kashmir", "midterm", "dems", "netflix" was being wrongly predicted by the model as "negative", whereas "wingstop", "panera", "minister", "popeyes" as "positive". all 161 global-level problematic features would then be passed to the next step to guide the instance-level error identification. to assess the method's effectiveness, we applied the proposed error detection framework first on the test dataset of semeval2016 task 4 subtask a. we found 932 instances containing at least one of the globally identified problematic features. thus, we only calculated the local erroneous score e for the 932 cases. given that the semeval dataset only covers a subset (60/161) of the erroneous unigrams, we prepared another dataset customized just for better understanding of the precision of the proposed framework. specifically, we adopted the 161 problematic unigrams as search keywords to collect tweets using twitter search api. for each keyword we collected up to 50 most recent tweets. we cleaned the collected dataset by removing duplicate tweets and tweets with url, assuming that they have a higher probability of being spam. in total, we collected 3,111 instances in this customized twitter testing dataset. for both datasets, we extracted all instances with @xmath0. for the self-collected twitter data, we acquired the ground truth labels from the crowd on figure eight . we reported the precision of the proposed method at different settings of the threshold τ , to understand its impacts on the framework's performance. we adopted uncertainty sampling as the baseline to evaluate the performance of the proposed error detection framework. we adopted the least confidence as the measurement in this experiment, which is based on the difference between the most confident prediction and 100% confidence. in other words, for a three-class classification task, the model is most unconfident when having the maximum prediction probability around 0.33. we applied uncertainty sampling just on the complete semeval test data, assuming no filtering at all was applied on the original dataset. we extracted in-stances with the lowest prediction confidence as under-trained cases and compared the precision@k for both the baseline method and the proposed method. ## results in figure 4 , we plotted the error detection precision for the proposed method on both datasets, along with varied thresholds of τ ranging from 0 to 0.4. we set the upper bound τ to 0.4 instead of 1, since only very limited number of instances (sizes with no or little statistical meaning) were detected for the semeval dataset when τ ≥ 0.5. as can be seen from figure 5 , even when the threshold τ was set to a low score of 0, the proposed error detection framework can still achieve relatively high precision scores for both datasets. specifically, for the semeval dataset, with τ = 0, the proposed framework indicated 48.9% of all prediction instances as erroneous predictions. among these flagged instances, 67.7% were proved to be truly problematic according to the ground truth labels. the same pattern was also noticed for the self-collected twitter dataset, when τ = 0, the proposed method detected in total 57.9% suspicious predictions, and 70.5% of them were proved to be truly problematic. looking further, we observed that the error detection framework became even more precise, as we incrementally increased the value of τ . it reached the highest precision of 85.9% for the semeval and 90.8% for the twitter dataset when τ = 0.4. but obviously these increases in precision were achieved with a tradeoff of the degradation in the number of detected problematic predictions. considering our ultimate goal of altering users of potential prediction errors, we next compared the proposed error detection framework with uncertainty sampling based on precision@k. precision@k is a widely adopted evaluation metric in information retrieval tasks. it is being defined here as the proportion of identified erroneous cases that are real errors in the top k retrieved results. we chose k ranging from 100 to 400, as only 455 predictions are being identified as problematic with τ ≥ 0. as can be noticed in in addition to precision, we were also interested in knowing if the proposed method was able to catch errors that can not be detected by the uncertainty sampling baseline. to achieve this goal, we plotted in figure 5 the distribution of the prediction probabilities of the erroneous cases identified by the proposed framework when τ = 0.4. from figure 5 , we found that about 40% of the erroneous instances detected by the proposed method were associated with prediction probabilities of larger than 0.7. in other words, this means that the model is quite confident about those predictions and uncertainty sampling would hardly treat them as potential errors. in that sense, we conclude that the proposed framework is able to detect errors even when the prediction confidence is high. ## discussion and conclusion our work was motivated by the practical concerns of not knowing the limitations or potential errors of a sentiment model prior to deployment, and not being able to notify or even rectify when erroneous predictions were made once deployed. driven by this demand, in this paper, we presented a framework for identifying prediction errors in an interpretable manner for sentiment analysis tasks. we validated the proposed error detection framework on two different datasets. results showed that the proposed method can identify problematic model predictions with high precision, which is critical to continuous model refinement. while comparing the proposed approach with the baseline, we noticed that our method can also be adopted as a selective sampling approach, in addition to uncertainty sampling. besides, given that the proposed error detection framework can be applied on unseen data without ground truth, it can proactively notify users about possible erroneous decisions made by the model in prediction run-time. in addition to its effectiveness, the proposed error detection framework can also be easily explained to the users. globally, the proposed framework allows the users to understand the overall contribution of a word to the final predictions made by the black-box sentiment model. besides, as demonstrated in our results, global-level contributions can also be useful for identifying potential bias existed in the model. for instance, we noticed that "netflix", "dems", and "palestine" were being learned as "negative" in our pre-trained sentiment model. while integrating global-level problematic features with local-level predictions, the proposed erroneous score enables users to know why that specific prediction could be wrong or biased and how much the problematic global feature contributed to the erroneous or biased prediction. certainly, more work is needed on how the proposed framework can be generalized to the task of bias detection. furthermore, the proposed framework efficiently integrated human into the loop of model validation and refinement. comparing with the previous methods, our approach allows the human annota-tors to label on the explainable feature level, rather than on the instance level, which can significantly save their time and effort. regarding the annotation quality of the feature level labeling, our results showed that even non-expert crowd workers can accurately finish the assessment tasks with high inter-rater agreement in a very short period of time. finally, our work comes with certain limitations. one of them is the relatively small number of global features (2,000) that were labeled by the human assessors in this work. to some extent, this limited us from evaluating the presented error detection framework from more angels other than precision, although precision is the most important measurement for error detection. annotations on larger scales will be conducted in later studies and the effectiveness of the proposed framework will be evaluated from more angels. besides, future works will also be conducted on investigating how these identified erroneous instances or features could be used for further fixing or debugging the pre-trained models.