Outlier-Aware Training for Improving Group Accuracy Disparities (2022)

Abstract: Methods addressing spurious correlations such as Just Train Twice (JTT, Liu et al. 2021) involve reweighting a subset of the training set to maximize the worst-group accuracy. However, the reweighted set of examples may potentially contain unlearnable examples that hamper the model’s learning. We propose mitigating this by detecting outliers to the training set and removing them before reweighting. Our experiments show that our method achieves competitive or better accuracy compared with JTT and can detect and remove annotation errors in the subset being reweighted in JTT.

## introduction machine learning models trained with empirical risk minimization @xcite ) can achieve a high average accuracy by minimizing the overall loss during training. despite this, erm models are also known to perform poorly on certain minority groups of examples. when specific attributes in a dataset frequently co-occur with a class label, erm models often learn to correlate the co-occurring attributes and the label, using the attributes as "shortcuts" for classifying examples. these "shortcuts" are also called spurious correlations, because model performance can significantly decrease when the model encounters examples that belong to a minority group where the correlations between the attributes and class label do not hold. more specifically, each class in a dataset can be divided by whether their examples contain such spurious attributes. each set of examples with a class-attribute combination is called a "group". the worst group is characterized by having the poorest erm model performance among other groups. as an example, figure 1 shows accuracy disparities among groups in the fever dataset. the @xcite . the data are divided into six groups in accordance with class-attribute combinations, where @xmath0, supports (sup), not enough info (nei)} and @xmath1, neg}, indicating the presence of a negation word in the claim. both methods perform well on groups with strong spurious correlations (e.g., @xcite ). our proposed method (jtt-m) helps improve accuracies for groups where such spurious correlations do not appear (e.g., @xcite and @xcite ). erm-trained model can achieve close to perfect accuracy on the group with a spurious correlation (the refutes class with negation), but only half the accuracy on the worst group (the supports class with negation). improving the worst-group performance of erm models while maintaining the overall accuracy is an active topic of research that has applications in fair machine learning classifiers or robustness against adversarial examples @xcite . methods aiming to maximize worst-group accuracy can be roughly categorized into two categories: those that utilize group information and those that do not. group distributionally robust optimization (group dro, sagawa et al. 2020) uses attribute (and thus group) information during training to dynamically minimize the loss of each group. while group dro achieves a high worstgroup and overall accuracy, it requires annotation on group information during training, which can be expensive to obtain and unavailable for less popular datasets. on the other hand, methods such as dro with conditional value-at-risk (cvar dro, @xcite , learning from failure (lff, nam et al. 2020), predict then interpolate (pi, @xcite , spectral decoupling (sd, @xcite , just train twice (jtt, @xcite @xcite all aim to minimize worstgroup loss without group information. cvar dro minimizes worst-case loss over all subpopulations of a specific size and requires computing the worst-case loss at each step. lff trains an intentionally biased model and upweights the minority examples. pi interpolates distributions of correct and incorrect predictions and can minimize worst-case loss over all interpolations. sd replaces the l 2 weight decay in the cross entropy loss function with logits. rwy reweights sampling probabilities so that mini-batches are classbalanced. suby subsamples large classes so that every class is the same size as the smallest class. jtt simply obtains misclassified examples (the error set) from the training set once and upweights the fixed set of erroneous examples. we focus on jtt due to its simplicity and relative effectiveness and because it does not require group information for improving worst-group accuracy. while idrissi et al. (2022)'s suby and rwy also follow jtt in improving worst-group accuracies, their methods target only datasets with imbalanced classes, and are not applicable to class-balanced datasets such as multinli @xcite . we propose further enhancing jtt by removing outliers from the error set before upweighting it. the outliers might be examples that are difficult to learn, such as annotation errors. keeping them from being upweighted allows the model to train on a cleaner error set and thus better show the intended effect of the original jtt. we focus on worst-group performance caused by the spurious correlations of negation words and evaluate on datasets susceptible to spurious correlations of this type. our experiments on the fever and multinli datasets show that our method can outperform jtt in terms of either the average or the worst-group accuracy while maintaining the same level of performance for the other groups. our contributions are as follows. we devise a method for improving worst-group accuracy with-out group information during training based on jtt (section 3). we show that by removing outliers from the error set being upweighted, we can achieve similar or better overall and worst-group performance (section 4.2). our examination of the outliers being removed also suggests that the improvement may come from removing annotation errors in the upweighted error set (section 4.3). ## background spurious correlations and minority groups we investigate the spurious correlations occurring in two natural-language datasets: fever @xcite and multinli @xcite . the task for fever involves retrieving documents related to a given claim, finding sentences to form evidence against the claim, and then classifying the claim on the basis of the evidence into three classes: supports (sup), refutes (ref), or not enough information (nei). we focus on improving the worst-group classification performance for the final part of the task. the task for multinli is to classify whether the hypothesis is entailed by, neutral with, or contradicted by the premise. we use @xcite 's preprocessing of both datasets, containing 178,059/11,620/11,710 training/dev/test examples for fever and 392,702/9,832 training/test examples for multinli. attributes known to cause spurious correlations for these datasets are negation words @xcite and verbs that suggest negating actions @xcite . we merge these two sources of negation words into a single set: {no, never, nothing, nobody, not, yet, refuse, refuses, refused, fail, fails, failed, only, incapable, unable, neither, none}. each class can be split into two groups based on whether each claim/hypothesis contains a spurious attribute (i.e., the negation words listed above). models tend to perform well on groups where the attributes are highly correlated with the label. groups where the correlation between the label and the attribute does not hold are called minority groups or worst groups, since models often fail to classify their examples correctly. for example, the claim "luis fonsi does not go by his given name on stage.", labeled supports, belongs to the worst group @xcite . table @xcite shows that most claims containing negation are from the class refutes. the relatively small amount of examples from the groups no negation negation ref 27,575 (17.1%) 14,275 (86.3%) 41,850 (23.5%) sup 99,303 (61.5%) 1,267 (7.7%) 100,570 (56.5%) nei 34,633 (21.4%) 1,006 (6.0%) 35,639 (20.0%) (a) fever no negation negation contr 88,180 (27.3%) 42,723 (61.2%) 130,903 (33.3%) ent 118,554 (36.7%) 12,345 (17.7%) 130,899 (33.3%) neut 116,185 (36.0%) 14,715 (21.1%) 130,900 (33.3%) (b) multinli ## proposed method even though the upweighted erm error set can improve worst-group accuracy, it is possible that the error set contains unlearnable or out-of-distribution (ood) examples, e.g., annotation errors. when upweighting the entire error set, these examples will get amplified along with the rest of the error set, lessening the overall benefits of upweighting and retraining. we propose modifying the jtt algorithm by removing outliers in the erm error set before training the second time. we adopt a similar approach from @xcite for detecting outliers. let x be the output of the penultimate layer (i.e., the last layer before the logits) and belong to class y. first, we calculate the mahalanobis distance for each x from the mean of each class y: correct, neg wrong, neg outlier, neg correct, no neg wrong, no neg outlier, no neg where s in and s out are the sets of in-distribution and ood training examples, and p i is the p-value of the i-th example. we show the t-sne visualization in figure 2 . once the ood examples are identified, we remove the subset of misclassified ood examples from the error set e, forming a new error set e in : ## conclusion we have shown that the jtt algorithm can benefit from pruning the error set before upweighting and training a second time, improving worst-group accuracy or overall accuracy on two popular datasets. we also showed that annotation errors may occur in the error set, hampering jtt's effectiveness. these annotation errors can be mitigated by detecting and removing them with our mahalanobis distance method. investigating the effects of using other ood-detection methods and finding a more effective way to tune the additional hyperparameters are directions for our future work.