When in Doubt: Improving Classification Performance with Alternating Normalization (2021)

Abstract: We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification. CAN improves classification accuracy for challenging examples by re-adjusting their predicted class probability distribution using the predicted class distributions of high-confidence validation examples. CAN is easily applicable to any probabilistic classifier, with minimal computation overhead. We analyze the properties of CAN using simulated experiments, and empirically demonstrate its effectiveness across a diverse set of classification tasks.

## introduction classification is core to nlp, and many language problems can be effectively addressed as supervised classification tasks. however, even the most effective classifier can suffer when given examples to classify that are close to its decision boundary. the reasons for such failures vary, and include lack of training data coverage, limited representation expressivity, or over-fitting the training data. despite significant progress, including using pre-trained models @xcite to address these issues, every classifier has its weak spots, and some examples will be hard to classify correctly. in this paper, we study a simple nonparameterized post-processing step to improve classifier accuracy on difficult examples. at the core of our approach is using alternating normalization (an; @xcite to re-adjust the prediction of low-confidence examples using the predicted class distributions of a reference set of high-confidence validation examples. our process, classification with alternating normalization (can), is applicable to any classifier that generates a distribution over target classes. we first 1 our code for the work are open sourced at github.com/kmnp/can. identify challenging examples and a disambiguating reference set using the ambiguity level of the predicted class probability distributions. then we perform a series of normalizations, alternating between normalizing across examples for each class and for each example across classes. for example, in figure 1 , we classify an input example ("smile") to one of two labels ( and ). the example sits at the decision boundary between the two target labels, and is completely ambiguous. the reference set includes a single example ("hat"), which the classifier can resolve with high confidence. here, we use a single alternating normalization step, including normalizing across rows (examples) and columns (target labels), which disambiguates this simple example to classify it correctly. we study can on randomly generated matrices (section 3) and evaluate it on several text classification tasks (section 4). in general, we find that can is most effective when the original predictions are of higher ambiguity. our experiments also suggest can is more beneficial for tasks with many labels. all code will be released upon publication. ## classification with alternating normalization we modify the output of a probabilistic classifier for an ambiguous input example at test time using alternating normalization with respect to an additional set of examples to re-adjust the input example's predicted probabilities. let x and y be random variables for input and labels, respectively. given a challenging example @xmath0, and a probability distribution p(@xmath1, we compute p (y|x) as an adjusted distribution in three steps, the second of which is an iterative process. our method requires a reference set of examples. we use the confident portion of the validation data commonly used in standard evaluation settings. given n confident examples, we create a row stochastic matrix a 0 ∈ r n×m by concatenating the predicted probability distributions. we describe the proposed method below. step 1: identify hard examples let b 0 ∈ r m be the source prediction, i.e., the predicted class probability distribution for a challenging example x. we identify examples that can benefit from can by computing the ambiguity level of its class probability distribution b 0 . with a selected example, we construct a matrix l 0 ∈ r (n+1)×m by concatenating a 0 and b 0 along the rows: 3 a common way to identify challenging examples is by measuring the entropy of their predicted class distribution h(b 0 ) =i b 0,i log m (b 0,i ). the higher the value of h(b 0 ) is, the more uniform the distribution v is, which indicates higher ambiguity level. we observe h(b 0 ) may not be ideal to capture ambiguity well for our classification purpose, which is concerned mainly with high probability events. for example, consider two distributions b 1 0 = 0.5 0.25 0.25 and b 2 0 = 0.5 0.5 0 , for which h(b 1 0 ) > h(b 2 0 ). however, b 2 0 expresses a more uncertain classification result. instead, we select challenging examples based on the entropy near the peak of the distribution. we define top-k-entropy to focus on the top of the distribution. let t : r @xmath2. we use a base of m so that 0 < h top-k (b 0 ) ≤ 1. given a scalar threshold 0 ≤ τ ≤ 1 and the number of classes k max , the ambiguity level of a probability distribution b 0 is larger than τ , if for any @xmath3, k max ], the top-k-entropy of b 0 is greater than τ . step 2: one iteration of an each iteration d of an normalizes l 0 , first across its rows (row norm) and then along its columns (column norm). let d : r @xmath4, and let e be a vector of ones. ## simulations on random matrices we study the effect of the ambiguity level of the source prediction and the reference set using monte carlo simulations. we randomly generate a 0 , b 0 , and λ q to evaluate the expected performance trix sizes, simulations, and values of α. we observe that (a) the expected accuracy gains are positive across all ambiguity levels of a 0 and b 0 ; (b) can tends to improve the performance of b 0 with high ambiguity level, especially using a reference set with low top-k entropy; and (c) the performance is robust to the ambiguity level of a 0 . ## empirical experiments we evaluate can using three classification tasks: ultrafine entity typing @xcite , and dialogue-based relation extraction (dialogre; @xcite . we compare off-the-shelf classifiers (baseline) and our method. 7 we select can's hyperparameters (α, d, τ ) using the official validation sets (val) and evaluate on the official test sets (). the challenging subsets of the val and test are identified as the source predictions using top-k entropy. the rest of val, which has low ambiguity, is used as the reference set a 0 . foot_6 results tuning can figure 3 shows the how the values of the hyperparameters (α, d, τ ) affect can on the val set using ultrafine entity typing multitask base-line. we observe that the effect of d and τ diminish gradually as α grows, because α controls how quickly can transforms the source predictions to high-confidence ones. for a fixed recursion depth d, the performance does not always improve using a larger α. this suggests that larger α can deteriorate the performance through over-calibration. we also see that a small value of recursion depth d yields the best results in general. ## analysis we hypothesize that classifiers with better performance require less re-adjustment. we test this hypothesis by controlling the number of "hard" examples, therefore controlling the performance of the baseline classifier. we use imagenet @xcite because of the existence of established ways for image perturbation without modifying the image semantics. we use a resnet-50 @xcite model from the default pretrained models in torchvision @xcite as baseline. we systematically make the task harder by convolving the images with a gaussian function with zero-mean and various values of standard deviation σ. higher values of σ results in more blurred image, which emulate examples with higher uncertainty and lower baseline classifier performance. figure 4 shows example images with different values of σ ∈ {2, 4, 8, 16, 32}. we assess the effect of can by comparing two variants of can with baseline using this setup: (a) can-naive: α = 1.0, recursion depth is 1; and (b) can-best: an upperbound-version of can with α and depth d optimized on the test data. figure 5 presents the absolute top-1 accuracy (%) gain on the hard subset of val. optimizing α and d (can-best) indeed offers larger performance gains across six different image settings comparing with can-naive. we observe a general positive correlation between the number of hard examples and the relative gain using can, confirming that our method benefits classifiers that receive harder data. however, this advantage diminishes when a classifier significantly under-performs, as the accuracy gain dropped for blur (σ = 8) onward and its top-1 accuracy is only 3.25% on hard subset. previous work has applied rsa to systems that generate and understand language @xcite @xcite @xcite in both referential games @xcite @xcite and sequential decisionmaking systems @xcite . our method departs from these applications by focusing on the ambiguity avoidance property of the listener agent as applied to generic classification tasks. confidence calibration similar to confidence calibration techniques @xcite @xcite , our method rescales the posterior distribution produced by the classifier at test time. however, the aim of calibration is to make the output probabilities more representative of the correctness likelihood, whereas our's is to resolve ambiguity. ## conclusions and future work we propose classification with alternating normalization (can) as a simple and light-weight postprocessing step. our method adjusts the predicted class distribution of "edge cases" for a generic classifier during test time. via experiments on both simulated and real-world nlp tasks, we show that can helps improve performance using a fixed reference set with low ambiguity and increases the performance of standard classifiers. future work may further investigate the properties of can. for example, appendix 5 describes an initial study on vision examples showing the benefit of can increases as examples become noisier, even with the same classifier. advancing this study and generalizing it to language is an important direction for future work. finally, one could study improving can, for example by selecting a 0 strategically, or applying it during training.