Measuring and Mitigating Local Instability in Deep Neural Networks (2023)

Abstract: Deep Neural Networks (DNNs) are becoming integral components of real world services relied upon by millions of users. Unfortunately, architects of these systems can find it difficult to ensure reliable performance as irrelevant details like random initialization can unexpectedly change the outputs of a trained system with potentially disastrous consequences. We formulate the model stability problem by studying how the predictions of a model change, even when it is retrained on the same data, as a consequence of stochasticity in the training process. For Natural Language Understanding (NLU) tasks, we find instability in predictions for a significant fraction of queries. We formulate principled metrics, like per-sample “label entropy” across training runs or within a single training run, to quantify this phenomenon. Intriguingly, we find that unstable predictions do not appear at random, but rather appear to be clustered in data-specific ways. We study data-agnostic regularization methods to improve stability and propose new data-centric methods that exploit our local stability estimates. We find that our localized data-specific mitigation strategy dramatically outperforms data-agnostic methods, and comes within 90% of the gold standard, achieved by ensembling, at a fraction of the computational cost.

## introduction while training large deep neural networks on the same data and hyperparameters may lead to many distinct solutions with similar loss, we say that the model is underspecified (d' @xcite . one tangible manifestation of underspecification is that a model prediction on a single data point can change across different training runs, without any change in the training data or hyperparameter settings, due to stochasticity in the training procedure. this extreme sensitivity of model output, which has been termed as model variance/instability or model jitter/churn @xcite , is highly undesirable as it prohibits comparing models across different experiments @xcite . we refer to this problem as local instability foot_0 , a term that highlights our focus on the non-uniformity of instability across data points. local instability can lead to highly undesirable consequences for deployed industrial systems, as it can cause inconsistent model behavior across time, eroding trust on ai systems @xcite . the problem is further exacerbated by the fact that industry models are typically more complex and trained on diverse datasets with potentially higher proportion of noise. massive dataset. this shows different predictions over 50 model runs with different seeds. p is the prediction score on gold labels and σ m is the standard deviation over multiple model outputs p1 , . . . , p 50 . for example, start house cleanup with gold label iot is predicted to label lists 26 out of the 50 model runs. its prediction score on iot ranges between 0.002 and 0.97. green: low variability, predictions match gold label, red: high predicted label flipping or switching. measuring local model instability while detecting and quantifying local instability across multiple runs that is trivial for toy problems, it becomes infeasible when dealing with much larger industrial datasets. previous research @xcite suggested the use of single-run training dynamics to estimate the variance in prediction scores over multiple epochs for a model. however, as shown in ## related work sophisticated, real-world applications of deep neural networks (dnns) introduce challenges that require going beyond a myopic focus on accuracy. uncertainty estimation is increasingly important for deciding when a dnn's prediction should be trusted, by designing calibrated confidence measures that may even account for differences between training and test data @xcite . progress on uncertainty estimation is largely orthogonal to another critical goal for many engineered systems: consistency and reliability. will a system that works for a particular task today continue to work in the same way tomorrow? one reason for inconsistent performance in realworld systems is that even if a system is re-trained with the same data, predictions may significantly change, a phenomenon that has been called model churn @xcite . the reason for this variability is that neural networks are underspecified (d'amour et al., 2020b), in the sense that there are many different neural networks that have nearly equivalent average performance for the target task. while randomness could be trivially removed by fixing seeds, in practice tiny changes to data will still significantly alter stochasticity and results. we will explore the case of altering training data in future studies. studying how stochasticity affects model churn addresses a key obstacle in re-training engineered systems while maintaining consistency with previous results. the most common thread for reducing model churn focuses on adding constraints to a system so that predictions for re-trained system match some reference model. this can be accomplished by adding hard constraints @xcite or distillation @xcite @xcite . we adopt a subtly different goal which is to train at the outset in a way that reduces variability in predictions due to stochasticity in training. previous research @xcite have suggested a co-distillation procedure to achieve this. label smoothing, which reduces over-confidence @xcite , has also been suggested to reduce variance, with a local smoothing approach to reduce model churn appearing in @xcite . a distinctive feature of our approach is a focus on how properties of the data lead to instability. inspired by dataset cartography @xcite which explored variance in predictions over time during training of a single model, we investigate how different data points vary in predictions across training runs. non-trivial patterns emerge, and we use sample-specific instability to motivate a new approach to reducing model churn. our work draws connections between model stability and recent tractable approximations for bayesian learning @xcite . recent bayesian learning work focuses on the benefits of bayesian model ensembling for confidence calibration, but an optimal bayesian ensemble would also be stable. bayesian approximations exploit the fact that sgd training dynamics approximate mcmc sampling, and therefore samples of models over a single training run can approximate samples of models across training runs, although not perfectly @xcite @xcite . we have studied connections between prediction variability within a training run and across training runs, and used this connection to devise practical metrics and mitigation strategies. similar to banns @xcite , our teacher and corresponding student models use the same model architecture with same no. of parameters rather than using a high-capacity teacher model, however, unlike banns, our work is geared towards addressing model instability. architecturally, our methodology (tgtss) uses a temperature scaled temporally smoothed vector that is obtained from the last n checkpoints from the teacher model instead of the finalized teacher model and not use the annotated labels for the utterances. ## model instability measurement the examples in therefore, we computed the label switching entropy. given a setup with training data {x i , y i } ∈ x where x are utterances, @xmath1, ..., k} are the corresponding gold labels, the multi-run label entropy (le m ) over n independent runs for an utterance x i can be computed as, ## model instability mitigation in our study, we have explored 3 baseline mitigation strategies to address model instability: ensembling, stochastic weight averaging (swa) and uniform label smoothing. these methodologies have been used in numerous other works to improve generalization as well as predictive accuracy across a diverse range of applications. performance of the ensembling strategy serves as our upper bound in reducing model instability. we propose a novel model instability mitigation strategy, temporal guided temperature scaled label smoothing, that is able to recover 90% of the reduction in model instability as ensembling at a fraction of model training time and computational cost. we describe all the mitigation strategies below. ## conclusion in this work, we studied the problem of model instability/churn in deep neural networks in the context of large scale nlu systems. assigning different labels to the same training data over multiple training runs can be detrimental to many applications based on dnns. we noticed that the instability of model predictions are non-uniform over the data, hence we call it local instability. we proposed a new metric, label switching entropy, that is able to quantify model instability over multi-runs as well as a single training run. we also introduced temporal guided temperature scaled smoothing that reduces model