Reducing Model Churn: Stable Re-training of Conversational Agents (2022)

Abstract: Retraining modern deep learning systems can lead to variations in model performance even when trained using the same data and hyper-parameters by simply using different random seeds. This phenomenon is known as model churn or model jitter. This issue is often exacerbated in real world settings, where noise may be introduced in the data collection process. In this work we tackle the problem of stable retraining with a novel focus on structured prediction for conversational semantic parsing. We first quantify the model churn by introducing metrics for agreement between predictions across multiple retrainings. Next, we devise realistic scenarios for noise injection and demonstrate the effectiveness of various churn reduction techniques such as ensembling and distillation. Lastly, we discuss practical trade-offs between such techniques and show that co-distillation provides a sweet spot in terms of churn reduction with only a modest increase in resource usage.

## introduction deep learning systems can perform inconsistently across multiple runs, even when trained on the same data with the same hyper-parameters. deployment in real-world environments presents a challenge, where constantly changing production systems require frequent re-training of models. for a conversational semantic parsing system such as google assistant or amazon alexa, where the goal is to convert users' commands into executable forms, this erratic behavior can have some unfortunate practical consequences. some examples include irreproducibility, which limits the ability to make meaningful comparisons between experiments @xcite , bias, which creates credibility issues if systems consistently struggle with members of a certain class (d'amour et al., 2020), and user frustration, which can arise due to unpredictable interactions over time. query will i need snow tires to drive the sierra nevada mountains this afternoon? model run 1 [in:get_weather [sl:weather_attribute snow tires ] [sl:location sierra mountains ] [sl:date_time this afternoon ] ] model run 2 [in:get_info_road_condition [sl:road_condition snow tires ] [sl:location sierra mountains ] [sl:date_time this afternoon ] ] the root cause of widely divergent behavior is underspecification @xcite , where there are many equivalent but distinct solutions to a problem. non-determinism in model training (e.g. different data orders or weight initializations) can lead to finding local minima that obtain the same measurements on a held-out test set but make different predictions (also known as model churn). even in an academic setting, controlling for all non-determinism is unrealistic - we thus conduct experiments to evaluate and reduce churn across multiple model re-training runs. our contributions are as follows: to the best of our knowledge, we are the first to study model churn for the structured prediction task of spoken language understanding (slu). ## background and related work the problem of model churn @xcite , defined as the difference in predictions observed across runs when re-training models, has traditionally been studied for classification tasks. in contrast with previous work, we study the problem of model churn for structured prediction, specifically for slu. @xcite introduced "anti-distillation" to increase diversity in ensemble predictions and @xcite introduced the smooth-relu activation function; however, in our initial experiments we did not find significant improvement using these methods when applied to structured prediction. other work has explored forms of smoothing to reduce churn, either by computing soft labels using the nearest neighbors @xcite or by weighting the loss term of individual examples using the predicted probabilities from a teacher model @xcite . as these methods were developed for classification, we leave the task of adapting them to structured prediction for future work. other research has focused on related problems such as reproducibility @xcite and calibration @xcite . @xcite argue that this phenomenon is due to underlying task complexity and annotator disagreement. d'amour et al. (2020) claim that reproducibility is primarily due to underspecification, where there are many distinct solutions to the same problem. while these problems are related to churn, both reproducibility and calibration metrics are computed relative to a target, rather than accounting for agreement across re-training runs. it has been well known that ensembling increases reproducibility and model calibration @xcite . since ensembles increase inference times, distillation @xcite is commonly used to train a student model with similar inference resource usage. @xcite show that ensemble distillation improves calibration for machine translation and named entitity recognition. for our distillation baselines, we follow the recipe by @xcite . for co-distillation, we follow the recipe developed by @xcite . in our work, we look at the aforementioned approaches and compare them in terms of resource usage, churn reduction, and effectiveness on the task of conversational semantic parsing @xcite @xcite @xcite . ## task definition and evaluation we follow recent work @xcite and treat conversational semantic parsing as sequence generation using auto-regressive neural models. the goal is to make a structured prediction given a user command such as the example in our aim is to reduce churn across runs while maintaining high accuracy on the gold labels. thus, we report exact match accuracy (em) with the mean over n runs. while our goal is not to obtain the state of the art, we do want to show which methods reduce churn without a loss in performance. to measure churn, we need a way to compare predictions across runs, independent of the gold labels. while previous work @xcite has used metrics such as prediction difference (similar to hamming distance), the focus was on classification tasks only, making it necessary to compute an alternative measure. metrics such as edit distance or multiple sequence alignment would be appropriate for sequence generation tasks such as machine translation or paraphrasing, where churn across output may differ locally by only a few tokens. comparatively, the meaning of these metrics is unclear for structured prediction tasks such as semantic parsing. for example, computing a token-level distance between a prediction such as "[in:unsupported ]" and "[in:get_event [sl:date_time this weekend ] ]" would not be a useful measure. thus, we report sequence-level model agreement (agr) across n runs, where each example has a score of 1 if all n runs agree on the exact same predicted sequence and 0 otherwise. however, it is possible for all runs to agree but make an incorrect prediction; the goal ultimately is to consistently make correct predictions. consequently, we further extend this metric to include the case where the predictions from all n runs agree and the predictions match the target. we refer to this metric as exact match agreement (em@n). ## methods for churn reduction for our experiments, we explore three techniques which have been effective on related problems such as model calibration: ensembling, which combines the predictions of multiple models, distillation, which pre-trains a teacher model and uses its predictions to train a student, and co-distillation, which trains two or more peer models in parallel and allows each model to learn from the predictions of the other. figure 1 displays these techniques. ## results we test the effectiveness of the methods described in section 4 over @xmath0. we compile results in ensemble superior at the cost of much increased computational cost first, ensemble sets a high bar in almost all settings regardless of artificial noise. while impressive, this approach requires significantly more computation at inference time and is sometimes deemed infeasible to deploy when accounting for resource usage (see co-distillation best among distillation-based methods regardless of noise for label smoothing ( on the other hand, soft/hard distillation perform merely on-par with the baseline or worse. surprisingly, in the 10% random/systematic noise setting, co-distillation not only narrows the gap for em@10/agr compared to the ensemble, but also occasionally outperforms the ensemble in em for top/mtop and top, respectively, which may be due to increased robustness to noise during training, rather than only during inference in the ensemble. ## conclusion our experiments showed that there exists substantial churn across runs when re-training models on the same conversational semantic parsing datasets. we showed that for "production-sized" models, co-distillation with label smoothing increases agreement without loss of accuracy. furthermore, on noisy data simulating a real-world environment, the improvement is even more drastic. when we account for resource usage along with accuracy, we provide strong evidence that co-distillation provides the sweet spot compared to methods like hard/soft distillation and ensembling. in future work, we plan to explore how other modeling decisions can increase or decrease model churn. in this work, we limited our focus to bert encoders with different number of layers. other questions to explore include whether the choice of pre-training technique affects churn or whether pre-trained encoder-decoders show the same effects. finally, we will examine whether alternative decoding algorithms, such as non-autoregressive approaches @xcite , can reduce churn. iulia turc, ming-wei chang, kenton lee, and kristina toutanova. 2019. well-read students learn better: the impact of student initialization on knowledge distillation. corr, abs/1908.08962. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, volume 30. curran associates, inc. ronald j. williams and david zipser. 1989. a learning algorithm for continually running fully recurrent neural networks. neural computation, 1(2):270-280.