Iterative Stratified Testing and Measurement for Automated Model Updates (2022)

Abstract: Automating updates to machine learning systems is an important but understudied challenge in AutoML. The high model variance of many cutting-edge deep learning architectures means that retraining a model provides no guarantee of accurate inference on all sample types. To address this concern, we present Automated Data-Shape Stratified Model Updates (ADSMU), a novel framework that relies on iterative model building coupled with data-shape stratified model testing and improvement. Using ADSMU, we observed a 26% (relative) improvement in accuracy for new model use cases on a large-scale NLU system, compared to a naive (manually) retrained baseline and current cutting-edge methods.

## introduction research in automated machine learning, including auto ml and auto ai, has primarily focused on automating the steps between data preparation and creating a purpose-relevant machine learning model, which can include choosing model architecture, tuning hyperparameters, and data preprocessing @xcite @xcite . however, in production machine learning systems, there is an equally important task of automating the maintenance and updates of existing models currently in use. this task embodies two essential requirements: (i) sufficiently covering new use cases; and (ii) not regressing already working use cases. in academic literature, there exist a range of methods to meet these requirements, including bayesian approaches @xcite , mixture of experts models @xcite , finetuning @xcite @xcite , and * edekeyse@amazon.com † kmardpl@amazon.com training data selection @xcite @xcite . however, production systems present unique challenges that are not addressed by these methods. first, in deep learning models, high model variance can result in changing interpretations across model builds @xcite @xcite . because of this, simply retraining a model with new training data can result in uneven coverage across a new use case, i.e. failing performance requirements for a subset and, thus not providing sufficient coverage for the new use case. furthermore, retraining the model for the new use case can shift formerly correct interpretations, i.e. failing performance requirements for previously existing use cases. a standard solution to this challenge in the industry is repeated manual retraining and failure analysis, requiring weeks of modeler work for a simple model update. second, experimentation in production systems is further hampered by the costs of changing model architectures. production systems can be in use by hundreds of millions of customers and are supported by complex engineering systems. changes to these systems require extensive testing of both inference and engineering performance. because of this, introducing architectural solutions to challenges of model variance or data diversity can be prohibitively costly. to address these challenges, we present automated data-shape stratified model updates (adsmu). adsmu builds and iteratively improves models to optimize toward data-shape stratified metrics. it can be applied on top of pre-existing model architectures in any production system, making it compatible with both legacy and cutting-edge model architectures and methods for model updating. through stratified optimization, adsmu ensures that models cover a diverse range of the data space without requiring manual intervention. auto-mated model updates using this framework resulted in a 26% (relative) decrease in new user experiences error rate compared to alternate model update approaches (refer to in the following sections, we discuss related work, examine the details of adsmu and how it can be applied to automated model updates, and discuss the limitations of adsmu. research on optimized model updates, separate from automl, has focused on two primary fields: first, model architectures, such as finetuning @xcite @xcite or bayesian approaches @xcite , and second, training data selection, which encompasses work on submodular optimization @xcite @xcite , cross-entropy comparison (moore and lewis, 2010), selection by proxy @xcite , and active learning @xcite @xcite . many of these methods are premised on developing rich training data that performs well across feature spaces. however, despite this focus on data diversity, performance is tested, measured, and improved based on average performance across development test sets. this inherently biases models to perform well on the majority data type, allowing more challenging or rarer test cases (corresponding to newly launched user experiences) to fall through the cracks. ## automated data-shape stratified model updating adsmu provides a framework for (1) automating data stratification, (2) using this stratification for model measurement, and (3) boosting performance in failing stratification groups using model iteration. we discuss an application of this method using an iterative approach to model retraining, which is compatible with any model architecture. ## conclusions and next steps adsmu presents a framework for using stratified testing and measurement to iteratively improve models in the setting of automated model updates. this highlights the need that focuses on automating model creation while largely ignoring the challenges associated with automated model refresh processes @xcite @xcite . one key extension of this approach is applications that remove the need for model iteration and improve performance within stratified sub-groups in a single model build. for instance, optimizing on stratified loss functions might help ensure that models cover diverse use cases in a more consistent manner. more simplistically, a model could have training data added incrementally in between epochs based on weaker performing stratification groups.