Local-to-global learning for iterative training of production SLU models on new features (2022)

Abstract: In production SLU systems, new training data becomes available with time so that ML models need to be updated on a regular basis. Specifically, releasing new features adds new classes of data while the old data remains constant. However, retraining the full model each time from scratch is computationally expensive. To address this problem, we propose to consider production releases from the curriculum learning perspective and to adapt the local-to-global learning (LGL) schedule (Cheng et. al, 2019) for a statistical model that starts with fewer output classes and adds more classes with each iteration. We report experiments for the tasks of intent classification and slot filling in the context of a production voice-assistant. First, we apply the original LGL schedule on our data and then adapt LGL to the production setting where the full data is not available at initial training iterations. We demonstrate that our method improves model error rates by 7.3% and saves up to 25% training time for individual iterations.

## introduction in many real-world nlp systems with ml models, new data becomes available with time and there is a need to refresh the model @xcite . in some cases it is a passive flow, when new data arrives due to the properties of the application (e.g. daily search queries) or an active act of collecting new data to be incorporated into the system (e.g. a new feature). in this paper, we regard the use case of an active extension of data to incorporate a new customer-facing feature into a production nlp model. we consider a spoken language understanding (slu) model that is used to interpret user requests in a commercial task-oriented voiceassistant. the model is a joint intent classification (ic) and slot filling (sf) architecture that is used to process utterances in a single domain. 1 we select one data-rich domain for our experiments, music, and construct a scenario when an existing ic+sf model is extended with a new user-facing feature that comprises a set of intents and slots to be now recognized by the model. it is conventional to re-train the original ml model on a combination of the old training data and the additional data for the new feature, starting from the same randomly initialized or pre-trained architecture as the previous time. the practitioners tend to use pre-trained models (language modeling and transfer learning are widely used here) to improve the generalization performance of the model. it seems logical also to re-use the previous iteration of the model trained on the old data in the previous model release to warm-start the next iteration. this could result in a reduced training time and a smaller computational and environmental footprint of the model updates in a scenario where new features are added regularly. yet, in practice it is usually considered 'safer' to start training from scratch or from the same general-purpose pre-trained model every time, the main concern being that repeated warm-starting would lead to overfitting and poorer generalization @xcite . re-training the same model architecture on nearly the same data with minimal changes, but extending the output space with a new class is a unique problem for industry applications. many previous works on continual learning have focused on learning from a continuous stream of data @xcite or on an incremental learning of new tasks @xcite and languages @xcite . @xcite discuss a single-task continual learning setup and simulated a passive data extension scenario where new examples are coming in for all output classes on a public dataset. similarly, ash and adams (2020) evaluate a batch-learning setup, where each model iteration is warm-started from the previous step and the whole training data is always available, while some new data is added across all output classes in each batch. in our scenario, we consider active data extension for new features and we do not restrict the access to the old training data. we rely on an offline training paradigm, where each model release is trained on the latest batch of data until convergence. if each release adds data for new features with the old data being constant, we can view a sequence of model releases as a subclass of curriculum learning, a machine learning paradigm that aims to arrange training data into a meaningful order to improve model training. in our scenario, the data is arranged by feature. @xcite describe a local-to-global learning (lgl) schedule for a statistical model that starts with fewer output classes and adds more classes with each iteration. we build upon their results in our work, but remove their assumption that the whole data is available in the first iteration. in this paper, we repeatedly apply a warm-start for training a set of subsequent ic+sf model releases, each one being extended with a new set of features. we define a single feature as a set of new intents and slots to be recognized by the model that are added to the output space. we focus on a real-life production setting and report results on a dataset sampled from a commercial german voice assistant. 2 as our main contribution, we show that warm-start is an effective strategy to reduce training time for later model releases and improve overall model performance in a scenario when the added training data pertains to new features only. ## conclusion we applied lgl to the tasks of intent classification and slot filling in the context of slu and studied the impact of lgl on intent classification error rate and semantic error rate. we conducted the experiments using different class selection strategies and showed that lgl improves intent classification performance for slu by -9.1% relative icer, without requiring any new training data or modified model architecture. in addition, we adapted original lgl setup to slu production scenario when new features are gradually added within fixed release cycle, and showed that it is able to improve model accuracy by up to -7.3% relative semer while reducing average training time by up to 25% for individual iterations. lgl application to feature expansion problem, apply it to other domains and investigate the impact of batch size on the model performance. in addition, we would track the impact of lgl training on model's generalization performance and computational cost over time. yi tay, shuohang wang, anh tuan luu, jie fu, minh c. phan, xingdi yuan, jinfeng rao, siu cheung hui, and aston zhang. 2019. simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4922-4931, florence, italy. association for computational linguistics. h. weld, x. huang, s. long, j. poon, and s. c. han. 2021. a survey of joint intent detection and slotfilling models in natural language understanding. xiaodong zhang and houfeng wang. 2016. a joint model of intent determination and slot filling for spoken language understanding. in proceedings of the twenty-fifth ijcai, page 2993-2999, new york, ny, usa.