Towards Need-Based Spoken Language Understanding Model Updates: What Have We Learned? (2022)

Abstract: In productionized machine learning systems, online model performance is known to deteriorate over time when there is a distributional drift between offline training and online application data. As a remedy, models are typically retrained at fixed time intervals, implying high computational and manual costs. This work aims at decreasing such costs in productionized, large-scale Spoken Language Understanding systems. In particular, we develop a need-based re-training strategy guided by an efficient drift detector and discuss the arising challenges including system complexity, overlapping model releases, observation limitation and the absence of annotated resources at runtime. We present empirical results on historical data and confirm the utility of our design decisions via an online A/B experiment.

## introduction traditionally, a spoken language understanding (slu) system like google assistant, siri or alexa, is a cascade of an automatic speech recognition (asr) component converting speech into text followed by a natural language understanding (nlu) component that interprets the meaning of the text through domain classification (dc), intent classification (ic) and slot filling (sf). once deployed to customers, the machine learning (ml) models implemented for dc, ic and sf may experience distributional drifts between offline training and online application data over time which leads to serious performance degrades. this is known as a model drift phenomenon. in most real-world cases, model drift can be avoided by retraining the ml models with regular cadence. however, this strategy often entails a significant amount of human, computational and environmental costs in each release cycle, especially for large systems. hence, it should be optimized with an intelligent decision * these authors contributed equally to this work making mechanism that is able to automatically predict if a particular model needs to be updated or can be left in place as is for another release cycle. given a deployed ml model, drift detection is a task to identify model drifts when the model is applied on a new data set, and therefore can be used to guide decisions on when to retrain the model. however, academic work in this field often makes simplifying assumptions that render the problem more manageable but do not hold in the industrial practice, e.g., that there is exactly one model that needs to be kept or updated, and that it is possible to create non-overlapping detection and reference data windows, which are aligned with a model release cycle. this makes productionizing drift detection nontrivial. this paper describes our effort to develop a drift detector to produce decisions whether to update the nlu models in the next release cycle for a productionized slu system. first, we generally discuss the challenges that may arise when applying drift detection on a large-scale slu production system: i) the complexity when the slu system is supporting multiple domains with several ml models and architectures; ii) the possible overlapping model releases in production; iii) the limited number of observations for each individual ml model; iv) the absence of annotated slu data at runtime. moreover, for each challenge, we consider the necessary system design decisions and possible solutions that are needed to apply drift detection in practice. finally, we describe our offline and online experiments on real-world slu data to confirm the utility of our design decisions. while drift detection is considered as one of the stages in modern ai workflows, corresponding work in nlp and slu has been limited. recently, @xcite have proposed a regression model to detect temporal performance drop in slu. the authors evaluated their approach via small-scale simulated release cycles for a joint ic+sf model in isolation, thus abstracting away from the complexity of a production slu system. they built one regression model per domain, assuming the availability of a large number of previous model releases for training the regression model, which is unrealistic for many real-world scenarios. ## nlu drift detection definition in this work, we consider multi-domain nlu as one part of a more complex production slu system, leaving out asr and other components. the main nlu tasks include dc, ic and sf, and there are different ways to approach them. usually, pipelined systems are constructed with dc being applied as the first step to determine the domain for a given utterance. subsequently, the utterance is fed into the corresponding domain-specific ic+sf model that jointly detects the intent and extracts semantic slots from the utterance. for instance, an asr transcribed utterance "play hello by adele" can be parsed into {domain: music, intent: "play", song name (slot): "hello", artist (slot): "adele"}. to simplify the problem, we focus on building a drift detector to decide on ic+sf model updates. in our experiments, we make an assumption that dc models also face distributional drifts when their corresponding ic+sf models do, since their data ages are usually similar. given a trained ic+sf model m, a data reference window, w ref , containing the testing instances considered belonging to the data distribution at the time m was developed offline, and a detection window, w detect , which contains the testing utterances representing the data distribution when the model update decision needs to be made. then, we define that m has suffered a drift if the error rate on the detection window exceeds the error rate on the reference window by a certain threshold: a drift detector should be able to identify automatically whether m has suffered a drift or not. ## nlu system and challenges in this section, we describe the considered nlu system, and discuss the potential challenges arising when applying drift detection on such a system. ## offline experiment we evaluate the proposed feature set and the selected learning algorithms for gaz and non-gaz architectures using historic data. ## online a/b experiment we conducted an online a/b experiment to evaluate our drift detection approach in the context of a complex large-scale production slu system (i.e., including several components in addition to nlu, such as asr). we picked a point in time in which all domain nlu models were scheduled for re-training and re-deployment. the production system with all models updated served as the a model. our b system comprises exactly the same components, except that we re-trained only a subset of the domain nlu models, according to the decision of our drift detector. we acknowledge that it would be desirable to include another baseline model with none of the domain models updated into the comparison. yet, simply leaving all domain models in place as they are increases the risk of model drift, thus potentially increasing friction for the customers being exposed to such a system. we generated the features for each domain ic+sf model and applied our mlr drift detection model with feature selection and drift threshold α = 0.0 (see section 6). a low threshold for the semer drop also reduces the risk of negatively impacting customers. if the predicted class was no-drop, the ic+sf model for the domain was left in place, otherwise it was retrained on current traffic. roughly half of the domain models were re-trained vs left in place. in this work, we assumed that a dc model faces model drift simultaneously with its corresponding ic+sf model as their data ages were usually similar. therefore, the dc model also followed the same retraining decision as its corresponding ic+sf unless there was a special circumstance like the appearance of a new domain in the next release. both, the a and the b systems were deployed, and we ran our experiment over 10 days. by applying a need-based approach to domain model re-training in the b system, our main goal is to decrease costs for model re-training compared to the a system (in which all domain models are re-trained), while keeping model performance the same. to measure the impact on performance, we have monitored online friction metrics that reflect the overall end-to-end system performance (as opposed to the nlu only component in the offline experiments). unlike in the offline experiments, all metrics were computed automatically and concern overall system performance rather than nlu in isolation. we compared the a and b system's performances using the selected online metrics, indicating that there was no significant increase recorded in any of the error metrics for the b system compared to the a system. thus, we conclude that even though we left around half of the domain models in place according to the decision of our drift detector, there was no negative effect on overall system performance. due to re-training fewer models, we observed a decrease in costs for expensive gpu instances of 46.4% for training ic+sf models (no gpus were used for detector building). ## conclusions we presented an efficient drift detection approach to guide ic+sf model retraining decisions, which requires only unlabeled data during the application phase in a multi-domain large-scale slu system. we discussed the challenges that we faced while developing the approach and the corresponding design decisions to address them. we presented experimental results using historical data and we evaluated our approach via both offline and online experiments with a large-scale slu system, confirming the utility of our design decisions.