Compute-Efficient Churn Reduction for Conversational Agents (2023)

Abstract: Model churn occurs when re-training a model yields different predictions despite using the same data and hyper-parameters. Churn reduction is crucial for industry conversational systems where users expect consistent results for the same queries. In this setting, compute resources are often limited due to latency requirements during serving and overall time constraints during re-training. To address this issue, we propose a compute-efficient method that mitigates churn without requiring extra resources for training or inference. Our approach involves a lightweight data pre-processing step that pairs semantic parses based on their “function call signature” and encourages similarity through an additional loss based on Jensen-Shannon Divergence. We validate the effectiveness of our method in three scenarios: academic (+3.93 percent improvement on average in a churn reduction metric), simulated noisy data (+8.09), and industry (+5.28) settings.

## introduction many industry natural language processing systems rely on deep learning models. however, these models can be brittle, leading to different predictions on the same queries after re-training. this is known as model churn @xcite and can be caused by non-determinism in training due to data ordering and initialization. for conversational agents like google assistant or amazon alexa, this problem can erode user trust if the system behaves unexpectedly over time. common approaches to address this issue include ensembling @xcite , distillation @xcite @xcite , or co-distillation @xcite . all these techniques involve training or serving one or more models in addition to the primary model, and may be unfeasible in @xcite with the same function call signature. intuitively, the representations of the spans for these slots should be close in vector space and the model should be consistent about predicting the same non-span tokens for both queries. our approach relies on the observation that similar training examples should have similar representations and predictions. @xcite demonstrated that using nearest neighbors to obtain soft labels for discriminative tasks can reduce churn. however, in structured prediction tasks like conversational semantic parsing, smoothing labels in this way is challenging. to address this, we propose pairing examples based on their function call signature, i.e. the nonterminal nodes in a parse tree. our contributions are as follows: ## related work model churn occurs when multiple re-training runs with the same model architecture yield different predictions, even when trained on the same data @xcite . this problem has traditionally been studied for classification tasks @xcite @xcite although we recently explored churn reduction for structured prediction @xcite . techniques such as ensembling and distillation are known to improve model calibration @xcite and reduce model churn @xcite . however, these techniques may not always be practical due to the computational cost -ensembling requires k trained models for inference. distillation, the process of training a "student" model based on the predictions of a "teacher" @xcite , traditionally requires only a single model for inference but an ensemble is often used as the teacher @xcite , meaning that the cost of training is the same. co-distillation @xcite was developed as an alternative where k peer models are trained in parallel and while training time may be less than kx this approach still requires a kx increase in compute resources. conversational semantic parsing is the task of converting a user query into an executable form that can be understood by a conversational assistant. in our previous work @xcite , we studied the problem of model churn for this structured prediction task and reported that codistillation reduces churn on the top @xcite , topv2 @xcite , mtop @xcite , and snips @xcite datasets for conversational agents. this work is a direct extension of @xcite -we show that we can reduce model churn using a single trained model, unlike co-distillation which requires training a peer model in parallel. our approach is similar to the work of @xcite to smooth labels using nearest neighbors. however, their approach does not directly extend to structured prediction tasks. instead, we create neighbors according to their "function call signature," where the idea is that a trained model should perform the same on training examples with the same shallow structure. ## methods to enhance robustness against churn, we propose two key contributions. first, we balance training data by pairing examples based on their "function call signature," specifically the non-terminal nodes in the semantic parse (e.g. ## qualitative analysis we conducted an analysis on massive comparing ls + pairwise/jsd to ls and noticed better results for calendar/event/date/time intents. 7 additionally, it showed improved ability to learn entity identification. given the query "let me know the recipe for preparing pasta," the baseline may predict "[in:cooking-recipe [sl:ingredient pasta ]]" whereas the target is [in:cooking-recipe [sl:foodtype pasta ]]. we also observed "overtriggering" (i.e. predicting extra slots for argument-less intents), likely due to encouraging span alignment. for instance, the query "is there a chance the hurricane will make landfall" indicates interest in weather conditions rather than a weather event. the correct semantic parse for such queries is [in:unsupported-weather. however, due to the term "hurricane," the model mistakenly predicts [in:get-weather [sl:weatherattribute hurricane ]]. 8 furthermore, the slot "[sl:date-time" was often associated with the intent "[in:update-reminder-date-time" rather than as discussed in section 6, encouraging span alignment may result in the model predicting extraneous slots. on the datasets we used for our experiments, the reduction in churn did not result in a decrease in exact match accuracy. in top/topv2 and the internal dataset, there are relatively complex function call signatures with many slots and nested intents. however, given a dataset with shallow trees where many examples consist of only a single intent and no slot, there would be no benefit to span alignment and our approach could therefore be detrimental to both churn reduction and accuracy. one possible cause for the limited gains on massive, other than the dataset size, is that there are no nested intents and relatively simple function call signatures. thus, practitioners should consider the structure of the semantic parse trees in the training data when considering whether to use this approach. ## conclusion: practical considerations and recommendation when deciding on the best approach for churn reduction, there are three factors to consider: 1) training data size 2) training data quality and 3) compute resource constraints. given a sufficiently large training dataset, we recommend the ls + pairwise/jsd approach regardless of the amount of noise in the data. ls + pairwise/jsd outperformed co-distillation alone on all topv2 settings and on the internal dataset. when accounting for resource usage, co-distillation requires 2x resources for training an extra model. pairwise/jsd requires only an additional preprocessing step over ls/cd and paired data can be cached and updated easily. if there are no resource constraints, cd + pairwise/jsd can achieve better or comparable results over ls + pairwise/jsd. overall, our approach is especially effective on our internal data due to the large available quantity of un-annotated system output. in a noisy production setting such as ours, we recommend combining our approach with label smoothing or codistillation, depending on how much data is available and what constraints exist on training time. in the one of the sequences in the pair. this setting was designed to force the model to generate dissimilar embeddings for the same span appearing in different contexts. however, we did not obtain additional improvement with this approach. the results can be seen in hyperparameter/ dataset / setting value learning rate / -/ without cd 5e-6 learning rate / -/ cd 1e-5 batch size / -/ without cd 32 batch size / -/ cd 128 train steps / top (gupta et al., 2018) / without cd 500k train steps / topv2 (chen et al., 2020) / without cd 800k train steps / massive (fitzgerald et al., 2022) / without cd 800k train steps / -/ cd 272k num decoder heads / -/ -8 num decoder layers / -/ -4 max decode length / -/ -51 span loss weight(λ) / -/ -1 cd loss weight / -/ cd 1 backoff smoothing(α) / top (gupta et al., 2018) / -0.25 backoff smoothing(α) / topv2 (chen et al., 2020) / -0.25 backoff smoothing(α) / massive (fitzgerald et al., 2022) / -0.5