Temporal Generalization for Spoken Language Understanding (2022)

Abstract: Spoken Language Understanding (SLU) models in industry applications are usually trained offline on historic data, but have to perform well on incoming user requests after deployment. Since the application data is not available at training time, this is formally similar to the domain generalization problem, where domains correspond to different temporal segments of the data, and the goal is to build a model that performs well on unseen domains, e.g., upcoming data. In this paper, we explore different strategies for achieving good temporal generalization, including instance weighting, temporal fine-tuning, learning temporal features and building a temporally-invariant model. Our results on data of large-scale SLU systems show that temporal information can be leveraged to improve temporal generalization for SLU models.

## introduction spoken language understanding (slu) models play an important role in voice-controlled devices, such as alexa or google home. two common slu tasks are intent classification (ic) and slot filling (sf). given a user request, ic aims to extract the user's intent, while sf is a sequence labeling task which assigns a slot label to each of the tokens. for example, the user request "play volbeat" should be classified as playmusic by the ic task, while sf should assign the labels o and artist to "play" and "volbeat", respectively. state-of-the-art approaches typically model the two tasks jointly via dnns @xcite . in deployed industry slu systems, new data continuously flows into the system, and the underlying data distributions keep drifting over time. in this paper, we focus on the setting of temporal covariate drift, where the distribution of utterances may change, but the correct label for an utterance or token remains fixed (i.e., no concept drift) @xcite . such data drifts happen, for example, because customer usage patterns change over time, as new movies are being released or new artists and songs become popular. another cause of data drifts are seasonal changes or changes related to (re-)occurring events. for instance, the utterance "will it snow tomorrow" is more likely to appear during the winter than the summer season, and the utterance "put on the christmas lighting" is likely uttered around christmas. to accommodate for temporal distributional changes, industry slu models are typically retrained and redeployed over time; in the following sections, we also refer to this process as model release, and we assume that model releases are executed at fixed time intervals, e.g., once per month. we further assume that for each release, new labeled data become available, which were collected since the previous release, yielding data belonging to different time periods. the common approach to utilize new data is to simply combine them with all previously available data, and subsequently split them into training, validation, and test datasets. we can then build and evaluate a model on these datasets, which we also refer to as offline data in this paper. in industry applications, slu models are subsequently deployed to customers and have to perform well on incoming customer requests, which we also refer to as online data. importantly, aiming to provide the best possible experience for our customers, our main goal is building models which perform well on the online rather than the offline data. since the online data are not available for model building and evaluation, we need to utilize the offline data to build a model which generalizes well to unseen online data from the upcoming time period. in this paper, we study this temporal generalization task assuming that data from several consecutive time periods, i.e., months in our case, are available, and we aim to build a slu model which yields high performance on data from an upcoming time period. we aim to improve performance over the common approach of simply combining all offline data, which ignores the temporal nature of the data and implicitly assumes that data from all time periods are equally useful for model training. instead, relating back to the previous examples, we assume that modeling the temporal nature of the data may be beneficial and that data from certain time periods may be particularly valuable. for instance, data from recent time periods may better reflect upcoming trends, and w.r.t. seasonality patterns, data from the same period in previous years may be particularly valuable. to tackle the task, we explore four directions: i) instance weighting based on our assumptions about the task, ii) temporal finetuning, iii) learning temporal features and iv) building a temporallyinvariant model. we present extensive experiments on real-world slu data of german and portuguese voice-controlled devices. our results indicate that temporal information can be leveraged to improve temporal generalization of slu models. we also show that simple temporal fine-tuning is not very effective and in fact leads to performance drop in certain cases. ## related work to the best of our knowledge, the temporal generalization scenario studied in this paper has not yet been explored for slu in the literature, which may be due to the fact that common academic slu datasets are rather small and do not have a temporal notation. in general, work addressing the temporal nature of slu datasets has been limited. @xcite address temporal data drift by adapting from stale to current data with an adversarial domain adaptation approach, treating the stale and current dataset as source and target domain, respectively. contrasting with our work, they assume the availability of data from the target and they focus on two time periods only. other work has explored short-term temporal information, e.g., the utterance context provided by a couple of prior utterances to resolve ambiguities @xcite . in nlp, previous research has explored the significance of temporal drift for several tasks, such as headline generation @xcite , sentiment analysis @xcite and named entity recognition (ner) @xcite , providing evidence that model performance drops when training data age increases compared to the test data. however, despite this evidence, the vast majority of nlp research does not take the temporal nature of data into account for evaluation @xcite . @xcite show that (downstream task) performance of pre-trained language models suffers when performance is measured on future data. since (re-)training of larger language models is costly, the authors propose to update model parameters by executing few steps of gradient decent on new data. other approaches to mitigate temporal drift include predictive feature selection for sentiment analysis @xcite , and selecting data based on frequent n-grams for ner @xcite . the most similar to our work is the study conducted by rijhwani and preotiuc-pietro (2020) who tackle temporal drift for a smallscale ner task, i.e., including only 3 named entities. they consider data from several consecutive years and aim to build a model which performs well on data of the following year, focusing -in line with the previously described work -on the effects of data receny. the best performance is achieved by using instance weighting of recent data and temporal finetuning for a bi-lstm-crf with flair and glove embeddings, respectively. by contrast, we study temporal generalization in the context of a large-scale slu production system covering a large numbers of labels. we focus on smaller time periods, i.e., spanning one month instead of a year, and we consider cyclic/seasonal changes in addition to data recency. for this purpose, we include methods which have not yet been explored in temporal generalization tasks, such as building models that leverage temporally-invariant representations. ## method given labeled slu data from several consecutive time periods, our goal is to build a model which generalizes well to unseen data from an upcoming time period. in the following, we first provide a formal definition for our tasks and subsequently present the modeling approaches. ## results and discussion the results on the "online" test data for using either all offline validation data val all or recent validation data val rec to select the best model are shown in rq 2: what is the impact of seasonality and reoccurring events vs recency effects? previous work in nlp on temporal adaptation and generalization has focused on larger time periods and the effects of data recency, showing strong performance for instance weighting of recent data and temporal finetuning on a small-scale ner task (rijhwani and preotiuc-pietro, 2020). by contrast, our domain datasets cover smaller time periods, with different seasonal effects and re-occruring patterns. on our task, temporal finetuning gives mixed results, with decreasing performance in several cases. we assume that the models may overfit to the recent data, and some previously acquired knowledge related to older time periods might have been forgotten. however, unlike in the ner task which included only the three named entities per, loc, and org and in which there might be mostly gradual temporal trends, in our task seasonal drifts exist, potentially making certain older knowledge more relevant. by finetuning on recent data, the models may lose too much relevant seasonal knowledge, harming performance for domains with changes related to seasons or re-occurring events. instance weighting of recent data gives consistent improvements, which is in line with previous findings, while instance weighting of the same time period gives mixed results, i.e., it helps in some cases, but decreases performance in others. to some extent this may be due to domain-specific differences. however, an issue might also be that there can be conflicting seasonal and gradual drifts. in particular, weighting is performed at the dataset level and a dataset from a year ago might include relevant seasonal data instances, but also less useful data instances such as data related to older (already outdated) trends. the negative effects can be mitigated to some extent by selecting the best model on recent validation data, which yields consistent gains in performance across all domains and languages. future work may explore how to disentangle these effects, and in temporal da scenarios one may select utterances based on the similarity to the target. however, in our scenario which does not assume the availability of target period data, modeling seasonal and re-occurring patterns indirectly via an auxiliary month prediction task appears to be a better choice in most cases, yielding consistent -and in most cases higher -gains. how to create a validation dataset without having access to target data? for half of the domain-language pairings, performance is improved by using a recent offline validation dataset. the choice of the best validation dataset may be both method-specific and domain-specific, as drifts differ across domains. interestingly, performance of the concat baseline model, which ignores the temporal nature of the data, is consistently improved across domains and languages when using recent validation data. this shows that model performance can also be improved by taking the temporal nature of the data into account for creating a validation dataset in-stead of model building. should we aim for a temporally aware or invariant model? in this paper, we have explored both building temporally-aware and temporallyinvariant models, since there may be different kinds of temporal features and artifacts related to our data, out of which we may want to leverage some, but abstract away from others. our results show consistent gains for temporally-aware models using an auxiliary month classifier as well as gains in all but one case for temporally-invariant models, with temporally-aware models giving better performance in most, but not all cases. thus, both directions appear to be generally promising. future work may explore different approaches to learning temporally aware or invariant models, for instance, by exploring others dg approaches in the latter case. one potentially interesting direction is to learn disentangled representations that separate temporally-invariant and seasonal components. ## conclusion we studied a temporal generalization task in which we used offline data of time periods spanning one month each to build a model that performs well on future online data. we explored four directions to leverage temporal information which are rather easy to apply in production, i.e., i) instance weighting based on our assumptions about the task, ii) temporal finetuning, iii) learning temporal features and iv) building a temporally-invariant model. our results on real-world slu data covering two languages and three domains each show that temporal information can be leveraged to improve temporal generalization for slu. while several of the explored methods provide consistent gains across all domain-language pairings, the best methods differ, as different domain datasets have different gradual and seasonal drifts. moreover, our results indicate that methods, such as temporal finetuning, which have been previously shown to provide strong performance on small-scale academic tasks with longer time periods and mostly gradual temporal drifts, do not necessarily yield the best performance in our large-scale slu task including seasonality patterns.