Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment
Abstract
The goal of sequential event prediction is to estimate the next event based on a sequence of historical events, with applications to sequential recommendation, user behavior analysis and clinical treatment.
In practice, the next-event prediction models are trained with sequential data collected at one time and need to generalize to newly arrived sequences in remote future, which requires models to handle temporal distribution shift from training to testing.
In this paper, we ﬁrst take a data-generating perspective to reveal a negative result that existing approaches with maximum likelihood estimation would fail for distribution shift due to the latent context confounder, i.e., the common cause for the historical events and the next event.
Then we devise a new learning objective based on backdoor adjustment and further harness variational inference to make it tractable for sequence learning problems.
On top of that, we propose a framework with hierarchical branching structures for learning context-speciﬁc representations.
Comprehensive experiments on diverse tasks (e.g., sequential recommendation) demonstrate the effectiveness, applicability and scalability of our method with various off-the-shelf models as backbones.
Introduction
Real-world problem scenarios are ﬂooded with sequential event data consisting of chronologically arrived events which reﬂect certain behaviors, activities or responses.
A typical example is user activity prediction [56; 12; 62; 20; 44; 64] that aims to harness user’s recent activities to estimate future ones, which could help downstream target advertisement in online platforms (e g., e-commerce or social networks).
Predicting future events also plays a central role in some real situations such as clinical treatment [3] for promoting social welfare.
A common nature of (sequential) event prediction lies in the different time intervals within which training and testing data are generated.
Namely, models trained with data collected at one time are supposed to predict next event in the future [59; 36; 41] where the underlying data-generating distributions may have gone through variation due to environmental changes.
However, most existing approaches [20; 14; 37; 24; 44; 39; 19; 18] overlook this issue in both problem formulation and empirical evaluation, which may leave the real problems under-resolved with model mis-speciﬁcation, and result in over-estimation for model performance on real data.
As a concrete example in sequential recommenders [11], the data-generating distribution for user clicking behaviors over items is highly dependent on user preferences that are normally correlated
Winter ... ...
Past Events
Future Event
Spurious Correlation ... ...
Past Events
Prediction
Spurious Correlation ... ...
Past Events
Future Event
Causality ... ...
Past Events
Prediction
Causality
Time Line
Train (Offline) Test (Online)
T i m e
G a p
Summer
T i m e
G a p
P r i o r
A r t
O u r s v .
s .
with some external factors, like the dynamic fashion trends in different years [13] or seasonal inﬂuences on item popularity [43].
These highly time-sensitive external factors would induce distinct user preferences leading to different behavioral data as time goes by.
This issue could also partially explain the common phenomenon of model performance drop after adaptation from ofﬂine to online environments [41; 36; 22].
Handling distribution shift in sequential event data poses several non-trivial challenges.
First , the temporal shift requires models’ capability for out-of-distribution (OoD) generalization [30], i.e., extrapolating from training environments to new unseen environments in remote future.
Prior art that focus on model learning and evaluation over in-distribution data may yield sub-optimal results on OoD testing instances.
Second , as mentioned before, there exist external factors that impact the generation of events.
These external factors, which we term as contexts , are unobserved in practice.
To eliminate their effects, one may also need the latent distributions that characterize how the contexts affect events generation.
Unfortunately, such information is often inaccessible due to constrained data collection, which requires the models to learn from pure observed sequences.
1.1 Our Contributions
To resolve these difﬁculties, in this paper, we adopt a generative perspective to investigate the temporal distribution shift problem and propose a new variational context adjustment approach with instantiations to solve the issue for sequential event prediction.
A generative perspective on temporal distribution shift.
We use proof-of-concept Structural
Causal Models (SCMs) to characterize the dependency among contexts, historical sequences and labels (i.e., next event type) in terms of both data generation and model prediction.
We show that contexts essentially act as a confounder, which leads the model to leverage spurious correlations and fail to generalize to data from new distributions.
See Fig.
1 for a toy example to illustrate the issue.
Variational context adjustment.
We propose a variational context adjustment approach to resolve the issue.
The resulting new learning objective has two-fold effects: 1) helping to uncover the underlying relations between latent contexts and sequences in a data-driven way, and 2) canceling out the confounding effect of contexts to facilitate the model to explore the true causality of interest (as illustrated in Fig. 1).
We also propose a mixture of variational posteriors to approximate context prior via randomly simulating pseudo input sequences.
Instantiation by a ﬂexible framework.
We propose a framework named as CaseQ to instantiate ingredients in the objective, which could combine with most off-the-shelf sequence backbone models.
To accommodate temporal patterns under different environments, we devise a novel hierarchical branching structure for learning context-speciﬁc representations of sequences.
It could dynamically evolves its architecture to adapt for variable contexts.
Empirical results.
We carry out comprehensive experiments on three sequential event prediction tasks with valuation protocols designed for testing model performance under temporal distribution shift.
Speciﬁcally, when we enlarge time gap between training and testing data, CaseQ can alleviate performance drop by 47 .
77% w.r.t.
Normalized Discounted Cumulative Gain (NDCG) and 35 .
73% w.r.t.
Hit Ratio (HR) for sequential recommendation, which shows its robustness against temporal distribution shift.
1.2 Related Works
Sequential Event Prediction [25] aims to predict the next event given a historical event sequence with applications including item recommendation in online commercial platforms, user behavior prediction in social networks, and symptom prediction for clinical treatment.
Early works [39; 19; 18] rely on Markov chains and Bayesian networks to learn event associations.
Later, deep learning based methods are proposed to capture non-linear temporal dynamics using CNN [45], RNN [20; 27; 54], attention models [24; 50; 51], etc.
These models are usually trained ofﬂine using event sequences collected during a certain period of time.
Due to temporal distribution shifts, they may not generalize well to online environments [41; 36; 22].
Our work develops a principled approach for enhancing sequential event prediction with better robustness against temporal distribution shifts via pursing causality behind data.
Out-of-Distribution Generalization [30; 4; 31; 63] deals with distinct training and testing distributions.
It is a common yet challenging problem, and has received increasing attention due to its signiﬁcance [26; 42; 52; 55].
For event prediction, distribution shift inherently exists because of different time intervals from which training and testing data are generated.
Most existing sequence learning models assume that data are independent and identically distributed [20; 14; 37].
Despite various approaches of OoD generalization designed for machine learning problems in other research areas (e.g., vision and texts), it still remains under-explored for sequential event prediction and sequential recommender systems.
Causal Inference [33; 35] is a fundamental way to identity causal relations among variables, and pursuit stable and robust learning and inference.
It has received wide attention and been applied in various domains e.g., computer vision [57; 28], natural language processing [40; 32] and recommender systems [58; 1].
Some existing causal frameworks for user behavior modeling [38; 16; 60; 61; 17] aim to extract causal relations based on observed or predeﬁned patterns, yet they often require domain knowledge or side information for guidance and also do not consider temporal distribution shift.
[1; 2; 6; 48] adopt counterfactual learning to overcome the effect of an ad-doc bias in recommendation task (e.g., exposure bias, popularity bias) or mitigate clickbait issue, whereas they do not focus on modeling sequential events and are still based on MLE as learning objectives.
Problem and Model Formulation
We denote X
= { 1 , 2 , · · · , M } as the space of event types, and assume each event is assigned with an event type x i 2 X .
Events occurred in chronological order consist of a sequence S
= { x 1 , x 2 , · · · , x |S| } .
As mentioned before, the data distribution is normally affected by time-dependent external factors, i.e., context c .
We use S , Y , ˆ Y and C to denote the random variables of historical event sequence S , ground-truth next event y , predicted next event ˆ y and context c , respectively.
The data distribution can be characterized as P ( S, Y | C ) =
P ( S | C ) P
( Y | S, C ) .
Problem Formulation.
Given training data { ( S i , y i ) } Ni =1 generated from data distributions with ( S i , y i )
⇠ P ( S, Y | C = c ( i ) tr ) , where c ( i ) tr denotes the speciﬁc context when the i -th training sample is generated, we are to learn a prediction model
ˆ y i =
f ( S i ; ✓ ) that can generalize to testing data { ( S j , y j ) } N 0 j =1 from new distributions with ( S j , y j ) ⇠ P ( S, Y | C
= c ( j ) te ) , where c ( j ) te denotes the speciﬁc context when the j -th testing sample is generated.
The distribution shift stems from different contexts that change over time, which we call temporal distribution shift in this paper.
2.1 Understanding the Limitations of Maximum Likelihood Estimation
Most of existing approaches target maximizing the likelihood P ✓ ( y |S ) as the objective for optimization.
Here we use P ✓ ( · ) to denote the distribution induced by prediction model f ✓ .
Based on the deﬁnitions, we can build two Structural Causal Models (SCMs) that interpret the causal relations among 1) C , S , Y (given by data-generating process) and 2) C , S and ˆ Y (given by model learning), as shown in Fig. 2(a) and (b).
For Fig. 2(a), the three causal relations are given by the deﬁnitions for data generation, i.e., P ( S, Y | C ) =
P ( S | C ) P
( Y | S, C ) .
We next illustrate the rationales behind the other two causal relations S ! ˆ Y and C ! ˆ Y . [Figure]
S ! ˆ Y :
This relation is induced by the prediction model ˆ y =
f
( S ; ✓ ) that takes a historical event sequence S as input and outputs the prediction for next event ˆ y .
The relation from S to ˆ y is deterministic given ﬁxed model parameters ✓ .
C ! ˆ Y :
This relation is implicitly embodied in the learning process that optimizes the model parameters with a given dataset collected at one time.
By our deﬁnition, the training dataset is generated from a latent distribution affected by context c tr and the MLE algorithm yields
[Equation]
where l ( · , · ) denotes a certain loss function, e.g., cross-entropy.
This indicates that the learned model parameters ✓ ⇤ is dependent on the distribution of c tr .
Also, due to the fact ˆ y = f
( S ; ✓ ) , we conclude the proof for relation from C to ˆ Y .
In short, intuitive explanations for such a causal relation lie in two facts: 1) C affects the generation of data used for model training, and 2) ˆ Y is the output of the trained model given input sequence S .
Confounding Effect of C .
The key observation is that C acts as the confounder in both Fig. 2(a) and (b), which could play a crucial role leading to undesirable testing performance of MLE-based approaches once the distribution is shifted.
As implied by the causal relations in Fig.
2(a), there exists partial information in S that is predictive for Y yet highly sensitive to C , a usually fast-changing variable in real-world scenarios.
As a result, the correlation between S and Y in previous contexts may become spurious in future ones.
This also explains the failure of MLE-based models for generalizing to data from new distributions, according to the similar causal pattern in Fig. 2(b).
We can reuse the toy example in Fig.
1 as an intuitive interpretation for the failure.
The ‘summer’ season (a context) acts as a confounder that correlates buying ‘ice cream’ (a historical event) and buying ‘T-shirt’ (a label), between which exists no obvious causal relation.
However, the model would ‘memorize’ their correlation and tend to improperly recommend ‘T-shirt’ in the ‘winter’ season (new context).
Whereas in fact the user purchases ice cream in winter most possibly because he/she is a dessert lover, in which case recommending other desserts would be a better decision.
Intervention.
To address the confounding effect of C and endow the model with robustness to temporal distribution shift, we propose to target model learning with P ✓ ( Y | do ( S )) instead of the conventional P ✓ ( Y | S ) .
As shown in Fig. 5(c), the do -operator cuts off the arrow (i.e., causal relation) coming from C to S , which essentially simulates an ideal data-generating process where sequences are generated independently from contexts.
This operation blocks the backdoor path S
C ! ˆ Y that spuriously correlates S and Y , and enables the model to learn the desired causal relation S ! Y which is invariant to environmental change.
2.2 Variational Context Adjustment
An ideal way to compute P ✓ ( Y | do ( S )) is to carry out randomized controlled trial (RCT)
[34] by recollecting data from a prohibitively large quantity of random samples under any possible context, which is infeasible since we could neither control the environment nor collect data in the future.
Fortunately, there exists a statistical estimation of P ✓ ( Y | do ( S )) by leveraging backdoor adjustment [34], wherein the confounder C is stratiﬁed into discrete pieces C
= { c i } |C| i =1 .
By using basic rules induced by the do -operator (see derivation in Appendix A), we have:
[Equation]
Intuitively, the backdoor adjustment approximates an ideal situation where c i is enumerated according to the context prior P ( C ) independent from input sequences, which serves to counteract the effect of C on the generation of S .
Nevertheless, optimizing with Eq.
(2) is intractable since c i is usually unobserved or even undeﬁned, and its prior distribution P ( C ) is also unknown.
Furthermore, computing Eq.
(2) requires awareness of the relation between contexts and generated sequences, which is implicit in the data-generating process behind the data.
To address the difﬁculty for targeting Eq.
(2), we introduce a variational distribution Q ( C | S ) as the estimation for latent contexts given input sequences.
By treating C as a latent variable and using variational inference technique, we can obtain the following tractable evidence lower bound (ELBO) as learning objective (see derivation in Appendix B):
[Equation]
where the last step is given by Jensen’s Inequality and the equality holds if and only if Q ( C | S ) exactly ﬁts the true posterior P ( C | S, Y ) , which suggests it successfully uncovers the latent context from observed data.
The ﬁrst component in Eq.
(3) is the negative reconstruction error (i.e. prediction error).
The second component is the Kullback–Leibler (KL) divergence of the variational distribution and context prior distribution.
Similar with [29], we could re-write the second term as a summation of the entropy of Q ( C | S ) , i.e., H [ Q ( C | S = S )] and the cross-entropy between Q ( C | S ) and prior distribution, i.e., H [ Q ( C ) , P ( C )] .
The former enforces high variance of contexts for each event sequence, and the latter aligns the aggregated posterior with the prior.
Learnable Prior via Mixture of Posteriors.
The distribution P ( C ) characterizes the true context prior in real world.
It serves to regularize Q ( C | S ) through the KL term in Eq.
Choosing an appropriate P ( C ) is important yet challenging.
One straightforward solution adopted by some previous works [8; 57] is to use a pre-deﬁned prior such as uniform distribution.
However, such a simplistic distribution may not reﬂect the true context prior, and potentially lead to over-regularization [5].
An alternative way is to estimate the prior by computing the average of the variational posterior [21], i.e.,
P ( C ) ⇡ 1 /N
P Ni =1 Q ( C | S = S i ) .
However, such a method is computationally expensive [46] and would often result in biased estimation given a limited quantity of training data collected within a certain time interval.
Inspired by [46] using mixture of Gaussian as a ﬂexible and learnable prior, we propose to use a mixture of pseudo variational posteriors as an estimation:
[Equation]
where S {+
+}
j is a randomly generated pseudo event sequence and we set R ⌧ N to reduce the computational cost.
Note that the variational distribution Q ( C | S ) is given by the prediction model (see details in Section 3).
Therefore, the prior estimation by Eq.
(4) is a general reﬂection of how the model favors each context given uninformative inputs.
Also, the estimated prior is learned with the model in a fully data-driven manner.
Model Instantiations
We next parameterize the components in Eq.
(3), i.e., P ✓ ( Y | S, C ) and Q ( C | S ) , and propose a ﬂexible framework named as CaseQ by taking ‘cas’ from ‘causal’ ‘seq’ ‘sequence’.
Event Embedding.
Given an event sequence S = { x 1 , x 2 , · · · , x t } as input, we use an embedding to represent each type of event.
We consider a global embedding matrix H x 2 R M ⇥ d to map each type of event into a d -dimensional embedding space:
[Equation]
where M denotes the number of event types, OneHot ( · ) : Z + ! { 0 , 1 } M transforms an event into a one-hot column vector, and h 1 m 2 R d denotes the initial representation for an event x m .
Inference Unit: Context-Speciﬁc Encoder.
To encode input sequences and accommodate the effect of C on prediction, we consider an encoder  ( · ) conditioned on a speciﬁc context.
It takes a sequence of event embeddings and estimated context representation c ( t ) at time step t (given by the branching unit detailed next) as input, and outputs a sequence of hidden states:
[Equation]
where h 2 m is the m -th hidden state for the event sequence.
Assume that there are K types of contexts, i.e., C = { c k } Kk =1 and c ( t ) 2 C .
Each type of context c k is represented as a K -dimensional one-hot column vector c k 2 { 0 , 1 } K whose k -th dimension is 1 .
Based on the SCM in Fig.
2, the context would change the underlying mechanism of how the next event is generated in real-world.
Therefore, we consider context-speciﬁc inference units {  k ( · ; ✓ k ) }
Kk =1 as sub-networks of the encoder  ( · ) to learn context-aware sequence representations, where ⇥ = { ✓ k } Kk =1 .
Then, we have
[Equation]
where c ( t ) [ k ] returns the k -th entry of the vector c ( t ) .
The inference unit can be speciﬁed as arbitrary off-the-shelf sequence models such as recurrent neural network (RNN) or self-attention (SA) [47].
Branching Unit: Dynamic Model Selection.
We next introduce a branching unit ( · ) to parame- terize the variational posterior Q ( C | S ) that aims to obtain c ( t ) given the sequence.
At each time step, it takes the sequence representation as input and outputs a probability vector q t 2 [0 , 1] K , whose k -th entry represents the probability of the corresponding context c k , i.e.,
[Equation]
, where
[Equation]
We use a d 0 -dimensional context embedding w k 2 R d 0 to accommodate the information of each context c k by an embedding matrix H c 2 R K ⇥ d 0 , where w k = c k > H c and d 0 = d ( d + 1) .
Then, we split each context embedding into several ﬁx-sized parameters W
[Equation]
2 R d
[Equation]
d , a k 2 R d :
[Equation]
The attribution score s tk that measures how likely a sequence up to time step t belongs to context c k can be calculated via
[Equation]
Namely, we ﬁrst project the sequence representation into a new d -dimensional space and then use dot product to measure the similarity.
The variational distribution Q ( C | S = S ) can be speciﬁed as a probability vector q t that is computed by using Softmax function over s tk , i.e., q t =
Softmax ([ s tk ] Kk =1 ; ⌧ ) , where ⌧ controls the conﬁdence level.
To implement the random sampling procedure c ⇠ Q ( C | S ) in Eq.
(3), one can also apply the categorical reparameterization trick that uses differentiable samples from Gumbel-Softmax [23] distribution, i.e.,
[Equation]
Then, we can revise the right term of Eq.
(7) as
[Equation]
CaseQ: a Hierarchical Branching Structure.
The aforementioned single-layered architecture requires an independent inference unit for each type of context, i.e., |C| = K , which has several drawbacks: 1) It is impractical to accommodate a large amount of context types due to limited computational resources; 2) Real-world contexts are not entirely isolated, and thus independently parameterizing them may lead to over-ﬁtting and undesired generalization performance given limited training data.
To address these limitations, we further devise a hierarchical branching structure for CaseQ framework.
Instead of using a one-hot vector, we denote a context as a 0-1 matrix, i.e.,
[Equation]
[Equation]
where D denotes the number of layers and K denotes the number of parallel inference units in each layer.
Each row in c k is a one-hot vector indicating a certain inference unit in a layer, and each type of context corresponds to a certain combination of inference units.
The entire network could then be represented as the stack of multiple parallel inference units and branching units:
[Equation]
where q lt = l ( h lt ) , and l 2 { 1 , 2 , · · · , D } .
The ﬁnal prediction result at each time step can be obtained by ranking the relevance scores of current hidden states and event embeddings:
[Equation]
The above computation acts as a realization of P ✓ ( Y | S = S , C = c ) and c ⇠ Q ( C | S = S ) , required by the ﬁrst term in Eq.
Besides, the KL term in Eq.
(3) also requires density estimation from
Q ( C | S = S ) .
To achieve this, we use the continued Kronecker product of the probability vector q lt in each layer to represent the ﬁnal variational posterior: [Equation] Flatten
[Equation]
where q lt 2 R K and ⌦ denotes Kronecker product.
The produced variational posterior is a K Ddimensional vector, where each entry is the probability of a certain path (i.e., a combination of inference units across layers) in the network.
This design has some merits.
First, we could accommodate |C| =
K D types of contexts that grow exponentially w.r.t.
model depth D .
This helps to endow the model with larger capacity for learning diversiﬁed contexts.
Second, different contexts may associate with each other by sharing the same inference unit in some layers, which plays as an effective inductive bias that guides the model to generalize to new environments (see more details and justiﬁcations in Appendix C.
According to Eq.
(3), the ﬁnal loss function can be written as
[Equation]
where ↵ is a weight parameter for balance, q ( S 0 j ) denotes the produced variational posterior distribution with a pseudo sequence S {+
+} j as input, and l ( · , · ) can be arbitrary loss functions (e.g., cross-entropy) depending on speciﬁc tasks.
Experiments
The goal of our experiments is to test whether CaseQ can effectively handle temporal event prediction under distribution shift and we consider several temporal prediction tasks with particular evaluation protocols for the purpose.
The codes are available at https://github.com/chr26195/Caseq .
Datasets.
We evaluate the effectiveness of CaseQ with different applications for event prediction, including sequential recommendation, ATM maintenance and user awarded badges prediction.
We use four datasets with variable length and number of event types:
Movielens , Yelp , Stack Overﬂow and ATM .
The detailed description of datasets and their statistics are deferred to Appendix E.
Evaluation Protocol.
The commonly adopted leave-one-out evaluation for sequential event prediction [24; 44; 45] uses the latest event for testing, the event right before the last for validation, and the rest for training.
To test the effectiveness of CaseQ for handling temporal distribution shift, we generalize such evaluation protocol by enlarging the time gap between training and testing data.
We use the last G + 1 events for testing, the ﬁrst |S|  G  2 events for training, and the ( |S|  G  1) -th event for validation.
We call G maximal gap size and g 2 [0 , G ] gap size which means the number of events between the validation event and a speciﬁc testing event.
When G = 0 and g = 0 , it degrades to the conventional leave-one-out evaluation.
By increasing g in our case, we could enlarge the time gap between training and testing data, so as to simulate temporal shift in real-world scenarios where data collection and online serving are during different periods of time.
More details about the evaluation protocol are in Appendix G. [Table] [Table] [Figure]
Evaluation Metrics.
For recommendation datasets, i.e., Movielens and Yelp, we adopt two widely used evaluation metrics:
Hit Ratio at K (HR@ K ), and Normalized Discounted Cumulative Gain at K (NDCG@ K ), where K = 10 in practice.
For other two low-dimensional datasets with fewer number of event types, i.e., Stack Overﬂow and ATM, we use Accuracy (i.e., Precision) as the metric.
4.1 Sequential Recommendation
We adopt three sequence models as backbones for sequential recommendation: 1) GRU4Rec [20] uses RNNs to model users’ sequential behaviors and predict the next clicked item; 2) SASRec [24] adopts Self-Attention (SA) in Transformer’s encoder for sequential recommendation; 3) SSE-PT [49] further considers user’s personalized embedding which incorporates user information into the input sequence.
We use MLE for training these models.
For fairness, our model uses the same number of layers as the baselines.
We set gap size g as 0 , 30 for Movielens and 0 , 20 for Yelp since the average sequence length of Movielens is greater than Yelp’s.
As shown in Table 1, our approach consistently outperforms the counterparts with different backbones throughout all the cases.
While the performance degrades when we increase the gap size, CaseQ is signiﬁcantly more robust to temporal distribution shift than conventional models.
The percentage of the performance drop for CaseQ is is signiﬁcantly lower than the counterpart model.
Also, CaseQ is agnostic to the sequential prediction backbones and can be further combined with other advanced architectures.
4.2 Event Prediction
We further study the performance of CaseQ in Stack Overﬂow and ATM datasets, which have distinct statistical features compared to sequential recommendation datasets.
We consider three commonly adopted models for predicting discrete sequential events, i.e., GRU, LSTM and Transformer [47].
We also consider two ensemble frameworks that has the same model size with ours and are built on parallel structures: 1) Ensemble [7] adopts mean pooling over the hidden states of different inference units; 2) DMoE [10] combines the hidden states by learning a gating network in each layer. Also, [Table]
(cid:47) (cid:68) (cid:92) (cid:72) (cid:85) (cid:3) (cid:20) (cid:47) (cid:68) (cid:92) (cid:72) (cid:85) (cid:3) (cid:21) 3 (cid:85) (cid:82) (cid:69) (cid:68) (cid:69) L(cid:79)L (cid:87) (cid:92) (cid:19)(cid:17)(cid:19)(cid:19)(cid:19)(cid:17)(cid:19)(cid:24)(cid:19)(cid:17)(cid:20)(cid:19)(cid:19)(cid:17)(cid:20)(cid:24)(cid:19)(cid:17)(cid:21)(cid:19)(cid:19)(cid:17)(cid:21)(cid:24) (cid:55)LP(cid:72)(cid:3)(cid:54)(cid:87)(cid:72)S(cid:3)(cid:29)(cid:3)(cid:22)(cid:19)
[Figure] we use multiple heads for Transformer to enlarge its model size.
For Stack Overﬂow, the gap size varies from 0 to 50 .
For ATM, it varies from 0 to 20 .
As shown in Fig.
3, CaseQ achieves the best results throughout all the cases in both datasets.
As the gap size increases, the overall accuracy of baseline models exhibit a sharp downward trend in general, which shows the severe negative impact of temporal distribution shift.
Despite with larger model size, both ensemble-based baselines show little improvement over the single models and also suffer a serious performance drop with larger gap sizes.
In comparison, when the gap size goes large, the performance improvement of CaseQ over other competitors becomes more signiﬁcant.
This suggests that our approach is indeed effective for improving the robustness of sequence models w.r.t.
temporal shift.
4.3 Case Study
Visualization of Temporal Shift.
To further investigate temporal distribution shift, in Fig.
4 we visualize the probabilities for each type of context at different time steps in a sequence.
We set
K = 4 , D = 2 .
The two sub-ﬁgures show the results with different base models (LSTM and
Transformer).
The probability value reﬂects the intensity of a certain context at one time.
As shown in the ﬁgure, while some contexts dominate at the beginning, they exhibit clear variation over time.
The results imply that there indeed exists temporal distribution shift behind the data collected at different time intervals.
Therefore, prior art use the past sequential data for training may not successfully transfer to new environments in the future.
By contrast, our approach could ﬁrst identiﬁes latent contexts behind data and then alleviate its impact on learning a stable prediction model.
Learning of Context Embeddings.
Context embeddings are latent representations for each type of context and dynamically affect the branching unit for selecting the proper inference unit.
Conceptually, each type of context is expected to represent an environment in physical world, so each context embedding is supposed to be informative and distinctive.
To verify whether our model could learn meaningful context representations without collapse or over-regularization, we visualize the context embeddings ( K = 3 , D = 2 ) through the training process in Fig.
5 by using dimensional reduction.
Each color corresponds to the embedding of a certain context.
While the context embeddings have the same initialized positions, they gradually separate and move towards different directions as the epochs increase.
This indicates that the model manages to learn informative context representations.
4.4 Scalability Test
We further test the scalability of CaseQ w.r.t.
the number of context types.
Fig.
6 shows the training time and inference time per mini-batch of data.
We use a single layer when the number of context types is within [1 , 6] and two layers when it further increases.
As we can see in the range of [1 , 6] , the training and inference time exhibit linear increase with a single-layer architecture.
The increasing trend goes down when we increase the model depth.
The results suggest CaseQ’s decent scalability. [Figure] [Figure] [Table]
[Figure]
[Figure]
[Table] [Figure] [Table] 4.5 Hyper-Parameter Analysis
An appropriate number of context types are important for guaranteeing the performance and generalizability of our model.
Recall that there are in total K D types of contexts, given a model whose depth (i.e. number of layers) is D and width (i.e. number of parallel inference units in a layer) is K .
Fig.shows the performance of CaseQ w.r.t.
different K and D in sequential recommendation task.
The results show that increasing K and D in a certain range is effective for improving the performance.
Notably, the improvements are more signiﬁcant with larger gap size, which again veriﬁes that CaseQ is helpful for addressing temporal shift.
Still, out-of-range K and D would harm the performance, which is presumably due to over-ﬁtting.
Limitations and Conclusion
Limitations and Potential Impacts.
One of the main limitations of our variational context adjustment framework is that it stratiﬁes discrete contexts (which may be inﬁnitely many) into a ﬁxed context set, in order to make the objective tractable.
However, this remains a common issue for all works that are dealing with inﬁnitely many discrete confounders, and our framework has already made a progress in this direction increasing the size of context set to K D .
We expect to see more works to be done in this direction.
Also, due to the nature of the problem setting where the context is deﬁned as a latent and abstract factor (which is a major challenge we aim to tackle and serves as our contribution), it would be hard or even impossible to identify its meaning in the real-world.
Finally, we do not foresee any direct negative societal impacts.
Conclusion.
In this paper, we endeavor to deal with temporal distribution shift in sequential event prediction.
Namely, the training and testing data are generated from distinct distributions with different contexts.
We ﬁrst show that existing approaches with maximum likelihood estimation would leverage spurious correlation and produce undesirable prediction.
Then we propose a new learning objective with backdoor adjustment and variation inference techniques.
Based on this, we devise a novel model framework for learning context-speciﬁc representations.
Extensive experiments demonstrate the practical efﬁcacy of proposed method.
We hope our work could enlighten new research focus for subsequent works and inspire more frontier research in sequential event prediction against distribution shift.
Acknowledgement
This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Shanghai Municipal Science and Technology (Major) Project (2021SHZDZX0102, 22511105100).
[-All the models are trained from scratch without any pre-training on GTX 
-]