C O LES: C ONTRASTIVE L EARNING FOR E VENT S EQUENCES WITH S ELF -S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
We address the problem of self-supervised learning on discrete event sequences generated by real-world users.
Self-supervised learning incorporates complex information from the raw data in low-dimensional ﬁxed-length vector representations that could be easily applied in various downstream machine learning tasks.
In this paper, we propose a new method CoLES, which adopts contrastive learning, previously used for audio and computer vision domains, to the discrete event sequences domain in a self-supervised setting.
Unlike most previous studies, we theoretically justify under mild conditions that the augmentation method underlying CoLES provides representative samples of discrete event sequences.
We evaluated CoLES on several public datasets and showed that CoLES representations consistently outperform other methods on different downstream tasks.
I NTRODUCTION
A promising and rapidly growing approach known as self-supervised learning 1 is the main choice for pre-training in situations where the amount of labeled data for the target task of interest is limited.
Most of the research in the area of self-supervised learning has been focused on the core machine learning domains, including NLP (e.g., ELMO (Peters et al., 2018), BERT (Devlin et al., 2019)), speech (e.g., CPC (van den Oord et al., 2018)) and computer vision (Doersch et al., 2015; van den
Oord et al., 2018).
However, there has been very little research on self-supervised learning in the domain of discrete event sequences, including user behavior sequences (Ni et al., 2018) such as credit card transactions at banks, phone calls and messages at telecom, purchase history at retail and click-stream data of online services.
Produced in many business applications, such data is a major key to the growth of modern companies.
User behavior sequence is attributed to a person and captures regular and routine actions of a certain type.
The analysis of these sequences constitutes an important sub-ﬁeld of machine learning (Laxman et al., 2008; Wiese and Omlin, 2009; Zhang et al., 2017;
Bigon et al., 2019).
NLP, audio and computer vision domains are similar in the sense that the data of this type is "continuous": a short term in NLP can be accurately reconstructed from its context (like a pixel from its neighboring pixels).
This fact underlies popular NLP approaches for self-supervision such as
BERT’s Cloze task (Devlin et al., 2019) and approaches for self-supervision in audio and computer vision, like CPC (van den Oord et al., 2018).
In contrast, for many types of event sequence data, a single token cannot be determined using its nearby tokens, because the mutual information between a token and its context is small.
For this reason, most state-of-the-art self-supervised methods are not applicable to event sequence data.
In this paper, we propose the COntrastive Learning for Event Sequences (CoLES) method that learns low-dimensional representations of discrete event sequences.
It is based on a novel theoretically grounded data augmentation strategy, which adapts the ideas of contrastive learning (Xing et al., 2002; Hadsell et al., 2006) to the discrete event sequences domain in a self-supervised setting.
The aim of contrastive learning is to represent semantically similar objects ( positive pairs of images, video, audio, etc.) closer to each other, while dissimilar ones ( negative pairs ) further away.
Positive pairs are obtained for training either explicitly , e.g., in a manual labeling process or implicitly using different data augmentation strategies (Falcon and Cho (2020)).
We treat explicit cases as a supervised approach and implicit cases as a self-supervised one.
In most applications, where each person is represented by one sequence of events, there are no explicit positive pairs, and thus only self-supervised approaches are applicable.
Our CoLES method is self-supervised and based on the observation that event sequences usually possess periodicity and repeatability of their events.
We propose and theoretically justify a new augmentation algorithm, which generates sub-sequences of an observed event sequence and uses them as different high-dimensional views of the same (sequence) object for contrastive learning.
Representations produced by the CoLES model can be used directly as a ﬁxed vector of features in some supervised downstream task (e. g. classiﬁcation task) similarly to (Mikolov et al., 2013; Song et al., 2017; Zhai et al., 2019).
Alternatively, the trained CoLES model can be ﬁne-tuned (Devlin et al., 2019) for the speciﬁc downstream task.
We applied CoLES to several user behavior sequence datasets with different downstream classiﬁcation tasks.
When used directly as feature vectors, CoLES representations achieve strong performance comparable to the hand-crafted features produced by data scientists.
We demonstrate that ﬁne-tuned CoLES representations consistently outperform methods based on other representations by a signiﬁcant margin.
We provide the full source code for all the experiments described in the paper 2 .
This paper makes the following contributions: (1) We present the CoLES method that adapts contrastive learning in the self-supervised setting to the discrete event sequence domain.
(2) We propose a novel theoretically grounded augmentation method for discrete event sequences.
(3) We demonstrate that CoLES consistently outperforms previously introduced supervised, self-supervised and semi-supervised learning baselines adapted to the event sequence domain.
We also conducted a pilot study on event sequence data of a large European bank.
We tested CoLES against the baselines and achieved superior performance on downstream tasks which produced signiﬁcant ﬁnancial gains, measured in hundreds of millions of dollars yearly.
The rest of the paper is organized as follows.
In the next section, we discuss related studies on self-supervised and contrastive learning.
In Section 3 we introduce our new method CoLES for discrete event sequences.
In Section 4 we demonstrate that CoLES outperforms several strong baselines including previously proposed contrastive learning methods adapted to event sequence datasets.
Section 5 is dedicated to the discussion of our results and conclusions.
R ELATED WORK
Contrastive learning has been successfully applied to constructing low-dimensional representations (embeddings) of various objects, such as images (Chopra et al., 2005; Schroff et al., 2015), texts (Reimers and Gurevych, 2019), and audio recordings (Wan et al., 2018).
The aim of these studies is to identify the object based on its sample (Schroff et al., 2015; Hu et al., 2014; Wan et al., 2018).
Therefore, their training datasets explicitly contain several independent samples per each particular object, which form positive pairs as a critical component for learning.
These supervised approaches are not applicable to our setting.
For situations when positive pairs are not available or their amount is limited, augmentation techniques were proposed in the computer vision domain.
One of the ﬁrst frameworks with augmentation was proposed by Dosovitskiy et al.
In this work, surrogate classes for model training were introduced using augmentations of the same image.
Several recent works (Bachman et al., 2019;
He et al., 2019; Chen et al., 2020) extended this idea by applying contrastive learning methods, they are nicely summarised by Falcon and Cho (2020).
Although augmentation techniques proposed in these studies provide good performance empirically, we note that no theoretical background behind different augmentation approaches has been proposed so far.
Contrastive Predictive Coding (CPC) is a self-supervised learning approach proposed for non-discrete sequential data (van den Oord et al., 2018).
CPC extracts meaningful representations by predicting latent representations of future observations of the input sequence and using autoregressive methods.
CPC representations demonstrated strong performance on four distinct domains: audio, computer vision, natural language and reinforcement learning.
We adapted the CPC based approach to the domain of discrete event sequences and compared it with our CoLES approach (see Section 4.2).
Independently of our study, several papers appeared in the past few months on self-supervision for user behavior sequences in the recommender systems domain.
Zhou et al.
(2020a) proposed to use a CPC-like approach for self-supervised learning on user clicks history.
Ma et al.
(2020) used an auxiliary self-supervised loss on click sequences.
Zhou et al.
(2020b) proposed "Cloze" task from
BERT (Devlin et al., 2019) for self-supervision on purchase sequences.
Finally, Yao et al.
(2020) adapts a SimCLR-like approach for text-based tasks and tabular data.
Although there has been signiﬁcant progress on contrastive learning, augmentation for contrastive learning does not have any theoretical grounding and is understudied in the domain of discrete event sequences.
P ROBLEM FORMULATION AND OVERVIEW OF THE C O LES METHOD 3. P ROBLEM FORMULATION
While the method proposed in this paper could be studied in different domains, in this paper we focus on discrete sequences of events.
Assume there are some entities e , and the life activity of each entity e is observed as a sequence of events x e := { x e ( t ) } T e t =1 .
Entities could be people or organizations or some other abstractions.
Events x e ( t ) may have any nature and structure (e.g., transactions of a client, click logs of a user), and their components may contain numerical, categorical, and textual ﬁelds (see datasets description in Section 4).
According to theoretical framework of contrastive learning proposed in Saunshi et al.
(2019), each entity e is a latent class, which is associated with a distribution P e over its possible samples (event sequences).
However, unlike the problem setting of Saunshi et al.
(2019), we have no positive pairs, i.e. pairs of event sequences representing the same entity e .
Instead, we have only one sequence x e per entity e .
Formally, each entity e is associated with a latent stochastic process { X e ( t ) } T e t =1 , and we observe only one realisation { x e ( t ) } T e t =1 generated by the process { X e ( t ) } .
Our goal is to learn an encoder M that maps event sequences into a feature space R d in such a way that the obtained embedding c e =
M ( { x e } ) ∈
R d of sequence { x e ( t ) } T e t =1 encodes essential properties of e and disregards any randomness and noise contained in the sequence.
That is, embeddings M ( { x 1 } ) and
M ( { x 2 } ) should be close to each other, if x 1 and x 2 are sequences generated by the same process { X e ( t ) } , and they should be further away, when generated by distinct processes.
The quality of representations can be examined by downstream tasks in the two ways: (1) c e can be used as a feature vector for a task–speciﬁc model, and (2) encoder M can also be (jointly) ﬁne-tuned (Yosinski et al., 2014).
3. S AMPLING OF SURROGATE SEQUENCES AS AN AUGMENTATION PROCEDURE
While we have no access to the latent processes { X e ( t ) } , we need to use augmentation.
Most augmentation techniques proposed earlier for continuous domains (such as image jitter, color jitter or random gray scale in computer vision, see Falcon and Cho (2020)) are not applicable to discrete events.
A possible approach for augmentation is generating sub-sequences of the same event sequence { x e ( t ) } .
The idea proposed below resembles the bootstrap method (Efron and Tibshirani, 1994), which enables to generate several bootstrap samples using only one sample of independent datapoints of a latent distribution.
However, our setting is different, since we have no independent observations, so we should rely on different data assumptions.
The key property of event sequences that represent life activity is periodicity and repeatability of its events (see Figure 4 in the Appendix D for the empyrical observations of these properties for the considered datasets).
This is a motivation for the
Random slices sampling method applied in CoLES, as presented in Algorithm 1.
Each sub-sequence is generated from the initial sequence as its connected segment ("slice") using the following three steps.
First, the length of the slice is chosen uniformly from possible values.
Second, its starting position is uniformly chosen from all possible values.
Third, too short (and optionally too long) sub-sequences are discarded.
It could seem that the mean length of obtained sub-sequences are less than the mean length of sequences in the dataset.
However, we show in the next section that the distribution of sub-sequences is close to the initial distribution in some realistic assumptions.
The overview of the CoLES method is presented in Figure 1.
[Figure]
sub
[Figure]
CoLESEncoder
User 2 event sequence sub-
[Figure]
sub-sequence
Minimizedistance
Maximizedistance
Minimizedistance
[Figure]
CoLESEncoder hyperparameters: m, M : minimal and maximal possible length of a sub-sequence k : number of trials.
input: A sequence S of length T .
output: S : sub-sequences of S .
for i ← 1 to k do
Generate random integer T i uniformly from [1 , T ]; if T i ∈ [ m, M ] then
Generate random integer s from [0 , T − T i − 1] ;
Add S i := S
[ s : s + T
i − 1] to S end
3. T HEORETICAL ANALYSIS
Assume that process { X e ( t ) } T e t =1 is a segment of a latent process { (cid:98) X e ( t ) } ∞ t =1 , which generates sequence of all events in the potentially inﬁnite life of entity e .
That is, we assume that X e ( t )
= (cid:98)
X e
( t + s e ) for some random starting point s e ∈ { 0 , 1 , . . . } and horizon T e .
Thus we observe, in our data, segment [ s e + 1 , s e + T e ] of the life of e .
We also make the following Assumptions:
Process { (cid:98) X
e ( t ) } ∞ t =1 is cyclostationary (in the strict sense)
(Gardner et al., 2006) with some period (cid:98) T .
Starting s e is independent, and the distribution of ( s e mod (cid:98) T ) is uniform over [0 , (cid:98) T − 1] .
Horizon T e is independent and follows a power–law distribution on [ m, ∞ ] .
These assumptions correspond to a scenario where some persons become clients of a service at a random moment and for some random time span and their behaviour obey some periodicity.
Theorem 1.
If sequences { x e ( t ) } in the dataset are generated from latent processes { (cid:98)
X e ( t ) } as described above with a lower bound m for the length of a sequence { x e ( t ) } , then sub-sequences obtained by Algorithm 1 from { x e ( t ) } follow the same distribution as { x e ( t ) } up to a slight alteration of the distribution of the length T e .
Namely, if T e follows power law with an exponent α < − 1 , then the density function for the length T (cid:48) e of a sub-sequence satisfy
[Equation]
This theorem means that a sub-sequence obtained by Algorithm 1 is a representative sample of entity e and follows its latent distribution P e .
Combining this result with generalization guarantees proved for setting with explicitly observed positive pairs (Saunshi et al.
, we obtain theoretical background for our implicit self-supervised setting.
See Appendix C for the proof of Theorem 1.
3. M ODEL TRAINING
Batch generation.
The following procedure creates a batch during CoLES training.
N initial sequences are randomly taken and K sub-sequences are produced for each of them.
Pairs of subsequences of the same sequence are used as positive samples and pairs from different sequences are used as negative ones.
We consider several baseline empirical strategies for the sub-sequence generation to compare with
Algorithm 1.
The simplest strategy is random sampling without replacement.
One more strategy is to produce sub-sequences by the random splitting of the initial sequence to several connected segments without intersection between them (see Appendix A).
Contrastive loss We consider a classical variant of the contrastive loss, proposed by (Hadsell et al., 2006): L = (1 − Y )
( D iW )
+ Y ∗ 12 { max (0 , m − D iW ) } 2 , where D iW is a distance function between embeddings in i-th labeled sample pair
, Y is a binary variable identifying that the pair is positive.
As proposed in (Hadsell et al., 2006), we use euclidean distance function:
D iW
= D
[Equation]
A
[Equation]
= (cid:112)(cid:80) i
[Equation]
i
[Equation]
Pair distance calculation.
In order to select negative samples, we need to compute the pairwise distance between all possible pairs of embedding vectors of a batch.
For the purpose of making this procedure more computationally effective we perform normalization of the embedding vectors, i.e. project them onto a hyper-sphere of the unit radius (see Appendix B).
Negative sampling is a way to address the following challenge of the contrastive learning approach: using all pairs of samples can be inefﬁcient: for example, some of the negative pairs are already distant enough, thus these pairs are not valuable for the training (Simo-Serra et al., 2015; Schroff et al., 2015).
Hence, only a part of possible negative pairs in the batch are used during loss calculation.
We compared the most popular choices for negative sampling applied for CoLES, see Section 4.2 for details.
3. E NCODER ARCHITECTURE
To embed a sequence of events to the ﬁxed-size vector, we use an encoder network, which consists of two conceptual parts: the event encoder and the sequence encoder subnetworks.
The event encoder e takes the set of attributes of each single event x t and outputs its representation in the latent space R d : z t = e ( x t ) .
The event encoder consists of several embedding layers and batch normalization layers.
Each categorical attribute is encoded by its corresponding embedding layer.
Batch normalization is applied to numerical attributes of events.
Outputs of all embedding and batch normalization layers are concatenated to produce latent representation z t .
The sequence encoder s takes latent representations of the sequence of events: z 1: T = z 1 , z 2 , · · · z T and outputs the representation of the whole sequence c t in the time-step t : c t =
s ( z 1: t ) .
Several approaches can be used to encode a sequence (Cho et al., 2014; Vaswani et al., 2017) .
In our experiments we use the recurrent network (RNN) similarly to (Sutskever et al., 2014).
The output produced for the last event is used to represent the whole sequence of events.
In the case of RNN the last output h t is a representation of the sequence.
To summarise, the CoLES method consists of three major ingredients: event sequence encoder, positive and negative pair generation strategy and the loss function for contrastive learning.
E XPERIMENTS
We compare our method with existing baselines on several publicly available datasets from various data science competitions.
We chose datasets with sufﬁcient amounts of discrete events per user.
Age group prediction competition 3 .
The dataset of 44M anonymized credit card transactions representing 50k persons was used to predict the age group of a person.
The label is known for 30k persons, other 20k are unlabelled.
The group ratio is balanced in the dataset.
Each transaction includes the date, type, and amount being charged.
Churn prediction competition 4 .
The dataset of 1M anonymized card transactions representing 10K clients was used to predict a churn probability.
Each transaction is characterized by date, type, amount and Merchant Category Code.
5k clients have labels, 5.2k clients haven’t labels.
Target is binary, almost balanced with proportions 0.55 and 0.45.
Assessment prediction competition 5 .
The task is to predict the in-game assessment results based on the history of children’s gameplay data.
Target is one of 4 grades, with proportions 0.50, 0.24, 0.14, 0.12.
The dataset consists of 12M gameplay events combined in 330k gameplays representing 18k children.
17.7k gameplays are labeled, the remaining 312k gameplays are not labeled.
Each gameplay event is characterized by timestamp, event code, the incremental counter of events within a game session, time since the start of the game session, etc.
Retail purchase history age group prediction 6 .
The task is to predict the age group of a client based on its retail purchase history.
The group ratio is balanced in the dataset.
Only labeled data is used.
The dataset consists of 45,8M retail purchases representing 400k clients.
Each purchase is characterized by time, product level, segment, amount, value, loyalty program points received.
As we can see in Figure 3 (Appendix D), these datasets satisfy the power law assumption for the sequence length distribution of Theorem 1.
Also, as shown in Figure 4 (Appendix D) the datasets satisfy the periodicity and repeatability assumption.
Dataset split.
For each dataset, we set apart 10% persons from the labeled part of the data as the test set that we used for evaluation of different models.
The rest 90% of labeled data and unlabeled data constitute our training set used for learning.
For all methods, a random search on 5-fold crossvalidation over the training set is used for hyper-parameter selection.
The hyper-parameters with the best out-of-fold performance are then chosen.
For the learning of semi-supervised/self-supervised techniques (including CoLES), we used all transactions of training sets including unlabeled data.
The unlabelled parts of the datasets were ignored while training supervised models.
Performance.
Neural network training was performed on a single Tesla P-100 GPU card.
For the training part of CoLES, the single training batch is processed in 142 milliseconds.
For example, in the age group prediction dataset the single training batch contains 64 unique persons withsub-sequences per person, i.e. 320 training sub-sequences in total, the mean number of transactions in a sub-sequence is 90, hence each batch contains about 28800 transactions.
Hyperparameters Unless we explicitly specify, we use contrastive loss and random slices pair generation strategy for CoLES in our experiments (see Section 4.2 for motivation).
The ﬁnal set of hyper-parameters used for CoLES is shown in the Appendix E, Table 5.
4. B ASELINES
LightGBM.
We consider the Gradient Boosting Machine (GBM) method (Friedman, 2001) on hand-crafted features.
GBM can be considered as a strong baseline in cases of tabular data with heterogeneous features.
(Wu et al., 2009; Vorobev et al., 2019; Zhang and Haghani, 2015; Niu et al., 2019).
GBM based model requires a large number of hand-crafted aggregate features produced from the raw transactional data.
An example of an aggregate feature is an average spending amount in some categories of merchants, such as hotels of the entire transaction history.
We used LightGBM (Ke et al., 2017) implementation of the GBM algorithm with nearly 1,000 hand-crafted features for the application.
The details of producing hand-crafted features can be found in the Appendix E.1.
Self-supervised baselines.
NSP.
We consider a simple baseline inspired by the next sentence prediction task used in BERT (Devlin et al., 2019).
Speciﬁcally, we generate two sub-sequences A and B, in a way that 50% of the time B is the sub-sequence from the same sequence as A and follows it (positive pair), and 50% of the time it is a random sub-sequence taken from another sequence (negative pair).
[Table]
SOP.
Another simple baseline is the same as sequence order prediction task from ALBERT (Lan et al., 2020).
It uses two consecutive sub-sequences as a positive pair, and two consecutive sub-sequences with swapped order as a negative pair.
RTD.
We also adapt the replaced token detection approach from ELECTRA (Clark et al., 2020) for event sequences as a baseline for our research.
We replaced 15% of events from the sequence with random events, taken from other sequences and train a model to predict whether an event is replaced or not.
CPC.
As the last self-supervised baseline, we selected the recently proposed Contrastive Predictive
Coding (CPC) (van den Oord et al., 2018), a self-supervised learning method that produced an excellent performance on sequential data of such traditional domains as audio, computer vision, reinforcement learning and recommender systems (Zhou et al., 2020a).
Supervised learning .
In addition to the aforementioned baselines, we compare our method with a supervised learning approach where the encoder network e (see Section 3.5) and the classiﬁcation sub-network h are jointly trained on the downstream task target, i. e. the classiﬁcation sub-network takes encoder output and produces a prediction:
ˆ y =
h ( e ( x )) .
One-layer neural net with softmax activation is used as h .
Note that no pre-training is used in this case.
Note that all neural network baselines use the same architecture of the encoder model as CoLES.
4. R ESULTS
Features of CoLES.
To evaluate the proposed method of sub-sequence generation we compared it with two alternative strategies described in Section 3.4.
The results are presented in Table 1.
The proposed random slices sub-sequence generation strategy signiﬁcantly outperforms alternative strategies, what conﬁrm theoretical results (see Section 3.3).
Also, note that the random samples strategy is similar to the augmentation strategy proposed by Yao et al.
(2020), and the random disjoint samples strategy is similar to sub-sequence generation proposed by Ma et al.
We evaluated several possible loss functions and found that contrastive loss that can be considered as the basic variant of contrastive learning loss, performs on par or better than other losses on the downstream tasks (see Appendix F.1, Table 7).
This means that improvements obtained by more recent losses in object recognition tasks does not necessarily lead to gains in other downstream tasks.
We also compared popular negative sampling strategies (distance-weighted sampling (Manmatha et al., 2017), and hard-negative mining (Schroff et al., 2015)) with random negative sampling strategy.
The results are shown in the Appendix F.1, Table 8.
We found that hard negative mining leads to a signiﬁcant increase in quality on downstream tasks in comparison to random negative sampling.
Comparison with baselines.
We compared CoLES with baselines described in Section 4.1 in two scenarios.
First, we compared embeddings produced by the CoLES encoder with other types of embeddings and with manually created aggregates by using them as input features of a downstream task model.
The downstream task model is trained by LightGBM (Ke et al., 2017) independently from the sequence encoder.
As Table 2 demonstrates, our method generates sequence embeddings of sequential data that achieve strong performance results in comparison to the case of manually crafted features when used on the downstream tasks.
In particular, Table 2 shows that even unsupervised
CoLES embeddings perform on par and sometimes even better than hand-crafted features.
Also note, that CoLES embeddings outperform embeddings produced by the other self-supervised baselines on each dataset.
[Figure]
The rightmost point corresponds to labels and
[Table]
Designed
[Table]
SOP embeddings − 21 .
9% ± 0 .
6% − 5 .
3% ± 0 .
8% − 4 .
1% ±
[Table] [Table]
% − 22 .
8% ±
[Table] [Table]
NSP embeddings − 1 .
5% ± 0 .
9% +0 .
6% ± 0 .
7% − 3 .
5% ± 1 .
1% − 22 .
3% ± 0 .
RTD embeddings +0 .
1% ± 0 .
6% − 2 .
9% ± 0 .
8% − 3 .
6% ± 1 .
1% − 5 .
0% ± 0 .
CPC embeddings − 5 .
9% ± 0 .
6% − 2 .
9% ± 0 .
6% − 2 .
3% ± 0 .
9% − 4 . 0% ± 0 .
[Table]
−
[Table]
−
[Table]
±
[Table]
% −
[Table]
9% ±
[Table] [Table]
[Table]
% ±
[Table]
[Table] [Table]
2% ±
[Table]
[Equation]
[Table]
[Equation] [Equation]
[Table]
[Equation]
[Table] [Table]
In the second scenario, we ﬁne-tune pre-trained models for speciﬁc downstream tasks.
The models are pre-trained using CoLES and other self-supervised learning approaches and then are additionally trained on the labeled data for the speciﬁc task in the same way as we trained a neural net for the supervised learning (see Section 4.1).
A neural net without pre-training is also added to the comparison.
As Table 2 shows, ﬁne-tuned representations obtained by our method achieve superior performance on all the considered datasets, outperforming all other methods by statistically signiﬁcant margins.
Semi-supervised setup.
To evaluate our method in case of the restricted amount of labeled data, we performed the series of experiments where only a fraction of available labels are used to train the downstream task model.
As in the case of the supervised setup, we compare the proposed method with LigthGBM over hand-crafted features, CPC, and supervised learning without pre-training (see
Section 4.1).
The results of this comparison are presented in Figure 2.
Note that the difference in performance between CoLES and supervised-only methods increases
Business applications.
In addition to the described experiments on public datasets, we have performed extensive testing of our method on the private data in a large European bank.
We’ve observed a signiﬁcant increase in model performance (+ 2-10% AUROC) after the addition of CoLES embeddings to the existing models in many downstream tasks, including credit scoring, marketing campaign targeting, product recommendations cold start, fraud detection and legal entities connections prediction.
C ONCLUSIONS
In this paper, we present Contrastive Learning for Event Sequences (CoLES) , a novel self-supervised method for building embeddings of discrete event sequences.
In particular, the CoLES method can be effectively used for pre-training neural networks in semi-supervised settings.
It can also be used to produce embeddings of complex event sequences that can be effectively used in various downstream tasks.
We also empirically demonstrate that our approach achieves strong performance results on several downstream tasks and consistently outperforms both classical machine learning baselines on handcrafted features, as well as other previously introduced self-supervised and semi-supervised learning baselines adapted to the event sequence domain.
In the semi-supervised setting, where the number of labeled data is limited, our method demonstrates even stronger results: the lesser is the labeled data the larger is performance margin between CoLES and supervised-only methods.
The method is especially adapted for event sequence data which is extensively used by the core businesses of many large companies, including ﬁnancial institutions, internet companies, retail and telecom.
A B ATCH GENERATION
We consider several empirical strategies for the sub-sequence generation to compare with the random slices algorithm described in section "Sampling of surrogate sequences" of the main paper.
The simplest strategy is random sampling without replacement.
One more strategy is to produce subsequences by the random splitting of the initial sequence to several connected segments without an intersection between them.
To generate k sub-sequences, the following procedure should be repeated k times: take a random number of elements from the sequence without replacement .
Algorithm 2: Disjointed sub sequences generation strategy hyperparameters: k : number of sub-sequences to be produced.
input: A sequence S of length l .
output: S
, ..., S k
: sub-
sequences of S .
Generate vector inds of length l with random integers from [1,k].
for i ← 1 to k do
S i =
S [ inds =
= i ] end
B P ERFORMANCE OPTIMIZATIONS
Here we describe several performance optimizations that can be used for model training and inference.
Pair distance calculation.
In order to select negative samples, we need to compute pair-wise distance between all possible pairs of embedding vectors of a batch.
For the purpose of making this procedure more computationally effective we perform normalization of the embedding vectors, i.e. project them on a hyper-sphere of unit radius.
Since D ( A, B ) = (cid:112)(cid:80) i
( A i − B i )
= (cid:112)(cid:80) i
A 2 i + (cid:80) i B
i − 2 (cid:80) i
A i B i and ||
A
|| =
|| B
||
= 1 , to compute the euclidean distance we only need to compute: (cid:112)
− 2( A · B ) .
To compute the dot product between all pairs in a batch we just need to multiply the matrix of all embedding vectors of a batch by itself transposed, which is a highly optimized computational procedure in most modern deep learning frameworks.
Hence, the computational complexity of the negative pair selection is O ( n 2 h ) where h is the size of the output embeddings and n is the size of the batch.
Embedding update calculation.
Encoder, based on RNN-type architecture like GRU (Cho et al., 2014), allows to calculate embedding c t + k by updating embedding c t instead of calculating embedding c t
+ k from the whole sequence of past events z 1: t : c k = rnn ( c t , z t +1: k ) .
We use this optimization to reduce inference time to update already existing person embeddings with new events, occurred after the calculation of embeddings.
This is possible due to the recurrent nature of RNN-like networks.
C P ROOF OF T HEOREM
In this section, we provide proof of Theorem 1 (Section 3.3) that justiﬁes the Random Slices subsequence generation strategy proposed in the paper.
Proof.
First, we state the following straightforward lemma:
Lemma 1.
Let a stochastic process { Y ( t ) } ∞ t =1 be a shift of another stochastic process { (cid:98) Y ( t ) } ∞ t = by independent random time s , i.e. Y ( t )
= (cid:98) Y ( t + s ) with integer s ≥ 0 .
If process { (cid:98) Y ( t ) } ∞ t =1 is cyclostationary with period (cid:98) T and ( s e mod (cid:98) T ) is uniform over [0 , (cid:98) T − 1] , then process { Y ( t ) } ∞ t = is stationary.
Lemma 1 implies that process { X e ( t ) } T e t =1 is stationary, and all its segments { X e ( t ) } T (cid:48)
e + s (cid:48) t
= s (cid:48)
+1 of a given length T (cid:48)
e deﬁne the same distribution over sequences as its starting segment { X e ( t ) } T (cid:48)
e t =1 does.
Furthermore, integrating over s (cid:48) , we conclude that the conditional distribution of a sub-sequence obtained via Random Slices generation strategy given its length T (cid:48) e follows the process { X e ( t ) } T (cid:48)
e t =1 .
To ﬁnish the proof, it remains to prove Equation 1.
Assume P
( T e = k )
∝ k α for k ∈ [ m, ∞ ] .
By the law of total probability, we have P ( T (cid:48)
e = k 0 )
= (cid:80) k P
( T e = k ) P ( T (cid:48)
e
= k 0 | T e = k ) , that is,
[Equation]
where C is the normalization constant.
To estimate the sum of the series, notice that
[Equation]
where the former inequality follows from the fact that (cid:82) k +1 / 2 k − 1 / 2 x α − 1 > k α − 1 as long as function f ( x )
= x α − 1 is convex.
After integration, we rewrite Equation 2 as follows:
[Equation]
Using these inequalities, we obtain the upper bound for P ( T (cid:48)
e = k ) in the following way:
[Equation]
At last, the lower bound for P ( T (cid:48)
e = k ) can be obtained using Equation 3 as follows:
[Equation]
The latter inequality in these calculations follows from the fact that (cid:16) l − 12 l (cid:17) α < (cid:16) m − 12 m (cid:17) α for l > m .
[Table]
[Table]
D D ATASETS
We designed the method specially for the user behavior sequences (Ni et al., 2018).
These sequences consist of discrete events per person in continuous time, for example, behavior on websites, credit card transactions, etc.
Considering credit card transactions, each transaction has a set of attributes, either categorical or numerical including the timestamp of the transaction.
An example of the sequence of three transactions with their attributes is presented in Table 3.
The merchant type ﬁeld represents the category of a merchant, such as "airline", "hotel", "restaurant", etc.
Another example of user behavior data is click-stream: the log of internet page visits.
The example of a click-stream log of a single user is presented in Table 4.
In our research we chose several publicly available datasets from data science competitions.
Age group prediction competition 7 - the task is to predict the age group of a person.
The group ratio is balanced in the dataset.
The dataset consists of 44M anonymized transactions representing 50k persons with a target labeled for only 30k of them (27M out of 44M transactions), for the other 20k persons (17M out of 44M transactions) label is unknown.
Each transaction includes date, type (for example, grocery store, clothes, gas station, children’s goods, etc.) and amount.
We use all available 44M transactions for contrastive learning, excluding 10% - for the test part of the dataset, and 5% for the contrastive learning validation. 2.
Churn prediction competition 8 .
The dataset of 1M anonymized
card transactions representing 10K clients was used to predict a churn probability.
5k clients have labels (0.49M out of 1M transactions), 5.2k clients haven’t labels (0.52M out of 1M transactions).
Target is binary, almost balanced with proportions 0.55 and 0.45.
Transactions of the same type and month are grouped and represented as a single pseudo-transaction, which amount is the sum of grouped transactions.
Each transaction is characterized by date, type, amount and
Merchant Category Code.
Assessment prediction competition 9 - the task is to predict the results of the in-game assessment based on the history of children gameplay data.
Target is one of 4 grades, with proportions 0.50, 0.24, 0.14, 0.12.
The dataset consists of 12M gameplay events combined in 330k gameplays representing 18k children.
17.7k gameplays (0.9M out of 12M gameplay events) are labeled, the remaining 312k gameplays (11.6M out of 12M gameplay events) are not labeled.
Each gameplay event is characterized by timestamp, event code, the incremental counter of events within a game session, time since the start of the game session, etc.
Events per client 0.
F r e q u e n c y
Age group
Churn
Assessment
Retail 4.
Retail purchase history age group prediction 10 - the task is to predict the age group of a client based on its retail purchase history.
The group ratio is balanced in the dataset.
The dataset consists of 45,8M retail purchases representing 400k clients.
Only labeled data is used.
Each purchase is characterized by time, product level, segment, amount, value, points received.
To check that sequence lengths of the considered datasets follow the power-low distribution we measured their distribution of lengths.
As Figure 3 shows, the event sequence length distribution is close to the power-law distribution for every considered dataset.
To check that considered datasets follow our repeatability and periodicity assumption made in
Section 3.2 and used for theoretical analysis in Section 3.3 we performed the following experiments.
We measure the KL-divergence two kinds of samples: (1) between random sub-samples of the same sequence, generated using a modiﬁed version of Algorithm 1 where overlapping events are dropped and (2) between random sub-samples taken from different sequences.
The results are shown in
Figure 4.
As Figure 4 shows, the KL-divergence between sub-sequences of the same sequence of events is relatively small compared to the typical KL-divergence between sub-samples of different sequences of events.
This observation supports our repeatability and periodicity assumption.
Also note that additional plot (e) is provided as an example for data without any repeatable structure.
E E XPERIMENT SETUP
For all methods, a random search on 5-fold cross-validation over the train set is used for hyperparameter selection.
The hyper-parameters with the best out-of-fold performance on the train set are then chosen.
The ﬁnal set of hyper-parameters used for CoLES is shown in Table 5.
The number of sub-sequences generated for each sequence was always 5 for each dataset.
E. H AND - CRAFTED FEATURES
Here we describe the details of producing hand-crafted features.
All attributes of each transaction are either numerical (e. g. amount) or categorical (e.g. merchant type (MCC code), transaction type, etc.).
For the numerical type of attribute we apply aggregation functions, such as ’sum’, ’mean’, ’std’, ’min’, ’max’, over all transactions per user.
For example, if we apply ’sum’ for the numerical ﬁeld ’amount’
[Figure]
[Table] [Table]
we obtain a feature ’sum of all transaction amounts per user’.
For the categorical type of attribute we apply aggregation functions in a slightly different way.
For each unique value of categorical attribute we apply aggregation functions, such as ’count’, ’mean’, ’std’ over all transactions per user’ numerical attribute.
For example, if we apply ’mean’ for the numerical attribute ’amount’ grouped by categorical attribute ’MCC code’ we obtain a feature ’mean amount of all transactions for each
MCC code per user’.
For example, for age prediction task we have one categorical attribute (small group) with 200 unique values, combining it with amount we can produce 200 ∗ 3 features (’group x amount x count’, ’group1 x amount x count’, ..., ’group199 x amount x count’, ’group0 x amount x mean’, ...).
In total we use approx 605 features for this task.
Note, that hand-crafted features contain information about user spending proﬁle but omit information about transactions temporal order.
F R ESULTS F. D ESIGN CHOICES OBSERVATIONS
We consider several contrastive learning losses that showed promising performance on different datasets (Kaya and ¸Sakir Bilge, 2019) and some classical variants: contrastive loss (Hadsell et al., 2006), binomial deviance loss (Yi et al., 2014), triplet loss (Hoffer and Ailon, 2015), histogram loss (Ustinova and Lempitsky, 2016), and margin loss (Manmatha et al., 2017).
The results of comparison are shown in the Table 7.
It is interesting to observe that even contrastive loss that can be considered as the basic variant of contrastive learning loss allows to get strong results on the downstream tasks (see Table 7).
Our hypothesis is that an increase in the model performance on contrastive learning task does not always lead to an increase in performance on downstream tasks.
As shown in Table 6, different choices of encoder architectures show comparable performance on the downstream tasks.
F. E MBEDDING SIZE
Figure 5 shows that the performance quality on the downstream task increases with the dimensionality of an embedding.
After the best quality is achieved, a further increase in the dimensionality of an embedding dramatically reduces quality.
These results can be interpreted as the bias-variance trade-off.
When the embedding dimensionality is too small, too much information can be discarded (high bias).
On the other hand, when embedding dimensionality is too large, too much noise is added (high variance).
Note, that increasing the embedding size will also linearly increase the training time and the volume of consumed memory on the GPU.
[Figure]
[Table] [Table]
F. S EMI - SUPERVISED SETUP
To evaluate our method in case of the restricted amount of labeled data, we use only part of the available target labels for the semi-supervised experiment, see Section 4.2 for details.
As in the case of the supervised setup, we compare the proposed method with LigthGBM over hand-crafted features,
CPC, and supervised learning without pre-training.
In ﬁgure 6 we provide learning curves for all considered datasets.
F. E MBEDDING VISUALIZATION
In order to visualize CoLES embeddings in 2-dimensional space, we applied tSNE transformation (van der Maaten and Hinton, 2008) on them.
tSNE transforms high-dimensional space to lowdimensional based on local relationships between points, so neighbor vectors in high-dimensional embedding space are pushed to be close in 2-dimensional space.
We colorized 2-dimensional vectors using the target values of the datasets.
Note, that embeddings was learned in a fully self-supervised way from raw user transactions without any target information.
The sequence of transactions represent user’ behavior, thus the CoLES model captures behavioral patterns and outputs embeddings of users with similar patterns nearby.
As shown below, local clusters in embedding space correspond to the distribution of user’s attributes either age or churn fact.
tSNE vectors from the age prediction dataset are presented in Figure 7a.
We can observe 4 clusters: clusters for group ’1’ and ’2’ are on the opposite side of the cloud, clusters for groups ’2’ and ’3’ are in the middle.
Taking into account that age is an ordinal attribute, we can make an assumption about the ordering of age groups: age (1) < age (3) < age (0) <
age (2) or vice versa.
( age ( bin ) returns age of user for speciﬁc group).
tSNE points from the churn prediction dataset are presented in Figure 7b.
There are areas where one type of label dominates over the other.
[Figure] [Figure]