L EARNING U NIVERSAL U SER R EPRESENTATIONS VIA S ELF -S UPERVISED L IFELONG B EHAVIORS M ODELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Universal user representation is an important research topic in industry , and is widely used in diverse downstream user analysis tasks such as user proﬁling and user preference prediction.
With the rapid development of
Internet service platforms, extremely long user behavior sequences have been accumulated.
However, existing researches have little ability to model universal user representation based on lifelong sequences of user behavior since registration.
In this study, we propose a novel framework called Lifelong User Representation Model (LURM) to tackle this challenge
Speciﬁcally, LURM consists of two cascaded sub
-models: (i) Bag of Interests (BoI) encodes user behaviors in any time period into a sparse vector with super -high dimension ( e
g
; (ii) Self-supervised Multi-anchor Encoder
Network (SMEN) maps sequences of
BoI features to multiple low-dimensional user representations by contrastive learning.
SMEN achieves almost lossless dimensionality reduction, beneﬁting from a novel multi-anchor module which can learn different aspects of user preferences.
Experiments on several benchmark datasets show that our approach outperforms state-of-the-art unsupervised representation methods in downstream tasks
I NTRODUCTION
Inferring user intents through user behavior data has been extensively studied in industrial applications, such as recommendation systems, search engines, and online advertising Dupret & Piwowarski (2008);
He et al.
(2014); Elkahky et al.
Yu et al.
One key aspect in these systems is user modeling, which describes the process of building up and modifying a conceptual understanding of the user Fischer (2001).
Essentially, user modeling is to learn a user representation that helps to capture the user’s interests and preferences to improve the performance on downstream tasks.
In the literature, there have been many studies that focused on task-speciﬁc user modeling, such as user response prediction and personalized recommendation Ren et al.
; Yu et al.
; Ji et al.
However, the user representation learned by a speciﬁc task can hardly be generalized to other tasks.
As a result, speciﬁc user representation models need to be trained in each downstream task, which requires massive labeled data, training time, computing and storage resources.
Given these limitations, universal user representations that can serve a variety of downstream tasks are preferred.
Due to the sequential form of user behavior data, recurrent neural networks (RNNs) are usually used to encode the temporal dynamics of user behavior sequences Wu et al.
; Devooght &
Bersini (2017); Zhu et al. (2017); An et al. (2019).
Unfortunately, these approaches can only process user behavior sequences with a length of tens or hundreds, while the length can reach hundreds of thousands in many social networks and e-commerce services.
Moreover, we have veriﬁed by experiments that the performance will become better as the user behavior becomes richer in most downstream tasks (Fig. 3).
To solve this problem, many methods drown from natural language processing (NLP)
Kumar et al.
; Dai et al.
; Yang et al.
(2019) are proposed.
They model long sequential data based on hierarchical architectures and memory networks Ying et al.
(2018); Pi et al.
; Ren et al.
However, it is still hard for them to encode lifelong user behavior sequences when the length scales up to 1000.
And representations they learned through a speciﬁc task result in poor generalization capabilities.
In this work, we propose a novel framework called Lifelong User Representation Model (LURM) to model user behaviors since registration.
To meet the needs of extremely long sequence modeling, we ﬁrst introduce a model named Bag of Interests (BoI) to summarize items in behavior sequences similar to Bag of Visual Words.
In this way, we can use a sparse vector with super-high dimension to represent user behavior in any time period.
Then, a Self-supervised Multi-anchor Encoder
Network (SMEN) that maps sequences of BoI features to multiple low-dimensional user representations is proposed.
SMEN consists of three modules: a multi-anchor module which can learn different aspects of user preferences, a time aggregation module which can model evolution of user behaviors, and a multi-scale aggregation module which can learn and aggregate BoI features in different scales.
Considering the consistency between user behaviors in different time periods, we introduce a contrastive loss function to the self-supervised training of SMEN.
With the designs above, SMEN achieves almost lossless dimensionality reduction.
The main contribution of our work can be summarized as follows:
• In this work, a novel framework named LURM is proposed to model lifelong user behaviors of any length.
To the best of our knowledge, it is the ﬁrst method that has the ability to model lifelong behaviors in the ﬁeld of universal user representation learning.
• We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.
• We can obtain compressed user representations with little information loss with the help of a designed sub-model named SMEN.
• Extensive experiments are performed on several real-world datasets.
The results demonstrate the effectiveness and generalization ability of the learned user representation.
R ELATED W ORKS 2. U NIVERSAL U SER M ODELING
Compared with task-speciﬁc user modeling that requires more resources, universal user representations are preferred to serve different downstream tasks.
In recent years, some works dedicated to learning universal user representations have been proposed.
Ni et al.
(2018) proposed a representation learning method based on multi-task learning, which enabled the network to generalize universal user representations.
Extensive experiments showed the generality and transferability of the user representation.
However the effectiveness of this method may still suffer due to the selection of tasks and the need of labels.
To release the burden of labeling, Andrews & Bishop (2019) proposed a novel procedure to learn user embedding by using metric learning.
They learned a mapping from short episodes of user behaviors to a vector space in which the distance between points captures the similarity of the corresponding users’ invariant features.
Gu et al.
(2020) proposed a network named self-supervised user modeling network (SUMN) to encode user behavior data into universal representation.
They introduced a behavior consistency loss, which guided the model to fully identify and preserve valuable user information under a self-supervised learning framework.
Wu et al.
(2020) proposed pre-trained user models (PTUM), which can learn universal user models based on two self-supervision tasks for pre-training.
The ﬁrst one was masked behavior prediction, which can model the relatedness between historical behaviors.
The second one was next K behavior prediction, which can model the relatedness between past and future behaviors.
Unfortunately, these methods can only process user behavior sequences with a length of hundreds, and cannot leverage the rich information brought by lifelong user behaviors.
2. L IFELONG U SER M ODELING
Previous works have shown that considering long-term historical behavior sequences for user modeling can signiﬁcantly improve the performance of different tasks Ren et al.
; Pi et al.
Ren et al.
proposed a hierarchical periodic memory network for lifelong sequential modeling.
They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory.
Pi et al.
decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost [Figure] and the system latency.
Speciﬁcally, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally.
Pi et al.
also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.
Unfortunately, the length of the user behavior sequence that these models can handle is still limited.
Moreover, these models are all trained on speciﬁc tasks, which limits the generalization ability.
M ETHODOLOGY
In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length.
For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).
The overall architecture of LURM is shown in Fig.
3. B AG OF I NTERESTS
In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.
Inspired by Bag of Visual Words (BoVW) Fei-Fei & Perona (2005), item is the natural granularity for aggregation.
But there are billions of items on the entire e-commerce platform, which means that the item vocabulary is extremely large, making it infeasible in practice.
Therefore, we propose a model called
Bag of Interests (BoI) to aggregate user behavior data at ‘interest’ granularity.
Every ‘interest’ is a cluster of similar items and represents a certain kind of preference.
The size of ‘interest’ vocabulary is often selected at a level of about 10 5 for retaining enough details.
As shown in Fig.
1 (a), BoI consists of an item embedding module and a large-scale clustering module.
For convenience, we focus on text modality only in this work.
It should be noted that our method can be extended to multi-modal data easily.
3.1. I TEM E MBEDDING M ODULE
Like the BoVW model, an ‘interest’ vocabulary are supposed to be built in our BoI model.
Therefore, the embedding of each item is required, so that similar items with close distance in the embedding space can be clustered together.
Recently, in natural language and image processing, dis- criminative approaches based on contrastive learning in the latent space have shown great success in the ﬁeld of representation learning, achieving state-of-the-art results.
Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec
Barkan & Koenigstein (2016).
Given a set of users U
= { u 1 , u 2 , ..., u | U | } , each user u ∈ U corresponds to a behavior sequence
S = (cid:8) x 1 , x 2 , ..., x | S | (cid:9) , where x i ∈ S denotes the i -th item.
| U | and | S | denote the number of users and the length of u ’s behaviors respectively.
Generally, the content of an item x can be expressed as { w 1 , w 2 , ..., w | x | } , where w i denotes a word from a vocabulary V , and | x | denotes the number of words in the content of x .
Firstly, an encoder with average operation is used to generate item embedding e :
[Equation]
where W i ∈ R
d is the embedding of word w i and will be learned during training, proj ( · ) includes two residual blocks, and a L 2 normalization layer.
To construct the contrastive learning task, we sample positive pairs from behavior sequences of users randomly.
Speciﬁcally, two items ( x i , y i ) are similar, i.e. a positive pair, if they are drawn from the same user behavior sequence and the time interval between the occurrence of these two items is less than β , where β is the window size controlling the interval of the two user behaviors.
Without loss of generality, the sampled mini-batch with batch size n can be denoted as ∆ = { x 1 , y 1 , x 2 , y 2 , ..., x n , y n } , where ( x i , y i ) construct a positive pair drawn from the behavior sequence S it of the i -th user in batch.
Then, the contrastive prediction task is deﬁned to identify y i in ∆ \{ x i } for a given x i , and all other items in ∆ \{ x i , y i } are negatives.
The loss for the positive pair ( x i , y i ) is written as
[Equation]
where g ( x , y ) =
[Equation]
y [Equation] = e T x e y denotes the cosine similarity between the embedding e x and the embedding e y , and τ is the temperature parameter.
The ﬁnal objective is the average loss of all positive pairs in the mini-batch, which can be written as
[Equation]
Noting that, other methods which can be used to generate item embedding are feasible.
3.1. L ARGE - SCALE C LUSTERING M ODULE
After the encoder above has been trained, an item embedding set E = { e i } i ∈
I is obtained, where I is the complete collection of items at the billion level.
In order to retain details as many as possible, the size of the ‘interest’ vocabulary is set to be at 10 4 ∼ 10 5 level.
In other words, all items should be clustered into D (e.g., 10 5 ) categories.
Considering the large-scale item set, a subset E (cid:48) ⊂
E at the million level is sampled, and an efﬁcient clustering algorithm named HDSC Yi
et al.
(2014) on
E (cid:48) is employed to cluster similar items into the same ‘interest’.
After clustering, the cluster centers C make up an ‘interest’ vocabulary.
Therefore each item can be attributed to one/multiple ‘interest(s)’ by hard/soft cluster assignment.
Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature b t ∈ R D in time period t according to his/her behavior sequence S t = (cid:8) x 1 , x 2 , ..., x | S t | (cid:9) :
[Equation]
where x i ∈ c
j means item i is assigned to the cluster c j ∈ C , I is an indicator function.
3. S ELF - SUPERVISED M ULTI - ANCHOR E NCODER N ETWORK
As mentioned above, a BoI feature b t for each user is obtained, through the BoI model given behavior data in any time period t .
Most directly, a user representation with super-high dimension b T can be obtained by applying the BoI model to the whole lifelong time T .
However, there are two main disadvantages: 1) it is not friendly to the downstream tasks since the dimension of representation is too high, and 2) it is too crude to aggregate the information in the whole lifelong time without considering variations over time.
Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time.
Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence of BoI features.
The lifelong time T is divided into N parts at a ﬁxed time interval, i.e. T = { t 1 , t 2 , ..., t N } and t i denotes the i -th time period.
In our experiments, the time interval is usually set to be monthly/seasonly/yearly granularity.
In this way, a sequence of BoI features B = { b t 1 , b t 2 , ..., b t N } can be obtained, where b t i denotes the BoI feature corresponding to the i -th time period t i .
After that, SMEN is used to map B to low-dimensional user representations.
As shown in Fig.
1 (b), SMEN consists of a multi-anchor module, a time aggregation module, a multiscale aggregation module and a contrastive learning module.
Details of the model will be described in the following subsections.
[Figure]
𝑩𝑩
[Figure]
[Figure]
𝒕𝒕 𝟏𝟏 𝒓𝒓𝒕𝒕 𝟏𝟏 𝒓𝒓 𝒌𝒌 𝒓𝒓 𝑴𝑴 𝒓𝒓𝒓𝒓 𝒌𝒌 𝒓𝒓 𝑴𝑴 𝒕𝒕𝒕𝒕 𝒊𝒊 𝒕𝒕 𝒊𝒊 𝒕𝒕 𝒊𝒊 𝒓𝒓 𝟏𝟏𝒕𝒕 𝑵𝑵 𝒕𝒕 𝑵𝑵 𝒓𝒓 𝒌𝒌 𝒕𝒕 𝑵𝑵 𝒓𝒓 𝑴𝑴 … …
[Figure]
in SMEN.
The module outputs a group of diverse representations by assigning different portions of each ‘interest’ to different anchors.
The • denotes the dot product operation, ⊗ denotes the scalar multiply operation, and ⊕ denotes vector sum operation.
3.2. M ULTI - ANCHOR M ODULE
The data of user behavior, which is highly unstructured and complex, implies different preferences of the user.
To capture diverse aspects of user preferences, a novel multi-anchor module is proposed.
Speciﬁcally, suppose there are M anchors, and each of them indicates a certain preference of users.
Let b be a BoI feature, the module converts b to M low-dimensional representations as shown in
Fig.
Each representation is computed as
[Equation]
where f ( b j )
=
b j W ej is the ‘interest’ embedding function, b j is the j -th element of b , W e =
( W e 1 , W e 2 , ..., W eD ) T is the embedding matrix.
And α ij is the attention weight between the i -th anchor and the j -th ‘interest’, which measures the portion assigned to the i -th preference from the j -th behavior ‘interest’.
The weight α ij is deﬁned as
[Equation]
where W a i is the anchor vector corresponding to the i -th anchor, W a
= ( W a 1 , W a 2 , ..., W a M ) T is the anchor matrix, and k j = W p ReLU ( W ej ) is the interest vector corresponding to the j -th ‘interest’.
W e ∈ R D × H , W a ∈ R M × H , and W p ∈
R H
× H are learned parameters.
r i can be computed efﬁciently since b is a sparse vector.
Due to the different anchor vectors, different attention weights can be generated for each ‘interest’.
Finally, a group of different aggregated representations
R = { r 1 , r 2 , ..., r M } can be obtained ( M indicates the total number of anchors).
In this way, we can learn different aspects of user preferences.
Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module.
3.2. T IME A GGREGATION M ODULE
Through the multi-anchor module, a sequence of representation groups R = { R t 1 , R t 2 , ..., R t N } for each user is obtained, where R t i = { r 1 t i , r 2 t i , ...,
r Mt i } is a representation group generated by multi-anchor module corresponding to the BoI feature b t
i in time period t i .
In the time aggregation module, each sequence R i =
{ r it 1 , r it 2 , ..., r it N } is aggregated separately to yield a new representation ˜ r i ∈ R H .
Thus the variable-size representation sequence R can be transformed into M ﬁxed-size representations ˜ R = (cid:8)
[Equation]
There are various methods which can be used to aggregate variable-size sequences, such as average/max pooling and RNNs.
Compared to average and max pooling, RNNs are more appropriate to capture variations over time.
Among all RNN-based models, long short-term memory (LSTM) and gated recurrent units (GRU) are most commonly used.
Considering that GRU has fewer parameters and is less computationally intensive, GRU is adopted to the time aggregation module in this work.
3.2. M ULTI - SCALE A GGREGATION M ODULE
Under a time division T = { t 1 , t 2 , ..., t N } , we have obtained M representations ˜ R = (cid:8)
˜ r 1 , ˜ r 2 , ..., ˜ r M (cid:9) through the BoI, multi-anchor module and time aggregation module as introduced above.
It’s worth noting that if the time interval is too small, the input sequence of SMEN, i.e.
B = { b t 1 , b t 2 , ..., b t N } will become extremely long, which contains extensive details but causes modeling difﬁculties due to catastrophic forgetting.
On the other hand, details may be lost if the time interval is too large.
To address this trade-off, multi-scale aggregation module is designed.
The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities.
Generally, user behavior in this work is aggregated monthly and yearly, i.e.
T = { t 1 , t 2 , ..., t N } = (cid:8) t (cid:48) 1 , t (cid:48) 2 , ..., t (cid:48) N (cid:48) (cid:9) (e.g., if the length of time period is 5 years, N isfor monthly granularity, and N (cid:48) is 5 for yearly granularity).
Thus two sequences of BoI feature
B = { b t 1 , b t 2 , ..., b t N } and B (cid:48)
= (cid:110) b t (cid:48) 1 , b t (cid:48) 2 , ..., b t (cid:48) N (cid:48) (cid:111) at different scales correspondingly are obtained.
Then, two groups of representations ˜ R = (cid:8)
˜ r 1 , ˜ r 2 , ..., ˜ r M (cid:9) and ˜ R (cid:48)
=
(cid:110)
˜ r (cid:48) 1 , ˜ r (cid:48) 2 , ..., ˜ r (cid:48)
M (cid:111) are obtained after the multi-anchor module and time aggregation module with parameter sharing.
Finally we aggregate ˜ R and ˜ R (cid:48) with self-attention as follow:
[Equation]
where (cid:12) is the Hadamard product, [ · ] is a operation concatenating vectors along the last dimension, and s i ( · ) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function.
The switch function controls the fusion between BoI feature sequences of different scales.
The output of this module ˆ R =
[ ˆ r 1 , ˆ r 2 , ..., ˆ r M ] is served as the ﬁnal user representation.
3.2. C ONTRASTIVE L EARNING M ODULE
As mentioned in 3.1.1, contrastive loss is also used to learn user representation in this network.
In order to obtain a uniﬁed vector for contrastive learning, a nonlinear projection head is added, i.e. h ( · ) , as used in SimCLR Chen et al.
The function h ( · ) is computed as
[Equation]
The function mix ( v , W h 3 ) is to allow interaction between different preferences, and is implemented as
[Equation]
[Equation] where r 1 ( · ) reshapes v to a matrix of size M × H , and r 2 ( · ) reshapes W h 3 r 1 ( v ) to a vector.
W h 1 i ∈
R H
× H , W h 2 ∈ R MH × MH , and W h 3 ∈ R M × M are trainable parameters.
Two behavior sequences drawn from the same user are treated as a positive pair, and behavior sequences from other users are negatives.
All sequences are continuous sub-sequences randomly sampled from the whole life time T .
Finally, the contrastive loss is deﬁned the same as equation (3).
E XPERIMENTS
In this section, we present our experiments in detail.
4. D ATASETS
Model comparisons were conducted on a public datatset and an industrial dataset.
Table 1 shows the statistics of two datasets.
The public Amazon dataset Ni et al.
(2019) contains product reviews and metadata like product titles and categories from Amazon.
For each user, the reviewed product titles constitute a sequence of review behaviors.
The dataset for self-supervised user modeling on Amazon
D Apt was constructed by collecting users who have behaviors between 1997-01 and 2017-12, while taking behaviors occurred between 2018-01 and 2018-10 to form datasets { D Ads ( t ) } t ( t refers to tasks) for downstream tasks.
Industrial dataset was collected from visit/order logs on a popular ecommerce platform.
We constructed a training dataset D Ipt for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets { D Ids ( t ) } t .
[Table]
4. D OWNSTREAM T ASKS
Two kinds of downstream tasks were used to evaluate our method: category preference identiﬁcation and user proﬁling prediction.
In the experiments, we randomly selected 80% of the samples from
D Ads ( t ) / D Ids ( t ) for task t to train downstream models and the rest for performance validation.
Category preference identiﬁcation refers to the task of predicting whether users have short-term preferences in the target category.
For Amazon dataset,four categories were included:‘Books of Literature’, ‘Games of Sports’, ‘Outdoor-hunting’, and ‘Musical Instrument’.
For the industrial dataset, we considered three categories including clothing, shoe and coffee for evaluation.
User proﬁling prediction aims to identify user aspects such as gender and age.
We only conducted experiments on industrial dataset.
Two tasks were involved: (1) user age classiﬁcation task (age is divided into 6 classes) which predicts the age ranges of users, and (2) baby age classiﬁcation task ( class).
In both tasks, the ground-truth labels came from an online questionnaire.
4. C OMPETITORS
We compared the proposed LURM model against a rich set of user representation learning methods, including TF-IDF, Doc2Vec, TextCNN, HAN, PTUM and SUMN.
TF-IDF and Doc2Vec view the whole user behavior sequence as a document.
TF-IDF generates a sparse high-dimensional vector, while Doc2Vec learns to represent the document by a dense vector.
We also compared LURM with two methods named TextCNN and HAN which learn task-speciﬁc user representations and classiﬁers in a supervised manner.
Finally, we compared LURM with two newly self-supervised methods for user modeling mentioned before, PTUM and SUMN.
4. T RAINING D ETAILS
For item embedding learning, the number of words in each item x was truncated.
The principle of setting truncation threshold is that 95% data values can be covered by the threshold.
In experiments, word2vec and average pooling were used to obtain item embedding on Amazon dataset, while the method described in 3.1.1 was used on the industrial dataset.
The entire behavior sequences were input into SMEN because the length of input sequence only depends on the time interval.
On
Amazon dataset, TF-IDF and Doc2Vec also took the entire behavior sequence as input.
Meanwhile, the length of input of TextCNN, HAN, Doc2Vec, SUMN and PUTM was limited to 50 due to memory limitation.
On industrial dataset, all competitors used data of two months.
We set the latent dimensions of word/item embedding and all hidden layers to 128 in LURM.
In item embedding module, the window size β was set to 5 days, and the temperature τ was 0.1.
In order to retain details as many as possible, the size of the ‘interest’ vocabulary was set to 10 5 , and the number of anchors M was set to 10 in our experiments.
So the ﬁnal user embedding dimension was 1280.
In addition, the input of month-granularity and year-granularity were used.
For all user representation models, the Adam optimizer with a learning rate of 0.001, and a batch size of 256 was used for training.
The hyper-parameters of the supervised competitors were tuned on the validation set.
For downstream tasks, a simple MLP classiﬁers was applied after the derived user representation.
Note that LURM, SUMN and PUTM were all not ﬁne-tuned for downstream tasks in our experiments.
4. R ESULTS
Table 2 shows the comparison of category preference identiﬁcation on the public Amazon dataset.
The last two rows show the results of our models under different settings.
The ﬁrst one uses the highdimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM.
It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3.23%/1.49% average improvements than SUMN and 4.41%/2.09% average improvements than PUTM in terms of AUC and ACC respectively.
Even compared with supervised methods,
LURM outperforms TextCNN and HAN about 8.26%/3.47% and 3.82%/2.04% respectively.
In particular, it can be observed that LURM has a signiﬁcant improvement over BoI.
Table 3 shows the comparison of two user proﬁling prediction tasks and three category preference identiﬁcation tasks on the industrial dataset.
In this table, BoI and LURM using 2-month input are compared.
When using the same input, LURM(short) achieved better results than other methods on both user proﬁling prediction tasks and category preference identiﬁcation tasks.
When using longer input, we can obtain more improvements, e [Table] g., about 15.55%/17.55% average improvement on user proﬁling prediction tasks, and 4.19%/2.91% average improvement on category preference identiﬁcation tasks compared with SUMN.
[Table]
4. A BLATION S TUDIES AND D ISCUSSION
From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods.
Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.
Fig.
3 shows the performance of LURM on several downstream tasks of industrial dataset when using inputs of different lengths.
It can be seen that the richer the behavior, the better the performance on user proﬁling prediction tasks.
Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks.
While the beneﬁts of using richer behavioral data become small for category preference identiﬁcation tasks.
This is because these tasks are more focused on the short-term interests of users, and it can be observed that the best performance can be obtained when input with a length of one year is used on these tasks.
We also veriﬁed the necessity of multi-scale aggregation module.
The third and fourth rows of Table 4 show the results of LURM with input of monthly/yearly granularity only, respectively.
It can be seen that using multi-scale aggregation module can achieve signiﬁcant improvements on user proﬁling prediction tasks, while the beneﬁts disappear on category preference tasks.
The reason is the same as explained in Fig.
3, that is, there is no need to model representation with extremely long user behaviors for category preference identiﬁcation tasks.
The last two rows of Table 4 show the results of LURM with 1 anchor (LURM(1anchor-128)), and 1 anchor with 1280-dimensional output (LURM(1anchor-1280)).
Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which veriﬁes the effectiveness of the multi-anchor module.
[Table]
BoI 96 . 48 / 80 .93.29/83.
LURM 96.09/78.
60% 65% 70% 75% 80% 85% 90% 2 months 1 year 2 years 3 years 4 years 5 years
Age
Baby Age
Clothing
Coffee
Shoe 70% 75% 80% 85% 90% 95% 2 months 1 year 2 years 3 years 4 years 5 years
Age
Baby Age
Clothing
Coffee
Shoe
[Table]
[Table]
[Table]
[Table]
In addition, we also observed that the gap between BoI and LURM is very different on the two datasets.
It may be that the item embedding we learned is better on the industrial dataset and then
LURM achieved similar results compared with BoI, indicating the ability of SMEN to compress representation losslessly.
On the Amazon dataset, the item embedding is worse.
And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation.
C ONCLUSION
In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length.
With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets.
Although our method has made some progress, there is still space for improvement.
In the future research work, we will consider more types of tasks; input data in different modalities, such as images, video, and audio; more dedicated network architecture and so on.