Original: I TERATIVE M EMORY N ETWORK FOR L ONG S EQUENTIAL U SER B EHAVIOR M ODELING IN R ECOMMENDER S YSTEMS
Revised: I TERATIVE M EMORY N ETWORK FOR L ONG S EQUENTIAL U SER B EHAVIOR M ODELING IN R ECOMMENDER S YSTEMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sequential user behavior modeling is a key feature in modern recommender systems, seeking to capture users’ interest based on their past activities.
Revised: Sequential user behavior modeling is a key feature in modern recommender systems, seeking to capture users’ interest based on their past activities.

Original: There are two usual approaches to sequential modeling : Recurrent Neural Networks (RNNs) and the attention mechanism.
Revised: There are two usual approaches to sequential modeling : Recurrent Neural Networks (RNNs) and the attention mechanism.

Original: As the user behavior sequence gets longer, the usual approaches encounter problems.
Revised: As the user behavior sequence gets longer, the usual approaches encounter problems.

Original: RNN-based methods incur the problem of fast forgetting, making it difﬁcult to model the user’s interests long time ago.
Revised: RNN-based methods incur the problem of fast forgetting, making it difﬁcult to model the user’s interests long time ago.

Original: The self-attention mechanism and its variations such as the transformer structure have the unfortunate property of a quadratic cost with respect to the input length, which makes it difﬁcult to deal with long inputs.
Revised: The self-attention mechanism and its variations such as the transformer structure have the unfortunate property of a quadratic cost with respect to the input length, which makes it difﬁcult to deal with long inputs.

Original: The target attention mechanism, despite having only O ( L ) memory and time complexity, cannot model intra-sequence dependencies.
Revised: The target attention mechanism, despite having only O ( L ) memory and time complexity, cannot model intra-sequence dependencies.

Original: In this paper, we propose Iterative Memory Network (IMN), an endto-end differentiable framework for long sequential user behavior modeling.
Revised: In this paper, we propose Iterative Memory Network (IMN), an endto-end differentiable framework for long sequential user behavior modeling.

Original: In
Revised: In

Original: IMN, the target item acts as a memory trigger, continuously eliciting relevant information from the long sequence to represent the user’s memory on the particular target item.
Revised: IMN, the target item acts as a memory trigger, continuously eliciting relevant information from the long sequence to represent the user’s memory on the particular target item.

Original: In the Iterative Memory Update module, the model early crosses the user behavior embeddings and the item embedding, walks over the long sequence multiple iterations and keeps a memory vector to memorize the content walked over.
Revised: In the Iterative Memory Update module, the model early crosses the user behavior embeddings and the item embedding, walks over the long sequence multiple iterations and keeps a memory vector to memorize the content walked over.

Original: Within each iteration, the sequence interacts with both the target item and the current memory for both target-sequence relation modeling and intra-sequence relation modeling.
Revised: Within each iteration, the sequence interacts with both the target item and the current memory for both target-sequence relation modeling and intra-sequence relation modeling.

Original: The memory is updated after each iteration.
Revised: The memory is updated after each iteration.

Original: The framework incurs only O ( L ) memory and time complexity while reduces the maximum length of network signal travelling paths to O (1) , achieved by the self-attention mechanism with O ( L 2 ) complexity.
Revised: The framework incurs only O ( L ) memory and time complexity while reduces the maximum length of network signal travelling paths to O (1) , achieved by the self-attention mechanism with O ( L 2 ) complexity.

Original: IMN outperforms various state-of-the-art sequential modeling methods on both public and industrial datasets for long sequential user behavior modeling.
Revised: IMN outperforms various state-of-the-art sequential modeling methods on both public and industrial datasets for long sequential user behavior modeling.

Original: It is successfully deployed on an industrial E-commerce recommender system, with a signiﬁcant 7.29% improvement in the Click-Through
Revised: It is successfully deployed on an industrial E-commerce recommender system, with a signiﬁcant 7.29% improvement in the Click-Through

Original: Rate (CTR) for a 7-day A/B test.
Revised: Rate (CTR) for a 7-day A/B test.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Click-through rate (CTR) prediction is critical for recommender systems.
Revised: Click-through rate (CTR) prediction is critical for recommender systems.

Original: User sequential modeling is the key to mine users’ interest for accurate predictions.
Revised: User sequential modeling is the key to mine users’ interest for accurate predictions.

Original: As the user sequence gets longer, particularly with lengths longer than 1000, the prediction task requires extraordinary long-range dependency modeling, efﬁcient memory storage, acceptable training speed and real-time inference.
Revised: As the user sequence gets longer, particularly with lengths longer than 1000, the prediction task requires extraordinary long-range dependency modeling, efﬁcient memory storage, acceptable training speed and real-time inference.

Original: Recurrent Neural Networks (RNNs) and the long short-term memory (LSTM) are employed by the early sequential recommenders (Hidasi et al., 2016; Hochreiter & Schmidhuber, 1997).
Revised: Recurrent Neural Networks (RNNs) and the long short-term memory (LSTM) are employed by the early sequential recommenders (Hidasi et al., 2016; Hochreiter & Schmidhuber, 1997).

Original: Graves et al.
Revised: Graves et al.

Original: (2014) prove that LSTM forgets quickly and fails to generalize to sequences longer than 20.
Revised: (2014) prove that LSTM forgets quickly and fails to generalize to sequences longer than 20.

Original: Many empirical results also verify that RNN-based sequential recommenders are surpassed by attentionbased methods (Zhou et al., 2017; Kang & McAuley, 2018;
Revised: Many empirical results also verify that RNN-based sequential recommenders are surpassed by attentionbased methods (Zhou et al., 2017; Kang & McAuley, 2018;

Original: Zhou et al., 2018; Pi et al., 2019).
Revised: Zhou et al., 2018; Pi et al., 2019).

Original: Lately, the self-attention mechanism has proven to beneﬁt a wide range of application domains, such as machine translation (Vaswani et al., 2017), speech recognition (Chan et al., 2015), reading comprehension (Cui et al., 2016; Lin et al., 2017) and computer vision (Xu et al., 2015; Parmar et al., 2019).
Revised: Lately, the self-attention mechanism has proven to beneﬁt a wide range of application domains, such as machine translation (Vaswani et al., 2017), speech recognition (Chan et al., 2015), reading comprehension (Cui et al., 2016; Lin et al., 2017) and computer vision (Xu et al., 2015; Parmar et al., 2019).

Original: The self-attention mechanism attends to different positions in the sequence, captures the most important features and allows the model to handle long-range dependencies.
Revised: The self-attention mechanism attends to different positions in the sequence, captures the most important features and allows the model to handle long-range dependencies.

Original: SASRec adapts the self- attentive Transformer architecture for sequential recommenders and outperforms convolution-based and recurrence-based methods empirically (Kang & McAuley, 2018).
Revised: SASRec adapts the self- attentive Transformer architecture for sequential recommenders and outperforms convolution-based and recurrence-based methods empirically (Kang & McAuley, 2018).

Original: However, SASRec has a quadratic complexity with respect to the sequence length, limiting its scalability to long sequences.
Revised: However, SASRec has a quadratic complexity with respect to the sequence length, limiting its scalability to long sequences.

Original: Research on efﬁcient self-attention is based on either sparse attention (Li et al., 2020;
Revised: Research on efﬁcient self-attention is based on either sparse attention (Li et al., 2020;

Original: Zhou et al., 2020; Child et al., 2019; Kitaev et al., 2020) or approximated attention (Wang et al., 2020), and consequently incompetent against Transformer.
Revised: Zhou et al., 2020; Child et al., 2019; Kitaev et al., 2020) or approximated attention (Wang et al., 2020), and consequently incompetent against Transformer.

Original: SASRec is also unaware of the target item during the encoding process, which could harm its performance.
Revised: SASRec is also unaware of the target item during the encoding process, which could harm its performance.

Original: Deep Interest Network (DIN) is the ﬁrst architecture that adaptively learns the user interest representation from historical behaviors with respect to a particular target item (Zhou et al., 2018).
Revised: Deep Interest Network (DIN) is the ﬁrst architecture that adaptively learns the user interest representation from historical behaviors with respect to a particular target item (Zhou et al., 2018).

Original: However,
Revised: However,

Original: DIN does not model dependencies between elements in the sequence.
Revised: DIN does not model dependencies between elements in the sequence.

Original: Research has shown the importance of learning long-range intra-sequence dependencies in sequential modeling (Vaswani et al., 2017).
Revised: Research has shown the importance of learning long-range intra-sequence dependencies in sequential modeling (Vaswani et al., 2017).

Original: SASRec considers pairwise intra-sequence dependencies and the maximum length of signal traversal paths is O (1) , while DIN models no intra-sequence dependencies.
Revised: SASRec considers pairwise intra-sequence dependencies and the maximum length of signal traversal paths is O (1) , while DIN models no intra-sequence dependencies.

Original: In this paper, we propose the Iterative Memory Network (IMN) for long sequential user behavior modeling.
Revised: In this paper, we propose the Iterative Memory Network (IMN) for long sequential user behavior modeling.

Original: The target item is a memory trigger in IMN, continuously eliciting relevant information from the sequence.
Revised: The target item is a memory trigger in IMN, continuously eliciting relevant information from the sequence.

Original: IMN early crosses the target item and the user sequence, walks over the sequence for multiple iterations and repeatedly updates the memory vector with new information retrieved from the sequence.
Revised: IMN early crosses the target item and the user sequence, walks over the sequence for multiple iterations and repeatedly updates the memory vector with new information retrieved from the sequence.

Original: The contributions of this paper are summarized as follows:
Revised: The contributions of this paper are summarized as follows:

Original: • We propose the Iterative Memory Network (IMN), an end-to-end differentiable framework for sequential recommenders.
Revised: • We propose the Iterative Memory Network (IMN), an end-to-end differentiable framework for sequential recommenders.

Original: To the best of our knowledge, it outperforms the state-of-the-art models for equal sequence lengths, with even more signiﬁcant advantages on long sequential user behavior modeling.
Revised: To the best of our knowledge, it outperforms the state-of-the-art models for equal sequence lengths, with even more signiﬁcant advantages on long sequential user behavior modeling.

Original: The framework is scalable to sequential recommenders with other objective functions.
Revised: The framework is scalable to sequential recommenders with other objective functions.

Original: • IMN is efﬁcient with O ( L ) complexity and O (1) number of sequential operations, making it scalable to long sequences.
Revised: • IMN is efﬁcient with O ( L ) complexity and O (1) number of sequential operations, making it scalable to long sequences.

Original: IMN requires signiﬁcantly less training and inference time, with greater memory efﬁciency.
Revised: IMN requires signiﬁcantly less training and inference time, with greater memory efﬁciency.

Original: Implemented on user sequences of length 1000, IMN is deployed successfully on an industrial E-commerce platform with 1500 QPS.
Revised: Implemented on user sequences of length 1000, IMN is deployed successfully on an industrial E-commerce platform with 1500 QPS.

Original: The inference time is 30ms.
Revised: The inference time is 30ms.

Original: There is a signiﬁcant 7.29% CTR improvement over the DIN-based industrial baseline.
Revised: There is a signiﬁcant 7.29% CTR improvement over the DIN-based industrial baseline.

Original: • To the best of our knowledge, IMN is the ﬁrst sequential recommender that early crosses the user sequence and the target item.
Revised: • To the best of our knowledge, IMN is the ﬁrst sequential recommender that early crosses the user sequence and the target item.

Original: Our ablation study validates empirically that early crossing before further sequence encoding beneﬁts the model performance compared to delayed crossing.
Revised: Our ablation study validates empirically that early crossing before further sequence encoding beneﬁts the model performance compared to delayed crossing.

Original: • IMN models both long-range intra-sequence dependencies and target-sequence dependencies in O ( L ) complexity.
Revised: • IMN models both long-range intra-sequence dependencies and target-sequence dependencies in O ( L ) complexity.

Original: In IMN, the memory vector encapsulates the sequence content walked over.
Revised: In IMN, the memory vector encapsulates the sequence content walked over.

Original: The multi-way attention between the sequence, the target item and the memory allows for intrasequence signal passing.
Revised: The multi-way attention between the sequence, the target item and the memory allows for intrasequence signal passing.

Original: Self-attention achieves this with O ( L 2 ) complexity, and the target attention mechanism allows for no intra-sequence dependency modeling.
Revised: Self-attention achieves this with O ( L 2 ) complexity, and the target attention mechanism allows for no intra-sequence dependency modeling.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Sequential Recommender Systems .
Revised: Sequential Recommender Systems .

Original: Sequential recommenders predict the user’s next behavior based on his past activities.
Revised: Sequential recommenders predict the user’s next behavior based on his past activities.

Original: Recurrent Neural Networks (RNNs) are introduced for early sequential recommenders (Wu et al., 2016; Hidasi et al., 2016).
Revised: Recurrent Neural Networks (RNNs) are introduced for early sequential recommenders (Wu et al., 2016; Hidasi et al., 2016).

Original: Yet, RNNs are difﬁcult to parallelize and suffer from the problem of fast forgetting (Graves et al., 2014).
Revised: Yet, RNNs are difﬁcult to parallelize and suffer from the problem of fast forgetting (Graves et al., 2014).

Original: First introduced in the encoder-decoder framework, attention is shown effective to replace RNNs to encode sequences (Bahdanau et al., 2016; Vaswani et al., 2017).
Revised: First introduced in the encoder-decoder framework, attention is shown effective to replace RNNs to encode sequences (Bahdanau et al., 2016; Vaswani et al., 2017).

Original: Attention-based recommender systems include self-attention based methods (Kang & McAuley, 2018;
Revised: Attention-based recommender systems include self-attention based methods (Kang & McAuley, 2018;

Original: Zhou et al., 2017), target-attention based methods (Zhou et al., 2018) and methods based on the integration between RNN and attention (Zhou et al., 2019).
Revised: Zhou et al., 2017), target-attention based methods (Zhou et al., 2018) and methods based on the integration between RNN and attention (Zhou et al., 2019).

Original: Target-attention based methods learn the weights for sequence items with respect to the target item.
Revised: Target-attention based methods learn the weights for sequence items with respect to the target item.

Original: Memory Networks .
Revised: Memory Networks .

Original: Memory Networks have wide applications in Question Answering (QA), ﬁnding facts for a query from the knowledge database (Chaudhari et al., 2021).
Revised: Memory Networks have wide applications in Question Answering (QA), ﬁnding facts for a query from the knowledge database (Chaudhari et al., 2021).

Original: Neural Turing Machines (NTM) introduces the addressing-read-write mechanism for memory searching and update (Graves et al., 2014).
Revised: Neural Turing Machines (NTM) introduces the addressing-read-write mechanism for memory searching and update (Graves et al., 2014).

Original: Weston et al.
Revised: Weston et al.

Original: (2014) proposes the general architecture for Memory Networks.
Revised: (2014) proposes the general architecture for Memory Networks.

Original: DMN,
Revised: DMN,

Original: DMTN and DMN+ are the subsequent research (Kumar et al., 2016; Xiong et al., 2016; Ramachandran & Sohmshetty, 2017).
Revised: DMTN and DMN+ are the subsequent research (Kumar et al., 2016; Xiong et al., 2016; Ramachandran & Sohmshetty, 2017).

Original: In recommender systems, MIMN uses GRU as the controller to update user memory slots with each new clicked item (Pi et al., 2019).
Revised: In recommender systems, MIMN uses GRU as the controller to update user memory slots with each new clicked item (Pi et al., 2019).

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: The recommender system models the user-item interaction as a matrix C = { c mn } M × N , where M and N are the total number of users and items respectively.
Revised: The recommender system models the user-item interaction as a matrix C = { c mn } M × N , where M and N are the total number of users and items respectively.

Original: The interaction is either explicit ratings (Koren, 2009) or implicit feedback (Agarwal et al., 2009).
Revised: The interaction is either explicit ratings (Koren, 2009) or implicit feedback (Agarwal et al., 2009).

Original: The CTR prediction task is usually based on implicit feedback.
Revised: The CTR prediction task is usually based on implicit feedback.

Original: We denote u ∈ U as user and i ∈ I as item, and the user u m clicking on the item i n makes c mn 1 and others 0.
Revised: We denote u ∈ U as user and i ∈ I as item, and the user u m clicking on the item i n makes c mn 1 and others 0.

Original: User sequential modeling predicts the probability of a user u ∈ U clicking on the target item i ∈
Revised: User sequential modeling predicts the probability of a user u ∈ U clicking on the target item i ∈

Original: I based on his past behaviors, i 1 , i 2 , ..., i L , where L is the length of the user sequence.
Revised: I based on his past behaviors, i 1 , i 2 , ..., i L , where L is the length of the user sequence.

Original: The sequence is usually in chronological order.
Revised: The sequence is usually in chronological order.

Original: Our paper focuses on long sequential user behavior modeling, where the length of the user behaviors L is on the scale of thousands.
Revised: Our paper focuses on long sequential user behavior modeling, where the length of the user behaviors L is on the scale of thousands.

Original: We use the term ‘target-sequence dependencies’ to refer to dependencies between the target item and the sequence items and ‘intra-sequence dependencies’ to refer to dependencies between the items within the sequence.
Revised: We use the term ‘target-sequence dependencies’ to refer to dependencies between the target item and the sequence items and ‘intra-sequence dependencies’ to refer to dependencies between the items within the sequence.

Original:  I TERATIVE M EMORY N ETWORK
Revised:  I TERATIVE M EMORY N ETWORK

Original: We illustrate the Iterative Memory Network architecture in Fig.1.
Revised: We illustrate the Iterative Memory Network architecture in Fig.1.

Original: The encoder layer converts user behavior features, the temporal features, and the target item features to embedding vectors.
Revised: The encoder layer converts user behavior features, the temporal features, and the target item features to embedding vectors.

Original: The
Revised: The

Original: Iterative Memory Update module is the major component in our architecture, modeling both targetsequence dependencies and intra-sequence dependencies.
Revised: Iterative Memory Update module is the major component in our architecture, modeling both targetsequence dependencies and intra-sequence dependencies.

Original: The Memory Enhancement module repeatedly elicits clearer memory with the target item and outputs the ﬁnal user interest vector.
Revised: The Memory Enhancement module repeatedly elicits clearer memory with the target item and outputs the ﬁnal user interest vector.

Original: Next, we discuss the framework in detail.
Revised: Next, we discuss the framework in detail.

Original: 4. E NCODER L AYER
Revised: 4. E NCODER L AYER

Original: We use e i to denote the embedding for the sequence item i .
Revised: We use e i to denote the embedding for the sequence item i .

Original: Each user behavior consists of not only the clicked item e i , but also the action time.
Revised: Each user behavior consists of not only the clicked item e i , but also the action time.

Original: Different users have different action patterns, thus the action time contains important temporal information.
Revised: Different users have different action patterns, thus the action time contains important temporal information.

Original: Since it is difﬁcult to learn a good embedding directly with continuous time features, we bucketize the time feature into multiple granularities and perform categorical feature look-ups.
Revised: Since it is difﬁcult to learn a good embedding directly with continuous time features, we bucketize the time feature into multiple granularities and perform categorical feature look-ups.

Original: We slice the elapsed time with respect to the ranking time into intervals whose gap length increases exponentially.
Revised: We slice the elapsed time with respect to the ranking time into intervals whose gap length increases exponentially.

Original: In other words, we map the time in range[0,1), [1,2), [2,4), ..., [ 2 k , 2 k +1 ) to categorical features 0,1,2,...
Revised: In other words, we map the time in range[0,1), [1,2), [2,4), ..., [ 2 k , 2 k +1 ) to categorical features 0,1,2,...

Original: k
Revised: k

Original: + 1 .
Revised: + 1 .

Original: Positional encodings are added to represent the relative positions of sequence items.
Revised: Positional encodings are added to represent the relative positions of sequence items.

Original: The timestamp encoding and the positional encoding have the same dimension as that of the item embedding e i to be directly summed.
Revised: The timestamp encoding and the positional encoding have the same dimension as that of the item embedding e i to be directly summed.

Original: We encode the j th user behavior e jb as
Revised: We encode the j th user behavior e jb as

Original: [Equation]
Revised: [Equation]

Original: where ⊕ denotes element-wise sum-up.
Revised: where ⊕ denotes element-wise sum-up.

Original: We denote the embedding for the target item as v T .
Revised: We denote the embedding for the target item as v T .

Original: It shares the item i
Revised: It shares the item i

Original: d embedding look-up table with the item
Revised: d embedding look-up table with the item

Original: id embedding e i for the sequence.
Revised: id embedding e i for the sequence.

Original: 4. M ULTI - WAY A TTENTION
Revised: 4. M ULTI - WAY A TTENTION

Original: We calculate the similarities between the sequence item, the target item and the memory.
Revised: We calculate the similarities between the sequence item, the target item and the memory.

Original: Then we concatenate them into a single feature vector.
Revised: Then we concatenate them into a single feature vector.

Original: Similarities are in both Euclidean distance and the
Revised: Similarities are in both Euclidean distance and the

Original: Hadamard distance (denoted by ◦ ) to capture multi-faceted similarity measures.
Revised: Hadamard distance (denoted by ◦ ) to capture multi-faceted similarity measures.

Original: [Equation]
Revised: [Equation]

Original: where e is the embedding vector for sequence items, m is the memory embedding vector and v is the embedding vector for the target item.
Revised: where e is the embedding vector for sequence items, m is the memory embedding vector and v is the embedding vector for the target item.

Original: We input the feature vector into a two-layer point-wise feed-forward network, with sigmoid as the activation function.
Revised: We input the feature vector into a two-layer point-wise feed-forward network, with sigmoid as the activation function.

Original: Here point-wise feed-forward means the fully-connected feed-forward network is applied to each sequence item separately and identically.
Revised: Here point-wise feed-forward means the fully-connected feed-forward network is applied to each sequence item separately and identically.

Original: [Equation]
Revised: [Equation]

Original: We have also experimented with the softmax activation function for the second layer.
Revised: We have also experimented with the softmax activation function for the second layer.

Original: There is very slight change in model performance.
Revised: There is very slight change in model performance.

Original: The multi-way attention mechanism encodes similarities between the user behavior sequence and the target item to extract the user’s interest speciﬁc to a particular target item.
Revised: The multi-way attention mechanism encodes similarities between the user behavior sequence and the target item to extract the user’s interest speciﬁc to a particular target item.

Original: It also encodes similarities between sequence items and the memory.
Revised: It also encodes similarities between sequence items and the memory.

Original: As the memory vector encapsulates the sequence information, the multi-way attention mechanism can model intra-sequence dependencies between items in the sequence and reduce the maximum length of signal traversal paths to O (1) .
Revised: As the memory vector encapsulates the sequence information, the multi-way attention mechanism can model intra-sequence dependencies between items in the sequence and reduce the maximum length of signal traversal paths to O (1) .

Original: 4. I TERATIVE M EMORY U PDATE M ODULE
Revised: 4. I TERATIVE M EMORY U PDATE M ODULE

Original: In the Iterative Memory Update module, the target item acts as the memory trigger, continuously waking up relevant memory in the sequence.
Revised: In the Iterative Memory Update module, the target item acts as the memory trigger, continuously waking up relevant memory in the sequence.

Original: The memory vector is updated after each iteration with the relevant information retrieved from this iteration.
Revised: The memory vector is updated after each iteration with the relevant information retrieved from this iteration.

Original: We use a GRU to model the memory update process.
Revised: We use a GRU to model the memory update process.

Original: The initial memory m 0 is initialized from the target item vector, to represent that the user’s initial memory on the target item is the target item itself.
Revised: The initial memory m 0 is initialized from the target item vector, to represent that the user’s initial memory on the target item is the target item itself.

Original: [Equation]
Revised: [Equation]

Original: For iteration t , we calculate the user interest representation v tu as a weighted sum pooling of sequence item vectors, with weights derived from Eq.(3).
Revised: For iteration t , we calculate the user interest representation v tu as a weighted sum pooling of sequence item vectors, with weights derived from Eq.(3).

Original: [Equation]
Revised: [Equation]

Original: where e 1 b , e 2 b , ..., e Lb is the list of user behavior embedding vectors, v T is the embedding vector of target item T and m is the memory embedding vector.
Revised: where e 1 b , e 2 b , ..., e Lb is the list of user behavior embedding vectors, v T is the embedding vector of target item T and m is the memory embedding vector.

Original: We use the user interest representation v tu after iteration t as the input to update the memory m t − 1 .
Revised: We use the user interest representation v tu after iteration t as the input to update the memory m t − 1 .

Original: [Equation]
Revised: [Equation]

Original: Since the memory is updated with a weighted sum pooling of sequence item embeddings, the memory contains information on the sequence items.
Revised: Since the memory is updated with a weighted sum pooling of sequence item embeddings, the memory contains information on the sequence items.

Original: As both the memory and the sequence contains information about the sequence items, the multi-way attention models intra-sequence dependencies.
Revised: As both the memory and the sequence contains information about the sequence items, the multi-way attention models intra-sequence dependencies.

Original: The architecture models co-occurence beyond the ( target item, sequence item ) pair.
Revised: The architecture models co-occurence beyond the ( target item, sequence item ) pair.

Original: For example, the target item is rum and the user sequence contains lime and peppermint .
Revised: For example, the target item is rum and the user sequence contains lime and peppermint .

Original: With the target attention mechanism, both the attention weight between the pair ( peppermint, rum ) and that between the pair ( lime, rum ) are not high.
Revised: With the target attention mechanism, both the attention weight between the pair ( peppermint, rum ) and that between the pair ( lime, rum ) are not high.

Original: With our IMN architecture, after the ﬁrst pass of sequence walk, the memory vector contains information about both the peppermint and rum .
Revised: With our IMN architecture, after the ﬁrst pass of sequence walk, the memory vector contains information about both the peppermint and rum .

Original: When it meets the item lime again, the memory of peppermint and rum awakens the item lime since the triplet ( rum, peppermint, lime ) is the recipe for Mojito and likely to co-occur multiple times in different samples.
Revised: When it meets the item lime again, the memory of peppermint and rum awakens the item lime since the triplet ( rum, peppermint, lime ) is the recipe for Mojito and likely to co-occur multiple times in different samples.

Original: Hence, with lime and peppermint in the sequence, the likelihood to click rum increases.
Revised: Hence, with lime and peppermint in the sequence, the likelihood to click rum increases.

Original: While the target attention mechanism ﬁnds the items that co-occur frequently with the target item, our IMN ﬁnds the composite group of the user’s behavior items for the user to click the target item.
Revised: While the target attention mechanism ﬁnds the items that co-occur frequently with the target item, our IMN ﬁnds the composite group of the user’s behavior items for the user to click the target item.

Original: Furthermore, IMN is structurally different from Memory Networks, which updates the memory with each incoming input and uses the updated memory for computations on the next input.
Revised: Furthermore, IMN is structurally different from Memory Networks, which updates the memory with each incoming input and uses the updated memory for computations on the next input.

Original: The sequential nature makes it difﬁcult to parallelize, hence infeasible to deploy on recommmender systems that need real-time responses.
Revised: The sequential nature makes it difﬁcult to parallelize, hence infeasible to deploy on recommmender systems that need real-time responses.

Original: 4. M EMORY E NHANCEMENT M ODULE
Revised: 4. M EMORY E NHANCEMENT M ODULE

Original: After N iterations of the memory update process, the user memory vector encapsulates both the information in the sequence relevant to the target item v T and the intra-sequence information.
Revised: After N iterations of the memory update process, the user memory vector encapsulates both the information in the sequence relevant to the target item v T and the intra-sequence information.

Original: The
Revised: The

Original: Memory Enhancement Module enhances the user memory with the target item repeatedly to elicit more clear memory speciﬁc to the target item and remove noises.
Revised: Memory Enhancement Module enhances the user memory with the target item repeatedly to elicit more clear memory speciﬁc to the target item and remove noises.

Original: We use a GRU to model the repeated elicitation process.
Revised: We use a GRU to model the repeated elicitation process.

Original: The GRU’s initial state is initialized from the memory after the Iterative Memory Update module, u 0 = m N .
Revised: The GRU’s initial state is initialized from the memory after the Iterative Memory Update module, u 0 = m N .

Original: For each step, we apply a linear transformation W u on the GRU’s last hidden state, concatenate the transformed vector with the target item, and use the concatenated vector as the GRU’s input.
Revised: For each step, we apply a linear transformation W u on the GRU’s last hidden state, concatenate the transformed vector with the target item, and use the concatenated vector as the GRU’s input.

Original: [Equation]
Revised: [Equation]

Original: where u t is the user representation after t steps in the Memory Enhancement Module.
Revised: where u t is the user representation after t steps in the Memory Enhancement Module.

Original: 4. O THER R ELEVANT E XPERIMENTS
Revised: 4. O THER R ELEVANT E XPERIMENTS

Original: Dot Product Multi-way Attention .
Revised: Dot Product Multi-way Attention .

Original: We have experimented with the dot product distance for the multiway attention.
Revised: We have experimented with the dot product distance for the multiway attention.

Original: Model performances degrade as expected.
Revised: Model performances degrade as expected.

Original: Sequence Segmentation & Hierarchical Memory Network .
Revised: Sequence Segmentation & Hierarchical Memory Network .

Original: We segment the sequence of lengthinto sub-sequences of length 100.
Revised: We segment the sequence of lengthinto sub-sequences of length 100.

Original: We compute the user  vector with IMN for each sub-sequence.
Revised: We compute the user memory vector with IMN for each subsequence.

Original: The user  vectors produced are treated as another sequence for IMN.
Revised: The user memory vectors produced are treated as another sequence for IMN.

Original: There is no performance improvement.
Revised: There is no performance improvement.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present the experimental setups, experimental results, ablation study, model analysis, computational cost analysis, memory efﬁciency analysis and hyper-parameter choices in detail.
Revised: In this section, we present the experimental setups, experimental results, ablation study, model analysis, computational cost analysis, memory efﬁciency analysis and hyper-parameter choices in detail.

Original: 5. D ATASETS AND E XPERIMENTAL S ETUP
Revised: 5. D ATASETS AND E XPERIMENTAL S ETUP

Original: Amazon Dataset .
Revised: Amazon Dataset .

Original: We collect two subsets from the Amazon product data, Books and Movies (McAuley et al., 2015).
Revised: We collect two subsets from the Amazon product data, Books and Movies (McAuley et al., 2015).

Original: Books contains 295982 users, 647589 items and 6626872 samples.
Revised: Books contains 295982 users, 647589 items and 6626872 samples.

Original: Movies contains 233282 users, 165851 items and 4829693 samples.
Revised: Movies contains 233282 users, 165851 items and 4829693 samples.

Original: We split each dataset into 80% training and 20% test data according to the behavior timestamp.
Revised: We split each dataset into 80% training and 20% test data according to the behavior timestamp.

Original: We use the Adam optimizer, with 0.
Revised: We use the Adam optimizer, with 0.

Original: learning rate (Kingma & Ba, 2015).
Revised: learning rate (Kingma & Ba, 2015).

Original: The mini-batch size is 512.
Revised: The mini-batch size is 512.

Original: We use 2 parameter servers andworkers, with 10GiB memory for each worker.
Revised: We use 2 parameter servers andworkers, with 10GiB memory for each worker.

Original: The MLP layer size is 128 × 64 .
Revised: The MLP layer size is 128 × 64 .

Original: Industrial Dataset .
Revised: Industrial Dataset .

Original: We collect trafﬁc logs from a real-world E-commerce platform.
Revised: We collect trafﬁc logs from a real-world E-commerce platform.

Original: The Ecommerce platform has search and recommendation systems, with user click and purchase logs.
Revised: The Ecommerce platform has search and recommendation systems, with user click and purchase logs.

Original: We use 30-day samples for training and the samples of the following day for testing.
Revised: We use 30-day samples for training and the samples of the following day for testing.

Original: There are 1.
Revised: There are 1.

Original: billion training samples and 57 million test samples.
Revised: billion training samples and 57 million test samples.

Original: MLP layers are 512 × 256 × 128 .
Revised: MLP layers are 512 × 256 × 128 .

Original: The hidden state dimensions for GRUs are 32.
Revised: The hidden state dimensions for GRUs are 32.

Original: The mini-batch size is 512.
Revised: The mini-batch size is 512.

Original: We use the Adam optimizer, with 0.0001 as the learning rate.
Revised: We use the Adam optimizer, with 0.0001 as the learning rate.

Original: We use 5 parameter servers and 50 workers, with 75GiB memory for each worker.
Revised: We use 5 parameter servers and 50 workers, with 75GiB memory for each worker.

Original: Evaluation Metric .
Revised: Evaluation Metric .

Original: We use Area Under the Curve (AUC) as the performance measurement metric.
Revised: We use Area Under the Curve (AUC) as the performance measurement metric.

Original: 5. M ODEL C OMPARISON
Revised: 5. M ODEL C OMPARISON

Original: • YouTube DNN .
Revised: • YouTube DNN .

Original: YouTube DNN uses average pooling to integrate behavior embeddings to ﬁxedwidth vectors as the user’s interest representation (Covington et al., 2016).
Revised: YouTube DNN uses average pooling to integrate behavior embeddings to ﬁxedwidth vectors as the user’s interest representation (Covington et al., 2016).

Original: [Table]
Revised: [Table]

Original: • DIN .
Revised: • DIN .

Original: DIN proposes the target attention mechanism to soft-search user sequential behaviors with respect to the target item (Zhou et al., 2018).
Revised: DIN proposes the target attention mechanism to soft-search user sequential behaviors with respect to the target item (Zhou et al., 2018).

Original: • DIEN .
Revised: • DIEN .

Original: DIEN uses attention-based GRU to model user interest evolutions (Zhou et al., 2019).
Revised: DIEN uses attention-based GRU to model user interest evolutions (Zhou et al., 2019).

Original: • SASRec .
Revised: • SASRec .

Original: SASRec is a self-attentive model based on Transformer (Kang & McAuley, 2018).
Revised: SASRec is a self-attentive model based on Transformer (Kang & McAuley, 2018).

Original: • MIMN .
Revised: • MIMN .

Original: MIMN uses ﬁxed number of memory slots to represent user interests.
Revised: MIMN uses ﬁxed number of memory slots to represent user interests.

Original: When a new click takes place, it updates the user memory slots with GRU (Pi et al., 2019).
Revised: When a new click takes place, it updates the user memory slots with GRU (Pi et al., 2019).

Original: • UBR4CTR .
Revised: • UBR4CTR .

Original: UBR4CTR is a two-stage method.
Revised: UBR4CTR is a two-stage method.

Original: The ﬁrst stage retrieves relevant user behaviors from the sequence with a learnable search method.
Revised: The ﬁrst stage retrieves relevant user behaviors from the sequence with a learnable search method.

Original: The second stage feeds retrieved behaviors into a DIN-based deep model (Qin et al., 2020).
Revised: The second stage feeds retrieved behaviors into a DIN-based deep model (Qin et al., 2020).

Original: The Amazon datasets contain no item side information, therefore we use a strengthened version of sequence selection for the ﬁrst stage with multi-head self-attention on the sequence itself.
Revised: The Amazon datasets contain no item side information, therefore we use a strengthened version of sequence selection for the ﬁrst stage with multi-head self-attention on the sequence itself.

Original: • IMN 2P/3P/3P+ .
Revised: • IMN 2P/3P/3P+ .

Original: IMN models without the Memory Enhancement module.
Revised: IMN models without the Memory Enhancement module.

Original: 2P refers to 2 iterations of the memory update process, and 3P refers to 3 iterations.
Revised: 2P refers to 2 iterations of the memory update process, and 3P refers to 3 iterations.

Original: • IMN 3P+ .
Revised: • IMN 3P+ .

Original: 3 memory update iterations, with the Memory Enhancement module.
Revised: 3 memory update iterations, with the Memory Enhancement module.

Original: The number of steps t is 3 for the Memory Enhancement module.
Revised: The number of steps t is 3 for the Memory Enhancement module.

Original: 5. E XPERIMENTAL R ESULTS
Revised: 5. E XPERIMENTAL R ESULTS

Original: We report model performances on three datasets for compared methods with maximum affordable sequence lengths in Table 1.
Revised: We report model performances on three datasets for compared methods with maximum affordable sequence lengths in Table 1.

Original: We summarize model performances with increasing sequence lengths for Amazon Books in Fig.2.
Revised: We summarize model performances with increasing sequence lengths for Amazon Books in Fig.2.

Original: From Table 1 and Fig.
Revised: From Table 1 and Fig.

Original: we have the following important ﬁndings:
Revised: we have the following important ﬁndings:

Original: • IMN 3P consistently outperforms all state-of-the-art baselines over three datasets.
Revised: • IMN 3P consistently outperforms all state-of-the-art baselines over three datasets.

Original: This demonstrates the effectiveness of our proposed methodology, modeling intra-sequence dependencies and target-sequence dependencies simultaneously.
Revised: This demonstrates the effectiveness of our proposed methodology, modeling intra-sequence dependencies and target-sequence dependencies simultaneously.

Original: IMN 3P+, with the Memory Enhancement module, has slight improvements over IMN 3P. • IMN 3P constantly outperforms SASRec, the Transformer-based sequential recommender, over equal sequence lengths.
Revised: IMN 3P+, with the Memory Enhancement module, has slight improvements over IMN 3P. • IMN 3P constantly outperforms SASRec, the Transformer-based sequential recommender, over equal sequence lengths.

Original: As seen in Fig.2, IMN 3P outperforms the compared models over equal sequence lengths.
Revised: As seen in Fig.2, IMN 3P outperforms the compared models over equal sequence lengths.

Original: Noticeably, IMN 3P outperforms SASRec over equal sequence lengths affordable by SASRec.
Revised: Noticeably, IMN 3P outperforms SASRec over equal sequence lengths affordable by SASRec.

Original: This does make sense, considering that SASRec encodes the sequence with multi-head attention with no knowledge on the target item.
Revised: This does make sense, considering that SASRec encodes the sequence with multi-head attention with no knowledge on the target item.

Original: In contrast, IMN is aware of the target item throughout the encoding process.
Revised: In contrast, IMN is aware of the target item throughout the encoding process.

Original: The ablation study in Section 5.4 veriﬁes the beneﬁts of early cross with the target item.
Revised: The ablation study in Section 5.4 veriﬁes the beneﬁts of early cross with the target item.

Original: • In general, methods that emphatically perform sequential updates seem to have moderate performance gain .
Revised: • In general, methods that emphatically perform sequential updates seem to have moderate performance gain .

Original: DIEN uses attention-based GRUs to update the user interest with each sequence item.
Revised: DIEN uses attention-based GRUs to update the user interest with each sequence item.

Original: Similarly, MIMN
Revised: Similarly, MIMN

Original: sequentially updates the nearest user memory slots with each sequence item.
Revised: sequentially updates the nearest user memory slots with each sequence item.

Original: Fig.2 shows that DIEN constantly outperforms DIN, though the improvement is moderate on sequence length 100.
Revised: Fig.2 shows that DIEN constantly outperforms DIN, though the improvement is moderate on sequence length 100.

Original: Method
Revised: Method

Original: Books
Revised: Books

Original: Movies
Revised: Movies

Original: Industrial
Revised: Industrial

Original: AUC AUC
Revised: AUC AUC

Original: AUC w/o.
Revised: AUC w/o.

Original: attention 0.
Revised: attention 0.

Original: w/o.
Revised: w/o.

Original: iterative walk 0.
Revised: iterative walk 0.

Original: w/o.
Revised: w/o.

Original: Euclidean dist.
Revised: Euclidean dist.

Original: delayed cross w/o.
Revised: delayed cross w/o.

Original: m.e.
Revised: m.e.

Original: w/o.
Revised: w/o.

Original: m.e. full 0.
Revised: m.e. full 0.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: We conduct ablation study about the model structure and report the results in Fig.3.
Revised: We conduct ablation study about the model structure and report the results in Fig.3.

Original: We remove the
Revised: We remove the

Original: Memory Enhancement module to produce the ablation model IMN(w/o. m.e.).
Revised: Memory Enhancement module to produce the ablation model IMN(w/o. m.e.).

Original: We further remove the cross with the target item to product the ablation model IMN(delayed cross & w/o. m.e.).
Revised: We further remove the cross with the target item to product the ablation model IMN(delayed cross & w/o. m.e.).

Original: We replace the Euclidean distance with another Hadamard distance calculation for the ablation model
Revised: We replace the Euclidean distance with another Hadamard distance calculation for the ablation model

Original: IMN(w/o. Euclidean dist.)
Revised: IMN(w/o. Euclidean dist.)

Original: We remove the iterative update process to produce IMN(w/o. iterative walk), which becomes the same as DIN.
Revised: We remove the iterative update process to produce IMN(w/o. iterative walk), which becomes the same as DIN.

Original: We replace the attention with average pooling to produce
Revised: We replace the attention with average pooling to produce

Original: IMN (w/o. attention), which becomes identical to YouTube DNN.
Revised: IMN (w/o. attention), which becomes identical to YouTube DNN.

Original: The following are our ﬁndings:
Revised: The following are our ﬁndings:

Original: • The iterative update process models intra-sequence dependencies, which beneﬁts the model performance signiﬁcantly.
Revised: • The iterative update process models intra-sequence dependencies, which beneﬁts the model performance signiﬁcantly.

Original: IMN(full) has a remarkable improvement over IMN(w/o. iterative walk), on par with the improvement of IMN(w/o. iterative walk) over IMN(w/o.attention).
Revised: IMN(full) has a remarkable improvement over IMN(w/o. iterative walk), on par with the improvement of IMN(w/o. iterative walk) over IMN(w/o.attention).

Original: It demonstrates the effectiveness of our proposal to model intra-sequence dependencies in addition to target-sequence dependencies as in DIN.
Revised: It demonstrates the effectiveness of our proposal to model intra-sequence dependencies in addition to target-sequence dependencies as in DIN.

Original: • Delayed cross with the target item results in performance degradation .
Revised: • Delayed cross with the target item results in performance degradation .

Original: IMN(w/o. m.e.) outperforms IMN(delayed cross & w/o.
Revised: IMN(w/o. m.e.) outperforms IMN(delayed cross & w/o.

Original: m.e.) by a large extent, showing that early crossing the user sequence and the target item results in performance gain.
Revised: m.e.) by a large extent, showing that early crossing the user sequence and the target item results in performance gain.

Original: This also explains for IMN’s performance improvements over SASRec, which crosses the Transformer-encoded user sequence and the target item at the very top.
Revised: This also explains for IMN’s performance improvements over SASRec, which crosses the Transformer-encoded user sequence and the target item at the very top.

Original: • Multi-faceted distance modeling beneﬁts model performances.
Revised: • Multi-faceted distance modeling beneﬁts model performances.

Original: IMN(full) outperforms IMN(w.o.
Revised: IMN(full) outperforms IMN(w.o.

Original: Euclidean dist.)
Revised: Euclidean dist.)

Original: to a certain extent, showing that multi-faceted distance modeling is beneﬁcial.
Revised: to a certain extent, showing that multi-faceted distance modeling is beneﬁcial.

Original: 5. M ODEL A NALYSIS
Revised: 5. M ODEL A NALYSIS

Original: We analyze the compared models and summarize the encoding paradigm, complexity, minimum number of sequential operations and maximum path length in Table 2, with the observations below:
Revised: We analyze the compared models and summarize the encoding paradigm, complexity, minimum number of sequential operations and maximum path length in Table 2, with the observations below:

Original: • IMN is the ﬁrst architecture with a ( CROSS, ENC ) structure, which beneﬁts model performances.
Revised: • IMN is the ﬁrst architecture with a ( CROSS, ENC ) structure, which beneﬁts model performances.

Original: We denote the cross between the user sequence and the target item as CROSS .
Revised: We denote the cross between the user sequence and the target item as CROSS .

Original: We denote sequence encoding as ENC .
Revised: We denote sequence encoding as ENC .

Original: The encoding paradigm for DIN is ( CROSS ) , with no sequence encoding after the target attention cross.
Revised: The encoding paradigm for DIN is ( CROSS ) , with no sequence encoding after the target attention cross.

Original: DIEN is ( ENC, CROSS ) , encoding sequence items with GRU before interest evolution modeling with respect to the target item.
Revised: DIEN is ( ENC, CROSS ) , encoding sequence items with GRU before interest evolution modeling with respect to the target item.

Original: SASRec adapts the Transformer encoder for sequence encoding and is therefore ( ENC, CROSS ) .
Revised: SASRec adapts the Transformer encoder for sequence encoding and is therefore ( ENC, CROSS ) .

Original: Similarly, MIMN follows the ( ENC, CROSS ) paradigm, using a GRU-based controller to update memory slots with the sequence.
Revised: Similarly, MIMN follows the ( ENC, CROSS ) paradigm, using a GRU-based controller to update memory slots with the sequence.

Original: UBR4CTR is ( CROSS ) since sequence items cross the target item with no further sequence encoding for the second stage.
Revised: UBR4CTR is ( CROSS ) since sequence items cross the target item with no further sequence encoding for the second stage.

Original: IMN is the ﬁrst model with the ( CROSS, ENC ) paradigm.
Revised: IMN is the ﬁrst model with the ( CROSS, ENC ) paradigm.

Original: The user sequence crosses with the target item in Hadamard distance and Euclidean distance before sequence encoding.
Revised: The user sequence crosses with the target item in Hadamard distance and Euclidean distance before sequence encoding.

Original: Ablation study in Fig.3 validates empirically that the early cross beneﬁts model performances.
Revised: Ablation study in Fig.3 validates empirically that the early cross beneﬁts model performances.

Original: • IMN is efﬁcient with O ( L · d ) complexity and O (1) number of sequential operations, with theoretical shortest maximum path length
Revised: • IMN is efﬁcient with O ( L · d ) complexity and O (1) number of sequential operations, with theoretical shortest maximum path length

Original: O (1) .
Revised: O (1) .

Original: SASRec, the Transformer-based sequential recommender, incurs O
Revised: SASRec, the Transformer-based sequential recommender, incurs O

Original: ( L 2 · d ) complexity.
Revised: ( L 2 · d ) complexity.

Original: IMN incurs O
Revised: IMN incurs O

Original: ( L · d ) complexity, since the optimal number of iterative sequence walk is 3 as seen in Section 5.8.
Revised: ( L · d ) complexity, since the optimal number of iterative sequence walk is 3 as seen in Section 5.8.

Original: The minimum number of sequential operations measures the amount of parallelizable computations.
Revised: The minimum number of sequential operations measures the amount of parallelizable computations.

Original: Pure attention-based methods are at O (1) , while recurrence-based methods are at O ( L ) .
Revised: Pure attention-based methods are at O (1) , while recurrence-based methods are at O ( L ) .

Original: Maximum path length refers to the maximum length
Revised: Maximum path length refers to the maximum length

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of signal traversal paths.
Revised: of signal traversal paths.

Original: SASRec is O (1) due to the pairwise comparisons in the self-attention mechanism.
Revised: SASRec is O (1) due to the pairwise comparisons in the self-attention mechanism.

Original: Our IMN also is O (1) since the memory vector with the sequence information interacts with sequence items through the multi-way attention.
Revised: Our IMN also is O (1) since the memory vector with the sequence information interacts with sequence items through the multi-way attention.

Original: There is no intra-sequence signal passing in DIN and UBR4CTR.
Revised: There is no intra-sequence signal passing in DIN and UBR4CTR.

Original: DIEN and MIMN are at O ( L ) due to the sequential nature.
Revised: DIEN and MIMN are at O ( L ) due to the sequential nature.

Original: 5. C OMPUTATIONAL C OST A NALYSIS
Revised: 5. C OMPUTATIONAL C OST A NALYSIS

Original: We report the run-time for compared methods in Fig.4a and the inference time in Fig.4b.
Revised: We report the run-time for compared methods in Fig.4a and the inference time in Fig.4b.

Original: Note that
Revised: Note that

Original: Fig.4b has y-axis on the logarithmic scale.
Revised: Fig.4b has y-axis on the logarithmic scale.

Original: The run-time efﬁciency is measured by global steps per second during training.
Revised: The run-time efﬁciency is measured by global steps per second during training.

Original: The real-time efﬁciency is measured by the inference speed in milliseconds.
Revised: The real-time efﬁciency is measured by the inference speed in milliseconds.

Original: For the inference speed, we only measure the forward pass cost, excluding the input encoding cost.
Revised: For the inference speed, we only measure the forward pass cost, excluding the input encoding cost.

Original: We use inputs with lengths 50, 100, 200, 500 and 1000.
Revised: We use inputs with lengths 50, 100, 200, 500 and 1000.

Original: All experiments are conducted on Tesla
Revised: All experiments are conducted on Tesla

Original: A100 GPU with 10GiB memory.
Revised: A100 GPU with 10GiB memory.

Original: We summarize our ﬁndings below:
Revised: We summarize our ﬁndings below:

Original: • IMN is computationally efﬁcient with increasing sequence lengths.
Revised: • IMN is computationally efﬁcient with increasing sequence lengths.

Original: IMN involves matrix operations heavily, which are highly optimized to be be parallelizable on GPU.
Revised: IMN involves matrix operations heavily, which are highly optimized to be be parallelizable on GPU.

Original: On sequence length 1000, the training time for IMN and DIN is similar, not even doubling that for YouTube DNN.
Revised: On sequence length 1000, the training time for IMN and DIN is similar, not even doubling that for YouTube DNN.

Original: More importantly, the forward pass inference time at sequence length 1000 is only 3.6ms.
Revised: More importantly, the forward pass inference time at sequence length 1000 is only 3.6ms.

Original: • Methods based on sequential updates are computationally expensive for training and inference.
Revised: • Methods based on sequential updates are computationally expensive for training and inference.

Original: DIEN and MIMN, the two models based on sequential updates, have signiﬁcantly lower training and inference speeds.
Revised: DIEN and MIMN, the two models based on sequential updates, have signiﬁcantly lower training and inference speeds.

Original: The inference time for MIMN at sequence length 50 is 216ms, while the industrial norm is 30ms to 50ms.
Revised: The inference time for MIMN at sequence length 50 is 216ms, while the industrial norm is 30ms to 50ms.

Original: In-depth analysis shows that its addressing head is timeconsuming.
Revised: In-depth analysis shows that its addressing head is timeconsuming.

Original: The computational inefﬁciency forces MIMN to separate the user and the item side, performing user side inference prior to online scoring.
Revised: The computational inefﬁciency forces MIMN to separate the user and the item side, performing user side inference prior to online scoring.

Original: At sequence length 100, DIEN’s inference time is 77ms.
Revised: At sequence length 100, DIEN’s inference time is 77ms.

Original: Unlike MIMN, DIEN cannot separate the user and item sides since it performs target attention on top of the GRU encoder.
Revised: Unlike MIMN, DIEN cannot separate the user and item sides since it performs target attention on top of the GRU encoder.

Original: This limits DIEN’s scalability to longer sequences.
Revised: This limits DIEN’s scalability to longer sequences.

Original: • SASRec, the self-attentive method, is efﬁcient during inference time, but not during training time.
Revised: • SASRec, the self-attentive method, is efﬁcient during inference time, but not during training time.

Original: The inference speed for SASRec is signiﬁcantly lower compared to methods with sequential updates.
Revised: The inference speed for SASRec is signiﬁcantly lower compared to methods with sequential updates.

Original: This is expected since self-attention allows for signiﬁcantly more parallelizations.
Revised: This is expected since self-attention allows for signiﬁcantly more parallelizations.

Original: However, training is relatively slow for SASRec.
Revised: However, training is relatively slow for SASRec.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5. M EMORY C ONSUMPTION
Revised: 5. M EMORY C ONSUMPTION

Original: We evaluate the memory efﬁciency by measuring the peak GPU memory usage in GiB in Fig.4c.
Revised: We evaluate the memory efﬁciency by measuring the peak GPU memory usage in GiB in Fig.4c.

Original: The memory limit is 10GiB. Experiments above the horizontal dotted line in Fig.4c incur Out-ofMemory (OOM) errors.
Revised: The memory limit is 10GiB. Experiments above the horizontal dotted line in Fig.4c incur Out-ofMemory (OOM) errors.

Original: We summarize our ﬁndings below:
Revised: We summarize our ﬁndings below:

Original: • IMN is efﬁcient with memory consumption increasing linearly with sequence lengths.
Revised: • IMN is efﬁcient with memory consumption increasing linearly with sequence lengths.

Original: IMN incurs linear space complexity.
Revised: IMN incurs linear space complexity.

Original: The memory overhead is the user memory vector, the same size as the target item.
Revised: The memory overhead is the user memory vector, the same size as the target item.

Original: The low peak memory consumption at varying lengths testiﬁes its memory efﬁciency.
Revised: The low peak memory consumption at varying lengths testiﬁes its memory efﬁciency.

Original: • Self-attentive methods have the most memory usage increase with increasing sequence lengths.
Revised: • Self-attentive methods have the most memory usage increase with increasing sequence lengths.

Original: The NTM-based MIMN is also memory-hungry
Revised: The NTM-based MIMN is also memory-hungry

Original: SASRec
Revised: SASRec

Original: incurs
Revised: incurs

Original: Out-Of-Memory (OOM) errors
Revised: Out-Of-Memory (OOM) errors

Original: on sequences
Revised: on sequences

Original: longer than
Revised: longer than

Original: Fig.4c also shows that its
Revised: Fig.4c also shows that its

Original: memory consumption increase
Revised: memory consumption increase

Original: is the most
Revised: is the most

Original: substantial
Revised: substantial

Original: with increasing sequence lengths, testifying the O ( L 2 ) memory bottleneck.
Revised: with increasing sequence lengths, testifying the O ( L 2 ) memory bottleneck.

Original: MIMN
Revised: MIMN

Original: is also
Revised: is also

Original: memory-hungry, since keeping additional user memory slots results in memory overheads.
Revised: memory-hungry, since keeping additional user memory slots results in memory overheads.

Original: 5. S ENSITIVITY W . R . T N UMBER OF S EQUENCE W ALK I TERATIONS
Revised: 5. S ENSITIVITY W . R . T N UMBER OF S EQUENCE W ALK I TERATIONS

Original: We investigate the impact of sequence walk iterations.
Revised: We investigate the impact of sequence walk iterations.

Original: Fig.5a shows IMN’s performances against the sequence walk iterations.
Revised: Fig.5a shows IMN’s performances against the sequence walk iterations.

Original: AUCs increase with more iterations and stablize after 3 to 4 iterations.
Revised: AUCs increase with more iterations and stablize after 3 to 4 iterations.

Original: Fig.5b shows the average attention entropy against sequence walk iterations.
Revised: Fig.5b shows the average attention entropy against sequence walk iterations.

Original: The trend coincides with that on model performance, indicating that the optimal iteration hyper-parameter is 3.
Revised: The trend coincides with that on model performance, indicating that the optimal iteration hyper-parameter is 3.

Original:  O NLINE A/B P ERFORMANCE
Revised:  O NLINE A/B P ERFORMANCE

Original: We report the 7-day A/B test result on the industrial recommender in Table.3.
Revised: We report the 7-day A/B test result on the industrial recommender in Table.3.

Original: The baseline is a
Revised: The baseline is a

Original: DIN-based deep model with sequences of length 50.
Revised: DIN-based deep model with sequences of length 50.

Original: IMN is implemented on user click sequences of length 1000.
Revised: IMN is implemented on user click sequences of length 1000.

Original: UV-IPV refers to the average click count per user.
Revised: UV-IPV refers to the average click count per user.

Original: Total Clicked Items Count (TCIC) is the number of distinct items being clicked.
Revised: Total Clicked Items Count (TCIC) is the number of distinct items being clicked.

Original: Clicked Categories Count (CCC) is the number of categories clicked per user.
Revised: Clicked Categories Count (CCC) is the number of categories clicked per user.

Original: TCIC and CCC are diversity measures for recommender systems.
Revised: TCIC and CCC are diversity measures for recommender systems.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The self-attention mechanism incurs O
Revised: The self-attention mechanism incurs O

Original: ( L 2 ) memory complexity that limits the scalability to long sequences.
Revised: ( L 2 ) memory complexity that limits the scalability to long sequences.

Original: The target attention mechanism, despite having only O ( L ) complexity, can not model intra-sequence dependencies.
Revised: The target attention mechanism, despite having only O ( L ) complexity, can not model intra-sequence dependencies.

Original: Methods with sequential updates incur high computational costs.
Revised: Methods with sequential updates incur high computational costs.

Original: IMN models intra-sequence dependencies and target-sequence dependencies simultaneously in
Revised: IMN models intra-sequence dependencies and target-sequence dependencies simultaneously in

Original: O ( L ) memory and time complexities.
Revised: O ( L ) memory and time complexities.

Original: It is efﬁcient with computational costs and memory consumption.
Revised: It is efﬁcient with computational costs and memory consumption.

Original: It outperforms the state-of-the-art sequential recommenders for equal sequence lengths, with even more signiﬁcant advantages on long user sequential modeling.
Revised: It outperforms the state-of-the-art sequential recommenders for equal sequence lengths, with even more signiﬁcant advantages on long user sequential modeling.

Original: It is deployed successfully on a real-world E-commerce recommendation platform, with a signiﬁcant improvement of 7.29% CTR.
Revised: It is deployed successfully on a real-world E-commerce recommendation platform, with a signiﬁcant improvement of 7.29% CTR.