Original: Sequence-Graph Duality: Unifying User Modeling with Self-Attention for Sequential Recommendation
Revised: Sequence-Graph Duality: Unifying User Modeling with Self-Attention for Sequential Recommendation

Original: Abstract
Revised: Abstract

Original: User modeling is of great importance in personalization services.
Revised: User modeling is of great importance in personalization services.

Original: Many existingmethods treat users as  sequencesof interacted items to encode item transitionpatterns.
Revised: Many existing methods treat users as interaction sequences to capture users’ evolving interests.

Original: Another line of research models users as graphs in which interacted itemsare modelled as nodes, and item-item relations are modelled as edges.
Revised: Another line of research models each user as a user graph in which the users’ interactions are modeled as nodes.

Original:  The graphbased user modeling provides more flexibility to encode complex item relationshipsof different types (e.g. co-occurrence, similarity) but usually overlooks sequentialpatterns.
Revised: Nodes (interactions) in user graphs are connected via edges that reflect certain relations such as item similarity. The graph-based user modeling can flexibly store item relationships.

Original: Here, we introduce a novel user representation, Heterogeneous User Graph (HUG), which unifies sequence and graph-based user modeling to takeadvantage of both methods.
Revised: In this work, we introduce a novel user representation, Heterogeneous User Graph (HUG), which unifies sequenceand graph-based user modeling to take advantage of both methods.

Original: An HUG is associated with two types of edges:sequential edges that preserve the sequential information and collaborative edgesthat store different item-item relationship.
Revised: A HUG is associated with two types of edges: sequential edges that preserve the order of interactions and collaborative edges that connect collaboratively similar items (i.e., items interacted by similar sets of users).

Original: To learnusers’ latent representationsfrom their HUGs, we propose a multi-head attention based architecture called Heterogeneous User Graph Transformer (HUGT).
Revised: To learn latent user representations for recommendation tasks, we propose a multi-head attention-based architecture called Heterogeneous User Graph Transformer (HUGT).

Original: HUGT is developed on thebasis of SASRec and can concurrently capture the sequential transition patternand complex graph topology.
Revised: HUGT is developed on the basis of SASRec and can concurrently capture the sequential pattern and graph topology encoded in HUGs.

Original: We conduct experiments on four real world datasetsfrom three different application domains.
Revised: We conduct experiments on four real-world datasets from three different application domains.

Original: Experimental results show that (1) jointlymodeling users as sequences and graphs with HUG provides better recommendationperformance over sequence-only and graph-only user modeling; (2) HUGT iseffective in learning user latent representations from HUGs; (3) HUGT outperformsthe baselines by up to 10% on datasets with long sequences and aligns with thestate-of-the-art performance on datasets with short sequences.
Revised: Experimental results show that (1) jointly modeling users as sequences and graphs with HUG provides better recommendation performance over sequence-only and graph-only user modeling; (2) HUGT is effective in learning user latent representations from HUGs; (3) HUGT outperforms the baselines by up to 10% on datasets with long sequences and aligns with the state-of-the-art performance on datasets with short sequences.

Original:  Introduction
Revised:  Introduction

Original: As internet services grow fast, users are facing a vast amount of information and products in theirdaily life.
Revised: As internet services grow fast, users are facing a vast amount of information and products in their daily life.

Original: Recommender systems play a critical role in filtering information and presenting relevantcontent that matches users’ interests.
Revised: Recommender systems play a critical role in filtering information and presenting relevant content that matches users’ interests.

Original: Due to the dynamic nature and complexity of users’ interests,learning patterns in users’ interaction history and encoding them into the users’ latent representationsfor recommendation have been a prevalent research topic.
Revised: Due to the dynamic nature and complexity of users’ interests, learning patterns in users’ interaction history and encoding them into the users’ latent representations for recommendation have been a prevalent research topic.

Original: There have been a surge of methods that model users by their sequences of item interactions andapply sequential models such as RNNs [8] and Transformers [20] to learn user representations[14, 7, 11, 18, 3].
Revised: There has been a surge of methods that model users by their sequences of item interactions and apply sequential models such as RNNs [9] and Transformers [22] to learn user representations [17, 8, 13, 20, 3].

Original: These methods demonstrated that encoding item transition patterns in users’latent representations helps capture users’ evolving interests and improves recommendation accuracy.
Revised: These methods demonstrated that encoding item transition patterns in users’ latent representations helps capture users’ evolving interests and improves recommendation accuracy.

Original: Another line of research advocates modelingeach users as a graph in which the user’s interacteditems are modelled as nodes, and  item-item relations(such as frequent co-occurrence, consecutiveclicks) are modelled as edges [25, 1, 23].
Revised: Another line of research advocates modeling users as user graphs in which the users’ interacted items are modeled as nodes, and certain item-item relations are modeled as edges [27, 1, 25].

Original: The initial features of the same items areshared acrossgraphs that contains the items. These methods usually apply graph neural networks (GNNs) on each user’s graph to encode relationships among the items into latent item embeddings and use a readoutmodule (e.g., average) to convert the item embeddings to a latent user representation.
Revised: The item relations are usually similarity or item co-occurrence patterns extracted from user-item interaction matrices or item meta-data. Graph neural networks (GNNs) [15, 5] have been a popular choice for encoding topology from such graphs into latent user representations. Most of the existing literature on recommender systems focuses on sequence-only or graph-only user modeling.

Revised: It is underexplored whether a joint sequence-graph user modeling provides better recommendation performance than a standalone-modality user modeling.

Original: In this work, we propose a method to link the two worlds by jointly modelling users as sequences andgraphs to encode both sequential patterns and graph topology.
Revised: In this work, we propose a method to link the two worlds by jointly modeling users as sequences and graphs to encode both sequential patterns and graph topology.

Original: An HUG is a graph of user interacteditems, in which nodes are connected by two types of edges: sequential edges that preserve thesequential ordering of items (e.g., chronological precedence) and collaborative edges that characterizeitem-item relationships such as collaborative similarity.
Revised: A HUG is a graph of userinteracted items, in which nodes are connected by two types of edges: sequential edges that preserve the sequential ordering of interactions (e.g., chronological precedence) and collaborative edges that characterize item-item collaborative similarity, i.e., a measure of similarity in regards users’ preference on the items.

Original: On the basis of SASRec [11], we design atransformer model to concurrently fuse and encode the sequential patterns and the graph topologyinto users’ latent  representations for recommendation.
Revised: On the basis of SASRec [13], we design a transformer model to concurrently fuse and encode the sequential patterns and the graph topology into latent user representations for recommendation.

Original: We call our method Heterogeneous User Graph Transformers (HUGT).
Revised: We call our method Heterogeneous User Graph Transformers (HUGT).

Original: We summarize our contributions as the following: First, we unify the sequence modeling and thegraph modeling of users with HUG, which provides a flexible framework of injecting inductive biasinto self-attention-based recommendation model.
Revised: We summarize our contributions as the following: First, we propose HUG, an effective data structure that jointly encapsulates sequential patterns and collaborative graph topology for user modeling.

Original: Secondly, we design HUGT to effeciently learnuser representations from their HUGs.
Revised: Secondly, we design HUGT to efficiently learn user representations from their HUGs.

Original: We propose two variants of HUGT: HUGT-SE which modelsgraph structure with spatial encoding, and HUGT-MP which computes a sparse attention matrixto simulate message passing in GNNs.
Revised: We propose two variants of HUGT: HUGT-SE which learns graph structure with spatial encoding, and HUGT-MP which computes a sparse attention matrix to simulate message passing in GNNs.

Original: Finally, we conduct comprehensive experiments on fourreal-world datasets from three different domains.
Revised: Finally, we conduct comprehensive experiments on four real-world datasets from three different domains.

Original: We show that jointly modeling users as sequencesand graphs via HUGs outperforms standalone sequence modeling or graph modeling of the users, HUGT is effective in concurrently fusing and encoding the sequence and graph aspects of HUG, and HUGT achieves state-of-the-art performance  onall the four  datasets.
Revised: We empirically demonstrate that jointly modeling users as sequences and graphs via HUGs outperforms standalone sequence modeling or graph modeling of the users, HUGT is effective in concurrently fusing and encoding the sequence and graph aspects of HUG, and HUGT achieves state-of-the-art performance in sequential recommendation on the four baseline datasets.

Original:  Related Works 
Revised:  Related Works 2.1 Sequence-based User Modeling

Original: 2.1 Sequence-based User Modeling

Original: Sequential signal is important in user modeling and can improve recommendation performance.
Revised: Sequential signal is important in user modeling and can improve recommendation performance.

Original: Traditional sequential recommendation models [16, 5, 19] usually rely on factorization methods (e.g.,
Revised: Traditional sequential recommendation models [18, 6, 21] usually rely on factorization methods (e.g.,

Original: Factorized Item Similarity Models [10]) to capture users’ general tastes and Markov Chains (MCs) tolearn the sequential signal.
Revised: Factorized Item Similarity Models [12]) to capture users’ general tastes and Markov Chains (MCs) to learn the sequential signal.

Original: This class of methods hold the Markov assumption that the next action isdependent on the previous few interactions and cannot capture complex and long sequential patterns.
Revised: This class of methods holds the Markov assumption that the next action is dependent on the previous few interactions and cannot capture complex and long sequential patterns.

Original: Moreover, the performance of these methods are limited by the capacity of the factorization models.
Revised: Moreover, the performance of these methods is limited by the capacity of the factorization models.

Original: In the past decade, researchers developed RNN-based approaches to model users’ interaction trajectories.
Revised: In the past decade, researchers developed RNN-based approaches to model users’ interaction trajectories.

Original: Hidasi et al.
Revised: Hidasi et al.

Original: [7] proposed GRU4Rec that leverages a variants of RNNs, gated recurrentunits (GRU), to capture sequential patterns in sessions.
Revised: [8] proposed GRU4Rec that leverages a variant of RNNs, gated recurrent units (GRU), to capture sequential patterns in sessions.

Original: Ma et al.
Revised: Ma et al.

Original: [14] successfully built real worldcommercial recommender systems based on hierarchical RNNs.
Revised: [17] successfully built real-world commercial recommender systems based on hierarchical RNNs.

Original: The authors of [27] combined RNNs with CNNs to capture long and short term sequential patterns, respectively.
Revised: The authors of [29] combined RNNs with CNNs to capture long- and short-term sequential patterns, respectively.

Original: Despite the successstories, RNN-based recommendation models suffer from data efficiency (they require a lot of trainingdata to perform well) and data sparsity issues.
Revised: Despite the success stories, RNN-based recommendation models suffer from data efficiency (they require a lot of training data to perform well) and data sparsity issues.

Original: 2.2 Graph-based User Modeling
Revised: 2.2 Graph-based User Modeling

Original: Graph-based user modeling is a rising direction that models users as graphs in which their interacted items are nodes.
Revised: Graph-based user modeling is a rising direction that models users as graphs in which their interacted items are nodes.

Original: Nodes in these graphs are usually connected via edges that indicatetransition/similarity relations [25, 1, 23].
Revised: Nodes in these graphs are usually connected via edges that indicate transition/similarity relations [27, 1, 25].

Original: Methods that fall into this class usually use graph neural networks (GNNs) to encode the graph structured information into item embeddings and use alearnable function to readout user embeddings for recommendation.
Revised: Methods that fall into this class usually use graph neural networks (GNNs) to encode the graph-structured information into item embeddings and use a learnable function to readout user embeddings for recommendation.

Original: Although this class of methodscaptures graph structured information, they usually overlook sequential patterns that are importantfor recommendation.
Revised: Although this class of methods can effectively capture graph topology, they usually overlook sequential patterns which are important for recommendation accuracy.

Original: 2.3 Transformer and Self-Attention Mechanism
Revised: 2.3 Transformer and Self-Attention Mechanism

Original: Self-attention mechanism and the Transformer architecture have demonstrated their superior abilityfor sequence modeling and have become the default methodology in natural language processing tasks[20, 4].
Revised: Self-attention mechanism and the Transformer architecture have demonstrated their superior ability for sequence modeling and have become the default methodology in natural language processing [Figure] tasks [22, 4].

Original: In  recommendation community,  self-attention mechanism is widely adopted for sequentialuser modeling [11, 18, 24, 9, 13].
Revised: In the recommendation community, the self-attention mechanism is widely adopted for sequential user modeling [13, 20, 26, 11, 16].

Original: He et al [6] propose to augment BERT4Rec by enforcing items insequences to attend toitems that are sequentially close  to reinforce the influence of local items while
Revised: He et al [7] propose to augment BERT4Rec by enforcing items in sequences to attend to sequentially close items to reinforce the influence of local items.

Original: Chen et al [2] propose to learn a sparse mask over the attention matrix to reduce the influence of[Figure] noisy items.
Revised: Chen et al [2] propose to learn a sparse mask over the attention matrix to reduce the influence of noisy items.

Original: In our work, we propose a formulation that allows  more flexible design of inductive biasin self-attention-based recommendation models.
Revised: In our work, we propose a formulation that allows a more flexible design of inductive bias in self-attention-based recommendation models.

Original: Recently, there has been discussions that connect Transformers with GNNs.
Revised: Recently, transformers have been applied to and have shown great potential in graph learning.

Original: In [22], the authors showthat self-attention mechanism is a special form of graph convolutions.
Revised: In [24], the authors show that the self-attention mechanism is a special form of graph convolutions.

Original: The authors of [28] generalizesthe definition of self-attention by directed graphs.
Revised: The authors of [30] generalize the definition of self-attention by directed graphs.

Original: Ying et al.
Revised: Ying et al.

Original: [26] successfully apply transformerson graph prediction tasks and outperform major GNN variants by augmenting transformers withgraph structural knowledge.
Revised: [28] successfully apply transformers on graph prediction tasks and outperform major GNN variants by augmenting transformers with graph structural knowledge.

Revised: Hu et al.

Revised: [10] applied transformers to learn from heterogeneous web-scale large graphs.

Original: These works motivate us to use Transformer as the backbone for toconcurrently learn sequential patterns and  graph topology in users’ behaviours.
Revised: These works motivate us to use Transformer as the backbone to concurrently learn sequential patterns and collaborative graph topology from HUGs.

Original:  Heterogeneous User Graph
Revised:  Heterogeneous User Graph

Original: To unify the sequence representation and the graph representation of a user, we first introducea datastructure, heterogeneous user graph, that has the capacity to store both sequence transition patternsand graph topology.
Revised: We introduce heterogeneous user graph (HUG), a data structure that stores both sequence patterns and graph topology for user modeling.

Original: We construct a heterogeneous user graph (HUG) G =
Revised: A HUG G =

Original: ( V , {E seq , E sim } )  from each user’s interactionsequence.
Revised: ( V , {E seq , E sim } ) is constructed from an user’s interaction sequence.

Original: We define the nodes V in the HUG as the interacted items in the sequence.
Revised: We define the nodes V in the HUG as the interacted items in the sequence.

Original: With theheterogeneity of HUGs, there could be various types of edges to encode different transition patterns.
Revised: With the heterogeneity of HUGs, there could be various types of edges to encode different transition patterns.

Original: In this paper, we construct two types of edges, sequential edges E seq and collaborative edges E sim , tocapture sequential signal and collaborative similarity, respectively.
Revised: In this paper, we construct two types of edges, sequential edges E seq and collaborative edges E sim , to capture sequential signal and collaborative similarity, respectively.

Original: Collaborative similarity measuresthe similarity between two items in terms of the two sets of users that interacted with them.

Original: We showthe construction process in Figure 1.

Original: Note that the collaborative edge can be easily generalized tocapture other types of item-item relation as well.

Original: Sequential Edges .
Revised: Sequential Edges .

Original: The sequential edges are to store the chronological order of each pair of items.
Revised: The sequential edges are to store the chronological order of each pair of items.

Original: For i, j ∈ V , ( i, j ) ∈ E seq if t i < t j where t i denotes the position of i in the sequence.
Revised: For i, j ∈ V , ( i, j ) ∈ E seq if t i < t j where t i denotes the position of i in the sequence.

Original: Collaborative Edges .
Revised: Collaborative Edges .

Original: Collaborative edgesare to encode collaborative similarity, a measure ofsimilarity in regards to the users’ preference patterns on the items.
Revised: Collaborative edges encode collaborative similarity, a measure of similarity in regard to the users’ preference patterns on the items.

Original: In order to measure collaborativesimilarities among items, wefirst conduct a truncated singular value decomposition (SVD) on theuser-item interaction matrix R ∈ { 0 , 1 } |U|×|I| to obtain a set of low-dimensional item embeddings V ∈ R d ×|I| .
Revised: In order to measure collaborative similarities among items, we conduct a truncated singular value decomposition (SVD) on the user-item interaction matrix R ∈ { 0 , 1 } |U|×|I| to obtain a set of low-dimensional item embeddings V ∈ R d ×|I| .

Original: We lookup the embedding of each item  i ∈ V and compute its cosine similarity withall its predecessors, i.e., { j | j ∈ V , t j ≤ t i } .
Revised: We compute the cosine similarity between the embedding of each item in the HUG ( i ∈ V ) with its predecessors { j | j ∈ V , t j ≤ t i } .

Original: Each item is connected to its k -nearest predecessors.
Revised: Each item is connected to its k -nearest predecessors.

Revised: We show the overall construction process in Figure 1.

Revised: Note that the collaborative edge can be easily generalized to capture other types of item-item relations as well.

Original:  Heterogeneous User Graph Transformer
Revised:  Heterogeneous User Graph Transformer

Original: In this section, we present heterogeneous user graph transformer (HUGT) that operates on the HUGto concurrently encode sequential signal and graph topology into latent user representations.
Revised: In this section, we present heterogeneous user graph transformer (HUGT) that learns from HUGs to concurrently encode sequential signal and graph topology into latent user representations.

Original: HUGTis built on the basis of a multi-head attention user modeling framework. We divide the multipleattention heads into two groups, sequential modeling heads and graph modeling heads, to encodethe two types of item-item relations, respectively. We encode the sequential information via relative concat ......
Revised: concat ......

Original: Input embeddings
Revised: Input embeddings

Original: Relative positional encoding
Revised: Relative positional encoding

Original: Pairwise dot-product
Revised: Pairwise dot-product

Original: Unidirectional causal mask
Revised: Unidirectional causal mask

Original: Q K V
Revised: Q K V

Original: MatMul
Revised: MatMul

Original: Scale +
Revised: Scale +

Original: Softmax
Revised: Softmax

Original: MatMul
Revised: MatMul

Original: Q K V
Revised: Q K V

Original: MatMul
Revised: MatMul

Original: Scale +
Revised: Scale +

Original: Softmax
Revised: Softmax

Original: MatMul SequentialModeling Head
Revised: MatMul SequentialModeling Head

Original: V K Q
Revised: V K Q

Original: MatMul
Revised: MatMul

Original: Scale +
Revised: Scale +

Original: Softmax
Revised: Softmax

Original: MatMul
Revised: MatMul

Original: Collaborative Edges
Revised: Collaborative Edges

Original: Shortest path (Floyd Warshall)
Revised: Shortest path (Floyd Warshall)

Original: OR
Revised: OR

Original: Adjacency matrix
Revised: Adjacency matrix

Original: Message Passing （ sparse attention ）
Revised: Message Passing （ sparse attention ）

Original: Graph Modeling Head
Revised: Graph Modeling Head

Original: MLP
Revised: MLP

Original: Sequential Edges
Revised: Sequential Edges

Original: Multihead Self-attention Block in HUGT xL
Revised: Multihead Self-attention Block in HUGT xL

Original: Spatial Encoding
Revised: Spatial Encoding

Original: HUGT-MP
Revised: HUGT-MP

Original: HUGT-SEpositional encoding. We propose two strategies, Message Passing (HUGT-MP) and Spatial Encoding(HUGT-SE), to learn graph topology stored in the collaborative edges. The message passing strategyiteratively aggregates information from each node’s graph neighbor while the spatial encoding methodpre-compute structural information from the graph and learn to leverage this information into attentionscores. Both groups of attention heads utilize the generalized self-attention mechanism.
Revised: HUGT-SE

Original: 4.1 Generalized Self-Attention Mechanism
Revised: 4.1 Generalized Self-Attention Mechanism

Original: In the standard self-attention,  item at each position aggregates weighted information from all itemsin the sequence to selectively build the context.
Revised: In the standard self-attention, the item at each position aggregates weighted information from all items in the sequence to selectively build the context.

Original: Generalized self-attention (GSA) mechanismgeneralizes the standard self-attention by allowing each item to attend to any subset of the positionsin the sequence and enlarges the design space of self-attention.
Revised: Generalized self-attention (GSA) mechanism generalizes the standard self-attention by allowing each item to attend to any subset of the positions in the sequence and enlarges the design space of self-attention.

Original: Let H =
Revised: Let H =

Original: [ h 1 , h 2 , · · · , h n ] ∈ R d × n be the latent representation of a sequence S from  previous layer,where n and d denotes the length of the sequence and the dimension, respectively.
Revised: [ h 1 , h 2 , · · · , h n ] ∈ R d × n be the latent representation of a sequence S from the previous layer, where n and d denotes the length of the sequence and the dimension, respectively.

Original: GSA computesthe representation of an item i as
Revised: GSA computes the representation of an item i as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where W q , W k , and W v are learnable parameters that map the input embedding to query, key, andvalue domain, respectively.
Revised: where W q , W k , and W v are learnable parameters that map the input embedding to query, key, and value domain, respectively.

Original: N ( i ) denotes the set of items that item i attends to.
Revised: N ( i ) denotes the set of items that item i attends to.

Original: When N ( i ) =
Revised: When N ( i ) =

Original: S , the
Revised: S , the

Original: GSA fully recovers the standard self-attention.
Revised: GSA fully recovers the standard self-attention.

Original: In unidirectional attention with causal masking (i.e.,each item can only attend to its precedents) used in SASRec [11], N ( i ) =
Revised: In unidirectional attention with causal masking (i.e., each item can only attend to its precedents) used in SASRec [13], N ( i ) =

Original: { j | t j < t i } .
Revised: { j | t j < t i } .

Original: .2 User 
Revised: 4.2 User Modeling with Multi-head Self-Attention

Original: Modeling with Multi-head Self Attention

Original: Embedding Layer.
Revised: Embedding Layer.

Original: We use a learnable embedding table E ∈ R
Revised: We use a learnable embedding table E ∈ R

Original: d ×|I| to convert discrete itemids [ i 1 , i 2 , · · · , i n ] to a matrix representation, X =
Revised: d ×|I| to convert discrete item ids [ i 1 , i 2 , · · · , i n ] to a matrix representation, X =

Original: [ e i 1 , e i 2 , · · · , e i n ] ∈ R d × n where d is thedimensionality of the embeddings and n is the maximum sequence length that the model can take.
Revised: [ e i 1 , e i 2 , · · · , e i n ] ∈ R d × n where d is the dimensionality of the embeddings and n is the maximum sequence length that the model can take.

Original: The latent representation is fed into a stack of self-attention blocks to generate user embeddings.
Revised: The latent representation is fed into a stack of self-attention blocks to generate user embeddings.

Original: Self-Attention Block.
Revised: Self-Attention Block.

Original: A self-attention block (SAB) consists of a generalized self-attention (GSA)layer that is introduced in Section 4.1 and a feed-forward layer (MLP) that adds non-linearity andconsiders interactions between latent dimensions, i.e.,
Revised: A self-attention block (SAB) consists of a generalized self-attention (GSA) layer that is introduced in Section 4.1 and a feed-forward layer (MLP) that adds non-linearity and considers interactions between latent dimensions, i.e.,

Original: [Equation]
Revised: [Equation]

Original: A SAB takes the output from the previous SAB as input, and passes the result to the next block, i.e.,
Revised: A SAB takes the output from the previous SAB as input, and passes the result to the next block, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where h ( l ) i denotes the output of the l -th SAB block, and h (0)
Revised: where h ( l ) i denotes the output of the l -th SAB block, and h (0)

Original: i =
Revised: i =

Original: e i is the output of the initialembedding layer.
Revised: e i is the output of the initial embedding layer.

Original: Following [11], we use layer normalization to accelerate training and residualconnections to preserve low-level information.
Revised: Following [13], we use layer normalization to accelerate training and residual connections to preserve low-level information.

Original: We stack L SABs and use the output embedding atthe last position of the last SAB as the final user representation for recommendation, i.e., h u = h ( L ) n .
Revised: We stack L SABs and use the output embedding at the last position of the last SAB as the final user representation for recommendation, i.e., h u = h ( L ) n .

Original: 4.3 Sequential Modeling with Relative Positional Encoding
Revised: 4.3 Sequential Modeling with Relative Positional Encoding

Original: In contrast to RNNs that rely on recurrent operations to capture sequential signals, Transformers usepositional encoding to break the order equivariance.
Revised: In contrast to RNNs that rely on recurrent operations to capture sequential signals, Transformers use positional encoding to break the order equivariance.

Original: We use an embedding table P ∈ R d × n to mapthe positions to continuous representations [ p 1 , p 2 , · · · , p n ] .
Revised: We use an embedding table P ∈ R d × n to map the positions to continuous representations [ p 1 , p 2 , · · · , p n ] .

Original: The positional embedding table couldbe either learnbale or pre-computed by a set of sinusoid functions.
Revised: The positional embedding table could be either learnable or pre-computed by a set of sinusoid functions.

Original: In this paper we follow [11] anduse learnable positional embeddings.
Revised: In this paper we follow [13] and use learnable positional embeddings.

Original: There are two major ways to incorporate the positional embeddings into self-attention, adding them tothe input item features at the corresponding positions (absolute positional encoding) [20] and addinga relative score computed by the two embeddings of the corresponding positions to the attention scorecomputed in Equation 1 (relative positional encoding)
Revised: There are two major ways to incorporate the positional embeddings into self-attention, adding them to the input item features at the corresponding positions (absolute positional encoding) [22] and adding a relative score computed by the two embeddings of the corresponding positions to the attention score computed in Equation 1 (relative positional encoding)

Original: [17], i.e.,
Revised: [19], i.e.,

Original: [Equation]
Revised: [Equation]

Original: where ψ ( · , · ) is a learnable function.
Revised: where ψ ( · , · ) is a learnable function.

Original: Empirically, we found that relative positional encoding outperforms absolute positional embedding in sequential recommendation tasks.
Revised: Empirically, we found that relative positional encoding outperforms absolute positional embedding in sequential recommendation tasks.

Original: 4.4 Graph Modeling
Revised: 4.4 Graph Modeling

Original: Message Passing.
Revised: Message Passing.

Original: Graph neural networks that rely on message passing along graph edges haveshown their ability to encode graph topology into latent node/graph embeddings.
Revised: Graph neural networks that rely on message passing along graph edges have shown their ability to encode graph topology into latent node/graph embeddings.

Original: Note that, Equation1 could be viewed as a message passing operation on a graph G =
Revised: Note that, Equation 1 could be viewed as a message passing operation on a graph G =

Original: ( V , E sim ) by setting i ∈ V and
Revised: ( V , E sim ) by setting i ∈ V and

Original: N ( i ) =
Revised: N ( i ) =

Original: { j | ( i,
Revised: { j | ( i,

Original: j ) ∈ E sim } .
Revised: j ) ∈ E sim } .

Original: In fact, Equation 1 is a special implementation of graph attention networks(GAT).
Revised: In fact, Equation 1 is a special implementation of graph attention networks (GAT).

Original: We skip the details of GAT and refer the readers to [21] for more information.
Revised: We skip the details of GAT and refer the readers to [23] for more information.

Original: In this paper, we directly apply GSA on the collaborative edges in HUGs as one way to encode graphstructural information.
Revised: In this paper, we directly apply GSA on the collaborative edges in HUGs as one way to encode graph structural information.

Original: With such formulation, each node adaptively aggregates information fromits graph neighbors and updates its representation.
Revised: With such formulation, each node adaptively aggregates information from its graph neighbors and updates its representation.

Original: The stacked SABs allows nodes to accumulateinformation that are several hops away and encode graph topology into their latent representations.
Revised: The stacked SABs allow nodes to accumulate information that is several hops away and encode graph topology into their latent representations.

Original: Graph Spatial Encoding.
Revised: Graph Spatial Encoding.

Original: The standard self-attention allows each item to attend to all items inthe sequence.
Revised: The standard self-attention allows each item to attend to all items in the sequence.

Original: This brings the advantage of a global receptive field, but also raises the challengeof encoding graph structural information into item representations.
Revised: This brings the advantage of a global receptive field, but also raises the challenge of encoding graph structural information into item representations.

Original: We tackle this challenge bypre-extracting graph related properties and learning to incorporate the properties into the computationof self-attention.
Revised: We tackle this challenge by pre-extracting graph-related properties and learning to incorporate the properties into the computation of self-attention.

Original: In this paper, we first compute the length of the shortest-path s ij ∈ Z + between each pair of nodesi, j ∈ V via collaborative edges to measure the spatial distance between the two nodes.
Revised: In this paper, we first compute the length of the shortest-path s ij ∈ Z + between each pair of nodes i, j ∈ V via collaborative edges to measure the spatial distance between the two nodes.

Original: We encodethis information via a learnable mapping ϕ : Z + (cid:55)→ R that converts the distance to a real scalarϕ ( s ij ) and directly add this value to the attention base, i.e., π ij =
Revised: We encode this information via a learnable mapping ϕ : Z + (cid:55)→ R that converts the distance to a real scalar ϕ ( s ij ) and directly adds this value to the attention base, i.e., π ij =

Original: q Ti k j / √ d + ϕ ( s ij ) .
Revised: q Ti k j / √ d + ϕ ( s ij ) .

Original: We clip theshortest-path length to an upper bound S , i.e.,  length larger than S will be set to S .
Revised: We clip the shortest-path length to an upper bound S , i.e., a length larger than S will be set to S .

Original: The clippingenables the model to generalize to unseen lengths during training, controls the model size, andprovides encoding for disconnected item pairs.
Revised: The clipping enables the model to generalize to unseen lengths during training, controls the model size, and provides encoding for disconnected item pairs.

Original: 4.5 Information Fusion with Multi-head Attention
Revised: 4.5 Information Fusion with Multi-head Attention

Original: Recall that in Equation 1, the input matrix H to each layer is mapped to a key K , query Q , andvalue V domain for the computation of self-attention.
Revised: Recall that in Equation 1, the input matrix H to each layer is mapped to a key K , query Q , and value V domain for the computation of self-attention.

Original: In a multi-head self-attention mechanism,
Revised: In a multi-head self-attention mechanism,

Original: H is mapped by multiple sets of projection matrices to several key-query-value spaces (each spacecorresponds to one attention head) to compute self-attention in parallel.
Revised: H is mapped by multiple sets of projection matrices to several key-query-value spaces (each space corresponds to one attention head) to compute self-attention in parallel.

Original: The resulting representationfrom all attention heads are concatenated and projected to be the final representation, i.e.,
Revised: The resulting representation from all attention heads is concatenated and projected to be the final representation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] where m is the number of heads, W o ∈ R d × ( d × m ) is a projection matrix,
Revised: (4) where m is the number of heads, W o ∈ R d × ( d × m ) is a projection matrix,

Original: h i is the output of the i -thattention head.
Revised: h i is the output of the i -th attention head.

Original: The multi-head formulation allows each item to aggregate information from different attention pathsin one attention block.
Revised: The multi-head formulation allows each item to aggregate information from different attention paths in one attention block.

Original: This provides flexibility to cope with the heterogeneity of HUGs.
Revised: This provides flexibility to cope with the heterogeneity of HUGs.

Original: In eachattention block, We separate the attention heads into two groups, the sequential modeling heads thatlearn from the sequential edges and the graph modeling heads which learn from the collaborativeedges.
Revised: In each attention block, We separate the attention heads into two groups, the sequential modeling heads which learn from the sequential edges, and the graph modeling heads which learn from the collaborative edges.

Original: The two groups of attention heads adopt different mechanisms to capture the two sourcesof information.
Revised: The two groups of attention heads adopt different mechanisms to capture the two sources of information.

Original: By stacking such multi-head attention blocks, we can effectively fuse and encodeinformation from the two types of edges into user representations.
Revised: By stacking such multi-head attention blocks, we can effectively fuse and encode information from the two types of edges into user representations.

Original: 4.6 Prediction and Model Training
Revised: 4.6 Prediction and Model Training

Original: With the learned user representation at position t , i.e. h tu , the final prediction ˆ y tui of an item i iscomputed by
Revised: With the learned user representation at position t , i.e. h tu , the final prediction ˆ y tui of an item i is computed by

Original: [Equation]
Revised: [Equation]

Original:  where w i and b i are decoder weight and bias for item i .
Revised: 5) where w i and b i are decoder weight and bias for item i .

Original: We notice that the model generalizes betterby sharing weights of the input embedding table with the item decoder.
Revised: We notice that the model generalizes better by sharing weights of the input embedding table with the item decoder.

Original: We train the model by optimizing the cross-entropy loss
Revised: We train the model by optimizing the cross-entropy loss

Original: [Equation]
Revised: [Equation]

Original: where B is a batch of users and C tu is a set of candidate items for user u at t .
Revised: where B is a batch of users and C tu is a set of candidate items for user u at t .

Original: n is the maximum lengthof the sequence.
Revised: n is the maximum length of the sequence.

Original: 4.7 Complexity Analysis
Revised: 4.7 Complexity Analysis

Original: Let G be an user HUG with n nodes.
Revised: Let G be an user HUG with n nodes.

Original: The time and space complexity of a m -head HUGT is thesame with the standard Transformer, i.e., O ( mn 2 ) .
Revised: The time and space complexity of a m -head HUGT is the same as the standard Transformer, i.e., O ( mn 2 ) .

Original: In the pre-processing steps, we apply a truncated
Revised: In the pre-processing steps, we apply a truncated

Original: SVD on a sparse user-item interaction matrix which costs O (( |U| + |I| )
Revised: SVD on a sparse user-item interaction matrix which costs O (( |U| + |I| )

Original: d 2 + Td ) where T is thecost of a sparse matrix-vector multiplication and d is the dimension.
Revised: d 2 + Td ) where T is the cost of a sparse matrix-vector multiplication and d is the dimension.

Original: When applying HUGT-SE,pre-computing the shortest path between each pair of nodes costs O ( n 3 ) .
Revised: When applying HUGT-SE, pre-computing the shortest path between each pair of nodes costs O ( n 3 ) .

Original: Since these steps only needto be applied once and could be computed in parallel, they are not efficiency barrier to our method.
Revised: Since these steps only need to be applied once and could be computed in parallel, they are not efficiency barriers to our method

Original:  Experiments
Revised: . Experiments

Original: In this section, we carefully design experiments to investigate three research questions in regards ofthe heterogeneous user graph (HUG) modeling of users and the heterogeneous user graph transformers(HUGT):
Revised: In this section, we carefully design experiments to investigate three research questions in regard of the HUG modeling of users and the HUGT:

Original: • RQ1 : How does HUGT perform in sequential recommendation compared to the state-of- the-art methods
Revised: • RQ1 : How does HUGT perform in sequential recommendation compared to the state-ofthe-art methods?

Original: • RQ2 : Is jointly modeling users as graphs and sequences better than a standalone sequence/graph user model?
Revised: • RQ2 : Is jointly modeling users as graphs and sequences better than a standalone sequence/graph user model?

Original: • RQ3 : Is a concurrent sequence-graph model better than modeling a user as  graph first andthen a sequence?
Revised: • RQ3 : Is a concurrent sequence-graph model better than modeling a user as a graph first and then a sequence?

Original: 5.1 Experimental Setup 
Revised: 5.1 Experimental Setup 5.1.1 Datasets

Original: 5.1.1 Datasets

Original: We experiment with four benchmark datasets, ML1M , Netflix , Steam , Taobao , that fall into threedifferent application domains, movie recommendation, game recommendation, and E-commerce.
Revised: We experiment with four benchmark datasets, ML1M , Netflix , Steam , Taobao , that fall into three different application domains, movie recommendation, game recommendation, and E-commerce.

Original: Thedatasets are different in regards their size, shape, density, and sequence length.
Revised: The datasets are different in regard to their size, shape, density, and sequence length.

Original: The detailed statisticsof the datasets are shown in Table 1.
Revised: The detailed statistics of the datasets are shown in Table 1.

Original: For all the datasets, we convert the ratings/reviews/interactionsto implicit feedback (i.e., existing entries are assumed positive while missing entries are treated asnegative signal).
Revised: For all the datasets, we convert the ratings/reviews/interactions to implicit feedback (i.e., existing entries are assumed positive while missing entries are treated as negative signals). [Table]

Original: [Table]
Revised: [Table]

Original: 5.1.2 Baselines and Implementation Details
Revised: 5.1.2 Baselines and Implementation Details

Original: We compare the performance of HUGT with state-of-the-art sequential recommendation baselines that fall into the categories of factorization-based methods ( Fossil [5]), RNN-based methods( GRU4Rec [7]), GNN-based methods ( SR-GNN [25]), and Self-Attention-based methods ( FISSA[13], BERT4Rec [18], and SASRec [11]).
Revised: We compare the performance of HUGT with state-of-the-art sequential recommendation baselines that fall into the categories of factorization-based methods ( Fossil [6]), RNN-based methods ( GRU4Rec [8]), GNN-based methods ( SR-GNN [27]), and Self-Attention-based methods ( FISSA [16], BERT4Rec [20], and SASRec [13]).

Original: For Fossil 1 and FISSA 2 , we use the implementation associated with the original paper.
Revised: For Fossil 1 and FISSA 2 , we use the implementation associated with the original paper.

Original: For GRU4Rec,we use the implementation from [14], and we use an PyTorch implementation of BERT4Rec 3 .
Revised: For GRU4Rec, we use the implementation from [17], and we use a PyTorch implementation of BERT4Rec 3 .

Original: Weimplement SASRec using PyTorch an we find that our implementation greatly outperforms the original
Revised: We implement SASRec using PyTorch an we find that our implementation greatly outperforms the original

Original: SASRec implementation.
Revised: SASRec implementation.

Original: In our experiments, we select the hidden dimensions from { 50 , 300 } .
Revised: In our experiments, we select the hidden dimensions from { 50 , 300 } .

Original: For
Revised: For

Original: Markov chain(MC)-based methods, we consider the MC order from { 1 , 2 , 3 , 4 , 5 } .
Revised: Markov chain(MC)-based methods, we consider the MC order from { 1 , 2 , 3 , 4 , 5 } .

Original: For attentionrelated methods such as SASRec and BERT4Rec, we choose the number of attention heads from1 , 2 , 3 , 4  .
Revised: For attentionrelated methods such as SASRec and BERT4Rec, we choose the number of attention heads from { 1 , 2 , 3 , 4 } .

Original: For batch size and optimization related hyper-parameters, we follow the instructions fromthe original paper if provided.
Revised: For batch size and optimization related hyper-parameters, we follow the instructions from the original paper if provided.

Original: Otherwise, we set the batch size to 128 and optimize the model usingthe Adam [12] optimizer with a learning rate of 0 . 001 .
Revised: Otherwise, we set the batch size to 128 and optimize the model using the Adam [14] optimizer with a learning rate of 0 . 001 .

Original: We implement our HUGT model using PyTorch[15].

Original: The maximum length for each dataset is setto be able to cover the average sequence lengths of the dataset.
Revised: The maximum length for each dataset is set to be able to cover the average sequence lengths of the dataset.

Original: We choose 200 for ML1M, 100 for Netflix and Taobao, and 50 for Steam.
Revised: We choose 200 for ML1M, 100 for Netflix and Taobao, and 50 for Steam.

Original: For constructing collaborative edges in HUG, we considerthe k value from { 5 , 10 , 20 , 30 , 50 , 70 } for ML1M, Netflix and Taobao.
Revised: For constructing collaborative edges in HUG, we consider the k value from { 5 , 10 , 20 , 30 , 50 , 70 } for ML1M, Netflix, and Taobao.

Original: For the Steam dataset, weonly consider { 2 , 3 } since its average sequence length is 5.
Revised: For the Steam dataset, we only consider { 2 , 3 } since its average sequence length is 5.

Original: When computing the ranking objective in Equation 6, we use all the items as candidate items for ML1M, Netflix, and Steam.
Revised: When computing the ranking objective in Equation 6, we use all the items as candidate items for ML1M, Netflix, and Steam.

Original: Since the Taobaodataset has a large number of items, we compute the loss on a randomly selected 0 .
Revised: Since the Taobao dataset has a large number of items, we compute the loss on a randomly selected 0 .

Original: 5 (cid:112) |I| candidateitems to reduce GPU memory consumption.
Revised: 5 (cid:112) |I| candidate items to reduce GPU memory consumption.

Original: We use the Adam algorithm to optimize the parametersof HUGT with a learning rate of 0 . 001 .
Revised: We use the Adam algorithm to optimize the parameters of HUGT with a learning rate of 0 . 001 .

Original: We set the batch size for ML1M, Netflix,  Steam to 128, and Taobao to 32.
Revised: We set the batch size for ML1M, Netflix, and Steam to 128, and Taobao to 32.

Original: In our experiments, we set the number of attention heads to be 4 and consider differentcombinations of sequence heads and graph heads.
Revised: In our experiments, we set the number of attention heads to be 4 and consider different combinations of sequence heads and graph heads.

Original: 5.2 Experimental Settings
Revised: 5.2 Experimental Settings

Original: We use the first 90% of the interactions of each user as  training set and the last 10% of the interactionsfor evaluation purpose.
Revised: We use the first 90% of the interactions of each user as the training set and the last 10% of the interactions for evaluation purposes.

Original: We follow an auto-regressive manner in evaluating the recommendationmodels, i.e., we compute a user’s state/representation from her current interaction sequence, makea prediction (a ranked list of items) for the next interaction, evaluate the recommendation list withthe ground truth item, append the ground truth item to the tail of her sequence, and update the user’sstate/representation for next prediction.
Revised: We follow an auto-regressive manner in evaluating the recommendation models, i.e., we compute a user’s state/representation from her current interaction sequence, make a prediction (a ranked list of items) for the next interaction, evaluate the recommendation list with the ground truth item, append the ground truth item to the tail of her sequence, and update the user’s state/representation for next prediction.

Original: We validate on 30% of the users for model selection andearly stopping and report test performance on the remaining 70% of the users.
Revised: We validate on 30% of the users for model selection and early stopping and report test performance on the remaining 70% of the users.

Original: We train HUGT and all the baselines on the training set and evaluate their performance on thevalidation set after each training epoch.
Revised: We train HUGT and all the baselines on the training set and evaluate their performance on the validation set after each training epoch.

Original: We stop training if the validation score does not decreasein ten epochs or the number of training epochs reaches 100.
Revised: We stop training if the validation score does not decrease in ten epochs or the number of training epochs reaches 100.

Original: We report the test performance of thecheckpoint that performs the best on the validation set.
Revised: We report the test performance of the checkpoint that performs the best on the validation set.

Original: 5.2.1 Evaluation Metrics
Revised: 5.2.1 Evaluation Metrics

Original: In this paper, we use HR@ k and NDCG@ k to evaluate the performance of the models.
Revised: In this paper, we use HR@ k and NDCG@ k to evaluate the performance of the models.

Original: HR@ k and
Revised: HR@ k and

Original: NDCG@ k compute the proportion of cases when the ground truth item is in the top- k prediction [Table]
Revised: NDCG@ k compute the proportion of cases when the ground truth item is in the top- k prediction list.

Original: list. HR equally weighs the hit items while NDCG weighs the hit items decreasingly by theircorresponding rank in the top- k list.
Revised: HR equally weighs the hit items while NDCG weighs the hit items decreasingly by their corresponding rank in the top- k list.

Original: 5.3 Recommendation Performance (RQ1)
Revised: 5.3 Recommendation Performance (RQ1)

Original: [Table]
Revised: [Table]

Original: We compare recommendation performance ofthe proposed HUGT with the state-of-the-art sequential recommendation methods on the fourbenchmark datasets.
Revised: We compare recommendation performance of the proposed HUGT with the state-of-the-art sequential recommendation methods on the four benchmark datasets.

Original: The overall performanceis shown in Table 2.
Revised: The overall performance is shown in Table 2.

Original: We observe that our proposed HUGT significantly outperforms all thebaselines on ML1M, Netflix, and Taobao whileits performance on Steam is similar to SASRec.
Revised: We observe that our proposed HUGT significantly outperforms all the baselines on ML1M, Netflix, and Taobao while its performance on Steam is similar to SASRec.

Original: This provides evidence that the collaborativegraph topology is helpful for user modeling andour HUGT can effectively capture this signal toimprove recommendation performance.
Revised: This provides evidence that the collaborative graph topology is helpful for user modeling and our HUGT can effectively capture this signal to improve recommendation performance.

Original: Notethat, the ML1M, Netflix, and Taobao datasethave an average sequence length of 165, 83,and 98, respectively, while the average lengthof Steam is 5.
Revised: Note that, the ML1M, Netflix, and Taobao datasets have an average sequence length of 165, 83, and 98, respectively, while the average length of Steam is 5.

Original: This shows that our proposedmethod works well in settings where the interaction sequences are long.
Revised: This shows that our proposed method works well in settings where the interaction sequences are long.

Original: When the sequencesare in general short, our method performs on parwith SASRec.
Revised: When the sequences are generally short, our method performs on par with SASRec.

Original: We observe that the two graph learning schemes,message passing (MP) and spatial encoding (SE), show advantages in different datasets.
Revised: We observe that the two graph learning schemes, message passing (MP) and spatial encoding (SE), show advantages in different datasets.

Original: Thisdemonstrates that some datasets could benefit from the global receptive field of the SE method whileothers may enjoy the local structure of the MP scheme.
Revised: This demonstrates that some datasets could benefit from the global receptive field of the SE method while others may enjoy the local structure of the MP scheme.

Original: 5.4 HUG vs. Standalone User Modeling (RQ2)
Revised: 5.4 HUG vs. Standalone User Modeling (RQ2)

Original: In this section, we design experiments to investigate the necessity of the unified sequence-graph userrepresentation and the graph learning scheme.
Revised: In this section, we design experiments to investigate the necessity of the unified sequence-graph user representation and the graph learning scheme.

Original: For each dataset, we train five HUGT-MP models thateach has four attention heads.
Revised: For each dataset, we train five HUGT-MP models that each has four attention heads.

Original: The five HUGT-MP models have a number of sequential heads rangingfrom zero to four while  rest of the heads are graph heads.
Revised: The five HUGT-MP models have a number of sequential heads ranging from zero to four while the rest of the heads are graph heads.

Original: We show the recommendation performanceof the models in Figure 3.
Revised: We show the recommendation performance of the models in Figure 3.

Original: We observe a performance drop on all the datasets when we set the numberof sequence heads tobe zero.
Revised: We observe a performance drop on all the datasets when we set the number of sequence heads to zero.

Original: This shows the importance of sequence pattern in user modeling.
Revised: This shows the importance of sequence patterns in user modeling.

Original: On the three datasets with long sequences (ML1M, Netflix, Taobao), we observe a performance dropwhen we set the number of graph heads to zero.
Revised: On the three datasets with long sequences (ML1M, Netflix, Taobao), we observe a performance drop when we set the number of graph heads to zero.

Original: This demonstrates that the collaborative graphtopology is important in a long sequence setting and the graph learning schemes that we proposedis effective in capturing this information.
Revised: This demonstrates that the collaborative graph topology is important in a long sequence setting and the graph learning schemes that we proposed are effective in capturing this information.

Original: On the Steam dataset that has short sequence lengths,the recommendation performance is not influenced by cutting the graph heads.
Revised: On the Steam dataset that has short sequence lengths, the recommendation performance is not influenced by cutting the graph heads.

Original: This suggests thatthe transition patterns in short sequence settings are not very sophisticated and could be very wellcaptured by  naive sequential model.
Revised: This suggests that the transition patterns in short sequence settings are not very sophisticated and could be very well captured by a naive sequential model.

Original: 5.5 Importance of Concurrent Modeling (RQ3)
Revised: 5.5 Importance of Concurrent Modeling (RQ3)

Original: In a HUGT block, to incorporate the heterogeneity of HUG, we split multiple attention heads tosequence heads and graph heads to concurrently encode sequential pattern and graph topology.
Revised: In a HUGT block, to incorporate the heterogeneity of HUG, we split multiple attention heads to sequence heads and graph heads to concurrently encode sequential patterns and graph topology.

Original: Inthis section, we conduct experiments to answer the question that whether this formulation is betterthan its asynchronous alternatives.
Revised: In this section, we conduct experiments to answer the question that whether this formulation is better than its asynchronous alternatives.

Original: We design two architectures that process the sequence headsand the graph heads sequentially.
Revised: We design two architectures that process the sequence heads and the graph heads sequentially.

Original: One that applies an HUGT with only graph heads on the HUG toencode only graph topology into the item embeddings, then applies an HUGT with only sequenceheads on the HUG whose initial node embeddings are the output of the previous graph-only HUGTto mix graph topology and sequential patterns.
Revised: One that applies an HUGT with only graph heads on the HUG to encode only graph topology into the item embeddings then applies an HUGT with only sequence heads on the HUG whose initial node embeddings are the output of the previous graph-only HUGT to mix graph topology and sequential patterns.

Original: The second architecture is similar but applies asequentce-only HUGT first and then a graph-only HUGT.
Revised: The second architecture is similar but applies a sequence-only HUGT first and then a graph-only HUGT.

Original: The recommendation performance of thedefault HUGT and the two alternative architectures are shown in Table 3.
Revised: The recommendation performance of the default HUGT and the two alternative architectures are shown in Table 3.

Original: We observe that on thethree datasets with long sequences (ML1M, Netflix, Taobao), concurrent modeling of sequencesand graphs outperforms the asynchronous alternatives.
Revised: We observe that on the three datasets with long sequences (ML1M, Netflix, Taobao), concurrent modeling of sequences and graphs outperforms the asynchronous alternatives.

Original: This demonstrates the effectiveness of theheterogeneous graph modeling of users and the proposed Transformers.
Revised: This demonstrates the effectiveness of the heterogeneous graph modeling of users and the proposed Transformers.

Original: On the Steam dataset, thereis no significant performance difference between the three architectures.
Revised: On the Steam dataset, there is no significant performance difference between the three architectures.

Original:  Conclusion
Revised:  Conclusion

Original: We introduce a novel heterogeneous user graph (HUG) that jointly models users as sequences andgraphs to encode item transition patterns and complex item relations.
Revised: We introduce a novel heterogeneous user graph (HUG) that jointly models users as sequences and graphs to encode item transition patterns and complex item relations.

Original: To leverage the item transitionpatterns and graph topology stored in HUGs for recommendation tasks, we develop heterogeneoususer graph transformer (HUGT) that extends the standard self-attention for concurrent sequencelearning and graph learning.
Revised: To leverage the item transition patterns and graph topology stored in HUGs for recommendation tasks, we develop heterogeneous user graph transformer (HUGT) that extends the standard self-attention for concurrent sequence learning and graph learning.

Original: Experimental results show that the unified sequence-graph user modelingis better than either sequence-only modeling or graph-only modeling and HUGT achieves state-ofthe-art performance on all benchmarks.
Revised: Experimental results show that the unified sequence-graph user modeling is better than either sequence-only modeling or graph-only modeling and HUGT achieves state-ofthe-art performance on all benchmarks.

Original: HUG and HUGT provide a flexible framework for future research to incorporate complex itemrelations into user modeling.
Revised: HUG and HUGT provide a flexible framework for future research to incorporate complex item relations into user modeling.

Original: A valuable direction to explore is the choice of graphs for user modeling.
Revised: A valuable direction to explore is the choice of graphs for user modeling.

Original: We demonstrated that item collaborative similarity graph could provide extra context information foruser modeling.
Revised: We demonstrated that the collaborative similarity graph could provide extra context information for user modeling.

Original: There remains a lot of choices such as item temporal graph, or item feature similaritygraph.
Revised: There remain a lot of choices such as item temporal graph, or item feature similarity graph.

Original: We leave them as future works.
Revised: We leave them as future works.