Graph Convolution Network based Recommender Systems: Learning Guarantee and Item Mixture Powered Strategy
Abstract
Inspired by their powerful representation ability on graph-structured data, Graph
Convolution Networks (GCNs) have been widely applied to recommender systems,and have shown superior performance.
Despite their empirical success, there isa lack of theoretical explorations such as generalization properties.
In this paper,we take a first step towards establishing a generalization guarantee for GCN-basedrecommendation models under inductive and transductive learning.
We mainlyinvestigate the roles of graph normalization and non-linear activation, providingsome theoretical understanding, and construct extensive experiments to furtherverify these findings empirically.
Furthermore, based on the proven generalizationbound and the challenge of existing models in discrete data learning, we propose
Item Mixture (IMix) to enhance recommendation.
It models discrete spaces in acontinuous manner by mixing the embeddings of positive-negative item pairs, andits effectiveness can be strictly guaranteed from empirical and theoretical aspects.
Introduction
With the explosive growth of online information, Internet users rely on recommender systems toalleviate such information overload [1, 2, 3].
In order to take full advantage of the rich graphstructure with high-hop neighbors in recommender systems, recent studies have begun to apply Graph
Convolution Networks (GCNs) [4] to recommender systems.
As expected, GCN-based models haveemerged as one of the most attractive approaches to recommender systems due to their powerfulgraph representation abilities.
However, these works only demonstrate the effectiveness of GCNsempirically and do not provide theoretical guarantees.
GCN-based Recommendation.
[5] was the first to introduce GCN into recommender systems.
Specifically, GCN in their approach was applied to the user-user and item-item graphs to learnauxiliary information, respectively.
GC-MC [6] proposed to apply GCN to the user-item interactiongraph to learn the relations between users and items.
SpectralCF [7] directly discovered all possibleconnectivity between users and items from the spectral domain of the user-item graph.
However,
SpectralCF fails to capture the information in high order.
NGCF [8] further proposed employing the element-wise product between users and embeddings to augment the item features that userscare about and the user features that items are selected by.
Some works [8, 9, 10, 11, 12] observedthat removing the non-linear activation not only simplifies the update operation, but also retains orenhances the recommendation performance.
LightGCN [9] further removed the feature transformationto simplify the design of GCN.
However, these changes only depend on the empirical results, withouttheoretical guarantees.
In addition to the user-item interaction matrix, there exist studies combiningthe social network [13] and knowledge graph [14] to enhance the representations of users and items.
In this paper, we mainly consider the user-item interaction matrix, i.e. bipartite graph.
Generalization Analysis for GNNs.
Due to the numerical success [15, 5, 16], several theoreticalstudies have been devoted to establishing generalization guarantees for GNNs.
In inductive learning,[17] studied the generalization capability of GNNs by the VC-dimension, which is polynomial w.r.t.the number of parameters and the number of nodes in the graph.
[18] provided the first data-dependentbounds via Rademacher Complexity for message passing GNNs and did not allow the analysis ofgraph information.
Subsequently, [19] derived a tighter bound for GNNs based on the PAC-Bayesframework.
In transductive learning, [20] analyzed the uniform stability of (single layer)
GCNs andthereby deriving their generalization guarantees.
They proved that the graph filter, which makes thelargest absolute eigenvalue independent of the graph size, satisfies this uniform stability criterion.[21] derived a Rademacher complexity based bound for two layers GCN model.
[22] provided amore informative bound according to the relations between normalized graph and input feature atthe first layer.
However, these works mentioned above only focused on node or graph classificationtasks, different from recommendations.
The recent works [23, 24] have studied the link predictionand graph embedding tasks; however, they did not concern with the specific models.
Contributions and paper structure.
The main contributions are shown as follows:
1) We quantify the model complexity for GCN-based recommender systems on the bipartite graph
via l ∞ -Covering number.
And then, in Section 3, we relate the Covering number to the generalizationerror under inductive and transductive settings, leading to the generalization guarantees.
2) Motivated by the proven bound that generalization needs to compromise empirical risk and model
complexity and the fact that the recommendation model overfits the sampled discrete data, we propose
a simple but effective augmentation strategy, Item mixture (IMix) in Section 4.
It makes discretesample space continuous by mixing positive-negative item pairs about the same user.
Notably, we
theoretically demonstrate that it is beneficial for recommendation generalization.
In Section 5, extensive evaluations are conducted
on two datasets, Gowalla and Yelp2018.
First, toconfirm the previous theoretical foundation, we present the corresponding numerical results, mainlyincluding graph normalization and non-linear activation.
Second, we validate the effectiveness ofthe proposed IMix on different models, and the experimental results consistently show that therecommendation performance is significantly improved after configuring IMix.
We conclude in Section 6.
All proofs and more experimental results are provided in the appendix.
Preliminary
In this section, we first formulate the problem of collaborative filtering (CF) based recommendersystem.
We then describe the GCN-based CF models in detail.
2.1 Problem Formulation
We use u ∈ U and i ∈ I to denote the user and item, and use R = { ( u, i ) | u ∈ U , i ∈ I} to denotethe collected user-item interactions.
Let Z = { ( u, i ) | u ∈ U , i ∈ I} , D to be a fixed but unknowndistribution over Z ×Z ×{− 1 , +1 } .
For any triplet ( z, z ′ , y ) ∼ D , y =
+1 if the preference betweenuser-item pair z is higher than z ′ , otherwise y = − 1 .
Let f ∈ F : U
× I → R be a recommendersystem, the predicted preference score f ( z ) for any user-item pair z ∈
Z depend on the learneduser embedding e u and item embedding e i .
In this paper, we consider f ( z ) =
e Tu e i and a pairwiseranking loss for optimization.
For any ( z, z ′ , y ) ∼ D , the loss function is formalized as followsloss =
l ( y ( f ( z ) − f ( z ′ ))) .
2.2 GCN-based Recommendation
Let x u , x i ∈
R
d be the user and item feature, where d is the input dimension.
Without loss ofgenerality, we fix the collected user-item interactions as matrix R ∈ R |U|×|I| , where each entry
R ui = 1 if user u has interacted with item i .
Let A and ˜ A to be the adjacency matrix the user-itemgraph and the corresponding laplacian matrix, which are both block anti-diagonal matrices since thegraph is bipartite.
We formalize these two matrices as follows.
[Equation]
Following [25], the existing GCN-based collaborative filtering models can be generalized into twosteps.
Note that we omit the update formula of items for the sake of brevity.(1) Update operation.
The most straightforward layer-wise updations are inspired by the GNNtechniques, and they can be summarized as follows,
[Equation]
where ˜ a ui =
[ (cid:101) R u ] ui represents the assigned weights of user u to item i .
Some recent works [8]further adopt the element-wise product between nodes with their neighbors to leverage the consistencybetween user preferences and item attributes.
This operation is formulated as,
[Equation]
In this paper, we focus on the more complex operation defined in Eq.
2 since the Eq.
1 can be seen asa special case of it when W l 2 = O .(2)
Final node representation.
The CF-based recommender systems require the representation ofusers and items for the final prediction task.
The mainstream approaches include linear combinationand concatenation, respectively defined as,
[Equation]
combination:
[Equation]
where α l is a learnable parameter or hyper-parameter, L is the network depth.
In this paper, forconvenience, we only consider the embeddings of the last layer as the final representation, i.e.,e u = h
Lu .
We will prove that the generalization analysis can be easily extended to other integrationoperations in appendix.
Generalization for GCN-based Recommender Systems
We first investigate the model complexity of GCN-based recommender system via Covering numberwith specific radius in Section 3.1.
Then we link the Covering number and generalization errors forinductive and transductive learning in Section 3.2, which finally yield the meaningful bounds.
Assumptions.
To establish the generalization guarantee, let us immediately make some mildassumptions that are easy to implement.
For the loss function, assuming l : R → [ − B, B ] isany bounded C l - lipschitz continuous function.
For the non-linear activation function, we assume ϕis C ϕ - lipschitz continuous with some C ϕ > 0 , ϕ (0)
= 0 .
For the weight matrices, we assume that
W l 1 , W l 2 ∈ R
d × d are shared across layers, and have bounded norms ∥ W l 1 ∥ 2 ≤ B 1 and ∥ W l 2 ∥ 2 ≤ B 2 .
For the input feature, we assume max u ∈U ∥
x u ∥ 2 ≤ B u and max i ∈I ∥
x i ∥
≤ B i .
3.1 Covering Number for GCN-based Recommender Systems
In classical learning theory, the complexity of function class is closely related to the generalizationability [26], and the typical complexity measurements are VC-Dimension [27] and Rademachercomplexity [28].
In this paper, we adopt the Covering number [29] to investigate the model complexityof GCN-based recommender systems.
Based on the properties of bipartite graphs, we study thecomplexity by combining the recommendations of GCN-based models for any user-item pairs.
Lemma 1 (Covering number bound) .
Under Assumptions, we further assume that ∥ h lu ∥ ∞ , ∥ h li ∥ ∞ ≤b for any u ∈ U , i ∈ I , and l =
[ L ] , let γ = ∥ (cid:101)
A ∥ ∞ .
Given a sample set S with size n , the Coveringnumber of F over S with specific ϵ is bounded as
[Equation]
1 + 4 γbMB√ nd ϵ (cid:33) .
Moreover, when ϵ ≤ 4 M √ nd max { ( γ + 1) B 1 , γbB 2 } ,
[Equation]
, where
[Equation]
Remark 1.
The key idea of the proof is to exploit the properties of the bipartite graph in therecommender system.
To investigate the complexity of the recommendation for any user-item pairgenerated by GCN-based models, we establish the connection between the complexities of functionspace and weight space.
Here C describes the layer-wise update complexity, which is the importantterm for the following generalization analysis.
According to the norm bound of hidden states, theassumption can be naturally satisfied when the activation function is bounded.
And we will discussthe role of unbounded activation in Section 3.
Note that the layer-wise complexity reduces to
C = C ϕ ( γ + 1) B 1 when ignoring element-wise between users and items, which means that there isno need to discuss the value of b .
3.2 Generalization Bound for Inductive & Transductive Learning
Transductive and inductive learning are both commonly used in GNNs and recommender systems.
The primary difference between these two settings is whether the testing set depends on the trainingset.
The testing samples in inductive learning are drawn from some unknown distribution, while intransductive learning, they are sampled without replacement from a fixed data set.
In the following,we first relate the generalization error for two settings to the Covering number of GCN-basedrecommendations, then derive the final generalization guarantees using Lemma 1.
Inductive Learning.
Given a training set S m = { ( z i , z ′
i , y i ) } mi =1 , we assume that all samples in S mare i.i.d.
according to D , denoted as S m ∼ D m , and we aim to make accurate prediction for anyunobserved samples ( z, z ′ , y ) ∼ D .
Under this inductive setting, the empirical error over the trainingset S m and the corresponding generalization error is defined as follows,
Empirical error:
[Equation]
Under inductive learning, we can derive the following generalization error bound by the complexityterm according to the Covering number.
Lemma 2.
Let F be a real-valued function class taking values in [ − e, e ] , and assume that 0 ∈ F .
Under assumptions, for any function f in a class F , with probability of at least 1 − δ over an i.i.d.size- m training set, we have
[Equation]
Transductive Learning.
For the transductive learning setting, the full samples are observed prior tolearning.
Given a fixed set S
= { ( z i , z ′
i , y i ) } m + u i =1 , a labeled training set S m with size m is selectedfrom S uniformly at random.
The goal of transductive learning is to predict the labels of the samplesin testing set S u = S \ S m .
Under this transductive setting, the formulation of empirical error ˆ L m ( f )is similar to Eq. 3, and the generalization error is defined as follows,
[Equation]
Following the same strategy adopted in proving Lemma 2, we again need to perform a generalizationbound under transductive setting via Covering number of function class.
As shown above, thegeneralization and empirical error under transductive setting do not depend on any underlyingdistributions.
The important difference induces two versions of generalization analysis.
For thetransductive learning, we consider the transductive Rademacher complexity [30] defined on both thetraining and testing data.
Then the Covering number bounded generalization error can be derived bythe interdependence between model complexity measurements [31, 32].
Lemma 3.
Let F be a real-valued function class taking values in [ − e, e ] , and assume that 0 ∈ F .
Let Q 1 = 1 u + 1 m , Q 2 = m + u ( m + u − 1 / 2)(1 − 1 / 2(max( m,u ))) and c 0 < 5 . 05 .
Under assumptions, for anyfunction f in a class F , with probability of at least 1 − δ over random partitions of S , we have
[Equation]
Armed with Lemma 2, Lemma 3 and Lemma 1, we now present the generalization error bound forinductive and transductive learning.
Theorem 1 (Generalization Bound) .
Under assumptions, for any function f in a class F , in inductivelearning, with probability of at least 1 − δ , we have,
[Equation]
Accordingly, in transductive learning, with probability of at least 1 − δ , we have,
[Equation]
Remark 2.
Theorem 1 states that the gap between generalization error and empirical error convergesuniformly to 0 with the rate of (cid:112) (log m ) /m .
We first discuss several obvious findings aboutthe dependency of the bound on some terms.
For the hidden dimension d , the generalizationbound scales as O ( d √ log d ) ; for the input feature norms B u and B i , the bound scales as O (( B u +
B i ) 2 (cid:112) log( B u + B i )) .
The most important term is the layer-wise update complexity C , whichdepends on the value of γ according to the graph structure, the spectral norms of weight matrices
B 1 and B 2 , and the value of C ϕ and b according to the non-linear activation function.
A naturalidea is to constrain the weight matrices during training such that the generalization bound would notgrow exponentially with the network depth L , which is a regularization techniques commonly usedin machine learning.
The subsequent discussions focus on the γ , C ϕ and b to investigate the influenceof graph normalization and non-linear activation functions.
Role of graph normalization.
It is easy to see that γ is equal to the maximum degree D max ofthe user-item interaction graph without graph normalization.
Therefore, fixed other terms in C , thegeneralization bound grows as O ( D L max ) .
For the random-walk graph, i.e., (cid:101)
A
= D − 1 A , γ = 1 , andfor symmetric normalized graph, i.e., (cid:101)
A
= D − 1 / 2 AD − 1 / 2 , γ ≤ (cid:112) D max /D min , and hence, thegrowth is much smaller than the unnormalized graph.
Role of non-linear activation.
As pointed above, the activation functions affect the generalizationperformance through two quantities: the Lipschitz constant C ϕ and the norm bound of hidden statesb .
Due to the most activation functions satisfy the condition C ϕ ≤ 1 , and particularly LeakyReLU,
ReLU, and Tanh are 1 - Lipschitz , C ϕ presents negligible influence.
For the norm bound b , whilethe bounded activation naturally provides explicit value, the unbounded function can also satisfy theassumption by constraints on the weight matrices.
As discussed in Remark 2, weight regularization iswidely used in machine learning and can help improve model generalization performance to preventoverfitting.
In addition, one recent work [33] gives a new theoretical finding that non-linearity doesnot bring considerable expressive power to improve representation learning in GNNs.
Therefore, thisgeneralization bound provides theoretical support for removing the activation function to simplify themodel without incurring a performance penalty.
Item Mixture Powered Recommendation
The upper bound in Theorem 1 inspires us that the generalization needs to balance empirical risk andmodel complexity.
Most models [34, 8] inevitably lead to excessive complexity in pursuit of high expressiveness, but simple models may lead to higher empirical errors, so it is valuable to reducethe complexity while ensuring model expressiveness.
In addition, the recommendation algorithmis trained based on the limited discrete data obtained by sampling, making the model perform wellin the discrete space of sampling and prone to overfitting.
Considering the above and inspired by
Mixup [35], we propose Item Mixture (IMix) to power recommendation, which forces the model todeal with regions between discrete samples continuously by mixing any two positive-negative itempairs.
In addition, IMix does not make changes to the model, while we can theoretically prove that itreduces the model complexity, and then enjoys generalization.
Below we give specific details.
Since the original representations of items are discrete and unprocessable (e.g., orange or apple), wesuggest mixing their embeddings.
Specifically, given a triplet u, i, i ′ with a label
y
i , where y i =
+1 ifuser u prefers item i to i ′ ,
otherwise y i = − 1 .
The embeddings of ( u, i, i ′ ) are ( e u , e i , e i ′ ) computedaccording to Section 2.2, we arbitrarily sample another triplet ( u, j, j ′ ) with label y j of the same user.
If y i =
y j , we can obtain mixed embedding triplet ( e u , (cid:101)
e i , (cid:101) e i ′ ) =
( e u , λ e i + (1 − λ )
e j , λ e i ′
+ (1 −λ ) e j ′ ) , otherwise, ( e u , (cid:101) e i , (cid:101) e i ′ ) =
( e u , λ e i + (1 − λ )
e j ′ , λ e i ′
+ (1 − λ ) e j ) .
So in IMix, we willtrain the recommendation model on the new mixed embeddings.
The standard loss and IMix loss aredefined as
Standard loss:
[Equation]
Considering the motivation above, IMix preserves the model expressiveness without modification ofthe model structure.
Besides, it transforms the learning of discrete sample pairs into the learning ofcontinuous regions between sample pairs, which may helps to alleviate the overfitting.
Notably, although IMix is inspired by the generalization bound in Theorem 1, it is decoupled from themodel structure.
Therefore, it is theoretically applicable to any embedding-based recommendationmodel, and we will show its performance in non-GCN-based models in Appendix A.3.
Next, we theoretically guarantee the effectiveness of the proposed IMix.
Following a similar approachin [36], we adopt second-order Taylor expansion to state the IMix training loss is approximatelyequivalent to standard empirical error with a regularization term.
Consider a training set S , denote by D u all users and D i all item-pairs in S , where ( i, i ′ )
∈ D iif ( i ′ , i ) ∈
D i .
We assume that the samplings of user and item-pair are independent, and for∀ u ∈ D u , ∀
( i, i ′ )
∈
D i , ( u, i, i ′ ) ∈ S .
For convenience, let y u,i,i ′ denote the label of ( u, i, i ′ ) .
Lemma 4.
Suppose that ˆ E u =
[Equation]
k )
[Equation]
−
[Equation]
) T .
Ifthe loss function l is twice differentiable, the approximation of IMix loss defined in Eq.
5 is given by
[Equation]
where ω ( λ ) =
1 − λλ , η k
= y k e Tu k ( e i
k − e i ′ k ) .
Then we are ready to investigate the generalization bound considering the function class F mix τ relatedto the dual problem of the above regularized term.
The details for this function class are provided in
Appendix F.2.
Remark 3.
Assuming that the loss function l satisfies l ′ ( x ) E λ [ ω ( λ )] ≤ l ′′ ( x ) ≤ E λ [ ω ( λ ) 2 ] ≤ ρ ≤τ, ∀ x ∈ R .
Then with probability of at least 1 − δ , for any function f ∈ F mixτ , we have,
[Equation]
If we consider the function class with a general regularization technique on the weight matrices, as wediscussed above, the constraint on weights will ultimately control the node representation.
Therefore,for the general condition, we focus on the function class
[Equation]
,then the similar proof strategy would yield the generalization error bound L
[Equation]
) ≤
[Equation]
+
[Equation]
B
[Equation]
we can conclude that when rank (Σ X ) < τ , IMix has a better generalization.
It iseasily met because τ tends to be large to pursue high performance.
For example, in the experiment,the embedding size d is 64 (noted that rank (Σ X ) < d ), while the minimum value of τ is 130.8.
Experiments
In this section, we conduct extensive experiments to validate our theoretical findings and the effectiveness of the proposed IMix.
Dataset.
We follow existing work [8, 9], using two datasets: (1) Gowalla, contains 1,027,370 check-ininformation from 29,858 users to 40,981 locations; (2) Yelp2018, comprises 31,668 users who havereviewed 38,048 items (e.g., restaurants, bars) with 1,561,406 times.
The training set, validation set,and test set are randomly sampled in a ratio of 7 : 1 : 2 .
Model Structure.
Two competitive GCN-based algorithms, NGCF [8] and LightGCN [9], are usedto construct experiments.
Among them, NGCF explicitly models the high-order connectivity betweenusers and items through message construction and message aggregation, thereby improving theexpressiveness of embedding.
LightGCN, on the other hand, is keenly aware that the nonlinear activation function and feature transformation of NGCF have no specific semantics in the recommendation,so it is simplified to only a domain message aggregation module.
Correspondingly, these models areconfigured with the proposed IMix denoted as NGCF-IMix and LightGCN-IMix, respectively.
We use the public implementation version of LightGCN 1 and follow the default parameters of theiralgorithms unless otherwise specified.
More details are provided in Appendix A.1.
Metrics.
The recommendation performance is measured by the positive items’ rankings in the test set.
We use two ranking metrics widely used in recommender systems [37, 38, 39]: Recall and NDCG.
Recall at a cutoff K, denoted as Recall@K, denotes the proportion of positive items predicted to becorrect.
NDCG at a cutoff K, denoted as NDCG@K, measures the reward for the positive items’position in the top-K recommendation list.
For these two metrics, we truncate the ranked list K to 20.
5.1 Numerical Discussion
In this subsection, we discuss the implication of the generalization bound defined in Theorem.
1, andthe results are built on NGCF because it is more similar to traditional GCNs.     /D\HU1XPEHU     
   * DS 5HFDOO#.
      /D\HU1XPEHU        * DS 1'&*#.
 D7KHJHQHUDOL]DWLRQHIIHFWRI/D\HUV E7KHJHQHUDOL]DWLRQHIIHFWRIZHLJKWQRUPV  

   L 2 1RUPRI:HLJKW   
     * DS 5HFDOO#.
  
    L 2 1RUPRI:HLJKW        * DS 1'&*#. 
Quantitative Analysis.
We investigate the effects of two major values on the generalization gap,which represents the differences between recommendation performances in training and testing sets.
We consider the number of layers L , as shown in the left part of Fig.
As the number of layersincreases, the generalization gaps of ndcg@K and recall@K significantly increase.
The empiricalresults imply that the generalization error is proportional to the number of layers, consistent withthe theoretical analysis.
We also consider the spectral norms of weight matrices B 1 and B 2 , asshown in the right part of Fig.
We compare the generalization abilities with different weightnorms.
As the norms increase, the generalization gaps increase steadily.
Compared with the numberof layers, we can observe that the weight norms increase more slowly.
The theoretical reason isthat the generalization performance is L -th order with respect to the weight norms B 1 and B 2 , andexponential with respect to the number of layers L .
The Role of Normalized Graph.
We compare the generalization performances with different normalized graphs, as shown in the left part of Fig.
It is clear that the two normalized graphs have highergeneralizations than the unnormalized graph.
This result is consistent with the theoretical findings.
The comparison between two normalized graphs depends on the specific user-item interaction graph.
         (SRFKV       5 H F D OO #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I          (SRFKV     
 1'& * #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I D7KHJHQHUDOL]DWLRQHIIHFWRIQRUPDOL]HGJUDSK E7KHJHQHUDOL]DWLRQHIIHFWRIDFWLYDWLRQIXQFWLRQ          (SRFKV   
  5 H F D OO #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK          (SRFKV  



    1'& * #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK
The Role of Non-linear Activation Function.
We study the role of non-linear activation function in
GCN-based recommender systems.
Since most activation functions are all 1 -Lipschitz continuous, wecompare the generalization performances with the popular activation functions, including LeakyReLU,
ReLU, Sigmoid, and Tanh.
Furthermore, we compare these results with the model without activationfunction.
As shown in the right part of Fig.
2, these models present similar performances, whichprovide theoretical support for removing activation functions. Although the Sigmoid presents greatrecommendations at first, it overfitted as the training continued, while the performance of othermodels is still improved, and their optimal performance tends to be unified.
Performance Analysis of IMix [Table]
0644(+6.98%) 0.0527(+7.11%)          (SRFKV        5 H F D O O #  0HWKRG /LJKW*&1/LJKW*&1,0L[          (SRFKV  
  
1'& * #  0HWKRG /LJKW*&1/LJKW*&1,0L[ D*RZDOOD E<HOS          (SRFKV      5 H F D O O #  0HWKRG /LJKW*&1/LJKW*&1,0L[          (SRFKV        1'& * #  0HWKRG /LJKW*&1/LJKW*&1,0L[
Effectiveness Validation.
We apply the proposed IMix to NGCF and LightGCN and perform adetailed comparison, as shown in Table 1.
The percentage of relative improvement is shown inparentheses.
Obviously, the proposed augmentation strategy (corresponding to NGCF-IMix and
LightGCN-IMix) significantly improves the performance of the corresponding models (NGCF and
LightGCN).
For example, in the Gowalla dataset, the best performance of NGCF on Recall@20 is0.1308, while with IMix configured, the performance improves to an encouraging 0.1496.
In addition, the improvements of IMix on NGCF are larger than that on LightGCN.
We reasonably believe that
LightGCN is more expressive (better performance), and the benefit of the boosting strategy decreasesaccordingly.
Nonetheless, the performance on LightGCN is also a pleasant surprise.
Besides, Fig.
3 presents the performance curves regarding Recall and NDCG for the test set duringthe training of LightGCN-based models (LightGCN and LightGCN-IMix).
Throughout the trainingprocess, LightGCN-IMix consistently outperforms LightGCN.
After 50 epochs, LightGCN showsdifferent degrees of overfitting.
On the contrary, the performance of LightGCN-IMix is still furtherimproved, and their gap is gradually enlarged.
We reasonably believe that the proposed strategy addsa data-dependent regularization term to LightGCN (Lemma 4), which helps to alleviate overfitting.
[Figure]
  
[Figure]
    
[Figure]
H F D OO # 
[Figure]
 
   )HDWXUH6L]H 
        1'& * # 
[Figure]
D*RZDOOD E<HOS      )HDWXUH6L]H        
[Figure]
H F D OO # 
[Figure]
 
 
[Figure]
)HDWXUH6L]H
[Figure]
[Figure]

[Figure]

[Figure]
   5 H F D OO #  0HWKRG 1*&)1*&),0L[  
   +LGGHQ/D\HU6L]H          1'& *
#  0HWKRG 1*&)1*&),0L
[ D*RZDOOD E<HOS  
   +LGGHQ/D\HU6L]H      5 H F D OO #  0HWKRG 1*&)1*&),0L[  
   +LGGHQ/D\HU6L]H        
1'& * #  0HWKRG 1*&)1*&),0L[
Sensitivity w.r.t.
Model Hyperparameters.
We also explore the impact of the model’s hyperparameters: different feature sizes (dimension of x u and x i ) and hidden layer sizes (dimension of W l 1 and
W l 2 ).
Here, the number of layers is set to 1, and the results are shown in Fig.
4 and Fig.
In Fig.
4, we find that the larger the feature size, the more beneficial it is to the recommendation. Moreover, LightGCN-IMix is better than LightGCN, and even more, the better the performance of theoriginal model, the better the improvement after configuring IMix, which highlights the superiorityof the proposed IMix.
Fig.
5 reveals that the hidden layer size does not cause a significant change in model performance,while when the hidden layer size is only 8, the performance is sufficiently satisfactory, which providessupport for the practicality of GCN-based algorithms.
In addition, with similar conclusions to Fig.
4and Table 1, the IMix-configured models outperform the original models in all cases.
In conclusion,we validate the effectiveness of IMix.
Conclusion and Future Work
GCNs achieve superior empirical success in recommender systems but lack theoretical understanding.
In this paper, we provide a model complexity analysis for GCN-based recommendations via Coveringnumber, and then induce the generalization error bounds in inductive and transductive learning.
We report several theoretical findings and again verify them via experimental results.
Inspired bythe generalization analysis, we introduce an augmentation strategy, Item Mixture, to improve therecommendation generalization by mixing item embeddings.
There are several interesting future directions.
First, it would be interesting to further analyze theuser-item bipartite graph with social relationship and knowledge graph.
The second is to extend ouranalysis to investigate the plausibility of the other GNN-based variants.
The third is to study the roleof graph spectral properties in generalization ability.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
Notations
Let [ L ] = { 1 , . . .
, L } .
For a vector v ∈ R n , we denote its Euclidean norm by ∥ v ∥ 2 = √ v T v , andthe infinity norm by ∥ v ∥ ∞ =
max i | v i | .
For a matrix W ∈ R m × n , we denote its Frobenius norm by∥ W ∥ F = (cid:112) trace ( W T W ) , the infinity norm by ∥ W ∥ ∞ = max 1 ≤ i ≤ m (cid:80) nj =1 | W | ij , and the spectralnorm by ∥ W ∥ 2 = σ max ( W ) .
For ease of exposition, we summarize the notations used throughoutthe paper in Table 2.
[Table]
A ExperimentsA.1 Hyper-parameter Settings
Following LightGCN, we use its default parameters in most cases.
Specifically, unless otherwisespecified, the features are initialized using the Xavier method, and their sizes are fixed to 64.
For thenumber of layers, we learn the range from 1 to 3.
The regularization factor is fixed to 1 e − 4 , andthe node dropout ratio is set to 0.
For NGCF with hidden layer parameters, we set the hidden layersize to 64 and use LeakyRelu as its default activation function.
For the proposed strategy, only twoparameters, α and β about the beta distribution, both are set to 0 .
The learning rate of all modelswas set to 0.002 and trained with Adam optimizer for 200 epochs.
Additionally, an early stop strategyis used during training.
A.2 Additional Experiments for Numerical Discussion
The Role of Normalized Graph.
Fig.
6 shows the performance effect of different Normalizedgraphs (left) and non-linear activation functions (right) in Yelp2018.
Obviously, we can get a similarconclusion to Section 5.1 of the main text.
         (SRFKV         5 H F D OO #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I          (SRFKV      
1'& * #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I D7KHJHQHUDOL]DWLRQHIIHFWRIQRUPDOL]HGJUDSK E7KHJHQHUDOL]DWLRQHIIHFWRIDFWLYDWLRQIXQFWLRQ       
[Figure]
(SRFKV       5 H F D OO #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK          (SRFKV     1'& * #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK
[Figure]
The Role of Non-linear Activation Function.
In Section 5.1 of the main text, we explore theimpact of different non-linear activation functions on performance.
To take a deeper look at howthe activation function affects the generalization performance, we consider non-linear activationfunctions with different Lipschitz constants.
Specifically, we multiply the LeakyReLU by differentweights, then compare their effects on generalization abilities.
From Fig.
7, it can be observed thatthe generalization gap significantly increases with the Lipschitz constants.
These results once againconfirm our theoretical findings on the generalization error.
A.3 Performance Analysis on Non-GCN-based Models
[Table]
In theory, the proposed strategy is suitable for non-GCN recommendation models.
We experiment onthree factorization-based recommendation models: (1) BPRMF is a traditional matrix decompositionalgorithm that decomposes the user-commodity interaction matrix into user embedding and commodity embedding and uses BPR as the loss function.
(2) GRMF is a smoothed version of BPRMF thatadds graph Laplacian regularization to the loss function.
(3) NGRMF is a variant of GRMF that performs normalization on graph Laplacian.
Correspondingly, the models configured with the proposed
IMix are denoted as BPRMF-IMix, GRMF-IMix, and NGRMF-IMix, respectively.
The comparisonresults are shown in Table 3.
As expected, IMix still works well on non-GCN-based models, withan average improvement of 7.94% regarding Recall@20 and 8.29% regarding NDCG@20, whichreveals the potential of the proposed enhancement strategy.
         (SRFKV        7 U D L Q L QJ  /R VV 0HWKRG /LJKW*&1/LJKW*&1,0L[          (SRFKV      7 H
V W /R VV 0HWKRG /LJKW*&1/LJKW*&1,0L[ D*RZDOOD E<HOS          (SRFKV      
7 U D L Q L QJ  /R VV 0HWKRG /LJKW*&1/LJKW*&1,0L[          (SRFKV        7 H V W /R VV 0HWKRG /LJKW*&1/LJKW*&1,0L[
Fig.
8 presents the model training and testing curves with and without IMix.
During the wholetraining process, the training loss of LightGCN-IMix converges faster, which indicates that the linearand continuous processing of IMix is more suitable for the training data.
Secondly, consistent withthe find observed in Fig.
3 of the main text, LightGCN has serious overfitting, while LightGCN-IMixis much lighter, which confirms the positive role of IMix in alleviating overfitting.
It is also the keymotivation to propose IMix.
Furthermore, this observation also validates the theoretical analysisin Section 4 of the main text; that is, IMix essentially adds a regularization term to ensure modelgeneralization.
A.5 Comparison of Transductive Learning and Inductive Learning
[Figure]
  5 H F D OO #  0HWKRG 3LQ6DJH1*&)/LJKW*&          (SRFKV      
  1'& *
#  0HWKRG 3LQ6DJH1*&)/LJKW*& D*RZDOOD E<HOS          (SRFKV       5 H F D
OO #  0HWKRG 3LQ6DJH1*&)/LJKW*&          (SRFKV  
   1'& * #  0HWKRG 3LQ6DJH1*&)/LJKW*&
OD\HU OD\HU 1XPBOD\HU          5 H F D OO #  3LQ6DJH1*&)/LJKW*& OD\HU OD\HU 1XPBOD\HU         1'& * #  3LQ6DJH1*&)/LJKW*& D
*RZDOOD E<HOS OD\HU OD\HU 1XPBOD\HU    
  5 H F D OO #  3LQ6DJH1*&)/LJKW*& OD\HU OD\HU 1XPBOD\HU      1'& * #  3LQ6DJH1*&)/LJKW*&
This section compares the GCN-based recommender systems under transductive and inductivelearning.
In the recommendation field, typical models for these two settings are LightGCN [9] (and
NGCF [8]) and Pin-Sage [2], respectively.
The difference in model structure makes them unable tocarry out a fair numerical analysis.
therefore, we only compare their performance, and the results areshown in Fig.
9 and Fig.
In Fig.
9, we find that the performance of Pin-Sage with inductive learning is inferior to LightGCNand
NGCF throughout the training process, and this gap persists until the end of training, as shown in
Fig.
We reasonably suspect that in collaborative filtering, the recommendation results mainlydepend on historical interactions rather than node features [40], which reflects that the performance of
GCN-based models is more dependent on its neighbors.
However, Pin-Sage uses neighbor sampling to ensure model’s flexibility and generalization, which loses a large part of valuable information anddamages the recommendation performance.
A.6 Additional Experiments on GCN-based Recommendations
In addition to LightGCN and NGCF, we also use two other GCN-based models, SpectralCF [7] and
GCMC [6], to verify the effectiveness of our theoretical findings and the proposed IMix.
Amongthem, SpectralCF directly finds all possible connectivity between users and items from the spectraldomain of the user-item graph, while GCMC uses a graph auto-encoder to transmit messages on theinteraction graph to find the potential interests of users.
[Figure]
[Figure]
      (SRFKV      5 H F D OO #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I          (SRFKV       
1'& * #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I D7KHJHQHUDOL]DWLRQHIIHFWRIQRUPDOL]HGJUDSK E7KHJHQHUDOL]DWLRQHIIHFWRIDFWLYDWLRQIXQFWLRQ          (SRFKV       5 H F D OO #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK          (SRFKV      1'& * #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK
Numerical Discussion.
We evaluate the role of normalized graph and non-linear activation functionon SpectralCF as well as GCMC, and the results are illustrated in Fig.
11 and Fig.
We can getsimilar findings to Section 5.1 and A.2: (1) Graph normalization helps to improve recommendationperformance (Fig.
11 (a) and Fig.
12 (a)).
The activation function has little effect on the modelperformance (Fig.
11 (b) and Fig.
12 (b)).
This further supports the theoretical findings in Section3.2.
Effectiveness of IMix.
In Section 5.2, we have verified the effectiveness of IMIx on LightGCN and
NGCF.
In this section, we further apply IMix to SpectralCF and GCMC (denoted as SpectralCF-IMixand GCMC-IMix), and the results are shown in Table 4.
It can be clearly found that the models’recommendation performance equipped IMix can significantly outperform the original model underdifferent structures.
Specifically, the average improvement on Recall@20 is 11.86%, while theaverage improvement is 12.70% concerning NDCG@20.
These encouraging results emphasize thepotential of IMix on GCN-based recommender systems.
A.7 Experiments on LastFM
In this section, we introduce LastFM to enhance our work.
LastFM is a music recommendationdataset consisting of 1,027,370 ratings for 4449 artists from 1892 user ratings collected from themusic website of Last.fm.
The item is defined as an artist.
The training, validation, and test sets arerandomly divided in a ratio of 7:1:2.
Numerical Discussion.
We construct experiments on the effect of normalized graph and activationfunction on recommendation performance, as shown in Fig.
13 (a) and Fig.
13 (b).
We can get similar
[Table]
(+12.78%) 0.0550(+14.58%) 0.0447(+17.02%)
         (SRFKV         5 H F D
OO #  0HWKRG
A + I D 1 A + I D 1/2 AD 1/
+ I          (SRFKV       1'& * #  0HWKRG
A + I D 1 A
+ I D 1/2 AD 1/
+ I D7KHJHQHUDOL]DWLRQHIIHFWRIQRUPDOL]HGJUDSK E7KHJHQHUDOL]DWLRQHIIHFWRIDFWLYDWLRQIXQFWLRQ          (SRFKV         5 H F D
OO #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK          (SRFKV        1'& * #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK but more obvious findings as in Gowalla (in Fig. 2) and Yelp2018 (in Fig.
The performance onthe unnormalized graph is significantly worse than the other two normalized graphs, and it starts todecline after 5 rounds.
This underscores our point in Section 3.2 that the normalized graph can yieldgeneralization gains.
The effect of different activation functions on recommendation performanceis negligible.
Therefore, it is proved that removing the activation function can simplify the modelwithout incurring a performance penalty, which is consistent with the findings in [9].
[Table]
[Table]
Effectiveness of IMix.
We evaluate the performance of LightGCN and NGCF using IMix in LastFM.
[Table]
Although on 3-layer-LightGCN
[Table]
our method has a drop of about 5%
[Table]
in other cases
[Table]
the IMix models’
[Table]
performance has improved
[Table]
and the improvement is particularly obvious in NGCF
[Table]
with an average
[Table]
improvement is 11.83% concerning Recall@20.
[Table]
A.8 Comparison with Mixup-based Methods
Differences from Recent Mixup-based Methods.
More recently, some Mixup-based strategieshave been introduced in GNNs
[Table]
but our work is quite different from them: (1) G-Mixup proposed in
OD\HU OD\HU OD\HU 1XPBOD\HU         5 H F D OO #  OD\HU OD\HU OD\HU 1XPBOD\HU          1'& * #  D
*RZDOOD E<HOS OD\HU OD\HU OD\HU 1XPBOD\HU    
   5 H F D OO #  OD\HU OD\HU OD\HU 1XPBOD\HU    
  1'& * #  /LJKW*& /LJKW*&17%0L[ /LJKW*&18,0L[ /LJKW*&1,0L[
[41] studies graph classification tasks, while our recommendation task is essentially link prediction.
Besides, G-Mixup enhances the model by interpolating the graphon of different classes of graphs, butwe only mix the item embeddings of the same users; (2) [42] proposed to mix two-branch featuresand used Mixup at each layer, while our IMix does not introduce additional convolution branches andonly performs item mixture on the last layer.
In addition, we give the theoretical guarantee of IMix,which is lacking in [42].
Performance Comparison with Mixup-based Methods.
We compare IMix with the strategy in [42](two branch Mixup, denoted as TBMix) and UIMix (a variant of IMix that mixes both user and itemembeddings).
Note that we do not consider the version that only mixes user embeddings, because itmust ensure that the positive and negative items are the same, i.e., e i =
e j and e i ′
=
e j ′ , otherwise, aquintuple (˜ e u , e i , e i ′ , e j , e j ′ ) instead of triplet (˜ e u , e i , e i ′ ) .
We apply these strategies on LightGCN,denoted LightGCN-TBMix, and LightGCN-UIMix, respectively, and the results are shown in Fig.14.
First, although TBMix shows good performance on Yelp2018, it performs poorly on Gowalla,even inferior to LightGCN.
Second, the proposed IMix is the best among the compared methods.
Third, the performance of UIMix, which mixes both user and item embeddings, is not as good as
IMix, which only mixes item embeddings.
One possible reason is that the recommender system hastwo types of feature spaces: user space and item space.
IMix mixes in the item space of the sameuser (same user space), while UIMix mixes across spaces (different user space and different itemspace), which may lead to confusion in learning.
A.9 Experiments on node-features-included Models
In collaborative filtering, due to the lack of rich features, random embeddings are generally used asinitial features and learned during training, which is especially common in GCN-based recommendersystems [2, 7, 9, 8].
In this section, however, we explore whether recommendation models using rawfeatures can achieve similar findings.
To solve the misalignment of user and item features, we use a
Multilayer Perceptron to map them into the space with the same dimensionality and apply LightGCNand NGCF on it (denoted as F-LightGCN and F-NGCF).
[Figure]
I D7KHJHQHUDOL]DWLRQHIIHFWRIQRUPDOL]HGJUDSK E7KHJHQHUDOL]DWLRQHIIHFWRIDFWLYDWLRQIXQFWLRQ 

       (SRFKV        5 H F D OO #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK          (SRFKV       1'& * #  0HWKRG 1RQH/HDN\5HOX5HOX6LJPRLG7DQK
Numerical Discussion.
We construct experiments on the effect of the normalized graph and activationfunction based on F-LightGCN, as shown in Fig. 15.
In Fig.
15 (a), we find that the performancegap of whether the graph is normalized or not is much smaller than the model that does not use thenode features (Fig. 2).
This may be that the node features weaken the high dependence on the graphstructure.
But it is undeniable that our finding still holds that using graph normalization helps model generalization.
Second, the performance of different activation functions is still indistinguishable.
This provides support for simplifying the removal of nonlinear layers in GCN-based recommendersystems.
[Table]
[Table]
[Table]
Numerical Discussion.
We use the proposed augmentation strategy IMix (denoted as F-NGCF-IMixand F-LightGCN-IMix) on F-NGCF as well as F-LightGCN, and the results are shown in Table 6.
The performance of the models using IMix has been gratifyingly improved.
Considering Recall@20,the average improvement is 15.76%, and the maximum improvement is an astonishing 27.89%.
Theseresults again validate the ability of IMix to enhance recommendations.
A.10 Visualization
        
[Figure]
         
[Figure]
To intuitively measure the quality of model learning, we use the t-SNE tool [43] to visualize theembeddings of positive and negative items of the same user learned by NGCF and NGCF-IMix.
We experiment on Gowalla and randomly sample the same number of negative items as positiveitems.
The visualization results are shown in Fig.
Blue nodes denote the embeddings of positiveitems, and orange items represent the embeddings of negative ones.
We can observe that compared to
NGCF, NGCF-IMix can better distinguish positive and negative items, which indicates that using theproposed IMix has more potential to learn user preferences.
B Proof of Lemma 1
We first present a lemma according to the Covering number of matrices with bounded norm.
Lemma B.1 ([31]) .
Let G =
{ A ∈ R d 1 × d 2 : ∥
A ∥ 2 ≤ λ } be the set of matrices with boundedspectral norm and ϵ ≥ 0 be given.
The covering number N ( G , ϵ, ∥ · ∥ F ) is upper bounded by
[Equation]
For ease of understanding, we first summarize the proof strategy used in this section.
• Investigate the user and item representation spaces at each layer.
( Step 1 & Step 2 )
• Establish the correlation between Covering numbers of function class and parameters.
( Step 3 )
• Derive the matrix Covering number and yeild final result using Lemma B.1.
( Step 4 )
Then we provide the proof of Lemma 1.
Lemma 1 (Covering number bound) .
Under Assumptions, we further assume that ∥ h lu ∥ ∞ , ∥ h li ∥ ∞ ≤b for any u ∈ U , i ∈ I , and l =
[ L ] , let γ = ∥ (cid:101)
A ∥ ∞ .
Given a sample set S with size n , the Coveringnumber of F over S with specific ϵ is bounded aslog
[Equation]
|
[Equation]
[Equation]
, ∥ ·
[Equation]
ϵ
[Equation]
ϵ
[Equation]
Moreover, when ϵ ≤ 1 + 4 M √ nd max { ( γ + 1) B 1 , γbB 2 } ,log N ( F| S, ϵ, ∥ · ∥ 2 ) ≤ 2 d 2 log 8 M ( γ + 1) √ ndB 1 B 2 b ϵ , where
[Equation]
−
[Equation]
Proof.
Let f w to be a GCN-based recommendation model defined in Section 2.2 with a set of weightmatrices w = { W 1 , W 2 } , and f w ′ with w ′ =
{ W ′ 1 , W ′ 2 } .
For any user-item ( u, i ) , we first performthat the preference score predicted by f w and f w ′ can be bounded by w and w ′ .
Then we derive thecovering number bound for the function class F with specific radius. For convenience, we denotethe user representation for user u and the item representation for item w at layer l as u l and w l withweight w , and u ′ l and w ′ l are with weight w ′ .
Step 1: Max norm of user representation and item representation.
We first bound the maximumspectral norm of user embedding at layer l .
Let n l = (cid:80) w ∈N u ˜ a uw ( W 1 w l − 1 + W 2 ( u l − 1 ⊙ w l − 1 )) , since ϕ (0)
=
0 , we havemax
[Equation]
∈U
[Equation]
The first inequality is due to the activation function ϕ ( · ) is C ϕ − lipschitz .
Since ∥ W 1 ∥ 2 ≤
B 1 , ∥ W 2 ∥ 2 ≤ B 2 and
[Equation]
we have
[Equation]
The last inequality is due to ∥ a ⊙ b ∥ 2 ≤ ∥ a ∥ 2 ∥ b ∥ ∞ for any a , b ∈ R d .
Together with the assumptionabout the infinity norm bound of node representation at any layer, which means ∥
u l − 1 ∥ ∞ ≤ b , wecan get
[Equation]
Similarly,
[Equation]
Combining Eq.
6 and Eq.
7, we have
[Equation]
Let T l ≤ C T l − 1 , where C = C ϕ
[( γ + 1) B 1 + γB 2 b ] .
Since we assume that max u ∈U ∥ x u ∥ 2 ≤ B u andmax i ∈I
∥ x i ∥ 2 ≤ B i , then we have
[Equation]
and expanding the recursion T l ≤ C T l − 1 , we get
[Equation]
[Equation]
Step 2: Max change of user representation and item representation.
We first bound the maximumdifference between user representations at layer l generated by f w and f w ′ .
Let n l = (cid:80) w ∈N u ˜ a uw ( W 1 w l − 1 + W 2 ( u l − 1 ⊙ w l − 1 )) , then we have
[Equation]
[Equation]
Here the first inequality is due to the activation function ϕ ( · ) is C ϕ − lipschitz .
Then we proceed tobound the two terms in Eq. 9.
[Equation]
[Equation] The last inequality is due to ∥ W 1 ∥ 2 ≤ B 1 .∥ n l −
[Equation]
′ l ∥≤ (cid:88)
w
[Equation]
u
[Equation]
Similar to Eq.
[Equation]
And since ∥ W 2 ∥ 2 ≤ B 2 , we have
[Equation]
The last inequality is due to ∥ a ⊙ b ∥ 2 ≤ ∥ a ∥ 2 ∥ b ∥ ∞ for any a , b ∈ R d .
Together with the assumptionabout the infinity norm bound of node representation at any layer
[Equation]
which means ∥ u l − 1 ∥ ∞ ≤ b and(cid:13)(cid:13) w ′ l − 1 (cid:13)(cid:13) ∞ ≤ b .
Substituting Eq.
12 and Eq.13 into Eq.
11, and since
[Equation]
we can get
[Equation]
Combining Eq.
10 and Eq.
14, we have
[Equation]
Similarly, we have
[Equation]
Combining Eq.
15 and Eq.
16, we have
[Equation]
The last inequality is due to C ϕ
[( γ + 1) B 1 + 2 γbB 2 ] ≤ 2 C ϕ
[( γ + 1) B 1 + γbB 2 ] = 2 C .
Accordingto the norm bound for node representations completed in Eq. 8 and expanding the recursion, we canget the following result,
[Equation]
Step 3: Max change of recommendation.
For any user-item ( u, i ) , we first perform that thedifference between preference scores predicted by f w and f w ′ can be bounded by w and w ′ .
Then wederive the covering number bound for the function class F with specific radius.
[Equation]
There exists three types of integration operations to obtain the final node representations.
(Case 1) Final layer:
[Equation]
Similarly,
[Equation]
Combining Eq.
18 and Eq.
19, we have [Equation]
(Case 2)
Linear combination:
[Equation]
[Table] l = | α l | ( ∥ u l ∥ 2 + ∥ u ′ l ∥ 2 + ∥ w l ∥ 2 + ∥ w ′ l ∥ 2 ) (cid:33) (cid:32) L (cid:88) l = | α l | ( ∥ u l − u ′ l ∥ 2 + ∥ w l − w ′ l ∥ 2 ) (cid:33) ≤ (cid:32) L (cid:88) l = | α l | T l (cid:33) (cid:32) L (cid:88) l = | α l | ∆ l (cid:33) .
(Case 3) Concatenation:
[Equation]
For convenience, we only consider the outputs of the last layer as the final representations.
Substituting
Eq. 8 and Eq.
17 in to Eq.
20, we will derive the following change bound of recommendationspredicted by two GCN-based models with different parameters.
[Equation]
Step 4:
Covering number bound of the function class F over a sample set S with size n .
Let
S =
{ z i } ni =1 , we denote a vector f S =
( f ( z 1 ) , . . .
, f ( z n )) as the recommendations predicted by themodel f ∈ F for all sample in sample set S .
Since ∥ a ∥ 2 ≤ √ n ∥ a ∥ ∞ for any vector a ∈ R n , let
M = C ϕ C
L − 1 ( B u + B i ) 2 (2 C )
L − 1 2 C− 1 , we have
[Equation]
Since for any matrix A , ∥ A ∥ 2 ≤ ∥
A ∥ F , we can find a ϵ
[Equation]
M − cover for W 1 and aϵ 2 √ nγbM − cover for W 2 to bound
[Equation]
According to Lemma B.1, we have
[Equation]
Thus, we have a way of obtaining the covering number of the function class F to bound Λ L ≤ 2 ϵ ,log N ( F , ϵ, ∥ · ∥ 2 ) ≤
[Equation]
ϵ
[Equation]
Moreover, when ϵ ≤ 4 M √ nd max { ( γ +
[Equation]
B 1 ,
[Equation]
[Equation]
ϵ .
The proof is complete.
C Proof of Lemma
We first present the definition of the inductive empirical Rademacher complexity and some lemmasthat we will use in the later proofs.
Definition C.1 (Inductive empirical Rademacher complexity) .
Let D be a probability distributionover X .
Suppose that the examples S
= { x i } ni
=1 are sampled independently from X according to D .
Let F be a family of functions X to R .
The empirical Rademacher complexity of F with respect tothe samples S is defined as
[Equation]
where σ = ( σ 1 , · · · , σ m ) T , with σ i is independent uniform random variables taking values in{− 1 , +1 } .
Lemma C.1 ([44]) .
For any bounded loss function l : R → [ − B, B ] , given a training set D ∼ D m ,with probability of at least 1 − δ , for any function g in a class G ,
[Equation]
Lemma C.2 (Contraction Lemma [26]) .
Let Φ : R → R be an L − lipschitz function, Then for areal-valued function set G ,
[Equation]
Lemma C.3 (Extension of [32]) .
Let F be a real-valued function class taking values in [ − e, e ] , andassume that 0 ∈ F .
Then the empirical Rademacher complexity of F can be bounded as
[Equation]
We first illustrate the roles of the above lemmas in the proof.
• Derive a Rademacher complexity-based generalization error bound using Lemma C.
andLemma C.2.• Yield the final result with respect to discrete Covering number via chaining.
( Step 3 )
Then we
provide the proof of Lemma 2.
Lemma 2.
Let F be a real-valued function class taking values in [ − e, e ] , and assume that 0 ∈ F .
Under assumptions, for any function f in a class F , with probability of at least 1 − δ over an i.i.d.size- m training set, we have
[Equation]
Proof.
Let l ◦ H
= { ( z, z ′ , y )
→ l ( y ( f ( z ) − f ( z ′ ))) : f ∈ F} and H = { ( z, z ′ , y )
→ y ( f ( z ) −f ( z ′ )) : f ∈ F} .
We denote S 1 = {
( z i , y i ) } mi =1 as the sample derived from S m by keeping onlythe first element of each pair and S 2 = { ( z ′ i , y i ) } mi =1 the one obtained by keeping only the secondelement.
Using Lemma C.1, Lemma C.2 and the assumption about loss function, we have
[Equation]
Here, since σ i y i and σ i have same distribution, ℜ m ( H ) can be bounded as follows,
[Equation]
Considering the distribution D as symmetric, we have ˆ ℜ S 1 ( F ) =
ˆ ℜ S 2 ( F ) , and then ˆ ℜ S m ( H )
≤2ˆ ℜ S 1 ( F ) .
Together with Lemma C.3, we can derive the following generalization error bounded by
Covering number.
[Equation]
The proof is complete
[Equation]
D Proof of Lemma 3
Similarly, we first present the definition of the transductive Rademacher complexity (TRC) and some
TRC version of lemmas that we will use in the later proofs.
Definition D.1 (Transductive Rademacher complexity) .
Let F ⊆ R m + u and p ∈ [0 , 1 / 2] . Letσ =
( σ 1 , . . .
, σ m + u ) T be a vector of i.i.d.
random variables such that
[Equation]
The transductive Rademacher complexity with parameter p is
[Equation]
Lemma D.1 ([30]) .
Let B 1 ≤ 0 , B 2 ≥
and V be a (possibly infinite) set of real-valued vectorsin [ B 1 , B 2 ] m + u .
Let B = B 2 − B 1 and B max = max( | B 1 | , | B 2 | ) .
Let Q 1 = 1 u + 1 m , Q 2 =m + u ( m + u − 1 / 2)(1 − 1 / 2(max( m,u ))) and c 0 =
[Equation]
< 5 . 05 .
Then with probability of at least 1 − δover random partitions of S , for all v ∈ V ,
[Equation]
Lemma D.2 (Contraction lemma for TRC [30]) .
Let Φ : R → R be an L − lipschitz function, Thenfor a real-valued function set G ,
[Equation]
Lemma D.
(Massart’s lemma for TRC) .
Let
A ⊆ R m be a finite set, with r = max x ∈A ∥ x ∥ 2 , then
[Equation]
where σ i s are transductive Rademacher random variables defined in Eq.
21 and x = { x 1 , . .
, x m } .
Lemma D.4.
Let F be a real-valued function class taking values in [ − e, e ] , and assume that 0 ∈ F .
Then the transductive Rademacher complexity of F can be bounded as
[Equation]
The proofs of Lemma D.3 and Lemma D.4 follow the proofs for inductive Rademacher complexity,and will be provided in Section D.1 and Section D.2.
Lemma 3.
Let F be a real-valued function class taking values in [ − e, e ] , and assume that 0 ∈ F .
Let Q 1 = 1 u + 1 m , Q 2 = m + u ( m + u − 1 / 2)(1 − 1 / 2(max( m,u ))) and c 0 < 5 . 05 .
Under assumptions, for anyfunction f in a class F , with probability of at least 1 − δ over random partitions of S , we have
[Equation]
Proof.
Let l ◦ H
= { ( z, z ′ , y )
→ l ( y ( f ( z ) − f ( z ′ ))) : f ∈ F} and H = { ( z, z ′ , y )
→ y ( f ( z ) −f ( z ′ )) : f ∈ F} .
We denote S 1 = {
( z i , y i ) } m + u i =1 as the sample derived from S by keeping onlythe first element of each pair and S 2 = { ( z ′ i , y i ) } m
+ u i =1 the one obtained by keeping only the secondelement.
Using Lemma D.1, Lemma C.2 and the assumption about loss function, we have
[Equation]
Here, since σ i y i and σ i have same distribution, ℜ m ( H ) can be bounded as follows,
[Equation]
Considering the distribution D as symmetric, we have ˆ ℜ S 1 ( F ) =
ˆ ℜ S 2 ( F ) , and then ˆ ℜ S m ( H )
≤2ˆ ℜ S 1 ( F ) .
Together with Lemma C.3, we can derive the following generalization error bounded by
Covering number.
[Equation]
The proof is complete.
D.1 Proof of Lemma D.
Proof.
The proof follows similar to the one for inductive Rademacher random variables [26].
For ∀ t > 0 , using Jensen’s inequality, we haveexp (cid:32) t E σ (cid:34) sup x ∈A m (cid:88) i =
σ i x i (cid:35)(cid:33) ≤ E σ (cid:34) exp (
cid:32) t sup x ∈A m (cid:88) i =
σ i x i (cid:33)(cid:35) =
E σ (cid:34) sup x ∈A exp (cid:32) t m (cid:88) i =
σ i x i (cid:33)(cid:35) ≤ (cid:88) x ∈A
E σ (cid:34) exp (cid:32) t m (cid:88)
i =
σ i x i (cid:33)(cid:35) = (cid:88) x ∈A
E σ (cid:34) m (cid:89) i = exp
( tσ i x i ) (cid:35) = (cid:88) x ∈A m (cid:89) i =
E σ [exp ( tσ i x i )] .
Since E σ [ σ i x i ] = 0 and −| x i | ≤ σ i x i ≤ | x i | , using Hoeffding’s lemma, we haveexp (cid:32) t E σ (cid:34) sup x ∈A m (cid:88) i =
σ i x i (cid:35)(cid:33) ≤ (cid:88) x ∈A m (cid:89) i =
E σ [exp ( tσ i x i )] ≤ (cid:88) x ∈A m (cid:89) i = exp (cid:18) t 2 (2 x i ) (cid:19) =
(cid:88) x ∈A exp (cid:32) t m (cid:88) i
= x 2 i (cid:33) ≤ (cid:88) x ∈A exp (cid:18) t 2 r (cid:19) ≤ |A| exp (cid:18) t 2 r (cid:19) .
Therefore,
E σ (cid:34) sup x ∈A m (cid:88) i =
σ i x i (cid:35) ≤ log |A| t
+ tr2 .
Set t = √ 2log |A| t , we can complete the proof.
[Equation]
The proof is complete.
D.2 Proof of Lemma D.
Proof.
The proof follows similar to the one for inductive empirical Rademacher complexity [32].
Let N be arbitrary and let ε i = 2 e √ n 2 −
( i − 1) for each i ∈ [ N ] .
We denote the outputs of functionf ∈ F over training set S
= { x 1 , . . .
, x n } as a vector f S =
( f ( x 1 ) , . . .
, f ( x n )) .
For each i let V idenote the cover achieving N ( F| S, ε i , ∥ · ∥ 2 ) , so that∀ f ∈ F , ∃ v ∈ V i , ∥ f S − v ∥ 2 ≤ ε i , and | V i | =
N ( F| S, ε i , ∥ · ∥ 2 ) .
For a fixed f ∈ F , let v i [ f ] denote the nearest element in V i .
Thenwe have
[Equation]
[Equation]
The last inequality is due to σ i ∈ {− 1 , +1 , 0 } .
For the first term in Eq.
22, using Cauchy-Schwarzinequality, since ( σ j ) 2 ≤ 1 , we have
[Equation]
For the second term in Eq.
22, let W i
= { v ij [ f ] − v
i +1 j
[ f ] | f ∈ F} , then we have
[Equation]
Since ε i +1 ≤ ε i , we can get | W i | ≤ | V
i
||
V
i +1 | ≤
| V
i +1 | 2 , which implies W i is a finite set.
Using
Lemma D.3, the following inequality holds,
[Equation]
And furthermore
[Equation]
Combining with Eq. 23 and Eq.
24, we have
[Equation]
For the third term in Eq.
22, if we set V 1 = { 0 } , ∥ f S −
∥ 2 ≤ 2 e √ n = ε 1 , which is a coverachieving N ( F| S,
ε 1 , ∥ · ∥ 2 ) .
Then the following equation holds,
[Equation]
Collecting all terms, we can get the final result.
[Equation]
Finally, select any α > 0 and take N be the largest integer with ε N +1 > α .
Then ε N < 4 ε N +2 < 4 α ,we have
[Equation]
The proof is complete.
E Proof of Theorem
The main the strategy in this proof is to derive the integral over discrete Covering number.
Theorem 1 (Generalization Bound) .
Under assumptions, for any function f in a class F , in inductivelearning, with probability of at least 1 − δ , we have,
[Equation]
) 2 d (cid:113) 2 log(8 mM ( γ + 1) (cid:112) dB 1 B 2 b )
+ 16 C l m + 4 B (cid:114) 2 log 4 /δ m .
Accordingly, in transductive learning, with probability of at least 1 − δ , we have,
[Equation]
Proof.
First, we need to derive the bound [ − e, e ] for any f ∈ F .
[Equation]
Using Eq.
8, we have e = 14 C 2 L ( B u + B i ) 2 .
Together with Lemma 1 and Lemma 2, the generalizationerror can be bounded as follows.
[Equation]
[Equation]
where
[Equation]
Set α = (cid:113) 1 m and substitute Eq. 26 into Eq.
25, we have
[Equation]
Similarly
[Equation]
we can derive the generalization error bound for transductive learning
[Equation]
[Equation]
The proof is complete.
F Proofs of Section
We consider a training set S m = { ( u k , i k , i ′ k ) } m k =1 with labels { ( y k ) } m k =1 , where y k = +1 if useru k prefers item i k to i ′
k , otherwise y = − 1 .
Denote by D u all users in training set and D i all itempairs in training set.
We assume
that ( u, i, i ′ ) ∈ S m for any u ∈ D u and ( i, i ′ )
∈
D i .
We furtherassume that the training set S m is symmetric, which implies if ( u, i, i ′ )
∈ S m with label y , then( u, i ′ , i ) ∈
S m with label − y .
We also assume that the samplings of user u and item-pair ( i, i ′ ) are independent
Recall the Item Mixture strategy proposed in Section 4.
Since the recommendation is invariant tothe scaling of the embedding, so it suffices to consider the following definition.
For a triplet ( u, i, i ′ )with label
y i , we arbitrarily sample another triplet ( u, j, j ′ ) with label y j .
Since the preferenceprediction is invariant to the scaling of the training data, so it suffices to consider ( e u , (cid:101) e i , (cid:101)
e i ′ )
=(
e u , e i + 1 − λλ e j , e i ′ + 1 − λλ e j ′ )
if y i = y j .
Otherwise, we compute the mixed embedding triplet( e u , (cid:101)
e i , (cid:101) e i ′ ) =
( e u , e i + 1 − λλ e j ′ , e i ′ + 1 − λλ e j ) .
F.1 Proof of Lemma 4
In this section, we prove by second-order Taylor expansion that the IMix loss approximates thestandard loss plus a regularization term.
Lemma 4.
Suppose that ˆ E u = 1 m m
[Equation]
If the loss function l is twice differentiable, then the IMix loss defined in Eq.
5 has the followingapproximation,
[Equation]
where ω ( λ ) =
1 − λλ , η k
= y k e Tu k ( e i
k − e i ′ k ) .
Proof.
The IMix loss function over training set S m is defined as
[Equation]
Denote the randomness (of λ and ( j, j ′ ) ) by ξ , using the second-order Taylor expansion
[Equation]
we canapproximate the IMix loss as follows,
[Equation]
For convenience
[Equation] let ω ( λ ) = 1 − λλ , η k =
y k e Tu k
( e i
k − e i ′ k ) .
Then we have
[Equation]
[Equation]
where ˆ E
[Equation]
=
[Equation] (cid:80)
[Equation]
e
[Equation]
e
[Equation]
k ) T .
The last equality isdue to the training set is symmetric and ( u, i, i ′ )
∈ S m for any u ∈ D u and ( i, i ′ )
∈
D i .
Then wehave the following second-order Taylor approximation of the IMix loss.
[Equation]
The proof is complete.
F.2 Proof of Remark
We first present a fundamental uniform convergence theory, then we establish the generalizationbounds for via Rademacher complexity.
Lemma F.1 ([44]) .
For any bounded loss function l : R → [ − B, B ] , with probability of at least1 − δ , for any function g in a class G ,
[Equation]
Remark 3.
Let τ > 0 , E u =
E y
( e i − e i ′ ) and Σ X = E (cid:2) ( e i − e i ′ )
( e i − e i ′ )
T (cid:3) , we shed light uponthe generalization bound by investigating the following function class:
[Equation]
Assuming that the loss function l satisfies l ′ ( x ) E λ [ ω ( λ )] ≤ l ′′ ( x ) ≤ E λ [ ω ( λ ) 2 ] ≤ ρ, ∀ x ∈ R on thebasis of the previous assumptions.
Then with probability of at least 1 − δ , for any function f ∈ F mix τ ,we have,
[Equation]
For the general condition, we focus on the function class F stdτ =
{F| E ( u,i,i ′ )
∥ e u ∥ 22 + ∥ e i ∥ 22 ≤ τ } ,then the generalization error bound is
[Equation]
log 2 /δ m .
Proof.
Let (cid:101) x k = Σ † / 2 X
( e i
k − e i ′ k ) and v k = Σ 1 / 2 X e u k , we have
E
[Equation]
⇒
[Equation]
⇒
[Equation]
⇒ E (cid:34)(cid:18)(cid:113)
[Equation]
⇒ E (cid:20)(cid:113) e
[Equation]
u
[Equation]
≤
[Equation]
+ (cid:114) τ ρ + Jensen’s inequality .
Here ρ ≥ 2 l ′′ ( x ) E λ [ ω ( λ ) 2 ] , ∀ x ∈ R .
We further assume that τ > ρ , then we can bound the empirical
Rademacher complexity as follows,
[Equation]
The last inequality uses Lemma C.2.
Then we have
[Equation]
Therefore
[Equation]
we have the following bounded Rademacher complexity.
[Equation]
Similarly, for the general condition
[Equation]
we can bound the empirical Rademacher complexity as follows,
[Equation]
Finally, based on Lemma F.1, we can derive the corresponding generalization error bound using thesetwo Rademcaher complexities.
The proof is complete.