L EARNING ON L ARGE - SCALE T EXT - ATTRIBUTED G RAPHS VIA V ARIATIONAL I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies learning on text-attributed graphs (TAGs), where each node is associated with a text description.
An ideal solution for such a problem would be integrating both the text and graph structure information with large language models and graph neural networks (GNNs).
However, the problem becomes very challenging when graphs are large due to the high computational complexity brought by large language models and training GNNs on big graphs.
In this paper, we propose an efficient and effective solution to learning on large text-attributed graphs by fusing graph structure and language learning with a variational ExpectationMaximization (EM) framework, called GLEM.
Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step.
Such a procedure allows to separately train the two modules but at the same time allows the two modules to interact and mutually enhance each other.
Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach.
I NTRODUCTION
Graphs are ubiquitous in the real world.
In many graphs, nodes are often associated with text attributes, resulting in text-attributed graphs (TAGs) (Yang et al., 2021).
For example, in social graphs, each user might have a text description; in paper citation graphs, each paper is associated with its textual content.
Learning on TAG has become an important research topic in multiple areas including graph learning, information retrieval, and natural language processing.
In this paper, we focus on a fundamental problem, learning effective node representations, which could be used for a variety of applications such as node classification and link prediction.
Intuitively, a TAG is rich in textual and structural information, both of which could be beneficial for learning good node representations.
The textual information presents rich semantics to characterize the property of each node, and one could use a pre-trained language model (LM) (e.g., BERT (Devlin et al., 2019)) as a text encoder.
Meanwhile, the structural information preserves the proximity between nodes, and connected nodes are more likely to have similar representations.
Such structural relationships could be effectively modeled by a graph neural network (GNN) via the message-passing mechanism.
In summary, LMs leverage the local textual information of individual nodes, while
GNNs use the global structural relationship among nodes.
An ideal approach for learning effective node representations is therefore to combine both the textual information and graph structure.
One straightforward solution is to cascade an LM-based text encoder and GNN-based message-passing module and train both modules together.
Such an approach is effective when graphs are small.
However, it becomes problematic once the graphs become very large.
This is because the size of the computational graph is proportional to the size of the graph structure between nodes as well as the language model capacity, and the memory complexity is proportional to the size of the observed graph and the number of parameters in the LM.
Therefore, on real-world TAGs where various nodes are densely connected, the memory cost of this method would become unaffordable.
To address such a problem, multiple solutions have been proposed.
These methods reduce either the capacity of LMs or the size of graph structures for GNNs.
More specifically, some studies choose to fix the parameters of LMs without fine-tuning them (Liu et al., 2020b).
Some other studies reduce graph structures via edge sampling, and perform message-passing only on the sampled edges (Zhu et al., 2021;
Li et al., 2021a; Yang et al., 2021).
Despite the improved scalability, reducing the LM capacity or graph size would sacrifice the model effectiveness, leading to degraded performance of learning effective node representation.
Therefore, we are wondering whether there exists a scalable and effective approach to integrate big LMs and GNNs on large text-attributed graphs.
In this paper, we propose such an approach, named G raph and L anguage Learning by E xpectation
M aximization GLEM.
In the GLEM, instead of simultaneously training both the LMs and GNNs, we leverage a variational EM framework (Neal & Hinton, 1998) to alternatively update the two modules.
Take the node classification task as an example.
The LM uses local textual information of each node to learn a good representation for label prediction, which thus models local marginal label distributions of nodes.
By contrast, for a node, the GNN leverages the labels and textual encodings of surrounding nodes for label prediction, and it essentially defines a global conditional label distribution.
The two components are optimized to maximize a variational lower bound of the log-likelihood function, which can be achieved by alternating between an E-step and an M-step, where at each step we fix one component to update the other one.
This separate training framework significantly improves the efficiency of GLEM, allowing it to scale up to real-world TAGs.
In each step, one component presents pseudolabels of nodes for the other component to mimic.
By doing this, GLEM can effectively distill the local textual information and global structural information into both components, and thus GLEM enjoys better effectiveness in node classification.
We conduct extensive experiments on 3 benchmark datasets to demonstrate the superior performance of
GLEM.
Notably, by leveraging the merits of both graph learning and language learning, GLEMLM achieves on par or even better performance than existing GNN models, GLEM-GNN achieves new state-of-the-art results on ogbn-arxiv, ogbn-product, and ogbn-papers100M.
R ELATED W ORK
Text-attributed graph (TAG)
(Yang et al., 2021) is a widely existing data format, where each node is associated with a text feature.
Representation learning on TAGs has been attracting growing attention in graph machine learning, and one of the most important problems is node classification.
The problem can be formalized as a text representation learning task, where the goal is to use the text feature of each node for learning.
Early works resort to convolutional neural networks (Kim, 2014;
Shen et al., 2014) or recurrent neural networks Tai et al.
Recently, with the superior performance of transformers (Vaswani et al., 2017) and pre-trained language models (Devlin et al., 2019; Yang et al., 2019), LMs have become the go-to model for encoding contextual semantics in the sentences for text representation learning.
At the same time, the graph learning area has been vastly developed by graph neural networks (GNNs) (Kipf & Welling, 2017; Velickovic et al., 2018;
Xu et al., 2019; Zhang & Chen, 2018; Teru et al., 2020).
A GNN takes numerical features as input and learns node representation by transforming representations and aggregating them according to the graph structure.
With the capability of considering both node attributes and graph structure,
GNNs have shown great performances in various applications, including node classification and link prediction.
Nevertheless, LMs and GNNs only use parts of observed information (i.e., textual or structural) for representation learning, and the results remain to be improved.
There are some recent efforts focusing on the combination of GNNs and LMs, which allows one to enjoy the merits of both models.
One widely adopted way is to encode the texts of nodes with
a fixed
LM, and further treat the LM embeddings as features to train a GNN for message passing.
Recently, a few methods propose to utilize domain-adaptive pretraining (Gururangan et al., 2020) on TAGs and predict the graph structure using LMs (Chien et al., 2022; Yasunaga et al., 2022) to provide better LM embeddings for GNN.
Despite better results, these LM embeddings still remain unlearnable in the
GNN training phase.
Such separated training paradigm ensures the model scalability as the number of trainable parameters in GNNs is relatively small.
However, its performance is hindered by task and topology-irrelevant semantic modeling process.
To overcome these limitations, endeavors have been made (Zhu et al., 2021;
Li et al., 2021a; Yang et al., 2021;
Bi et al., 2021; Pang et al., 2022) to co-train GNNs and LM under a joint learning framework.
However, such co-training approaches suffer from severe scalability issues as all the neighbors need to be encoded by language models from scratch, incurring huge extra computation cost.
In practise, these models restrict the messagepassing to very few (e.g. 3 (Li et al., 2021a; Zhu et al., 2021)) sampled first-hop neighbors, resulting in severe information loss.
To summarize, existing methods of fusing LMs and GNNs suffer from either unsatisfactory results or poor scalability.
In contrast to these methods, GLEM uses a pseudo-likelihood variational framework to integrate an LM and a GNN, which allows the two components to be trained separately, leading to good scalability.
Also, GLEM encourages the collaboration of both components, so that it is able to use both the textual semantics and structural semantics for representation learning, and thus enjoys better effectiveness.
Besides, there are also some efforts using GNNs for text classification.
Different from our approach, which uses GNNs to model the observed structural relationship between nodes, these methods assume graph structures are unobserved.
They apply GNNs on synthetic graphs generated by words in a text (Huang et al., 2019; Hu et al., 2019; Zhang et al., 2020) or co-occurrence pattern between texts and words (Huang et al., 2019; Liu et al., 2020a).
As structures are observed in the problem of node classification in text-attributed graphs, these methods cannot well address the studied problem.
Lastly, our work is also related to GMNN Qu et al.
(2019), which also uses a pseudo-likelihood variational framework for node representation learning.
However, GMNN aims to combine two
GNNs for general graphs and it does not consider modeling textual features.
Different from GMNN,
GLEM focuses on TAGs, which are more challenging to deal with.
Also, GLEM fuses a GNN and an LM, which can better leverage both structural features and textual features in a TAG, and thus achieves SOTA results on a few benchmarks.
B ACKGROUND
In this paper, we focus on learning representations for nodes in TAGs, where we take node classification as an example for illustration.
Before diving into the details of our proposed GLEM, we start with presenting a few basic concepts, including the definition of TAGs and how LMs and GNNs can be used for node classification in TAGs.
3. T EXT - ATTRIBUTED G RAPH
Formally, a TAG G S =
( V, A, s V ) is composed of nodes V and their adjacency matrix A ∈
R | V |×| V | , where each node n ∈ V is associated with a sequential text feature s n .
In this paper, we study the problem of node classification on TAGs.
Given a few labeled nodes y L of L ⊂ V , the goal is to predict the labels y U for the remaining unlabeled objects U =
V \ L .
Intuitively, node labels can be predicted by using either the textual information or the structural information, and representative methods are language models (LMs) and graph neural networks (GNNs) respectively.
Next, we introduce the high-level ideas of both methods.
3. L ANGUAGE M ODELS FOR N ODE C LASSIFICATION
Language models aim to use the sentence s n of each node n for label prediction, resulting in a text classification task (Socher et al., 2013; Williams et al., 2018).
The workflow of LMs can be characterized as below:
[Equation]
where SeqEnc θ is a text encoder such as a transformer-based model (Vaswani et al., 2017; Yang et al., 2019), which projects the sentence s n into a vector representation h n .
Afterward, the node label distribution y n can be simply predicted by applying an MLP θ with a softmax function to h n .
Leveraging deep architectures and pre-training on large-scale corpora, LMs achieve impressive results on many text classification tasks (Devlin et al., 2019; Liu et al., 2019).
Nevertheless, the memory cost is often high due to large model sizes.
Also, for each node, LMs solely use its own sentence for classification, and the interactions of nodes are ignored, leading to suboptimal results especially on nodes with insufficient text features.
3. G RAPH N EURAL N ETWORKS FOR N ODE C LASSIFICATION
Graph neural networks approach node classification by using the structural interactions between nodes.
Specifically, GNNs leverage a message-passing mechanism, which can be described as:
[Equation]
where σ is an activation function, MSG( · ) and AGG( · ) stand for the message and aggregation functions respectively, NB( n ) denotes the neighbor nodes of n .
Given the initial text encodings h (0) n , e.g. pre-trained LM embeddings, a GNN iteratively updates them by applying the message function and the aggregation function, so that the learned node representations can well capture the structural interactions between nodes.
Afterward, an MLP is applied to the final node representation for label prediction.
With the message-passing mechanism, GNNs are able to effectively leverage the structural information for node classification.
Despite the good performance on many graphs, GNNs are not able to well utilize the textual information, and thus GNNs often suffer on nodes with few neighbors.
[Figure]
M ETHODOLOGY
In this section, we introduce our proposed approach which combines GNN and LM for node representation learning in TAGs.
Existing methods either suffer from scalability issues or have poor results in downstream applications such as node classification.
Therefore, we are looking for an approach which enjoys both good scalability and capacity.
Towards this goal, we take node classification as an example and propose GLEM.
GLEM leverages a variational EM framework, where the LM uses the text information of each sole node to predict its label, which essentially models the local marginal label distribution of each node; whereas the
GNN leverages the text and label information of surrounding nodes for label prediction, which characterizes the global conditional label distribution.
The two modules are optimized by alternating between an E-step and an M-step.
In the E-step, we fix the LM, and the GNN is optimized by using the node representations learned by the LM as features and the node labels inferred by the LM as target.
By doing this, the GNN can effectively capture the global correlations of nodes for precise label prediction.
In the M-step, we fix the GNN and let the LM mimic the labels inferred by the
GNN, allowing the global knowledge learned by the GNN to be distilled into the LM.
With such a framework, the LM and GNN can be trained separately, leading to better scalability.
Meanwhile, the
LM and GNN are encouraged to benefit each other, without the sacrifice of model performance.
4. T HE P SEUDOLIKELIHOOD V ARIATIONAL F RAMEWORK
Our approach is based on a pseudo-likelihood variational framework, which offers a principled and flexible formalization for model design.
To be more specific, the framework tries to maximize the log-likelihood function of the observed node labels, i.e., p ( y L | s V , A ) .
Directly optimizing the function is often hard due to the unobserved node labels y U , and thus the framework instead optimizes the evidence lower bound as below:
[Equation]
(3) where q ( y U | s U ) is a variational distribution and the above inequality holds for any q .
The ELBO can be optimized by alternating between optimizing the distribution q (i.e., E-step) and the distribution p (i.e., M-step).
In the E-step, we aim at updating q to minimize the KL divergence between q ( y U | s U ) and p ( y U | s V , A, y L ) , so that the above lower bound can be tightened.
In the M-step, we then update p towards maximizing the following pseudolikelihood (Besag, 1975) function:
[Equation]
The pseudo-likelihood variational framework yields a formalization with two distributions to maximize data likelihood.
The two distributions are trained via a separate E-step and M-step, and thus we no longer need the end-to-end training paradigm, leading to better scalability which naturally fits our scenario.
Next, we introduce how we apply the framework to node classification in TAGs by instantiating the p and q distributions with GNNs and LMs respectively.
4. P ARAMETERIZATION
The distribution q aims to use the text information s U to define node label distribution.
In GLEM, we use a mean-field form, assuming the labels of different nodes are independent and the label of each node only depends on its own text information, yielding the following form of factorization:
[Equation]
As introduced in Section 3.2, each term q θ ( y n | s n ) can be modeled by a transformer-based LM q θ parameterized by θ , which effectively models the fine-grained token interactions by the attention mechanism (Vaswani et al., 2017).
On the other hand, the distribution p defines a conditional distribution p ϕ ( y n | s V , A, y V \ n ) , aiming to leverage the node features s V , graph structure A , and other node labels y V \ n to characterize the label distribution of each node n .
Such a formalization can be naturally captured by a GNN through the message-passing mechanism.
Thus, we parameterize p ϕ ( y n | s V , A, y V \ n ) as a GNN p ϕ parameterized by ϕ to effectively model the structural interactions between nodes.
Note that the
GNN p ϕ takes the node texts s V as input to output the node label distribution.
However, the node texts are discrete variables, which cannot be directly used by the GNN.
Thus, in practice we first encode the node texts with the LM q θ , and then use the obtained embeddings as a surrogate of node texts for the GNN p ϕ .
In the following sections, we further explain how we optimize the LM q θ and the GNN p ϕ to let them collaborate with each other.
4. E- STEP : LM O PTIMIZATION
In the E-step, we fix the GNN and aim to update the LM to maximize the evidence lower bound.
By doing this, the global semantic correlations between different nodes can be distilled into the LM.
Formally, maximizing the evidence lower bound with respect to the LM is equivalent to minimizing the KL divergence between the posterior distribution and the variational distribution, i.e.,
KL ( q θ ( y U | s U )
||
p ϕ
( y U | s V , A, y L )) .
However, directly optimizing the KL divergence is nontrivial, as the KL divergence relies on the entropy of q θ ( y U | s U ) , which is hard to deal with.
To overcome the challenge, we follow the wake-sleep algorithm (Hinton et al., 1995) to minimize the reverse KL divergence, yielding the following objective function to maximize with respect to the LM q θ :
[Equation]
which is more tractable as we no longer need to consider the entropy of q θ
( y U | s U ) .
Now, the sole difficulty lies in computing the distribution p ϕ
( y n | s V , A, y L ) .
Remember that in the original GNN which defines the distribution p ϕ
( y n | s V , A, y V \ n ) , we aim to predict the label distribution of a node n based on the surrounding node labels y V \ n .
However, in the above distribution p ϕ ( y n | s V , A, y L ) , we only condition on the observed node labels y L , and the labels of other nodes are unspecified, so we cannot compute the distribution directly with the GNN.
In order to solve the problem, we propose to annotate all the unlabeled nodes in the graph with the pseudo-labels predicted by the LM, so that we can approximate the distribution as follows:
[Equation]
where ˆy U \ n = { ˆy s } s ∈ U \ n with each ˆy s ∼ q θ ( y s | s s ) .
Besides, the labeled nodes can also be used for training the LM.
Combining it with the above objective function, we obtain the final objective function for training the LM:
[Equation]
where α is a hyperparameter.
Intuitively, the second term (cid:80) n ∈ L log q ( y n | s n ) is a supervised objective which uses the given labeled nodes for training.
Meanwhile, the first term could be viewed as a knowledge distilling process which teaches the LM by forcing it to predict the label distribution based on neighborhood text-information.
4. M- STEP : GNN O PTIMIZATION
During the GNN phase, we aim at fixing the language model q θ and optimizing the graph neural network p ϕ to maximize the pseudo-likelihood as introduced in equation 4.
To be more specific, we use the language model to generate node representations h V for all nodes and feed them into the graph neural network as text features for message passing.
Besides, note that equation 4 relies on the expectation with respect to q θ , which can be approximated by drawing a sample ˆy U from q θ ( y U | s U ) .
In other words, we use the language model q θ to predict a pseudo-label ˆy n for each unlabeled node n ∈ U , and combine all the labels { ˆy n } n ∈ U into ˆy U .
With both the node representations and pseudo-labels from the LM q θ , the pseudo-likelihood can be rewritten as follows:
[Equation]
where β is a hyperparameter which is added to balance the weight of the two terms.
Again, the first term can be viewed as a knowledge distillation process which injects the knowledge captured by the
LM into the GNN via all the pseudo-labels.
The second term is simply a supervised loss, where we use observed node labels for model training.
Finally, the workflow of the EM algorithm is summarized in Fig.
The optimization process iteratively does the E-step and the M-step.
In the E-step, the pseudo-labels predicted by the GNN together with the observed labels are fed into the LM for model training.
In the M-step, the LM provides both text embeddings and pseudo-labels for the GNN, which are treated as input and target respectively for label prediction.
Once trained, both the LM in E-step (denoted as GLEM-LM) and the GNN (denoted as GLEM-GNN) in M-step can be used for node label prediction.
In our experiments, we empirically compare the performance of them.
E XPERIMENTS
In this section, we conduct experiments to evaluate the proposed GLEM framework, where two settings are considered.
The first setting is transductive node classification, where given a few labeled nodes in a TAG, we aim to classify the rest of the nodes.
Besides that, we also consider a structurefree inductive setting, and the goal is to transfer models trained on labeled nodes to unseen nodes, for which we only observe the text attributes without knowing their connected neighbors.
5. E XPERIMENTAL S ETUP
Datasets.
Three TAG node classification benchmarks are used in our experiment, including ogbnarxiv, ogbn-products, and ogbn-papers100M (Hu et al., 2020).
The statistics of these dataset are shown in Table 1.
Compared Methods.
We compare GLEM-LM and GLEM-GNN against LMs, GNNs, and methods combining both of worlds.
For language models, we apply DeBERTa
He et al.
(2021) to our [Table] [Table] setting by fine-tuning it on labeled nodes, and we denote it as LM-Ft .
For GNNs, a few wellknown GNNs are selected, i.e., GCN (Kipf & Welling, 2017) and GraphSAGE (Hamilton et al., 2017).
Three top-ranked baselines on leaderboards are included, i.e., RevGAT (Li et al., 2021b),
GAMLP (Zhang et al., 2022), SAGN (Sun & Wu, 2021).
For each GNN, we try different kinds of node features, including (1) the raw feature of OGB, denoted as X OGB ; (2) the LM embedding inferenced by pre-trained LM, i.e. the DeBERTa-base checkpoint 1 , denoted as X PLM ; (3) the GIANT (Chien et al., 2022) feature, denoted as X GIANT .
Implementation Details.
We adopt the DeBERTa
(He et al., 2021) as the LM model and fine-tune it for node classification on each dataset to provide initial checkpoints for LMs.
During optimization,
GLEM can start with either the E-step or the M-step, and the priority is decided according to the performance.
For example, if GNN outperforms LM, we will start with the M-step for GNN training.
To provide embedding and predictions for the first GNN M-step, we use a pre-trained DeBERTa and fine-tune it on node classification on each tasks.
To provide embedding and predictions for the firstLM E-step, we use a pre-trained GNN predictions, e.g. the original GNN predictions, for the initial target labels.
The best EM-iteration is chosen based on the validation accuracy of GLEM-GNN.
For fair comparison against other feature learning methods such as GIANT, the hyper-parameters of GNNs are set to the best settings described in the paper or in the official repository, and other parameters are tuned by grid search. The code is available in the suplementary materials.
5. T RANSDUCTIVE N ODE C LASSIFICATION
Main Results.
Next, we evaluate GLEM in the transductive setting.
The results of three OGB datasets are presented in Table 2.
For LMs, we see that fine-tuned LMs (LM-Ft) have competitive results, showing the importance of text attributes in a TAG.
By further leveraging the structural information for message passing, our approach (GLEM-LM) achieves significant improvement over
LMs, which demonstrates its advantage over LMs.
For GNN-based methods, we see that for each GNN model, using OGB and GIANT node embeddings as GNN inputs ( X OGB and X GIANT ) yields strong results.
However, these embeddings remain [Table] [Table] unchanged during training.
By dynamically updating the LM to generate more useful node embeddings for the GNN, the GNN of our approach (GLEM-GNN) significantly outperforms all the other methods with fixed node embeddings in most cases.
Notably, GLEM-GNN achieves new state-of-the-art results on all three TAG datasets on the OGB benchmark.
Scalability.
One key challenge of fusing LMs and GNNs lies in the scalability.
When using large
LMs with numerous parameters, the combined method will suffer from severe scalability problems.
GLEM eases the challenge through the EM-based optimization paradigm, allowing it to be adapted to large LMs.
To verify this claim, we train GLEM with DeBERTa-large (He et al., 2021) on ogbn-arxiv.
The results are reported in Table 3.
We observe that GLEM is able to generalize to
DeBERTa-large with about 0.4B parameters, showing the appealing scalability.
Besides, for every
LM, applying GLEM yields consistent improvement, which proves the effectiveness of GLEM.
5. S TRUCTURE - FREE I NDUCTIVE N ODE C LASSIFICATION
Besides the transductive setting, inductive settings are also important, where we aim to train models on nodes of a training graph, and then generalize models to new nodes of test graphs.
In many real cases, these new nodes are often low-degree nodes or even isolated nodes, meaning that we can hardly use the structural information for node classification.
Therefore, we consider a challenging setting named structure-free inductive setting, where we assume for each test node, we only observe its text attributes without any connected neighbors.
For this setting, we consider different methods for label prediction, including GNN models (GNN), neural networks without using structural information (MLP), and GLEM.
The results are shown in Table 4.
We can see that, compared with that of MLPs and LMs, the structure-free inductive setting is a more challenging task especially for GNNs where a sheer performance drop is observed.
Meanwhile, by effectively fusing with graph learning, GLEM-LM is able to consider local semantics as well as neighboring structural information, leading to more accurate structure-free inductive predictions.
Besides, the generated embeddings are able to boost other models (e.g., see X GLEM -deep
in the
MLP and LM sections), enabling both MLP and GNNs with better structure-free inference ability.
[Table]
[Figure] [Figure]
ogbn
[Figure]
arxiv
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
ogbn
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
ogbn
[Figure]
products
[Figure]
[Figure]
[Figure]
GLEM
[Figure]
5. C OMPARISON OF D IFFERENT T RAINING P ARADIGMS
As discussed before, besides directly fine-tuning LM (denoted as LM-Ft ), a few training paradigms have been proposed for fusing GNNs and LMs.
One paradigm is to use fixed/static LMs to generate node embeddings for GNN to do label prediction (denoted as Static ).
Besides that, another paradigm is to restrict message passing to a few sampled neighbors, so that the memory cost can be reduced (denoted as Joint ).
In this section, we compare our proposed paradigm GLEM ) against the others.
For each paradigm, we choose two models trained with it.
The results are presented in Table 5.
we see that although static training has the best efficiency, its classification accuracy is not very high due to the restricted model capacity caused by the fixed LM.
On the other hand, the joint training paradigm has the worst efficiency and effectiveness due to the reduced graph structure.
Finally, our proposed paradigm achieves the optimal classification results, thanks to its ability to encourage the collaboration of LMs and GNNs.
Meanwhile, our paradigm remains close to static training in terms of efficiency (time/epoch).
To summarize, our proposed approach achieves much better results than other paradigms without the sacrifice of efficiency.
5. C ONVERGENCE A NALYSIS
In GLEM, we utilize the variational EM algorithm for optimization, which consists of an E-step training GLEM-LM and an M-step training GLEM-GNN in each iteration.
Here, we analyze the convergence of GLEM by looking into the training curves of validation accuracy on ogbn-arxiv and
OGB-Products.
From the results in Fig.
We can clearly see that with each E-step and M-step, both the performance of GLEM-GNN and the GLEM-LM consistently increase to a maximum point and converge in a few iterations.
Notably, GLEM takes only one iteration to converge on the ogbn-arxiv dataset, which is very efficient.
C ONCLUSION
This paper studies how to fuse LMs and GNNs together for node representation learning in TAGs.
We propose an approach GLEM based on a pseudo-likelihood variational framework.
GLEM alternatively updates the LM and GNN via an E-step and an M-step, allowing for better scalability.
In each step, both GNN and LM are mutually enhanced by learning from pseudo-labels predicted by the other module, fusing graph and language learning together.
Extensive experiments on multiple datasets in two settings demonstrate the effectiveness of GLEM.
A S ENSITIVITY A NALYSIS
Recall that GLEM fuses LMs and GNNs by training them separately with an E-step and an M-step, where at each step both the observed labels and pseudo-labels are used for model training as shown in Eq.8 and Eq.9. For both objectives, we introduce a coefficient, i [Figure] e., α and β ) denoted as LMPL-weight and GNN-PL-weight respectlively, to control the relative weight of pseudo-labels.
Next, we systematically analyze these two hyperparameters by investigating how the performance varies when changing the hyperparameters.
We treat GCN (Kipf & Welling, 2017) and RevGAT (Li et al., 2021b) as the backbone GNNs. The results on the OGBN-Arxiv dataset are shown in Figure 3 and Figure 4 respectively. On one hand, we can clearly see that the LM-PL-weight α is an important parameter, as GLEM-LM provides both node feature and pseudo-labels for GLEM-GNN.
We also observe that guiding LM with pseudo labels of GNN consistently boosts the performance of both LM and GNN compared with optimizing LM with gold label only, i.e. α = 0 , demonstrating the effectiveness of fusing the knowledge of GNN into LM.
On the other hand, the GNN-PL-weight α that balance the important of LM pseudo-labels in training GLEM-GNN is not very sensitive.
Lastly, we also observe that for different GNN and LM, the optimal α and β varies, indicating these parameters should be carefully selected.
B R EPRODUCIBILITY S TATEMENT
We provide our code in the supplementary material along with a detailed instructions on conducting experiments.
Our experimental setting is stated in Section 5.1, the important hyper-parameters are discussed in the Appendix.