S PEC TRA:
S PECTRAL
T RANSFORMER
FOR
G RAPH
R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Several positional and structural encodings are recently proposed into vanilla transformer architecture to overcome its inability to model graph’s positional invariance and topology.
However, in addition to graph topology, graph signals could be multi-channeled and contain heterogeneous information which cannot be inherently captured by the transformer.
Hence, we propose an approach to induce a spectral module into the transformer architecture to enable decomposition of graph spectrum and selectively learn useful information akin to filtering in the frequency domain. Empirical results suggest that SpecTRA provides homogeneous performance gain against vanilla transformer across all tasks on standard benchmarks.
Furthermore, incorporating SpecTRA instead of vanilla transformer model with recently proposed position encoding schemes have resulted in comparable or better performance than existing transformer and GNN based architectures.
I NTRODUCTION
Several graph neural network (GNN) approaches have been devised as generic and efficient framework to learn from graph-structured data for tasks such as graph classification, node classification, graph regression, and link property prediction (Zhou et al., 2020).
Among them, message-passing
GNNs (MPGNNs) have been prominently used to obtain latent encoding of graph structures, achieving good results on related tasks (Gilmer et al., 2017;
Veli ˇ ckovi ´ c et al., 2018; Xu et al., 2019a).
Although effective, these methods suffer from performance issues such as over-smoothing (Zhao &
Akoglu, 2020), suspended animation (Zhang & Meng, 2019), and over-squashing (Alon & Yahav, 2021).
Recently, researchers (Zhang et al., 2020; Li et al., 2018) have attempted to use transformers (Vaswani et al., 2017) for graph representation learning.
However, transformers are inherently incapable of learning the topological information of graphs (Dwivedi et al., 2020).
Hence, more recent works have added a gamut of effective positional and structural encoding methods (Mialon et al., 2021; Kreuzer et al., 2021) to alleviate limitation of transformers to learn topological information.
Background and Hypothesis: Viewed from classical signal processing domain, signals on a graph could contain heterogeneous information spread over a wider frequency domain (Ortega et al., 2018).
The message passing graph convolutional networks (MPGCNs) such as GCN (Kipf &
Welling, 2017) and GAT (Veliˇckovi´c et al., 2018) have low-pass characteristics (Muhammet et al., 2020).
Furthermore, researchers aim go beyond static low or highpass filters to design adaptive filters to capture entire graph spectrum in GNNs (Gao et al., 2021; Bo et al., 2021; Chien et al., 2021).
We argue that a similar issue plagues the transformers that cannot effectively segregate the noise from the signal for the entire spectral components of a graph.
Gleaning from the work on MPGNNs as transformer can be special case of GNNs (Joshi, 2020), we hypothesize that inducing transformers with the ability to selectively use the signals in the frequency domain will enable better representation learning on a broader range of tasks.
As graphs have different topologies, we further postulate that having an approach to dynamically select filters, as opposed to dataset-specific single static filter, will be more expansive to permit effective learning of graph representations using transformers.
Proposed Approach: This work, instead of devising a new position encoding scheme, proposes a novel direction. Our work aims to empower transformers to learn the essential components of the graph spectrum while filtering out the noise.
We propose SpecTRA that effectively integrates the attention of the transformer with the spectrum of the graph.
SpecTRA consists of a filtering module built on top of the vanilla transformer (Vaswani et al., 2017) to learn distinct filter coefficients for each input graph.
The transformer naturally gives diverse attention sub-spaces , which are utilized by SpecTRA to design multiple filters covering a broad spectrum of the graph.
Specifically, we utilize the spatial information learned by the attention heads of the transformer to dynamically decide the filter coefficients .
This enables interpreting the association between a particular frequency component with each head for a given graph.
Furthermore, it also helps select the useful range of information in the frequency domain for different sub-spaces (attention heads in SpecTRA’s case).
Our rationale emerges from the following: 1) If the graph spectrum consists of several underlying components, which are relevant for different sub-graphs, then it is paramount to dynamically learn filter coefficient.
This also provides an extra advantage concerning interpretability in spectral space per graph (cf., section 5.1).
In cases where certain classes are skewed, a task-specific filter would learn generic coefficients for the majority class, keeping a limit on the number of filters.
SpecTRA handles these special cases by learning filters after observing the input graph (cf., section A.5).
Contributions: To summarize, this work make the following key contributions:
• Our primary contribution is SpecTRA, a novel approach for empowering vanilla transformer with the ability to perform graph-specific filtering in multiple sub-spaces of the attention heads.
We study the efficacy of SpecTRA by conducting extensive experiments on standard benchmark datasets of graph classification/regression and node classification resulting in superior performance compared to vanilla transformer.
• We provide an exhaustive empirical study on the effect of inducing a variety of recently proposed transformer-based positional encoding schemes (Dwivedi et al., 2020; Kreuzer et al., 2021; Mialon et al., 2021) into the SpecTRA architecture. Empirical results show comparable or better performance against current state-of-the-art transformer and GNN based approaches. Hence, transformers, when induced with the ability to decompose and attend to signals spectrally, can effectively capture representations on a wide range of tasks complementing the effect of position encoding schemes.
The remainder of the paper is organized as follows: we describe the related works in section 2.
Section 3 provides the preliminaries and problem definition.
In section 4, we explain the proposed approach for graph specific dynamic filtering.
Dataset details, experimental results, and ablations are given in sections 5 and 5.1.
Section 6 concludes the paper.
R ELATED W ORK
GNNs and Graph Transformers: In this section, we stick to the work closely related to our approach (detailed survey in (Chen et al., 2020b)).
Since the early attempts for GNNs (Scarselli et al., 2008), many variants of the message passing scheme were developed for graph structures such as
GCN (Kipf & Welling, 2017), GIN (Xu et al., 2018), and GraphSAGE (Hamilton et al., 2017).
The message passing paradigm employs neural networks for updating representation of neighboring nodes by exchanging messages between them.
The use of transformer-style attention to GNNs for aggregating local information within the graphs is also an extensive research topic in recent literature (Thekumparampil et al., 2018; Shi et al., 2020; Li et al., 2018).
Dwivedi & Bresson (2020) use the eigenvectors of the graph laplacian to induce positional information into the graph.
Kreuzer et al.
(2021) propose a learnable position encoding module that applies a transformer on the eigenvectors and eigenvalues of the graph laplacian.
Ying et al.
(2021) provide the concept of relative positional encoding in which the positional information is induced in the attention weights rather than in the input by obtaining correlation matrices of the spatial, edge, and centrality encoding.
Similarly, Mialon et al.
(2021) induce relative position information in the form of diffusion and random walk kernels along with structural information using the Graph Convolutional Kernel network (GCKN).
In contrast with existing transformer-based attempts that inherit vanilla transformer model (Vaswani et al., 2017) for inducing position and topological encoding, we aim to improve the capabilities of the vanilla transformer model by empowering it to decompose the spectrum of a graph.
Filters on Graphs: Filtering in the frequency domain is generalized to graphs using the spectral graph theory (Chung et al., 1997; Shuman et al., 2013).
The GCN model (Kipf & Welling, 2017) and variants such as (Zhang & Meng, 2019) approximate convolution for graph structures in the spatial domain.
However, these models suffer from operating in the low-frequency regime, leaving rich information in graph data available in the middle- and high-frequency components (Gao et al.,
Other approaches in the spectral domain attempt to reduce the computationally complex eigen decomposition of the laplacian by adopting certain functions of the graph laplacian such as
Chebyshev polynomials (Defferrard et al., 2016), Cayley polynomials (Levie et al., 2018), and auto regressive moving average (ARMA) filters (Isufi et al., 2016).
These approaches focus on designing specific filters with desirable characteristics such as bandpass and highpass.
Our work focuses on the design of learnable filters, whose frequency response can be represented in polynomial functions in multiple sub-spaces of the signal.
Gao et al.
(2021) is closely related to part of our work which also designs filter banks, albeit task-specific, for heterogeneous and multi-channel signals on graphs.
However, our work differs from Gao et al.
(2021) in that we learn graph-specific filters to enable interpretability in spectral space per graph and filtering into the transformer architecture.
P RELIMINARIES AND P ROBLEM D EFINITION
Graph Fourier Transform: We denote a graph as ( V , ϵ ) where V is the set of N nodes and ϵ represents the edges between them.
The adjacency matrix is denoted by A .
Here, we consider the setting of an undirected graph, hence, A is symmetric.
The diagonal degree matrix D is defined as ( D ) ii =
(cid:80) j ( A ) ij .
The normalized laplacian L of the graph is defined as L =
I − D − 12 AD − 12 .
The laplacian L can be decomposed into its eigenvectors and eigenvalues as:
[Equation]
where U is the N × N matrix; the columns of which are the eigenvectors corresponding to the eigenvalues λ 1 , λ 2 , . .
. , λ N and Λ
= diag( λ 1 , λ 2 , . .
λ n ]) .
Let X ∈ R N ×
d
be the signal on the nodes of the graph.
The Fourier Transform ˆ X of X is then given as: ˆ X
= U ∗ X .
Similarly, the inverse Fourier Transform is defined as: X
= U ˆ X .
Note U ∗ is the transposed conjugate of U .
By the convolution theorem (Blackledge, 2005), the convolution of the signal X with a filter G having its frequency response as ˆ G is given by (below, v m is the m th node in the graph):
[Equation]
Transformers and Position encoding:
The recent works (Dwivedi & Bresson, 2020; Kreuzer et al., 2021) established the necessity of encoding schemes for inducing positional encoding into the transformer architecture.
The relative encoding schemes proposed by GraphiT (Mialon et al., 2021) use diffusion and random walk kernels for relative positional encodings.
The diffusion kernel K D for a graph with laplacian L is given by the equation below,
[Equation]
For physical interpretation, diffusion kernels can be interpreted as the amount of a substance that accumulates at a given node if injected into another node and allowed to diffuse through the graph.
Similarly, the random walk kernel generalizes this notion of diffusion for fixed-step walks on the graph.
It is described by the equation below,
[Equation]
We can see that above equation becomes the diffusion kernel if γ = βp and p −→ inf .
However, the difference with respect to the diffusion kernel is that the random walk kernel is sparse.
GraphiT also makes use of GCKN (Chen et al., 2020a) for learning structural position encoding and we inherit it in our approach describe in Section 4.
We refer the readers to (Mialon et al., 2021) for more details.
Spectral GNN: Spectral GNNs rely on the spectral graph theory (Chung et al., 1997).
Consider a graph with U as its eigen vectors, λ the eigen values, and L the laplacian.
The graph convolution operation in the frequency domain can be written as below,
[Equation]
where H lj is the feature vector of the j th node in the ( l ) th layer, d l is the dimension of the signal in the l th layer, G i,j,l is the learnable weight vector for the
i th and j th node in the l th layer.
This formulation is non-transferable to multigraph learning problem (Muhammet et al., 2020).
Thus,
G i,j,l is re-parametrized as below,
[Equation]
where B ∈ R N × S , S is the number of convolutional supports and W l,s is the learnable matrix.
The
Equation 4 depends on the computation of the eigen vectors U of L , which is computationally costly for large graphs.
In our work, we consider the polynomial approximations as proposed by Hammond et al.
Specifically, the frequency response of the desired filter can be approximated as,
[Equation]
where T ( k ) is the polynomial basis such as Chebyshev polynomials (Defferrard et al., 2016), ˜Λ = 2Λ λ max − I , λ max is the maximum eigen value and α k is the corresponding filter coefficients .
A recursive formulation could be used for the Chebyshev polynomials with basis T 0 ( x ) = 1 , T 1 ( x ) = x and beyond that T k ( x ) = 2 xT k − 1 ( x ) − T k − 2 ( x ) .
Thus, the convolution operation in Equationcan be approximated as
[Equation]
This corresponds to an FIR filter of order K (Smith et al., 1997).
Setting α as a learnable parameter help us learn the filter for the upstream task.
Problem definition In this work, we aim to learn the filter coefficients from the attention weights of the transformer.
Formally, given the transformer attention for the head h at layer l as A h,l ∈ R N × N , we aim to define a mapping M : R N × N −→ R K where K is the filter order.
The mapping M would take us from the space in the attention weights to the filter coefficient space.
S PEC TRA:S PETRAL T RANSFORMER
In this section, we present our approach, SpecTRA.
We first describe the notion of dynamic filtering module of SpecTRA architecture followed by its position encoding module (cf., Figure 1).
4. M ULTI H EADED A TTENTION WITH D YNAMIC F ILTERING MODULE
We view the graph as set of node features to be fed to the vanilla transformer (Vaswani et al., 2017) that learns the pairwise similarity between these nodes using its attention mechanism as follows:
[Equation]
Here, Q T = W hQ
X T and K T
= W hK X T where W hQ ,
W hK ∈ R d out × d in are the projection matrices for the query and key respectively for the head h .
The output X h at the head h can be obtained from:
[Equation]
Learning filter coefficients :
In order to obtain the filter coefficients from the weights of each attention head, we use a message passing framework.
The spectral and spatial domains are interchangeable, and transformers traditionally learn good latent representations of spatial domain (Vaswani et al., 2017; Devlin et al., 2019).
Hence, attention map provides inference to the connectivity of a graph implicitly, using which, we can deduce the frequency band to filter out the noise.
If each node’s signal x i is considered with its neighborhood N ( x i ) which represents the non-zero attention weights obtained from Equation 7, then the message passing is defined as:
[Equation]
where x li and x lim are the signals and the aggregated message at node x i in the layer l .
Here, A and U are the aggregation and update functions respectively.
This framework enables the usage of popular message passing schemes such as GCN (Kipf & Welling, 2017).
For example, in GCN, the aggregation is a simple projection of the signals followed by a summation weighted by the normalized laplacian with self loops.
Next, the updation is an activation function such as ReLU.
The final equation is now represented as below:
[Equation]
where x h,lj is the node embedding for the j th node in the l -th layer of the GNN(to learn coefficients) for the h -th head of the transformer,
L = I − D − [Equation] − 12 is the normalized laplacian and W lp is the learnable projection matrix at layer l .
This GNN is common for all the heads and at all layers of the transformer.
Here, the node embeddings reside in the coefficient space i.e. X ∈ R N × K , where K is the order of the filter.
For learning the filter coefficients we initialize the node embeddings with a prior of the filter depending on the task.
For an all-pass filter we could use a vector of all ones as the initialization.
This choice of prior is justified by property 4.1 (cf., Appendix A.1 for proof).
Property 4.1.
The filter coefficients consisting of the vector of all ones is an all-pass filter.
After the L -th message passing, the vector obtained from Equation 9 is given to a readout function such as global average pooling followed by a simple feed forward network to obtain the final filter coefficients α h , per attention head, as below:
[Equation]
where x hi is the vector at node x i in the h -th attention head obtained from Equation 9.
Dynamic Filtering Module : We use the filter coefficient α h to obtain the appropriate filter frequency response defined as:
[Equation]
where K is the filter order.
The desired filter response H h at head h can then be obtained from ˆ G h as observed in Eq 6 as:
[Equation]
The filter outputs from each head is concatenated to get the filtered output which is further concatenated with the attention output X , followed by an MLP, with appropriate normalizations, for the output of the encoder layer:
[Equation]
This could then be used in the upstream task of classification, regression, etc.
In order to learn distinct filter coefficients for each head we add a regularization term to the objective.
The regularization tries to keep the coefficient vectors orthogonal to each other.
It does so by taking the Frobenius norm (Horn & Johnson, 1990) of the gram matrix of X whose columns consist of the coefficient vector of each head.
Formally, define α i ∈ R k and X =
[ α 1 , α 2 , . . .
α h ] ∈ R k
×
h
where h is the number of heads and k is the filter order.
The regularization term is given by ∥ ( X T X ) ⊙ ( 1 − I ) ∥ 2 , where
I ∈
R h × h is the identity matrix and ⊙ is the hadamard product.
The below theorem (proof in
Appendix A.1) justifies the proposed method.
Theorem 4.1.
Assume the desired filter response G ( x ) has m + 1 continuous derivatives on the domain [ − 1 , 1] .
Let S Tn G ( x ) denote the n th order approximation by the polynomial(chebyshev) filter and S T ′ n G ( x ) denote the learned filter, C f be the first absolute moment of the distribution of the fourier magnitudes of f (function learned by the network), h the number of hidden units in the network and N the number of training samples.
Then the error between the learned and desired frequency response is bounded by the below expression
[Equation]
The above theorem argues that the Chebyshev polynomials are able to learn any smooth function.
Using injective aggregators in GNNs and universal approximators in MLP, we can approximate these coefficients to the desired precision given by the bounds.
The input to the GNNs could be divided into two parts: the input graph indicating the connectivity of nodes and the signals on the graph.
In this work, we aim to study if the spectral components could be identified by using the spatial connectivity pattern of the signals on the original graph obtained from the attention maps of the transformer.
This justifies the proposed architecture, in which a graph is constructed with the edge weights taken from the attention weights and fed to a GNN for learning the filter coefficients.
These coefficients are then used in a spectral GNN in which the input is the original graph with node embeddings as learned from the transformer.
The output of this is then fed to further transformer layers, and the process is repeated.
The input to the first transformer is the node/edge attributes itself.
We leave using the connectivity structure imposed by the original graph for future works.
4. P OSITIONAL ENCODING SCHEMES
In SpecTRA, we learn the pairwise similarity between graph nodes inheriting the attention mechanism of vanilla transformer as follows:
[Equation]
Here Q T =
W hQ
X T , K T =
W hK X T and V T
= W hV X T , where W hQ , W hK , W hV ∈ R d out × d in are the projection matrices for the query, key and values respectively for the head h .
Following GraphiT
(Mialon et al., 2021), we share the weight matrices for the query and key matrices for learning a positive semi-definite kernel.
For the input features we use the node attributes, if provided, along with the static laplacian position encoding, as in (Dwivedi & Bresson, 2020).
The static encoding in the form of the laplacian eigen vectors is simply added to the node embeddings.
For the relative positional encoding we follow GraphiT and use the diffusion ( K D ) and random walk ( K rw ) kernels (cf., Equations 2 and 3).
The attention using the relative positional encoding schemes is now:
[Equation]
[Equation]
Here, K p is the respective kernel being used i.e. K p ∈ K D , K rw .
We also borrow from the positional encoding scheme of SAN (Kreuzer et al., 2021) that allow usage of the edge features
E ∈ R N × N × d in .
Formally, the attention weights w kl ij between the nodes i and j in the l th layer and k th attention head is given by the below equations
[Equation]
, if i and j are connected in sparse graph [Equation] sparse graph
[Equation] where W 1 ,k,lQ , W 1 ,k,lK , W 1 ,k,lE are the projection matrices corresponding to the query, key and edge vectors for the real edges and W 2 ,k,lQ , W 2 ,k,
[Equation]
[Equation] [Equation] [Equation] are the projection matrices corresponding to the respective vectors for the added edges in the k th head and l th layer as in (Kreuzer et al., 2021).
We further employ GCKN (Chen et al., 2020a) to encode graph’s topological properties.
Limitations The space and time complexity of our method is O ( N 2 ) for full attention.
This could be alleviated by using sparse attention (cf., Table 2).
One may also use kernel methods as in (Choromanski et al., 2021) to reduce the number of nodes in the graph and then apply the transformer on the reduced graph.
Similar to vanilla transformer,SpecTRA cannot induce positional encoding on its own.
It is yet to be explored if learning position encodings could be incorporated into the proposed method.
We leave these directions of exploration to future works.
E XPERIMENT R ESULTS
We aim to answer following research questions: RQ1: Can multi-headed attention combined with dynamic graph filtering improve SpecTRA’s ability over base transformer for graph representation learning?
RQ2: What is the impact of recently proposed position encoding schemes on SpecTRA?
RQ3: What is the efficacy of graph-specific filters on the performance of SpecTRA?
Datasets, Settings and Baselines : We use widely popular datasets (Kreuzer et al., 2021; Mialon et al., 2021;
Hu et al., 2020): MUTAG, NCI1, and the OGBG-MolHIV for graph classification; PATTERN and CLUSTER for node classification; and ZINC for graph regression task.
Further details are in the Appendix A.2.
We borrow experiment settings from (Mialon et al., 2021; Kreuzer et al., 2021).
The GNN and transformer baselines are listed in Table 5.1.
SpecTRA Configurations: For a fair and exhaustive comparison, we provide six variants of SpecTRA: 1) SpecTra-Base to compare against vanilla transformer without position encoding module (cf., section 4.2), 2) SpecTRA+LapE contains static position encoding from Mialon et al.
SpecTRA+3RW consists of position encoding based on 3-step RW kernel from Mialon et al.
(2021), 4) SpecTRA+GCKN+3RW that uses GCKN
(Chen et al., 2020a)
in addition with 3-step RW kernel to induce graph topology, 5) SpecTRA+LPE+Full with learnable position encoding (Kreuzer et al., 2021) with full attention.
6) SpecTRA+LPE+Sparse inheriting learnable position encoding from
Kreuzer et al.
(2021), with sparse attention on the graphs.
5. R ESULTS
SpecTRA-Base outperforms vanilla transformer (Table 1) across all datasets establishing the positive impact of combining dynamic filtering with multi-headed attention into SpecTRA architecture (successfully answering RQ1) .
We observe (Table 2) that for the smaller MUTAG and NCI1 datasets,
SpecTRA achieves the best results against baselines using the structural encoding of GCKN.
It indicates that these datasets benefit more from structural information.
The results of the other SpecTRA variants are comparable or better than the transformer-based (SAN, GraphiT-LapE, GraphiT-3RW) and GNN baselines.
On OGBG-MolHIV and PATTERN/CLUSTER’s graph and node classification tasks, learnable position encoding has most positive impact on SpecTRA.
Graphformer reports the highest value on MolHIV.
However, its parameters are 47M compared to ≈ 500K from SpecTRA+LPE, GCN-based models, and SAN.
Notably, we see that SpecTRA performs exceptionally well on the graph regression task of ZINC with a relative decrease in the error of up to 50% .
Potential reason could be SpecTRA’s ability to learn diverse filters that are distinct for each graph (c.f, [Table] [Table]
[Table]
Appendix A.5). An important observation is that no position encoding scheme provides homogeneous performance gain across all datasets for SpecTRA and it supports our choice to use them as a plug-in (answering RQ2 ).
Hence, it is still an open research question to propose a position encoding scheme that provides homogeneous performance gain across all tasks agnostic of the datasets [Table]
Ablation Studies We created another SpecTRA configuration (SpecTRA-Static) where the filter is static per dataset based on attention heads.
Our idea here is to understand the impact of combining multi-headed attentions with a dynamic filter .
From Table 3, we clearly observe the empirical advantage of SpecTRA-Base.
More importantly, graph-specific filters aim for interpretability in spectral space per graph.
Similar to attention weights, learning graph-specific filters facilitates the understanding of which nodes interact in the spectral domain and which filter is helpful for the task.
In some cases, it may benefit from learning representations by aggregating from connected neighbors, which a low pass filter would do.
Whereas in other cases, the task would benefit learning from unconnected distant nodes, which may be graph dependent.
For instance, in Figure 3, we see that for sparse graphs (graph (a) - (c))
the filter response has a prominent magnitude for the lower components of the spectrum along with some components in the middle regions of the spectrum.
On the other hand, for relatively dense graphs, we see a relatively less prominent low frequency response and many heads learning to focus on the higher frequency components of the spectrum.
In a sense, this enables aggregation of nodes that are distant in the graph and helps in providing interpretation as to which nodes interact in the spectral domain .
Note that existing graph-specific attention mechanisms, such as GAT, learn only low pass filters (Muhammet et al., 2020) and cannot perform such an aggregation.
Also, the spectral GNNs (cf., section 2) that learn filters for the entire dataset are
[Figure]
unable to perform similar aggregation in a graph-specific manner which may be beneficial in some cases (cf., A.4).
Also, the learned filters have different frequency responses specific to the dataset as they are tuned to the data characteristics (cf., Figure 2).
These observations justify our rationale to propose graph-specific filters and successfully answering RQ3 .
We note, equation 11 represents [Figure] a polynomial filter using the Chebyshev polynomials (Defferrard et al., 2016).
Polynomial filters (Hammond et al., 2011) are smooth and have restrictions that they cannot model filter responses with sharp edges.
Hence, we devise SpecTRA-ARMA that uses rational filters such as ARMA (Isufi et al., 2016).
Our idea here is to study the empirical efficacy of these filters.
It can be observed from Table 3 that SpecTRA-ARMA resulted in lower performance than SpecTRA-Base on most of the datasets.
However, on ZINC, there is an improvement using ARMA filters as compared to Chebyshev polynomial filters.
Also on CLUSTER, we observer a sharp increase using SpecTRABase as compared to SpecTRA-Static but not with SpecTRA-ARMA.
Hence, we conclude that filter response and empirical predictive performance could be orthogonal objectives, and it becomes a trade-off to decide which type of filter to apply for a given task.
C ONCLUSION
In this work, we have introduced a novel transformer-based approach for graph representation learning.
Our model SpecTRA aims to effectively learn multiple filters per attention head to capture heterogeneous information spread over a wider frequency domain.
Furthermore, learning graphspecific filters provide interpretability in spectral space per graph, which is not the case in a datasetspecific static filter.
Experiments on standard datasets of a variety of tasks suggest a clear empirical edge on vanilla transformer.
Also, using recently proposed position encoding schemes as plug-ins in the flexible architecture of SpecTRA resulted in better or comparable state-of-the-art performance.
E THICS AND R EPRODUCIBILITY S TATEMENT
In this work, we present significant progress in graph representation learning using transformers.
Knowledge representation from the data (mainly graphs) is an important goal that human beings seek for reasoning along with the advancement of technology.
Many recently proposed transformer approaches rely on static frequency filters (including one of our configuration SpecTRA-Static) compared to our proposed dynamic filtering approach.
Furthermore, we employ widely used public datasets for graphs.
When it comes to who may be disadvantaged from this research, we do not think it is applicable since our study of addressing the dynamic filtering capabilities of the transformer, which is still at an early stage.
Additionally, there is no concrete evidence in the literature that signals on the graphs causes bias or ethical concerns for effectively learning graph representations.
Having said so, we are fully supporting the development of ethical and responsible AI.
The potential bias in the standard public datasets that may lead to wrong knowledge needs to be cleaned or corrected with validation mechanisms.
We are also aware of energy consumption of GPUs used in the empirical evaluations and support development of Green AI for sustainable development of ML research.
Regarding reproducibility: upon acceptance, we will publicly release code with detailed user instructions to reproduce our empirical values.
We have provided an additional appendix section including
GPU size and all other hyper parameters needed to reproduce this work.
Also, we have made sure that in methodology, all relevant equations are presented in the main section of the paper to clarify our proposed approach.
A A PPENDIX A. T HEORETICAL J USTIFICATION
Property A.1.
The filter coefficients consisting of the vector of all ones is an all-pass filter
Proof.
Consider the filter response given by the chebyshev coefficients T n of the first kind as below
[Equation]
where t i represent the coefficients for the i th polynomial.
It can be verified that the generating function for G above could be given by the below equation
[Equation]
setting t = 1 in this equation gives G ( x ) = 12 which does not depend on x .
Thus G would behave as an all-pass filter.
Note that t = 0 is also an alternative but this would cause a degenerate learning of the filter coefficients which may always remain 0 after aggregation and the MLP layers (in the absence of bias).
Theorem A.1.
Assume the desired filter response G ( x ) has m + 1 continuous derivatives on the domain [ − 1 , 1] .
Let S Tn G ( x ) denote the n th order approximation by the polynomial(chebyshev) filter and S T ′ n G ( x ) denote the learned filter, C f be the first absolute moment of the distribution of the fourier magnitudes of f (function learned by the network), h the number of hidden units in the network and N the number of training samples.
Then the error between the learned and desired frequency response is bounded by the below expression
[Equation]
Proof.
We begin by defining a bounded linear functional L on the space
C m +
[ − 1 , 1] with m +continuous derivatives as below
[Equation]
Since G ( x ) has m + 1 derivatives, it is approximated by a polynomial function of degree n > m .
By Peano’s kernel theorem (Peano, 1913), we can write LG as below
[Equation]
where
[Equation]
The notation ( . − t ) m + indicates
[Equation] if x > t otherwise
We note the n th order approximation of the chebyshev expansion of the function Gx is given by
[Equation]
where,
[Equation]
Thus we get S Tn ( x − t ) m + as
[Equation]
where,
[Equation]
If a Graph Neural Network approximates c km in the proposed method, then we would have an error due to this approximation.
We now find the bounds of this approximation.
As the current method relies on the connectivity pattern due to the signals of the nodes of the graph, we desire a different graph for different signal patterns and thus different filter coefficients.
Assuming an injective function (Xu et al., 2019b) for aggregation, we can always find a unique map for a unique graph corresponding to the frequency response.
Now we need to learn a function that maps the output of the previous network to the coefficients of the desired frequency response.
A classical neural network with h hidden units could approximate this by the below bound (Barron, 1991)
[Equation]
where d = n as the input dimension is the filter order, N is the number of training samples.
Intuitively the first term of the expression states that increasing the number of nodes h decreases the error bounds and the second term acts as a regularizer to limit the increase in h by the number of samples.
The latter term can be thought of as enforcing the bound to obey the Occam’s razor that parameters should not be increased beyond necessity.
[Equation]
| F | is the Fourier magnitude distribution of f which is the function to be learned by the network.
Thus we get,
[Equation]
We represent by S T ′ n ( x − t ) m
+ the approximation obtained by the network of the n th order chebyshev polynomial.
Thus
[Equation]
From equation 17 and the previous equation we get the below
[Equation]
We now bound the expression | (cid:80) ∞
k
= n +1 c km T k ( x ) | .
We note that
[Equation]
Using x = cos θ and t = cos ϕ we get
[Equation]
We now have to solve the integral I = (cid:82) ϕ 0 (cos θ − cos ϕ ) m cos kθdθ
We do this by integrating by parts
[Equation]
Continuing in this manner we get O (2 m ) integrals
[Equation]
one of which is as below
[Equation]
Thus I evaluates to O ( k − m ) and c km
= O ( k − m ) .
Thus we have,
[Equation]
Using this result in the expression for | S T ′ n ( x − t ) m + − ( x − t ) m
+ | we get
[Equation]
Finally using equations 16 and 15 for S T ′ n G we get
[Equation]
which concludes the proof.
The bound states that as the filter order n and the hidden dimension of the network h are increased (subject to n 3 C 2 f ≤ hn 2 ≤ N log N ) the approximation will converge to the desired filter response.
The condition can be thought to satisfy the statistical rule that the model parameters must be less than the sample size.
In the limit of N −→ ∞ , h and n could be increased to as large values as desired subject to h ≥ O ( n ) , which is the order of parameters to be approximated.
This is equivalent to say that if we do not consider the generalization error, we can theoretically take a large h .
Then, we are left with only the term containing the filter order i.e. the approximation error comes down to O ( n − m ) as expected.
One point to note is that we assume suitable coefficients c km can be learned from the input graph.
This assumption requires that the graph has the necessary information regarding spatial connectivity and signals.
In the current implementation, we only use the information from the signals residing on the graph nodes in the attention heat map and discard the spatial connectivity.
It is a straightforward exercise to extend this idea by using multi-relational graphs to include the graph signals and the spatial connectivity information which we leave for future works.
Nevertheless, the current implementation does well empirically, as is evident from the results on the real world and synthetic (A.4) datasets.
A.1. T RANSFORMERS AND WL TEST
WL-test has been used as a standard measure to study the expressivity of GNNs.
The k-WL test is the variant of the WL test that works on k-tuples instead of one-hop node neighbors compared to the standard 1-WL test.
Recently, with the rapid adoption of transformers on graph tasks, the equivalence of transformer and WL test naturally arises.
In the following section, we try to argue how a transformer can approximate the WL test.
Given a sequence, recent works by (Yun et al., 2019; 2020) have theoretically illustrated that Transformers are universal sequence-to-sequence approximators.
The core building block of the transformer is a self-attention layer; the self-attention layers compute dynamic attention values between the query and key vectors by attending to all the sequences.
This can be viewed as passing messages between all nodes, regardless of the input graph connectivity.
For position encoding, recently, many works have tried using eigenvectors and eigenvalues as PEs for GNNs (Dwivedi & Bresson, 2020; Kreuzer et al., 2021).
The recent work by DGN (Beaini et al., 2021) shows how using eigenvalues can distinguish non-isomorphic graphs which WL test cannot.
Transformers are universal approximators coupled with eigenvalues and eigenvectors as position encodings are powerful than the WL test given enough model parameters.
However, they can only
[Table]
[Table]
solution to the graph isomorphism problem with a specific error and not solve them fully which is also proven Kreuzer et al holds
A. D ATASET D ETAILS
We benchmark the widely used datasets for graph classification, node classification [Table] and graph regression.
Namely for graph classification we use MUTAG (Morris et al., 2020), NCI1 (Morris et al., 2020) and, the ogbg-MolHIV (Hu et al., 2020) dataset, for node classification we use the PATTERN and CLUSTER datasets (Dwivedi et al., 2020) and for graph regression we run our method on the
ZINC (Dwivedi et al., 2020) dataset.
MUTAG is a collection of nitroaromatic compounds.
Here, the goal is to predict the mutagenicity of these compounds.
Input graphs represent the compounds with atoms as the vertices and bonds as the edges.
Similarly, in NCI1, the graphs represent chemical compounds with nodes representing the atoms and the edges indicating their bonds.
The atoms are labeled as one-hot vectors as node features.
Ogbg-MolHIV is a molecular dataset in which each graph consists of a molecule, and the nodes have their features encoded as the atomic number, chirality, and other additional features.
PATTERN and CLUSTER are node classification datasets constructed using stochastic block models.
In PATTERN, the task is to identify subgraphs and CLUSTER aims at identifying clusters in a semi-supervised setting.
ZINC is a database of commercially available compounds.
The task is to predict the solubility of the compound formulated as a graph regression problem.
Each molecule has the type of heavy atom as node features and the type of bond as edge features.
A. E XPERIMENT S ETTINGS
Table 4 lists the hardware and the run time of the experiments on each dataset.
Table ?? lists out the model architecture parameters for each configurations.
For the configurations SpecTRA-Base , SpecTRA+LapE , SpecTRA+3RW and SpecTRA+GCKN+3RW we used the default hyper-parameters provided by GraphiT (Mialon et al., 2021) as we inherited position encodings from GraphiT. For instance, in ZINC, we don’t use edge features.
In the configurations SpecTRA+LPE+Sparse and
SpecTRA+LPE+Full we used the configurations from SAN (Kreuzer et al., 2021).
This is to ensure same experiment settings which these encoding schemes have used while inducing position encodings in the vanilla transformer models.
Means uncertainties are derived from four runs
[Table] 8 PATTERN RW8 CLUSTER RW8 ZINC
RWSpecTRA+GCKN +3RW
MUTAG
GCKN+RW      8 NCI
GCKN+RW      8 Molhiv
GCKN+RW      8 PATTERN
GCKN+RW      8 CLUSTER GCKN+RW      8 ZINC
GCKN+RW
GraphiT+GCKN +3RW
MUTAG
GCKN+RW      8 NCI
GCKN+RW      8 Molhiv
GCKN+RW      8 PATTERN
GCKN+RW      8 CLUSTER GCKN+RW      8 ZINC
GCKN+RW
[Table]
MUTAG [Table] [Table] [Table]
[Table] with different seeds, same as SAN.
Additionally, with the final optimized parameters, we reranexperiments with identical seeds, which is same as SAN’s/Mialon et al/Dwivedi et al’s experiment settings. Vanilla transformer implementation and its values in main paper is taken from Mialon et al.
A. S YNTHETIC D ATASET
To show the benefit of graph-specific filtering compared to static filtering, we run experiments on synthetic datasets that have different spectral components for different graphs.
We begin by noting a few basic properties of the Laplacian and its spectral decomposition before detailing the data generation process.
The below equation gives the unnormalized Laplacian ( L = D − A ) of a graph:
L ( i, j ) =
  deg ( i ) if i = j −if ( i, j
) ∈ E otherwise where deg ( i ) is the degree of the node i and E is the set of edges in the graph.
Multiplying L by a vector v gives the below expression
[Equation]
The expression v T Lv gives
[Equation]
Thus we can see that the expression v T Lv evaluates to the sum of squared distances between neighboring nodes in the graph.
We also note that due to this property, the laplacian is a positive semidefinite matrix.
If v were the eigenvector of the graph, we know that v T Lv would be the eigenvalue corresponding to that vector by the spectral decomposition theory.
Thus all the eigenvalues of the laplacian are non-negative.
We now try to develop an intuition of the laplacian’s lowest and largest eigenvalues/vectors.
We see from the above equations that v T Lv is the sum of squared differences between values on the nodes.
Thus the smallest eigenvalue would correspond to the eigenvector that assigns the same value to all the neighboring nodes, subject to ∥ v ∥ = 1 .
The second eigenvalue would correspond to the orthonormal vector to the first vector and minimizes the sum of squared differences of nodes within a cluster.
Thus the second eigenvector would try to keep values of its components similar/closer for
[Figure]
nodes that belong to the same cluster. Similarly, the vector corresponding to the largest eigenvalue would try to maximize the sum of squared differences between neighboring nodes and have its nodes(components) belonging to the same take different values subject to ∥ v ∥ = 1 while also being orthonormal to the other vectors.
With the above background, we generate synthetic datasets with node signals exhibiting different spectral components for different graphs.
Consider L to be the graph laplacian with eigenvalues λ and eigenvectors V .
We generate stochastic block matrices (SBMs) with B number of blocks and N number of nodes per block having an intracluster density of edges as p i and the inter-cluster edge density as p o .
The signals are assigned to the graphs according to the eigenvectors of the selected components of the spectrum.
To keep it simple, we use only a single eigenvector ( V i ) to generate signals per graph.
Specifically, out of the N G = E ( NB ) eigenvalues, we select one and take the components of the corresponding eigenvectors.
The nodes are then clustered using standard clustering algorithms, such as K-means, into C classes that we keep equal to the number of blocks.
Each node is assigned a one-hot vector at the position of the class it belongs to.
We then drop this attribute of 50% of the nodes i.e., these nodes are assigned a 0 vector, and the task is to find the correct assignment for the unknown class based on the connectivity pattern and the spectrum of the known signals on the graph.
Thus this task boils down to finding the suitable spectral component in the graph signal and using this information for classification.
This should benefit from a graph-specific decomposition and filtering approach, which we confirm from the empirical results.
We generate 3 datasets namely Synthetic 1 , Synthetic 2 , and Synthetic 3 using different values of N , B , p i , p o , V i .
Synthetic 1 has N = 10 , B = 2 , p i = 0 .
9 , p o = 0 .
05 , { V i | i ∈ { 1 , N G }} , with 1000, 100, 100 graphs in the train, test and valid graphs respectively.
Synthetic 2 has N = 10 , B = 6 , p i = 0 .
9 , p o = 0 .
05 , { V i | i ∈ { 1 , N G }} , with 1000, 100,graphs in the train, test and valid graphs respectively.
Synthetic 1 has N = 10 , B = 6 , p i = 0 .
9 , p o = 0 .
05 , { V i | i ∈ { 1 , ⌈ N G 2 ⌉ , N G }} , with 1000, 100, 100 graphs in the train, test and valid graphs respectively.
We train the synthetic datasets on the SpecTRA-Static and SpecTRA-Base models to study the effect of graph-specific dynamic filters on the graph.
For the Synthetic 1 dataset, we use the hidden dimension as 16, and for Synthetic 2 and Synthetic 3 we keep it to 64.
The number of layers is fixed to 1.
The number of heads h and filter order K for SpecTRA-Static are kept at 1 and 4 respectively and for varied for other settings as can be seen in table A.4 .
We can see from table A.4 that the performance on the synthetic datasets, using dynamic filters of SpecTRA-Base, has increased by a large margin as compared to the case of static dataset-specific filters in SpecTRA-Static using the same model parameters.
This justifies the benefit and necessity of graph-specific filter design in [Table] We study the effect of the order K of filters and the number of heads h for SpecTRA-Base. cases where the spectral information differs from graph to graph.
We also observe that as the filter order is increased for a given number of heads, the performance improves.
On the other hand, lower filter order is detrimental to the task.
The performance saturates if the filter order is increased beyond a specific limit, as is evident from the Synthetic 1 dataset.
Also, we do not observe any improvement by increasing the number of heads, keeping the filter order fixed, in this case.
This may be due to the nature of the dataset, where we have restricted each graph to contain a single spectral component.
We leave it to future studies to determine the effect of number of heads on multiple spectral components in the signal.
A. F ILTER F REQUENCY R ESPONSE ON O THER D ATASETS
Here we provide the plots of the frequency response of the filters learned on the other datasets which has not been included in the main paper due to page limit.
Filter Frequency response on ZINC : The frequency response of the filters learned on ZINC with filter order four is given in figure 5.
Each curve is the frequency response of a filter learned for a single head.
We see various filters being learned, such as few low pass, high pass, and band stop filters.
Figure 6 shows the frequency response for filters of order 8.
Here we observe the high pass and multi-modal band pass responses being learned.
Filter Frequency response on MolHIV : Figure 7 illustrates the frequency response for filters learned on the MolHIV.
Here, we observe the low pass and multi-modal band pass filter responses.
Filter Frequency response on PATTERN : Figure 8 shows the frequency response for filters learned on the PATTERN dataset.
Here we observe the filters allowing signals in the low and high frequency regime along with some magnitude assigned to the mid-frequency components.
The filter order eight (the two right-most plots) shows a surprising result of visually indistinct filters being learned despite the regularization.
This indicates the task has a high bias towards signals in the low-frequency region and some components in the middle and high frequency regime to an extent.
This could be interpreted as the model trying to learn the two components: the SBM, which corresponds to the low-frequency signal, and the underlying pattern itself, which may benefit from the middle and high frequency components in the spectrum.
Filter Frequency response on CLUSTER : Similar to PATTERN, the filter response plots for
CLUSTER as in figure 9 show low-frequency filters being learned along with bandpass filters.
This is intuitive as the task of CLUSTER would indeed benefit by learning low-frequency signals for grouping nodes belonging to the same cluster.
All these observations across all datasets validates our hypothesis studied in the scope of this paper.
A. I NTERPRETABILITY IN S PECTRAL S PACE
In this section, we look at the interpretability aspect induced by SpecTRA in the spectral space.
The figures 10 and 11 show the graphs with the eigenvectors on the nodes corresponding to the top eigenvalues selected by the filter.
From both figures, we see that in the case of the low pass filter (leftmost graph in sub-figure (a), (d) of figure 10 and subfigure (a)-(d) of figure 11), the nodes [Figure] [Figure]
[Figure]
(f) K =
[Figure]
(g) K =
[Figure]
(h) K =
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
in the neighborhood forming a cluster have similar eigenvalues.
Whereas in the highpass filter ( (rightmost graph in each subfigure (a)-(d)))
the eigenvalues of the nodes alternate, with neighboring nodes taking distinct values and far off nodes having similar values.
We could make two interpretations of this phenomenon.
The first one is closely related to attention in the spatial space where SpecTRA attends to select input features.
In this case, we can think of the model learning to pay more attention to the nodes with higher eigenvalues and lesser attention to the nodes with smaller eigenvalues in a graph and task-specific manner.
The second interpretation is related to the interaction between the nodes, i.e., for a given node which other nodes are considered for aggregation.
For example, in graph attention networks, the neighboring nodes are aggregated, and the values of nodes in the same cluster tend to be closer to each other (homophily).
This is a particular case of the low pass filter in which we can see from the figures 10 and 11 that the nodes belonging to the same cluster take on similar values.
For example, consider the Figure 10 (d).
The leftmost graph has nodes in a particular cluster taking similar eigenvalues showing short-range dependencies (interactions).
In the rightmost graph, nodes in the same clusters take on different eigenvalues, illustrating the need for long-range interactions.
However, depending on the task and graph, the model also learns to aggregate (interact with) distant nodes, as seen in the graph corresponding to the high pass filter (the rightmost graphs in each sub-figure of 10 and 11).
Thus SpecTRA can be interpreted as a generic(covering the entire spectrum of frequencies) attention network in the spectral space .