Vision GNN: An Image is Worth Graph of Nodes
Abstract
Network architecture plays a key role in the deep learning-based computer vision system.
The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not ﬂexible to capture irregular and complex objects.
In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graphlevel feature for visual tasks.
We ﬁrst split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors.
Based on the graph representation of images, we build our ViG model to transform and exchange information among all the nodes.
ViG consists of two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation.
Both isotropic and pyramid architectures of ViG are built with different model sizes.
Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture.
We hope this pioneering study of GNN on general visual tasks will provide useful inspiration and experience for future research.
The
PyTorch code is available at https://
github
.com/huawei
-noah
Efficient
AI-Backbones and the MindSpore
code is available at https:
gitee com
/mindspore/
models
Introduction
In the modern computer vision system, convolutional neural networks (CNNs) used to be the defacto standard network architecture [29, 27, 17].
Recently, transformer with attention mechanism was introduced for visual tasks [9, 3] and attained competitive performance.
MLP-based (multi-layer perceptron) vision models [49, 50] can also work well without using convolutions or self-attention.
These progresses are pushing the vision models towards an unprecedented height.
Different networks treat the input image in different ways.
As shown in Figure 1, the image data is usually represented as a regular grid of pixels in the Euclidean space.
CNNs [29] apply sliding window on the image and introduce the shift-invariance and locality.
The recent vision transformer [9] or MLP [49] treats the image as a sequence of patches.
For example, ViT [9] divides a 224 ×image into a number of 16 × 16 patches and forms a sequence with length of 196 as input.
Instead of the regular grid or sequence representation, we process the image in a more ﬂexible way.
One basic task of computer vision is to recognize the objects in an image.
Since the objects are usually not quadrate whose shape is irregular, the commonly-used grid or sequence structures in previous networks like ResNet and ViT are redundant and inﬂexible to process them.
An object can be viewed as a composition of parts, e.g. , a human can be roughly divided into head, upper body, ∗ Equal contribution.
[Figure]
arms and legs.
These parts linked by joints naturally form a graph structure.
By analyzing the graph, we are able to recognize the human.
Moreover, graph is a generalized data structure that grid and sequence can be viewed as a special case of graph.
Viewing an image as a graph is more ﬂexible and effective for visual perception.
Based on the graph representation of images, we build the vision graph neural network (ViG for short) for visual tasks.
Instead of treating each pixel as a node which will result in too many nodes (>10K), we divide the input image to a number of patches and view each patch as a node.
After constructing the graph of image patches, we use our ViG model to transform and exchange information among all the nodes.
The basic cells of ViG include two parts: Grapher and FFN (feed-forward network) modules.
Grapher module is constructed based on graph convolution for graph information processing.
To alleviate over-smoothing phenomenon of conventional GNN, a FFN module is utilized for node feature transformation and encouraging node diversity.
With Grapher and FFN modules, we build our ViG models in both isotropic and pyramid manners.
In the experiments, we demonstrate the effectiveness of ViG model on visual tasks like image classiﬁcation and object detection.
For instance, our Pyramid ViG-S achieves 82.1% top-1 accuracy on ImageNet classiﬁcation task, which outperforms the representative CNN (ResNet [17]), MLP (CycleMLP [5]) and transformer (Swin-T [35]) with similar FLOPs (about 4.5G).
To the best of our knowledge, our work is the ﬁrst to successfully apply graph neural network on large-scale visual tasks.
We hope our work will inspire the community to further explore more powerful network architectures.
Related Work
In this section, we ﬁrst revisit the backbone networks in computer vision.
Then we review the development of graph neural network, especially GCN and its applications on visual tasks.
2.1 CNN, Transformer and MLP for Vision
The mainstream network architecture in computer vision used to be convolutional network [29, 27, 17].
Starting from LeNet [29], CNNs have been successfully used in various visual tasks, e.g. , image classiﬁcation [27], object detection [42] and semantic segmentation [36].
The CNN architecture is evolving rapidly in the last ten years.
The representative works include ResNet [17], MobileNet [21] and NAS [75, 70].
Vision transformer was introduced for visual tasks from 2020 [14, 9, 3, 4].
From then on, a number of variants of ViT [9] were proposed to improve the performance on visual tasks.
The main improvements include pyramid architecture [57, 35], local attention [15, 35] and position encoding [61].
Inspired by vision transformer, MLP is also explored in computer vision [49, 50].
With specially designed modules [5, 32, 12, 48], MLP can achieve competitive performance and work on general visual tasks like object detection and segmentation.
2.2 Graph Neural Network
The earliest graph neural network was initially outlined in [11, 44].
Micheli [38] proposed the early form of spatial-based graph convolutional network by architecturally composite nonrecursive layers.
In recent several years, the variants of spatial-based GCNs have been introduced, such as [39, 1, 10].
Spectral-based GCN was ﬁrst presented by Bruna et al.
[2] that introduced graph convolution based on the spectral graph theory.
Since this time, a number of works to improve and extend spectralbased GCN have been proposed [18, 7, 26].
The GCNs are usually applied on graph data, such as social networks [13], citation networks [45] and biochemical graphs [55].
The applications of GCN in the ﬁeld of computer vision [63, 28, 56, 25] mainly include point clouds classiﬁcation, scene graph generation, and action recognition.
A point cloud is a set of 3D points in space which is usually collected by LiDAR scans.
GCN has been explored for classifying and segmenting points clouds [28, 58, 69].
Scene graph generation aims to parse the input image intro a graph with the objects and their relationship, which is usually solved by combining object detector and GCN [63, 68].
By processing the naturally formed graph of linked human joints, GCN was utilized on human action recognition task [24, 67].
GCN can only tackle speciﬁc visual tasks with naturally constructed graph.
For general applications in computer vision, we need a GCN-based backbone network that directly processes the image data.
Approach
In this section, we describe how to transform an image to a graph and introduce vision GNN architectures to learn visual representation.
[Figure]
3.1 ViG Block
Graph Structure of Image.
For an image with size of H × W × 3 , we divided it into N patches.
By transforming each patch into a feature vector x i ∈ R D , we have X =
[ x 1 , x 2 , · · · , x N ] where D is the feature dimension and i
= 1 , 2 , · · · , N .
These features can be viewed as a set of unordered nodes which are denoted as V = { v 1 , v 2 , · · · , v N } .
For each node v i , we ﬁnd its K nearest neighbors N ( v i ) and add an edge e ji directed from v j to v i for all v j ∈ N ( v i ) .
Then we obtain a graph G =
( V , E ) where E denote all the edges.
We denote the graph construction process as G = G ( X ) in the following.
By viewing the image as a graph data, we explore how to utilize GNN to extract its representation.
The advantages of graph representation of the image include: 1) graph is a generalized data structure that grid and sequence can be viewed as a special case of graph; 2) graph is more ﬂexible than grid or sequence to model the complex object as an object in the image is usually not quadrate whose shape is irregular; 3) an object can be viewed as a composition of parts ( e.g. , a human can be roughly divided into head, upper body, arms and legs), and graph structure can construct the connections among those parts; 4) the advanced research on GNN can be transferred to address visual tasks.
Graph-level processing.
To be general, we start from the features X ∈ R N × D .
We ﬁrst construct a graph based on the features: G = G ( X ) .
A graph convolutional layer can exchange information between nodes by aggregating features from its neighbor nodes.
Speciﬁcally, graph convolution operates as follows:
[Equation]
where W agg and W update are the learnable weights of the aggregation and update operations, respectively.
More concretely, the aggregation operation computes the representation of a node by aggregating features of neighbor nodes, and the update operation further merge the aggregated feature:
[Equation]
where N ( x li ) is the set of neighbor nodes of x li .
Here we adopt max-relative graph convolution [30] for its simplicity and efﬁciency:
[Equation]
where the bias term is omitted.
The above graph-level processing can be denoted as X ′ =
GraphConv ( X ) .
We further introduce multi-head update operation of graph convolution.
The aggregated feature x ′′ i is ﬁrst split into h heads, i.e. , head 1 , head 2 , · · · , head h
and then these heads are updated with different weights respectively.
All the heads can be updated in parallel and are concatenated as the ﬁnal values:
[Equation]
Multi-head update operation allows the model to update information in multiple representation subspaces, which is beneﬁcial to the feature diversity.
[Figure]
ViG block.
The previous GCNs usually repeatedly use several graph convolution layers to extract aggregated feature of the graph data.
The over-smoothing phenomenon in deep GCNs [31, 40] will decrease the distinctiveness of node features and lead to performance degradation for visual recognition, as shown in Figure 3 where diversity is measured as ∥ X − 1 ˜ x T ∥ with ˜ x = arg min ˜ x
∥ X − 1 ˜ x T ∥ [8].
To alleviate this issue, we introduce more feature transformations and nonlinear activations in our ViG block.
We apply a linear layer before and after the graph convolution to project the node features into the same domain and increase the feature diversity.
A nonlinear activation function is inserted after graph convolution to avoid layer collapse.
We call the upgraded module as Grapher module.
In practice, given the input feature X ∈ R N × D , the Grapher module can be expressed as
[Equation]
where Y ∈ R N × D , W in and W out are the weights of fully-connected layers, σ is the activation function, e.g. , ReLU and GeLU [19], and the bias term is omitted.
To further encourage the feature transformation capacity and relief the over-smoothing phenomenon, we utilize feed-forward network (FFN) on each node.
The FFN module is a simple multi-layer perceptron with two fully-connected layers:
[Equation]
where Z ∈ R N × D , W 1 and W 2 are the weights of fully-connected layers, and the bias term is omitted.
The hidden dimension of FFN is usually larger than D .
In both Grapher and FFN modules, batch normalization is applied after every fully-connected layer or graph convolution layer, which is omitted in Eq.
6 and 7 for concision.
A stack of Grapher module and FFN module constitutes the
ViG block which serves as the basic building unit for constructing a network.
Based on the graph representation of images and the proposed ViG block, we can build the ViG network for visual tasks as shown in Figure 2.
Compared to vanilla ResGCN [30], our ViG can maintain the feature diversity (Figure 3) as the layer goes deeper so as to learn discriminative representations.
3.2 Network Architecture
In the ﬁeld of computer vision, the commonly-used transformer usually has an isotropic architecture ( e.g. , ViT [9]), while CNNs prefer to use pyramid architecture ( i.e. , ResNet [17]).
To have a extensive comparison with other types of neural networks, we build two kinds of network architectures for
ViG, i.e. , isotropic architecture and pyramid architecture.
Isotropic architecture.
Isotropic architecture means the main body has features with equal size and shape throughout the network, such as ViT [9] and ResMLP [50].
We build three versions of isotropic ViG architecture with different models sizes, i.e. , ViG-Ti, S and B. The number of nodes is set as N = 196 .
To enlarge the receptive ﬁeld gradually, the number of neighbor nodes K increases from 9 to 18 linearly as the layer goes deep in these three models.
The number of heads is set as h = 4 by default.
The details are listed in Table 1.
[Table]
Pyramid architecture.
Pyramid architecture considers the multi-scale property of images by extracting features with gradually smaller spatial size as the layer goes deeper, such as ResNet [17] and PVT [57].
Empirical evidences show that pyramid architecture is effective for visual tasks [57].
Thus, we utilize the advanced design and build four versions of pyramid ViG models.
The details are shown in Table 2.
Note that we utilize the spatial reduction [57] in the ﬁrst two stages to handle large number of nodes.
[Table]
Positional encoding.
In order to represent the position information of the nodes, we add a positional encoding vector to each node feature:
[Equation]
where e i ∈ R D .
The absolute positional encoding as described in Eq. 8 is applied in both iostropic and pyramid architectures.
For pyramid ViG
[Table]
we further include relative positional encoding by following the advanced designs like Swin Transformer [35].
For node i and j , the relative positional distance between them is e Ti e j , which will be added into the feature distance for constructing the graph.
Experiments
In this section, we conduct experiments to demonstrate the effectiveness of ViG models on visual tasks including image recognition and object detection.
4.1 Datasets and Experimental Settings
Datasets.
In image classiﬁcation task, the widely-used benchmark ImageNet ILSVRC 2012 [43] is used in the following experiments.
ImageNet has 1.2M training images and 50K validation images, which belong to 1000 categories.
For the license of ImageNet dataset, please refer to http://www.
image-net.org/download .
For object detection, we use COCO 2017 [34] dataset with 80 object categories.
COCO 2017 contains 118K training images and 5K validation images.
For the licenses of these datasets, please refer to https://cocodataset.org/#home .
[Table]
Experimental Settings.
For all the
ViG models, we utilize dilated aggregation [30] in Grapher module and set the dilated rate as ⌈ l/ 4 ⌉ for the l -th layer.
GELU [19] is used as the nonlinear activation function in Eq.
6 and 7.
For ImageNet classiﬁcation, we use the commonly-used training strategy proposed in DeiT [51] for fair comparison.
The data augmentation includes
RandAugment [6], Mixup [73], Cutmix [72], random erasing [74] and repeated augment [20].
The details are shown in Table 3.
For COCO detection task, we take RetinaNet [33] and
Mask R-CNN [16] as the detection frameworks and use our Pyramid ViG as backbone.
All the models are trained on COCOtraining set in “1 × ” schedule and evaluated on validation set.
We implement the networks using
PyTroch [41] and MindSpore [23] and train all our models on 8 NVIDIA V100 GPUs.
[Table]
4.2 Main Results on ImageNet
Isotropic ViG The neural network with iostropic architecture keeps the feature size unchanged in its main computational body, which is easy to scale and is friendly for hardware acceleration.
This scheme is widely used in transformer models for natural language processing [53].
The recent neural networks in vision also explore it such as ConvMixer [49], ViT [9] and ResMLP [50].
We compare our isotropic ViG with the existing iostropic CNNs [50, 49], transformers [9, 51] and
MLPs [50, 49] in Table 4.
From the results, ViG performs better than other types of networks.
For example, our ViG-Ti achieves 73.9% top-1 accuracy which is 1.7% higher than DeiT-Ti model with similar computational cost.
Pyramid ViG
The pyramid architecture gradually shrinks the spatial size of feature maps as the network deepens, which can leverage the scale-invariant property of images and produce multiscale features.
The advanced networks usually adopt the pyramid architecture, such as ResNet [17],
Swin Transformer [35] and CycleMLP [5].
We compare our Pyramid ViG with those representative pyramid networks in Table 5.
Our Pyramid ViG series can outperform or be comparable to the stateof-the-art pyramid networks including CNN, MLP and transformer.
This indicates that graph neural network can work well on visual tasks and has the potential to be a basic component in computer vision system.
[Table]
Pyramid ViG-
[Table]
Pyramid ViG-
[Table]
ours)
[Table]
Pyramid ViG-
[Table]
4.3 Ablation Study
We conduct ablation study of the proposed method on ImageNet classiﬁcation task and use the isotropic ViG-Ti as the base architecture.
Type of graph convolution.
We test the representative variants of graph convolution, including
EdgeConv [58], GIN [64], GraphSAGE [13] and Max-Relative GraphConv [30].
From table 6, we can see that the top-1 accuracies of different graph convolutions are better than that of DeiT-Ti,
[Table]
indicating the ﬂexibility of ViG architecture.
Among them, Max-Relative achieves the best tradeoff between FLOPs and accuracy.
In rest of the experiments, we use Max-Relative GraphConv by default unless specially stated.
The effects of modules in ViG. To make graph neural network adaptive to visual task, we introduce FC layers in Grapher module and utilize FFN block for feature transformation.
We evaluate the effects of these modules by ablation study.
We change the feature dimension of the compared models to make their FLOPs similar, so as to have a fair comparison.
From Table 7, we can see that directly utilizing graph convolution for image classiﬁcation performs poorly.
Adding more feature transformation by introducing FC and FFN consistently increase the accuracy.
[Table]
The number of neighbors.
In the process of constructing graph, the number of neighbor nodes K is a hyperparameter controlling the aggregated range.
Too few neighbors will degrade information exchange, while too many neighbors will lead to over-smoothing.
We tune K from 3 to 20 and show the results in Table 8.
We can see that the number of neighbor nodes in the range from 9 to 15 can perform well on ImageNet classiﬁcation task.
accuracy
[Table]
The number of heads.
Multi-head update operation allows Grapher module to process node features in different subspaces.
The number of heads h in Eq.
5 controls the transformation diversity in subspaces and the FLOPs.
We tune h from 1 to 8 and show the results in Table 9.
The FLOPs and top-1 accuracy on ImageNet changes slightly for different h .
We select h = 4 as default value for the optimal trade-off between FLOPs and accuracy.
[Table]
4.4 Object Detection
We apply our ViG model on object detection task to evaluate its generalization.
To have a fair comparison, we utilize the ImageNet pretrained Pyramid ViG-S as the backbone of RetinaNet [33] and Mask R-CNN [16] detection frameworks.
The models are trained in the commonly-used “1x” schedule and FLOPs is calculated with 1280 × 800 input size.
From the results in Table 10, we can see that our Pyramid ViG-S performs better than the representative backbones of different types, including ResNet [17], CycleMLP [5] and Swin Transformer [35] on both RetinaNet and Mask
R-CNN.
The superior results demonstrate the generalization ability of ViG architecture.
[Table]
39.6 Swin-T [35] 47.8M 264.0B 42.
Pyramid ViG-S (ours) 45.8M 258.8B 42.
4.5 Visualization
To better understand how our ViG model works, we visualize the constructed graph structure in
ViG-S.
In Figure 4, we show the graphs of two samples in different depths (the 1st and the 12th blocks).
The pentagram is the center node, and the nodes with the same color are its neighbors.
Two center nodes are visualized as drawing all the edges will be messy.
We can observe that our model can select the content-related nodes as the ﬁrst order neighbors.
In the shallow layer, the neighbor nodes tend to be selected based on low-level and local features, such as color and texture.
In the deep layer, the neighbors of the center nodes are more semantic and belong to the same category.
Our ViG network can gradually link the nodes by its content and semantic representation and help to better recognize the objects.
[Figure]
Conclusion
In this work, we pioneer to study representing the image as graph data and leverage graph neural network for visual tasks.
We divide the image into a number of patches and view them as nodes.
Constructing graph based on these nodes can better represent the irregular and complex objects in the wild.
Directly using graph convolution on the image graph structure has over-smoothing problem and performs poorly.
We introduce more feature transformation inside each node to encourage the information diversity.
Based on the graph representation of images and improved graph block, we build our vision GNN (ViG) networks with both isotropic and pyramid architectures.
Extensive experiments on image recognition and object detection demonstrate the superiority of the proposed
ViG architecture.
We hope this pioneering work on vision GNN can serve as a basic architecture for general visual tasks
Acknowledgement
This research is supported by NSFC (62072449, 61872345), National Key R&D Program of China (2021YFB1715800) and Macau Science &Tech. Fund (0018/2019/AKP.
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend
AI Processor used for this research
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] No potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 4.1.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The common settings on ImageNet and COCO datasets.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.
(b) Did you mention the license of the assets?
[Yes] See Section 4.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Theoretical Analysis
In our ViG block, we propose to increase feature diversity in nodes by utilizing more feature transformation such as FFN module.
We show the empirical comparison between vanilla ResGCN and our
ViG model in our paper.
Here we make a simple theoretical analysis of the beneﬁt of FFN module in
ViG on increasing the feature diversity.
Given the output features of graph convolution X ∈ R N × D , the feature diversity [8] is measured as
[Equation]
where x = arg
[Equation]
9) where ∥ · ∥ is the ℓ 1 , ∞ norm of a matrix.
By applying FFN module on the features, we have the following theorem.
Theorem 1.
Given a FFN module, the diversity γ ( FFN ( X )) of its output features satisﬁes
[Equation]
where λ is the Lipschitz constant of FFN with respect to p -norm for p ∈ [1 , ∞ ] .
Proof.
The FFN includes weight matrix multiplication, bias addition and elementwise nonlinear function, which all preserve the constancy-across-rows property of FFN ( 1x T ) .
Therefore, we have
[Equation]
preserves constancy-across-rows.
[Equation]
▷ Deﬁnition of Lipschitz constant.
[Equation]
The Lipschitz constant of FFN is related to the norm of weight matrices and is usually much larger than 1 [54].
Thus, the Theorem 1 shows that introducing γ ( FFN ( X )) in our ViG block tends to improve the feature diversity in graph neural network.
A.2 Pseudocode
The proposed Vision GNN framework is easy to be implemented based on the commonly-used layers without introducing complex operations.
The pseudocode of the core part, i.e. , ViG block is shown in Algorithm 1.
Algorithm 1 PyTorch-like Code of ViG Block
[Table]
def forward(self, x): B, C, H, W = x.shape x = x.reshape(B, C, -1, 1).contiguous() shortcut = x x = self.fc
[Table]
x) x = self.graph
[Table]
(x) x = self.fc2(x) x = self
[Table]
x) + shortcut return x.reshape(B, C, H, W) class FFNModule(nn.
[Table]
[Table] [Table]
forward Network "
[Table]
[Table] [Table]
fc1 =
[Table] [Table]
[Table]
else
[Table]
[Table] [Table]
with
[Table]
and
[Table]
self
[Table] [Table]