GNN2GNN: Graph Neural Networks to Generate Neural Networks
Abstract
The success of neural networks (NNs) is tightly linked with their architectural design—a complex problem by itself.
We here introduce a novel framework leveraging Graph Neural Networks to
Generate Neural Networks (GNN2GNN) where powerful NN architectures can be learned out of a set of available architecture-performance pairs.
GNN2GNN relies on a three-way adversarial training of GNN, to optimise a generator model capable of producing predictions about powerful NN architectures.
Unlike Neural Architecture Search (NAS) techniques proposing efﬁcient searching algorithms over a set of NN architectures, GNN2GNN relies on learning NN architectural design criteria.
GNN2GNN learns to propose
NN architectures in a single step – i.e., training of the generator –, overcoming the recursive approach characterising NAS.
Therefore, GNN2GNN avoids the expensive and inﬂexible search of efﬁcient structures typical of NAS approaches.
Extensive experiments over two state-of-the-art datasets prove the strength of our framework, showing that it can generate powerful architectures with high probability.
Moreover, GNN2GNN outperforms possible counterparts for generating NN architectures, and shows ﬂexibility against dataset quality degradation.
Finally, GNN2GNN paves the way towards generalisation between datasets.
INTRODUCTION
Deep Learning (DL) techniques have recently seen an unstoppable rise in popularity: DL has changed the approach to most intelligence task, ranging from vision to text and audio processing.
Among those techniques, neural networks (NNs) have become the most popular solution.
NN performance is tightly linked with the operations they leverage and how these are connected – their architecture – whose design has been shown to be as complex as much as it is relevant.
On the other hand, there is no trivial way to ﬁnd out the best
NN design for a speciﬁc task.
Neural Architecture Search (NAS) techniques have recently emerged to tackle NN design issue [Elsken et al., 2019]: the underlying concept of NAS approaches is to efﬁciently search for the best NN architecture over a set of structures deﬁned as a search space.
Despite its success, NAS exhibits several drawbacks (see
Section 6), as it relies on looking for the best architecture rather than learning architectural criteria for building the optimal NN.
Moreover, NAS approaches are not ﬂexible with respect to slight changes of the application scenario, require huge amount of resources to run, and are limited by their search space speciﬁcations.
Also inspired by these limitations, in this work we present GNN2GNN, a novel tool leveraging graph neural networks to generate NN architectures.
GNN2GNN is a meta-learning framework exploiting Graph
Neural Networks (GNNs) to learn generating efﬁcient NN structures.
GNNs are particular models proposed to tackle graph -processing tasks via convolution-equivalent operation over graphs [Wu et al., 2021].
A NN structure can be seen as a Directed Acyclic Graph (DAG) where nodes represent layers – implementing common operations like convolution, pooling, etc. – and edges represent how the output of one layer is fed to the following one.
In this context, we propose a three-way adversarial learning setup to allow GNN to learn the features of an efﬁcient NN structure and generate novel architectures.
More in details, a generator GNN is trained to produce plausible architectures, while a discriminator
GNN is optimised to distinguish between generated and real architectures.
Finally, a valuer GNN aims at optimising the performance of the generated architectures.
During training, the generator loss is deﬁned as a mixture of the discriminator and valuer feedbacks, therefore aiming at enabling the learning of realistic – i.e., discriminator feedback – and powerful – i.e., valuer feedback – architectures.
While being embeddable into a broader NAS approach,
GNN2GNN represents a powerful approach to propose NN architectures by itself.
Indeed, differently from NAS techniques, which aim at efﬁciently searching NN architectures over a set of available ones, GNN2GNN aims at intrinsically learning architectural criteria from a set of available architecture-performance pairs.
While NAS consider to recursively propose, evaluate and optimise a set of NN structures (see Figure 1 left), we here consider learning to propose architectures from a set of NNs in a single step—i.e., training of the generator GNN (see Figure 1 right).
Once trained, GNN2GNN is capable of proposing multiple efﬁcient NN architectures at once, rather than focusing solely on the local optimum obtained from the deployed search algorithm.
Therefore, GNN2GNN signiﬁcantly shifts the paradigm of the approach to the problem of NN architecture design, from relying on searching architectures to learning design criteria.
To summarize, the contributions that our work brings are the following:
• We present GNN2GNN, the ﬁrst – up to our knowledge – framework leveraging GNN to generate powerful NN architectures, without relying on inefﬁcient searching procedure.
• We show the effectiveness of our framework over two state-of-the-art datasets, highlighting its ﬂexibility and generalization capability.
PRELIMINARIES ON GRAPH NEURAL NETWORKS
As the proposed framework relies on graph manipulation via GNNs, here we brieﬂy introduce Graph Neural Networks, presenting their fundamental concepts.
Graph Neural
Networks (GNNs) have been proposed as an extension of traditional NNs to enable processing of non-rigidly structured data such as graphs.
GNNs are mathematical models operating upon directed graphs, whose vertices (respectively, arcs) are labelled with vectors (or matrices, or tensors) of real numbers – x v ∈ R d for vertex v , and a v,w ∈ R c for the arc between vertex v and w –, each one carrying further numeric information about the corresponding vertex (resp., arc).
GNNs rely on graph convolution , which represents the generalisation of a 2-dimensional convolution over graphstructured data.
Graph convolution is deﬁned over a single vertex v and its neighbourhood N ( v ) , and relies on three successive phases:
propagation — the information x v (cid:48) belonging to each vertex v (cid:48)
∈ N ( v ) is weighted by the information a v,v (cid:48) belonging to the arc among v and v (cid:48) , then propagated to vertex v ; aggregation — the information propagated from each vertex v (cid:48) ∈
N ( v ) to v is aggregated via an aggregation function; transformation — the aggregated information corresponding to vertex v is transformed into a new embedding vector and assigned back to vertex v , as its new state x
(cid:48) v .
The single convolution operation is applied in parallel to each vertex in G , updating the whole graph representation.
GNNs have proven to be successful in many tasks involving graph structured data.
Most common applications concerns computational chemistry [Fung et al., 2021], social recommendations [Fan et al., 2019], computer vision [Wang et al., 2019], and many others [Hamilton et al., 2017, Yu et al., 2018].
However, a comprehensive review of GNNs and the underlying techniques is clearly out the scope of this paper: therefore, we refer interested readers to Wu et al.
Zhou et al.
GNN2GNN
In this section we present our framework, namely
GNN2GNN.
GNN2GNN leverages Graph Neural Networks
to Generate Neural Networks.
We ﬁrst present brieﬂy how
NN architectures can be mapped into graph structures (Section 3.1).
We then introduce the general pipeline for generating and processing NN architectures (Section 3.2), focusing speciﬁcally on its components.
3.1 NEURAL NETWORKS AS GRAPHS
NN architectures are uniquely deﬁned by a set of layers L , a set of operations O applied on layers, and a set I of interconnections between layers.
Each layer l v ∈ L , with v ∈ [0 , |L| ] , identiﬁes a speciﬁc component of the NN architecture and is characterised by a speciﬁc operation o v ∈ O .
Interconnections between layers, on the other hand, deﬁne how layers are linked to each other.
An interconnection i v,w ∈ I identiﬁes that layers l v and l w are connected, and more speciﬁcally, it identiﬁes that the output of the operation o v applied at layer l v is used as an input for the operation
o w applied at layer l w .
It is important to notice that, thanks to the feedforwarding nature of standard NNs, there exists total ordering among the layers in L and interconnections can only exist between successive layers.
Mathematically speaking, ∃ i v,w ∈
I ⇐⇒ w > v .
Following the above notations, NN architectures can be mapped easily into graph structures, speciﬁcally to Directed Acyclic Graphs (DAGs).
Layers in L are mapped into a set of vertices V characterised by a set of features X representing layers operations ( O ), while interconnections ( I ) are mapped into a set of directed edges E .
Vertices – i.e., layers –, along with their features – i.e., operations –, are deﬁned as vectors x v ∈ R d , where v enumerates the
Policy
Sample
Update
GNN2GNN
Train
G e n e r a t e G ene r a t e
NAS Approach
GNN2GNN Approach
Search Space
Train & Evaluate graph vertices, and d represents operations cardinality.
On the other hand, the set of edges – i.e., interconnections –, is denoted by the adjacency matrix E ∈ R |V|×|V| , where e v,w = 1 ⇐⇒ ∃
i v,w .
Therefore, a NN architecture can be mapped into a DAG deﬁned by X ∈ R |V|× d , where row k ( X ) = x v – i.e. a matrix of vertices characterised by the operations they apply – and E —i.e., the adjacency matrix deﬁning how operations are connected.
3.2 ADVERSARIALLY GENERATE ARCHITECTURES
The proposed framework relies on a generative adversarial approach (GAN)
[Goodfellow et al., 2014], applied over graph structured data leveraging Graph Neural Networks.
The proposed framework is presented in Figure 2 and relies on three components:
• A generator GNN G is in charge of proposing graph structures representing NN architectures.
• A discriminator model D is responsible for distinguishing between NN structures proposed by G and real architectures.
• A valuer network V is responsible for predicting the architecture performance, therefore optimising the generation towards powerful structures.
GNN2GNN relies on such triplets of GNNs to allow G to intrinsically learn optimal architectural criteria.
Indeed, the discriminator and valuer models are used during training to optimise the generator status.
More in details, the generator loss is deﬁned as a mixture of the discriminator and valuer feedbacks:
[Equation]
Here, λ represents the balancing factor between the two feedbacks.
Leveraging such mixture loss, G is capable of proposing realistic – i.e., D feedback – and powerful – i.e., V feedback – architectures.
Finally, once trained, the GNN2GNN framework exploits solely the generator component to propose signiﬁcant NN architectures.
3.2.1 Generator
Generating graph structures that satisfy speciﬁc properties is complex and represents an open research issue [You et al., 2018, Li et al., 2018].
This task complexity is three-folded:
Q1 Generate realistic structures.
For a generated structure to result realistic, the generative framework should learn which nodes should be linked and which not.
Q2 Generate realistic nodes.
The generated graph should be characterised by nodes having realistic features.
Q3 Stopping criteria.
In
the
generating process , it is important to identify when the generated graph structure has reached its ﬁnal structure,
which is
non- trivial. To tackle the aforementioned problems and generate realistic NN architectures, we here propose a novel generative
GNN.
Indeed, GNNs are particularly suited for handling interconnections and node features, while they exhibit limitations on stopping criteria identiﬁcation.
However, given the nature of available NAS techniques and datasets, this GNNs limitation is not an issue.
Indeed, available NAS techniques restrict their searching space, limiting the number of layers – vertices – that compose the NN architecture.
Therefore, publicly available NAS datasets build on top of this rationale, ﬁxing the number of NN layers.
Building on top of the same rationale, we here propose a generator model that receives as input a fully-connected
DAG – i.e., where every node is connected with every other
[Figure]
Strong
Weak
VALUER
Real
Fake
DISCRIMINATOR
DATASET node – having N nodes. N represents an hyperparameter of the framework, and can be arbitrarily set depending on the features of the NAS dataset at hand, or, on the complexity of the architecture to generate.
Fixing N immediately satisﬁes property Q3 , implicitly setting a stopping criteria for the graph generation process.
It is also important to notice that value N only provides an upper limit on the number of layers composing the generated architectures.
Indeed, architectures having n ≤ N can be generated by the proposed approach, thanks to edge removal and node isolation.
Finally, node features of the input graph are randomly initialised, mirroring the usual GAN approach.
The proposed generator framework relies on four successive steps, presented in Figure 3 along with an example of input graph and generated architecture, and explained in details below.
Graph convolution.
The generator applies µ layers of graph convolution to the randomly generated fullyconnected graph received in input.
Graph convolution layers allow elaboration of the random information received, building the backbone of the generated NN architecture.
Depending on the number µ of convolutional layers selected, we should expect more or less ﬁne-grained embeddings as output of this step.
However, given the fully-connected nature of the input graph, a small value of µ is enough to obtain a meaningful graph embedding.
Edge scoring & sampling.
Once a proposal of fullyconnected architecture is obtained from the graph convolution layers, the generator applies a learnable scoring function to each edge of the graph at hand.
This procedure allows different scores to be assigned to each edge of the architecture, depending on their relevance.
To score edges we ﬁrst build edge features vectors, through the concatenation of adjacent vertices features.
Mathematically speaking, the feature vector of edge connecting vertex v to vertex w is e v,w = x w (cid:107) x w , where (cid:107) denotes concatenation.
Once the edge feature is obtained, the edge relevance is scored using a standard densely-connected layer followed by normalisation in [0 , 1] , obtaining e (cid:48) v,w which represents the score given to the edge between v and w .
To avoid non-differentiability issues that may arise from scores thresholding, edges are then sampled depending on their scores using a gumbel softmax layer.
This procedure ensures the survival of relevant – from the generator perspective – edges only, aiming at satisfying Q1 .
Edge scoring and sampling are here presented as a unique step, given their logical bond.
However, it is also possible to conceive these two as separate steps, as done in
Figure 3 to ease reader understanding of the framework.
Layers operations generation.
The aim of this step is to assign one operation to each vertex – i.e., layer – of the graph corresponding to the NN architecture.
To do so, the graph embedding obtained from the graph convolution step is combined with the sampled edge scores and used as input for a new layer of graph convolution.
A softmax operation is then applied to the output of the convolutional layer to produce the one-hot encoding of the operations corresponding to each node.
This speciﬁc step, aiming at identifying realistic nodes features – i.e., operations –, is meant to satisfy Q2 .
Here, the layer generation step focus solely on the layer type – i.e., operation to deploy –, ignoring layers dimensioning issues.
Indeed, we consider layers size to be automatically inferrable from the overall NN architecture, as stated in Ying et al.
Graph reﬁnement.
Finally, the generator removes unsampled edges from the graph as well as isolated nodes, obtaining the ﬁnal NN architecture.
Possible cycles and pending nodes are also removed during this step, ensuring therefore to produce a DAG architecture.
The graphreﬁnement operation is left as the last operation to avoid possible non-differentiability issues which may arise from removal of nodes or edges.
However, this does not inﬂuence the generation of layer operations, since zero-scored edges do not propagate information in the previous convolution step.
[Figure]
3.2.2 Discriminator
The discriminator model aims at distinguishing between synthetically generated architectures and architectures available in the dataset at hand.
In the proposed framework we build the discriminator model stacking ν layers of graph convolution, followed by a single densely-connected classiﬁcation layer.
Graph convolutional layers extract graph-structured features from the input graph, while the classiﬁcation layer outputs a binary prediction.
The complexity of the discriminator model – i.e., the number of graph convolutions ν – depends on the complexity of the architectures under inspection.
Available NAS datasets consider fairly small architectures, as they deal with identical block structures, therefore in our experiments we set ν = 2 .
3.2.3 Valuer
The valuer model aims at identifying the performance of the architectures given as input.
Structurally speaking, we build the valuer model similarly to the discriminator, stacking few layers of graph convolution, followed by a single densely-connected layer.
The prediction of the valuer model over the structures generated by G are also used for the generator optimisation, aiming to push G toward the generation of more powerful NN architectures.
Indeed, the generator model is trained minimising a combination between the standard GAN loss and the reward loss obtained from the valuer NN:
[Equation]
where z represents the randomly initialised graph used as input for G , L R represents the reward loss and λ represents a balancing factor between the two loss terms.
The deﬁnition of L R depends on the role of the ﬁnal denselyconnected layer of V , which can be used either to regress the performance of the graph at hand or to binary classify graphs—strong vs. not-strong architecture.
In the ﬁrst approach, L R is represented via mean-squared error loss between the predicted performance of generated architecture and the best performing architecture.
In the second, the reward loss is represented via cross-entropy loss between predicted classiﬁcation and strong architecture labels.
Our experiments (see Section 4.4) show that the second approach is more consistent.
EXPERIMENTS AND RESULTS
In this section we propose a set of experiments to show the effectiveness of GNN2GNN for generating strong NNs.
Our source code is available at https://github.com/AndAgio/ GNN-2-GNN.
4.1 DATASETS
To test our framework performance we rely on the
NAS101 [Ying et al., 2019] and NATS [Dong et al., 2021] benchmark datasets.
Both datasets contain a set of NN architectures along with their recorded performance over a speciﬁc image classiﬁcation task.
Here, NN architectures are built from the repetition of identical cells, which are the target of our GNN2GNN approach.
NAS101 contains 423 k NN architectures trained multiple times over {+CIFAR
+}
[Krizhevsky and Hinton, 2009].
On the other hand, NATS contains a set of 15 k NN topologies trained over three different datasets: (i) CIFAR-10; (ii) CIFAR-100; (iii) ImageNet-16-120.
However, NATS represent operations over graph edges, while GNN2GNN and NAS101 represent operations over graph nodes, as introduced in Section 3.1.
Therefore, we translate NATS architectures into Section 3.1 form and remove possible duplicates, thus obtaining a reﬁned version of NATS consisting of 7 K unique architectures.
NAS101 and NATS datasets rely on similar search spaces used for the construction of NNs.
Indeed, both consider a small set of operations, containing: (i) 3 × 3 convolution, (ii) 5 × 5 convolution, and (iii) pooling —NAS101 considers max-pooling, while NATS examines average-pooling.
NAS101 contains NN cells with at most 7 nodes and 9 edges, while NATS examines cells with at most 8 nodes, without imposing any restriction on the number of edges.
4.2 EXPERIMENTAL SETUP
To test GNN2GNN ability to produce novel architectures and generate strong cells, we remove part of the architec- tures from the training dataset.
We eliminate some randomly picked cells from the dataset, as well as the best 10% of architectures—w.r.t.
their classiﬁcation accuracy.
Under these settings, the generator can not extract information from the strongest models during training, rendering the generation task more complex.
Therefore, a generator capable of producing the best 10% of architectures is to be considered a strong model.
N = 10 was selected since in
NAS101 there exists quite a signiﬁcant performance delta between the top- 10% architectures and the rest.
Each GNN2GNN instance is trained for 20 epochs over the training set using standard Stochastic Gradient Descent and setting the learning rate to 0 .
001 and the batch size to 32 .
Moreover, during the ﬁrst half of the training procedure we set λ = 1 .
This is done to allow V properly learning to distinguish between strong and weak architectures, before leveraging it to optimise G with backpropagation.
Indeed, backpropagating information from a partially trained V to the generator G may increase the noise of its training, slowing down or hindering its optimisation.
Therefore, in the ﬁrst 10 epochs the generator model is optimised only through the discriminator D .
After this setup period λ is set back to its desired value, enabling the interaction between G and V as described by Equation (2).
4.3 EVALUATION METRICS
Throughout our experiments, we consider only models which always output valid NN architectures, since they output DAGs thanks to some reﬁnement step.
Therefore, the metrics that we deﬁne refer solely to the quality of the generated architectures.
Moreover, since our framework is not directly comparable with NAS approaches, we avoid considering common NAS metrics—e.g., convergence time, etc.
Novelty measures the percentage of generated architectures not belonging to the training set used.
The Top- N metrics measure the percentage of generated architectures that belong to the best N % of architectures in terms of classiﬁcation accuracy.
Acc n measures the ratio between the number of generated architectures that reach an accuracy greater than n , and the number of generated models that belong to the dataset.
Finally, | Acc | measures the average accuracy reached by generated architectures.
4.4 ABLATION STUDY
To identify the best hyper-parameters setup for GNN2GNN, we propose a thorough ablation study.
The ablation study is performed over the NAS101 dataset, given its higher degree of expressiveness w.r.t.
NATS.
Hyper-parameters We consider the inﬂuence of the parametric values that may alter the generation of NN architectures.
We take into account the balancing factor λ used during training, the temperature τ of the gumbel softmax layer used to perform edge sampling, and the number of graph convolution layers used by the generator µ .
Tableshows the results of the ablation study on such parameters.
It is possible to notice that the model is highly affected by the balancing factor λ , which injects performance-critical information into the generator.
Indeed, leveraging smaller λ increases the performance of the proposed architectures, as the generator focuses more on the information received by V through backpropagation.
Smaller λ values also improve GNN2GNN ability to predict more complex models.
Architectures generated using λ = 0 .
1 have on average twice the number of parameters of their λ = 1 counterparts.
This phenomenon is encouraging, as it shows that GNN2GNN is capable of mapping the whole space, thanks to V .
However, smaller λ increases the risk of mode collapse issues, as highlighted by the slight drop in novelty obtained with λ = 0 .
Finally, µ and τ do not seem to heavily inﬂuence the GNN2GNN performance.
[Table]
Valuer mode The mechanism used by the valuer network V to identify strong and weak architectures may cause variation in the generation performance of GNN2GNN.
We distinguish between a classiﬁcation-based valuer C and a regression-based valuer
R. The former identiﬁes strong architectures as the cells belonging to the best half of the dataset.
On the other hand, in the regression-based setup, V aims at predicting precisely the classiﬁcation accuracy of a cell from its structure.
We pick the three best models in Table 1, retrain them using a regression-based V , and compare them against their classiﬁcation-based counterparts.
Table 2 shows the results of the ablation study, highlighting the superiority of the classiﬁcation-based setup.
Indeed, regressing exactly NN performance from its architecture is complex, mostly since few small architectural modiﬁcations may lead to relevant performance changes.
Such variability is complex to handle in a regression setup and hinders V ability to predict correctly cells strength.
[Table]
In the remainder of the experiments, we build the
GNN2GNN model employing a classiﬁcation-based V and the best hyperparameters values—i.e. µ = 2 , τ = 0 .
1 , λ = 0 .
1 , as highlighted in Table 1.
Indeed, these values represent a good starting point for deploying GNN2GNN over multiple scenarios, given NAS101 generality.
4.5 PERFORMANCE COMPARISON
To show the effectiveness of the proposed approach, we compare GNN2GNN against other generative mechanisms.
We ﬁrst consider generating random NN architectures using the Erdös–Rényi model [Erdös et al., 1960].
We then evaluate the strength of our approach against two GAN-based frameworks, relying on different generation strategies:
MOLGAN-like The model generates nodes and edges independently and simultaneously, recalling the approach by De Cao and Kipf
Two matrices representing node types and connections between them are generated from a random input vector and sampled using gumbel softmax.
RNN The model generates architectures starting from a single input node and appending new vertices – with corresponding edges – until a stopping criteria is met.
This approach resembles the one by Zhang et al.
[2019] and leverages Recurrent NNs to deal with graph construction via recursive node appending.
To make the comparison fair, both the MOLGAN-like and the RNN model are built using the three-way NNs adversarial approach that characterises GNN2GNN.
Therefore, the three approaches differ solely on the generation criteria embodied by the generator model G .
Table 3 shows the performance of the different models.
GNN2GNN vastly outperforms the counterparts, as it produces more accurate predictions for strong NN architectures.
Moreover, more than 80% of the predictions performed by our model are NNs characterised by an accuracy greater than 90% , while the best counterpart model – i.e., MOLGAN – fails to reach even 60% . This proves GNN2GNN’s generation consistency.
Indeed, even the simple random generation approach can sporadically generate powerful architectures, as shown also in Xie et al.
However, it suffers in terms of consistency, as it is uncommon to obtain articulate architectures starting from a random empty graph.
[Table]
4.6 RESISTANCE TO DATASET QUALITY DEGRADATION
To study the ﬂexibility of our approach against poorlyconstructed datasets, we analyse GNN2GNN performance when a high number of strong models are removed from the training dataset.
More in details, we ﬁrst remove the best
N % of models from the NAS101 training set, varying N between 10 and 90 , then retrain GNN2GNN.
Table 4 shows these tests results.
The performance loss between different setups is minimal, highlighting GNN2GNN strength against dataset quality degradation.
Indeed, even when almost all best models are removed from the training set, GNN2GNN produces strong predictions, showing just a 3% loss in the Top- 5 metric and a 0 .
82% decrease of the average accuracy reached by generated models.
[Table]
4.7 GENERALISATION BETWEEN DATASETS
We now consider the generalisation ability of our framework.
We start by training GNN2GNN over NATS and showing its performance.
As Table 5 shows, the performance obtained over NATS are poor, probably due to the small size of NATS—i.e., only 7 K NN architectures.
We then apply the generator model trained over NAS101 to NATS, analysing its prediction performance.
Table 5 shows the results of our generalisation study.
While still not being satisfactory, we notice that performance strongly increase when
GNN2GNN is transferred from NAS101 to NATS.
This is encouraging, especially if we consider the strong difference between NAS101 and NATS.
Indeed, only 576 NATS archi- tectures are available also in NAS101, and their performance vary on average by 16 .
183% between the two datasets.
[Table]
4.8 PRELIMINARY COMPARISON AGAINST NAS
GNN2GNN does not represent a traditional NAS technique, as it does not rely on search space exploration and focuses solely on the architecture generation procedure.
However, we can compare GNN2GNN against state-of-the-art NAS in terms of the performance obtained by the generated architectures over NAS101.
Results shown in Table 6 are extracted from Yu et al.
[2020] and consider 1000 GNN2GNN generation samples.
The average accuracy of GNN2GNN generation is comparable with other NAS approaches.
Meanwhile, results show that GNN2GNN vastly outperforms NAS techniques in terms of best accuracy.
Indeed, the best architecture generated by GNN2GNN achieves 94 .
32% , while NAO tops up at 93 .
Moreover, the architecture generated by GNN2GNN achieves a lower rank value, meaning that they are closer to the optimal architecture.
Indeed, the best achievable accuracy in NAS101 is 95 .
06% , which represents an increase of only 0 .
72% compared to what GNN2GNN achieves.
[Table]
DISCUSSION
GNN2GNN relies on an architecture-performance pairs dataset to remove part of the complexity burden of extracting architectures performance.
This might represent a possible drawback, as it requires the training of a set of hand-crafted
NNs.
However, results of Section 4.6 show how GNN2GNN learns to generate effective NNs even when most – i.e., 90% – of the arch-performance pairs are not available.
Moreover,
GNN2GNN does not impose any requirement on the dataset quality, as it can generate powerful architectures even when trained on the worst part of the dataset—i.e., worst 10%.
Finally, Section 4.7 hints how GNN2GNN can translate the generation process to a new setup, without requiring the extraction of a new dataset.
RELATED WORK
The propose framework is tightly related to state-of-the-art techniques for searching NN structures—namely, Neural
Architecture Search.
NAS techniques have been recently proposed to tackle NN design [Elsken et al., 2019, Ren et al., 2021].
NAS techniques deﬁne a search space S , containing
NN architectures.
A set of architectural rules identify the list of operations available at NN layers, as well as a list of rules deﬁning admissible connections between NN layers.
NAS approaches aims then at efﬁciently explore S to identify the strongest NN architecture.
While the search space is explored, architectures are sampled, the corresponding NNs are built, trained over the task at hand, and evaluated, depending on a performance estimation strategy.
Proposed
NAS approaches may vary for the selected search space and the exploration strategy they deploy [Real et al., 2019, Tan and Le, 2019, Chu et al., 2020,
Agiollo et al., 2021].
NAS approaches have proven to be successful in identifying powerful NN architectures.
However, these approaches present drawbacks, which raise concerns about their applicability, namely:
Learning to search vs. learning to build.
Whereas NAS approaches aim at efﬁciently searching a sub-optimal NN among the ones available in S – i.e., learning to search –, they fail to learn any proper architectural criteria — i.e., learning to build .
In contrast, GNN2GNN, relying on graph learning techniques, aims at implicitly learning NN design criteria, representing a step forward towards learning to build NNs.
Flexibility.
NAS techniques currently lack in ﬂexibility and generality, as they cannot identify architectural criteria and focus on a speciﬁc task and dataset.
Instead, GNN2GNN is limited only by the dataset at hand, being adaptable to the most diverse architectures, operations, and interconnections.
Moreover, aiming to learn architectural criteria, GNN2GNN paves the way towards generalisation between datasets.
Search space restrictions.
Most, if not all, NAS techniques limit the number of available operations or the way in which they can be connected to ease the searching procedure over
S .
GNN2GNN exploits GNN, which are applicable to any graph structure, avoiding restrictions on architectural rules.
Our work is also related to the application of GNNs to NN architectures.
Indeed, some works have exploited, with some success, the graph processing nature of GNNs to extract relevant information about NNs from their architecture.
A common approach here consists of predicting the performance of a NN from its design, aiming to avoid expensive training processes [Lukasik et al., 2020].
Few of them even integrate GNNs into NAS algorithms [Yan et al., 2020, Wen et al., 2020].
Such frameworks, however, rely on the GNN regressive power solely to evaluate rapidly the performance of the proposed architectures, removing the training process from the NAS loop.
Therefore, such approaches exploit
GNN mostly as a speed-up component for the NAS procedure, failing to capture the proper expressive power of GNNs, which is their capability to learn and sub-symbolically express NN architectural criteria.
Other works aim at ﬁnding relevant embeddings for the NN architectures at hand.
Such embeddings identify a continuous embedding space, which can then be used to optimize the NN structure [Luo et al., 2018] or propose quick search mechanisms [Li et al., 2020].
CONCLUSIONS AND FUTURE WORKS
In this paper we present a novel GNN-based three-way adversarial framework for learning to generate strong NN architectures.
The experiments completed over two state-ofthe-art datasets highlight the strength of our approach.
We show GNN2GNN ability to predict optimal NN architectures and its superiority against other available generation approaches.
Moreover, given its ﬂexibility against dataset quality degradation, the proposed framework represents a step forward towards learning architectural criteria for NNs design.
Indeed, the GNN2GNN generator is capable of predicting unseen strong architectures even when dealing with unsound dataset.
Finally, some experiments on knowledge transferability suggest the generalisability of our approach.
While aiming to overcome NAS limitations – via removal of search algorithms – GNN2GNN can also be integrated into a NAS approach as a proposal technique.
Here, the adversarial framework characterising GNN2GNN would require online training, similarly to other NAS approaches.
The implementation of a GNN2GNN-based NAS approach, along with thoroughly comparison with other NAS, is left for future works.
Finally, we also intend to focus on boosting GNN2GNN generalisation ability, by introducing dataset embedding techniques. Acknowledgements
This paper has been partially supported by the CHIST-ERA IV project “EXPECTATION” (G.A. CHIST-ERA-19-XAI005).