G ENERALIZING AND T ENSORIZING S UBGRAPH S EARCH IN THE S UPERNET
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, a special kind of graph, i.e., supernet, which allows two nodes connected by multi-choice edges, has exhibited its power in neural architecture search (NAS) by searching better architectures for computer vision (CV) and natural language processing (NLP) tasks.
In this paper, we discover that the design of such discrete architectures also appears in many other important learning tasks, e.g., logical chain inference in knowledge graphs (KGs) and meta-path discovery in heterogeneous information networks (HINs).
Thus, we are motivated to generalize the supernet search problem on a broader horizon.
However, none of the existing works are effective since the supernet’s topology is highly task-dependent and diverse.
To address this issue, we propose to tensorize the supernet , i.e., unify the subgraph search problems by a tensor formulation and encode the topology inside the supernet by a tensor network.
We further propose an efﬁcient algorithm that admits both stochastic and deterministic objectives to solve the search problem.
Finally, we perform extensive experiments on diverse learning tasks, i.e., architecture design for CV, logic inference for KG, and meta-path discovery for HIN.
Empirical results demonstrate that our method leads to better performance and architectures.
I NTRODUCTION
Deep learning (Goodfellow et al., 2017) has been successfully applied in many machine learning tasks, such as image classiﬁcation for computer vision (CV) (LeCun et al., 1998; Krizhevsky et al., 2012;
He et al., 2016; Huang et al., 2017) and language modeling for natural language processing (NLP) (Mikolov et al., 2013; Devlin et al., 2018).
While the architecture design is of great importance to deep learning, manually designing proper architectures for a certain task is hard and requires lots of human efforts or sometimes even impossible (Zoph & Le, 2017; Baker et al., 2016).
Recently, neural architecture search (NAS) techniques (Elsken et al., 2019) have been developed to alleviate this issue, which mainly focuses on CV and NLP tasks.
Behind existing NAS methods, a multi-graph (Skiena., 1992) structure, i.e., supernet (Zoph et al., 2017; Pham et al., 2018; Liu et al., 2018), where nodes are connected by edges with multiple choices, has played a central role.
In such context, the choices on each edge are different operations, and the subgraphs correspond to different neural architectures.
The objective here is to ﬁnd a suitable subgraph in this supernet, i.e. better neural architectures for a given task.
However, the supernet does not only arise in CV/NLP ﬁeld and we ﬁnd it also emerge in many other deep learning areas (see Table 1).
An example is logical chain inference on knowledge graphs (Yang et al., 2017; Sadeghian et al., 2019; Qu & Tang, 2019), where the construction logical rules can be modeled by a supernet.
Another example is meta-path discovery in heterogeneous information networks (Yun et al., 2019; Wan et al., 2020), where the discovery of meta-paths can also be modeled by a supernet.
Therefore, we propose to broaden the horizon of NAS, i.e., generalize it to many deep learning ﬁelds and solve the new NAS problem under a uniﬁed framework.
Since subgraphs are discrete objects (choices on each edge are discrete), it has been a common approach (Liu et al., 2018;
Sadeghian et al., 2019; Yun et al., 2019) to transform it into a continuous optimization problem.
Previous methods often introduce continuous parameters separately for each edge.
However, this formulation cannot generalize to different supernets as the topological structures of supernets are highly task-dependent and diverse.
Therefore, it will fail to capture the supernet’s topology and hence be ineffective.
[Table]
In this paper
[Table]
we propose a novel method TRACE (TensoRizing ArchiteCture sEarch) to introduce a continuous parameter for each subgraph (all these parameters will form a tensor).
And we propose to construct a tensor network (TN) (andrzej. et al., 2016; 2017) based on the topological structure of supernet.
For different tensor networks [Table] we introduce a uniﬁed efﬁcient algorithm for optimization on supernets.
Extensive experiments are conducted on diverse deep learning tasks.
And empirical results demonstrate that TRACE performs better than the state-of-the-art methods in each domain
[Table]
As a summary, our contributions are as follows:
• We broaden the horizon of existing supernet-based NAS methods.
Speciﬁcally, we generalize the concept of supernet in NAS to other deep learning tasks that have graph-like structures and propose to solve them in a uniﬁed framework of supernet.
• We propose a novel formulation for optimization on supernet by introducing a tensor on supernet ( tensorizing supernets ), and optimize this tensor to ﬁnd the optimal subgraph on the supernet • While existing supernet-based NAS methods ignore the topological structure of the supernet, we encode the supernet in a topology-aware manner based on the tensor network and propose an efﬁcient algorithm to solve the search problem.
• We conduct extensive experiments on various learning tasks,
i
e., architecture design for CV, logic inference for KG, and meta-path discovery for HIN.
Empirical results demonstrate that our method can ﬁnd better architectures, which lead to state-of-the-art performance on various applications.
R ELATED W ORKS 2. S UPERNET IN N EURAL A RCHITECTURE S EARCH (NAS)
There have been numerous algorithms proposed to solve the NAS problem.
The ﬁrst NAS work,
NASRL (Zoph & Le, 2017), models the NAS problem as a multiple decision making problem and proposes to use reinforcement learning (RL) (Sutton & Barto, 2018) to solve this problem.
However, this formulation does not consider the repetitively stacked nature of neural architectures and is very inefﬁcient as it has to train many different networks to converge.
To alleviate this issue, NASNet (Zoph et al., 2017)
ﬁrst models NAS as an optimization problem on supernet.
The supernet formulation enables searching for transferrable architectures across different datasets and improves the searching efﬁciency.
Later, based on the supernet formulation, ENAS (Pham et al., 2018) proposes weight-sharing techniques, which shares the weight of each subgraph in a supernet.
This technique further improves searching efﬁciency and many different methods have been proposed under this framework (see Table 1), including DARTS (Liu et al., 2018), SNAS (Xie et al., 2018), and NASP (Yao et al., 2020).
DARTS is the ﬁrst to introduce deterministic formulation to NAS ﬁeld, and SNAS uses a similar parametrized method with DARTS under stochastic formulation.
NASP improves upon DARTS by using proximal operator (Parikh & Boyd, 2014) and activates only one subgraphs in each iteration to avoid co-adaptation between subgraphs.
2. T ENSOR M ETHODS IN M ACHINE L EARNING
A tensor (Kolda & Bader, 2009) is a multi-dimensional array as an extension to a vector or matrix.
Tensor methods have found wide applications in machine learning and deep learning, including network compression (Novikov et al., 2015; Wang et al., 2018) and knowledge graph completion (Liu et al., 2020).
Recently, andrzej. et al.
(2016; 2017) have proposed a uniﬁed framework called tensor network (TN) , which uses an undirected graph to represent tensor decomposition methods.
By using different graphs, TN can cover many tensor decomposition methods as a special case, e.g., CP (Bro, 1997), Tucker (Tucker, 1966), tensor train (Oseledets, 2011), and tensor ring decomposition (Zhao et al., 2016).
However, it is not an easy task to construct a tensor network for a given problem, as the topological structure of tensor network is hard to design (Li & Sun, 2020).
P ROPOSED M ETHOD
Here, we describe our method for the supernet search problem.
Section 3.1-3.2 introduce how we tensorize the supernets.
Section 3.3 propose an optimization algorithm for the search problem which can be utilized for both stochastic and deterministic objectives.
Finally, how supernet appears beyond existing NAS works and can be generalized to these new tasks are presented in Section 3.4. Notations.
In this paper, we will use S to denote a supernet, and P to denote a subgraph in supernet. For a supernet S with T edges, we let all edges be indexed by e 1 , . . . , e T , and deﬁne C t to be the number of choices on edge e t with t ∈ { 1 , . . . , T } .
With subscript i 1 , . .
, i T denoted by i − for short, we use S i − to denote the subgraph with choices i 1 ∈ { 1 , . . .
, C 1 } , . .
, i T ∈ { 1 , . . .
, C T } .
And we use softmax ( o i ) =
exp( o i ) / (cid:80) nj =1 exp( o j ) to denote the softmax operation over a vector o i ∈ R n .
3. A T ENSOR F ORMULATION FOR S UPERNET
While existing works (Liu et al., 2018; Zoph et al., 2017;
Pham et al., 2018) introduce parameters separately for each edge e t , since edges may correlate with each other, a more general and natural approach is to introduce a continuous parameter directly for each subgraph P ∈ S .
Note that a subgraph P can be distinguished by its choices on each edge, we propose to encode all possible choices into a tensor T ∈ R C 1 ×···× C T and take these choices as indices, i.e., i − , to the tensor T .
As a consequence, the architecture of the subgraph P is indexed as S i − , and T i − ∈ [0 , 1] represent how “good” P can be.
Let f ( w , P ) stand for a learning model with the parameter w and subgraph P ∈ T , L be the loss of f on the training dataset D tr , and J measure the performance of f on the validation set D val .
Under above tensor formulation, the search objective here is:
[Equation]
Same as existing supernet search works, the subgraph P is searched in the upper-level, while the network weight w is trained in the lower-level. D tr is the training set and D val is the validation set.
However, we have the extra constraint (cid:80) i − T
i − =1 here, which is to ensure that the sum of probabilities for all subgraphs is one.
Next, we show how P and T can be parameterized with topological information from the supernet in
Section 3.2.
Then, a gradient-based algorithm which can effectively handle the constrained bi-level optimization problem (1) is proposed in Section 3.3.
3. E NCODING S UPERNET T OPOLOGY BY T ENSOR N ETWORK (TN)
Existing methods consider each edge separately and can be seen as rank- 1 factorization of the full tensor T (see Table 1) under (1), i.e. T i − =
θ 1 i 1 . . .
θ T i T where θ t i t ∈ R is the continuous parameter for [Figure] choice i t on edge t .
However, this formulation ignores the topological structure of different supernets, as it uses the same decomposition method for all supernets.
Motivated by such limitation, we propose to introduce tensor network (TN) to better encode the topological structure of supernet.
Our encoding process is described in Algorithm 1, where N ( S ) to denotes the set of nodes in this supernet and
N (cid:48) ( S ) ⊆
N ( S ) denotes the set of nodes that are connected to more than one edge.
Speciﬁcally, we introduce a third-order tensor α t for each edge, which is based on previous methods (e.g., DARTS and SNAS) but uses tensors instead of vectors to allow more ﬂexibility.
And we introduce hyper-parameters R N 1 ( t ) and R N 2 ( t ) , which are corresponding to the rank of tensor network. Then we use index summation to reﬂect the topological structure (common nodes) between different edges, i.e.
[Equation]
We also give two examples in 1 Figure 2 to illustrate our tensorizing process for two speciﬁc supernets.
Input: Supernet S ; 1: Introduce α t ∈ R R N 1( t ) × C t × R N 2( t ) for edge e t which connects nodes N 1 ( t ) and N 2 ( t ) ; 2: Compute T i − by (2); 3: return encoded supernet T i − ;
The reason of using N (cid:48) ( S ) instead of N ( S ) is 2 Proposition 1, which shows that using N (cid:48)
( S ) will not restrict the expressive power but allows the usage of less parameters.
Proposition 1.
Any tensors that can be expressed by T i − =
(cid:80) R n r n ,n ∈N ( S ) (cid:81) T t
{+=
+}
α t r N 1 ( t ) ,i t ,r N 2 ( t ) can also be expressed by T i − = (cid:80) [Equation] n
[Equation] n ,n ∈N [Equation] S )
[Equation] Tt = [Equation] α tr N 1( t ) ,i t ,r N 2( t ) .
3. S EARCH A LGORITHM
To handle subgraph search problems for various applications (Table 1), we need an algorithm that can solve the generalized supernet search problem in (1), which is parameterized by (2).
However, the resulting problem is hard to solve, as we need to handle the constraint on T , i.e., (cid:80) i − T
i − = 1 .
To address this issue, we propose to re-parameterize T using softmax trick, i.e.,
[Equation]
Then [Equation] the optimization objective J in (1) becomes [Equation]
[Equation]
where we have substitute discrete subgraphs P and normalization constraint on T with continuous parameters β .
As shown in Proposition 2, we can now solve a unconstrained problem on β , while keeping the constraint on T satisﬁed.
Proposition 2.
(cid:80) i − ¯ T
i − = 1 , where ¯ T is given in (3).
Thus, gradient-based training strategies can be reused, which makes the optimization very efﬁcient.
The complete steps for optimizing (4) are presented in Algorithm 2.
Note that, our algorithm can solve both deterministic and stochastic formulation (see Appendix A).
After the optimization of β is converged, we obtain P ∗ from tensor ¯ T and retrain the model to obtain w ∗ .
3. S UBGRAPH S EARCH BEYOND E XISTING NAS
Despite NAS, there are many important problems in deep learning that have a graph-like structure.
Examples include meta-path discovery (Yang et al., 2018;
Yun et al., 2019; Wan et al., 2020), logical chain inference (Yang et al., 2017; Sadeghian et al., 2019) and structure learning between data points (Franceschi et al., 2019).
Inspired by the recent work that exploits graph-like structures in
NAS (Li et al., 2020; You et al., 2020), we propose to model them also as a subgraph search problem on supernets.
Meta-path discovery.
Heterogeneous information networks (HINs) (Sun & Han, 2012; Shi et al., 2017) are networks whose nodes and edges have multiple types.
HIN has been widely used in many real-world network mining scenarios, e.g., node classiﬁcation (Wang et al., 2019) and recommendation (Zhao et al., 2017).
For a heterogeneous network, a meta-path (Sun et al., 2011) is a path deﬁned on it with multiple edge types.
Intuitively, different meta-paths capture different semantic information from a heterogeneous network and it is important to ﬁnd suitable meta-paths for different applications on HINs.
However, designing a meta-path on a HIN is not a trivial task and requires much human effort and domain knowledge (Zhao et al., 2017; Yang et al., 2018).
Thus, we propose to automatically discover informative meta-paths instead of to manually design.
To solve the meta-path discovery problem under the supernet framework, we ﬁrst construct a supernet
S (see Figure 2(a)) and a subgraph P ∈ S will be a meta-path on HIN.
While GTN (Yun et al., 2019) introduces weights separately for each edge, our model f ( w , P ) uses a tensor T to model P as a whole.
The performance metric L ( · ) and M ( · ) will depend on the downstream task.
In our experiments on node classiﬁcation, we use cross-entropy loss for L ( · ) and macro F1 score for M ( · ) .
Logical chain inference.
A knowledge graph (KG) (Singhal, 2012; Wang et al., 2017) is a multirelational graph composed of entities (nodes) and relations (different types of edges).
KG has found wide applications in many different areas, including question answering (Lukovnikov et al., 2017) and recommendation (Zhang et al., 2016).
An important method to understand semantics in KG is logical chain inference, which aims to ﬁnd underlying logic rules in KG.
Speciﬁcally, a logical chain is a path on knowledge graph with the following form x −→ B 1 z 1 −→ B 2 . . .
−→ B T y , where x, y, z 1 , . . .
are entities and B 1 , B 2 , . . .
, B T are different relations in the knowledge graph.
And logical chain inference is to use a logical chain to approximate a relation in KG.
Obviously, different logical chains can have critical inﬂuence on KG as incorrect logic rules will lead to wrong facts.
[Table]
However
[Table]
directly solving the inference problem will have a exponential complexity as we have to enumerate all relations (Hamilton et al., 2018).
Thus, we propose to model it as a supernet search problem to reduce complexity.
Since logical chain has a chain structure
[Table]
we construct a supernet as in Figure 2(a).
Denote our target relation as B r , the adjacency matrix of relation B r as A B r , and the one-hot vector corresponding to entity x as v x .
Our learning model f ( w , P ) now has no model parameter w , and the original bi-level problem is reduced to a single-level one with the following performance measure M ( f ( w , P ) , D ) =
(cid:80)
B r ( x,y )
=1 in D v (cid:62) x
( (cid:81)
Ti =1 A B i ) v y , which counts the number of pairs ( x, y ) that has relation B r and is predicted by logical chain x −→ B 1 z 1 −→ B 2 · · · −→ B T y in the KG D .
E XPERIMENTS
All experiments are implemented on PyTorch (Paszke et al., 2017) except for the logical chain inference, which is implemented on TensorFlow (Abadi et al., 2016) following DRUM (Sadeghian et al., 2019).
We have done all experiments on a single NVIDIA RTX 2080 Ti GPU.
4. B ENCHMARK P ERFORMANCE C OMPARISON
Here, we compare our proposed method with the state-of-the-art methods on three different applications that can be seen as subgraph search problems, i.e., neural architecture design for image classiﬁcation, logical chain inference from KG, and meta-path discovery in HIN.
4.1. D ESIGNING C ONVOLUTIONAL N EURAL N ETWORK (CNN) A RCHITECTURES
We ﬁrst apply TRACE to the architecture design problem on CNN for image classiﬁcation, which is currently the most famous application for supernet-based methods.
We consider the following two different settings for our NAS experiments: (i).
Stand-alone (Zoph & Le, 2017; Zoph et al., 2017): train each architecture to converge to obtain separate w ∗ ( P ) ; (ii).
Weight-sharing (Liu et al., 2018;
Xie et al., 2018; Yao et al., 2020): share the same parameter w across different architectures during searching.
And for both stand-alone and weight-sharing experiments, we repeat our method for ﬁve times and report the mean ± std of test accuracy of searched architectures.
Stand-alone setting.
To enable comparison under stand-alone setting, we use the NAS-Benchdataset (Dong & Yang, 2020) where the authors exhaustively trained all subgraphs in a supernet and obtain a complete record of each subgraph’s accuracy on three different datasets: CIFAR-10, CIFAR100 and ImageNet-16-120 (details in Appendix E.1).
We use the stochastic formulation and compare our method with (i) Random Search (Yu et al., 2020); (ii) REINFORCE (policy gradient) (Zoph & Le, 2017); (iii) BOHB (Falkner et al., 2018) and (iv) REA (regularized evolution) (Real et al., 2018).
Results are in Table 2, and we can see that our method achieves better results than all existing stand-alone NAS methods and even ﬁnds the optimal architecture on CIFAR-10 and CIFARdataset.
Weight-sharing setting.
We use the deterministic formulation, construct supernet follows (Liu et al., 2018), and evaluate all methods on CIFAR-10 dataset (details are in Appendix E.1). These are the
[Table]
[Table]
most popular setups for weight-sharing NAS.
Results are in Table 3, we can see that TRACE achieves comparable performances with existing weight-sharing NAS methods.
4.1. L OGIC C HAIN I NFERENCE FROM K NOWLEDGE G RAPH (KG)
For logical chain inference, we use the deterministic formulation and compare our method with the following methods: Neural LP (Yang et al., 2017), DRUM (Sadeghian et al., 2019)
[Table]
and GraIL (Teru et al., 2020).
Neural LP and DRUM are restricted to logical chain inference [Table] while GraIL considers more complex graph structures.
We also compare our method with random generated rules to better demonstrate the effectiveness of our proposed method.
We do not compare our method with embedding-based methods, e
[Table] g. RotatE (Sun et al., 2019) as those methods all need embeddings for entities and cannot generalize found rules to unseen entities.
Following the setting of DRUM
[Table]
we conduct experiments on three KG datasets: Family, UMLS and Kinship (details are in Appendix E.2) [Table] and report the best mean reciprocal rank (MRR), Hits at 1 , 3 , 10 across 5 different runs.
Results are in Table [Table] which demonstrates that our proposed method achieves better results than all existing methods.
Besides, case studies in Section 4.2 further demonstrate that TRACE can ﬁnd more accurate rules than others.
4.1. M ETA - PATH D ISCOVERY IN H ETEROGENEOUS I NFORMATION N ETWORK (HIN)
Finally, we apply TRACE to meta-path discovery problem on HINs.
Following existing works (Wang et al., 2019; Yun et al., 2019), we use the deterministic formulation and conduct experiments on three benchmark datasets: DBLP, ACM and IMDB (details are in Appendix E.3) and compare our methods with the 1) baselines in GTN (Yun et al., 2019), i.e., DeepWalk (Bryan et al., 2014), metapath2vec (Dong et al., 2017), GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´ c et al., 2018), [Table]
[Table] HAN (Wang et al., 2019) and 2) random generated meta-paths.
Results on different datasets are in Table 6, which demonstrate that TRACE performs better than other methods on different HINs.
4. C ASE S TUDY
To further investigate the performance of TRACE, we list the top rules found by TRACE and other methods in Table 7.
This result demonstrates that our method can ﬁnd more accurate logic rules than other baselines, which contributes to the superior performance of our method.
We also give the architectures and meta-paths found by
TRACE in Appendix F.
[Table]
4. A BLATION S TUDY 4.3. I MPACT OF ENCODING APPROACH
We compare TRACE with the following encoding methods on supernet: (i) DARTS [Table] which introduces continuous parameters for each edge separately; (ii) RNN, which uses a RNN to compute the weights for each edge; (iii) CP decomposition, which generalizes DARTS to higher rank; (iv) TRACE(Full), which does not adopt Proposition 1 in Algorithm 1.
Results on NAS-Bench-201 using CIFARare shown in Figure 3(a)
[Table]
and we can see that DARTS performs worse than other methods (CP and TRACE) due to insufﬁcient expressive power.
And TRACE achieves better results than CP by being topological aware.
It also shows that our simpliﬁed encoding scheme does not harm the ﬁnal performance, as veriﬁed in Proposition 1.
(a) Encoding methods.
[Figure]
4.3. I MPACT OF RANK OF TENSOR NETWORK
We also investigate the impact of R n ’s, which are ranks in tensor network (TN) on supernets.
For simplicity, we restrict R n to be equal for all nodes n ∈ N ( S ) and compare the performance of different ranks with previous state-of-the-art REA (see Table 2) in Figure 3(b).
Results demonstrate that while the rank can inﬂuence the ﬁnal performance, it is easy to set rank properly for TRACE to beat other methods.
We also adopt R n = 2 for all other experiments.
4.3. O PTIMIZATION ALGORITHMS
Finally, we compare TRACE with proximal algorithm (Parikh & Boyd, 2014), which is a popular and general algorithm for constrained optimization.
Speciﬁcally, proximal algorithm is used to solve (1) with the constraint (cid:80) i − T
i − = 1 without Proposition 2.
We solve the proximal step iteratively and numerically since there is no closed-form solutions.
The comparison is in Figure 3(c), and we can see that TRACE beats proximal algorithm by a large margin, which demonstrates that the re-parameterized by Proposition 2 is useful for optimization.
C ONCLUSION
In this paper, we generalize supernet from neural architecture search (NAS) to other machine learning tasks.
To expressively model the supernet, we introduce a tensor formulation to the supernet and represent it by tensor network ( tensorizing supernets ).
We further propose an efﬁcient gradient-based algorithm to solve the new supernet search problem.
Empirical results across various applications demonstrate that our approach has superior performance on these machine learning tasks.
A C OMPLETE A LGORITHMS OF O PTIMIZATION ON T ENSORIZED S UPERNET
Here, we give a more detailed description for our algorithm under deterministic and stochastic formulation in Algorithm 3 and 4, respectively.
In our experiments, we use the stochastic formulation for NAS under standalone setting, and deterministic formulation for others.
[Table]
[Table] [Table] [Table] [Table]
B C OMPARISON OF PARAMETERS FOR DIFFERENT METHODS
Here, we compare the number of parameters and computational cost (FLOPs) of different methods for logic rule inference. We use n as the length of logical chains, d as the dimension of embeddings, r as the rank of tensor networks and e as the number of all relations. Results are in Table 8. Since n and r is often signiﬁcantly smaller than e and d (typical values are n = 3 , r = 2 , d = 128 and e ≈ 20 − 30 ), TRACE has comparable number of parameters and computational cost with existing methods.
[Table]
For comparison on HIN, we use n as the length of meta-paths, d as the dimension of embeddings, and r as the rank of tensor networks.
Results are in Table 9.
Note that r is often relatively small compared with d (typical values are r = 2 and d = 128 ), thus TRACE has the similar number of parameters and computational cost with existing methods. [Table] [Table] C R EVIEW ON EXISTING METHODS THAT CAN BE COMBINED WITH TRACE D P ROOFS D. P ROPOSITION
Proof.
Denote those nodes in supernet whose degree is greater than one (connected by more than one edges) as N (cid:48) ( S ) .
Thus, N ( S ) \ N (cid:48) ( S ) denotes all degree- 1 nodes in supernet.
Suppose one node N 1 ( t ) is only connected by edge t , while N 2 ( t ) is not, 3 then we can rewrite T i − =
(cid:80) R n r n ,n ∈N ( S ) (cid:81)
Tt
=1 θ tr N 1( t ) ,i t ,r N 2( t ) as follows:
T i − = (cid:88) r n ,n ∈N ( S )
T (cid:89) t = α tr N 1( t ) ,i t ,r N 2( t ) =
(cid:88) r n ,n ∈N (cid:48) ( S ) (cid:88) r n ,n ∈N ( S ) \N (cid:48)
( S )
T (cid:89) t = α tr N 1( t ) ,i t ,r N 2( t ) = (cid:88) r n ,n ∈N (cid:48) ( S ) T (cid:89) [Equation]
where similar process is done for all edges.
Thus, only nodes whose degree is greater than 1 is actually needed for index contraction.
And for n ∈ N ( S ) \ N (cid:48) ( S ) , we can simply set R n =without loss of expressive power.
D. P ROPOSITION
Proof.
Following (3), we can have:
[Equation]
E E XPERIMENT DETAILS E. N EURAL ARCHITECTURE SEARCH (NAS)
Stand-alone setting The supernet used in NAS-Bench-201 has 3 nodes, and each pair of nodes is connected by a directed edge, which gives 6 edges in total.
And for each edge, we have 5 different operations (“choices”): zero, skip connect, 1 × 1 convolution, 3 × 3 convolution and 3 × 3 average pooling.
And the details of datasets used in NAS-Bench-201 are in Table 11.
[Table]
Weight-sharing setting Our construction of supernet follows (Liu et al., 2018).
The supernet has 7 nodes, where the ﬁrst two nodes are the output from previous two cells, respectively, and the last node performs depthwise concatenation to the output of the rest four nodes.
Thus, the supernet hasedges with multiple choices (operations), and for each edge, we consider 8 different operations: zero, skip connect, 3 × 3 and 5 × 5 separable convolution, 3 × 3 and 5 × 5 dilated separable convolution, 3 × 3 max pooling and 3 × 3 average pooling.
We evaluate all weight-sharing methods on CIFARdataset and the dataset division is the same as in Table 11.
E. L OGIC C HAIN I NFERENCE
In our experiments, we follow the setting in DRUM (Sadeghian et al., 2019) and set the max length of rules T to be 3 for all datasets.
And we set the rank L to be 4 in DRUM based on best validation performances.
The details of KG datasets used in experiments are in Table 12.
[Table]
E. M ETA - PATH D ISCOVERY
The details of HIN datasets used in our experiments are in Table 13.
[Table]
[Figure]
[Table]
Indeed, correlation is a good criterion to show the rationality of one-shot architecture search methods (Bender et al., 2018; Liu et al., 2018; Yu et al., 2020; Guo et al., 2020).
However, it is only a sufﬁcient not necessary condition. Speciﬁcally, the goal of tensor T here is to capture good subgraph in the whole supernet, thus we expect the possibilities of P i will concentrate on some top subgraphs, which is shown in below Figure
[Figure]
F. C ASE STUDIES
[Figure]
c_
[Figure] [Figure]
2 sep
[Figure]
conv
[Figure]
sep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x sep_conv_3x dil_conv
[Figure]
5x c_{k} (
[Figure]
Normal Cell c_{k-2} avg_pool_3x  avg_pool_3x  avg_pool
[Figure]
3x
[Figure]
avg
[Figure]
pool
[Figure]
c_{
[Figure] [Figure] [Figure]
[Table]
PSP
[Table]
IMDB
MAM, MDM
DM, AM, MDM
MDM, MDMAM, MAM
F. R UNNING PLOTS
[Figure] [Figure]
F. A BLATION STUDIES
[Figure] [Figure]