C OVARIANT C OMPOSITIONAL N ETWORKS F OR L EARNING G RAPHS
A BSTRACT
Most existing neural networks for learning graphs address permutation invariance by conceiving of the network as a message passing scheme, where each node sums the feature vectors coming from its neighbors.
We argue that this imposes a limitation on their representation power, and instead propose a new general architecture for representing objects consisting of a hierarchy of parts, which we call covariant compositional networks (CCNs).
Here, covariance means that the activation of each neuron must transform in a speciﬁc way under permutations, similarly to steerability in CNNs.
We achieve covariance by making each activation transform according to a tensor representation of the permutation group, and derive the corresponding tensor aggregation rules that each neuron must implement.
Experiments show that CCNs can outperform competing methods on standard graph learning benchmarks.
I NTRODUCTION
Learning on graphs has a long history in the kernels literature, including approaches based on random walks (G¨artner, 2002; Borgwardt & Kriegel, 2005; Feragen et al., 2013), counting subgraphs (Shervashidze et al., 2009), spectral ideas (Vishwanathan et al., 2010), label propagation schemes with hashing (Shervashidze et al., 2011; Neumann et al., 2016), and even algebraic ideas (Kondor & Borgwardt, 2008).
Many of these papers address moderate size problems in chemo- and bioinformatics, and the way they represent graphs is essentially ﬁxed.
Recently, with the advent of deep learning and much larger datasets, a sequence of neural network based approaches have appeared to address the same problem, starting with (Scarselli et al., 2009).
In contrast to the kernels framework, neural networks effectively integrate the classiﬁcation or regression problem at hand with learning the graph representation itself, in a single, end-to-end system.
In the last few years, there has been a veritable explosion in research activity in this area.
Some of the proposed graph learning architectures (Duvenaud et al., 2015; Kearnes et al., 2016; Niepert et al., 2016) directly seek inspiration from the type of classical CNNs that are used for image recognition (LeCun et al., 1998; Krizhevsky et al., 2012).
These methods involve ﬁrst ﬁxing a vertex ordering, then moving a ﬁlter across vertices while doing some computation as a function of the local neighborhood to generate a representation.
This process is then repeated multiple times like in classical CNNs to build a deep graph representation.
Other notable works on graph neural networks include (Li et al., 2016; Sch¨utt et al., 2017; Battaglia et al., 2016; Kipf & Welling, 2017).
Very recently, (Gilmer et al., 2017) showed that many of these approaches can be seen to be speciﬁc instances of a general message passing formalism, and coined the term message passing neural networks (MPNNs) to refer to them collectively.
While MPNNs have been very successful in applications and are an active ﬁeld of research, they differ from classical CNNs in a fundamental way: the internal feature representations in CNNs are equivariant to such transformations of the inputs as translation and rotations (Cohen & Welling, 2016; 2017), the internal representations in MPNNs are fully invariant.
This is a direct result of the fact that MPNNs deal with the permutation invariance issue in graphs simply by summing the messages coming from each neighbor.
In this paper we argue that this is a serious limitation that restricts the representation power of MPNNs.
MPNNs are ultimately compositional (part-based) models, that build up the representation of the graph from the representations of a hierarchy of subgraphs.
To address the covariance issue, we study the covariance behavior of such networks in general, introducing a new general class of neural network architectures, which we call compositional networks (comp-nets).
One advantage of this generalization is that instead of focusing attention on the mechanics of how information propagates from node to node, it emphasizes the connection to convolutional networks, in particular, it shows that what is missing from MPNNs is essentially the analog of steerability .
Steerability implies that the activations (feature vectors) at a given neuron must transform according to a speciﬁc representation (in the algebraic sense) of the symmetry group of its receptive ﬁeld, in our case, the group of permutations, S m .
In this paper we only consider the deﬁning representation and its tensor products, leading to ﬁrst, second, third etc.
order tensor activations .
We derive the general form of covariant tensor propagation in comp-nets, and ﬁnd that each “channel” in the network corresponds to a speciﬁc way of contracting a higher order tensor to a lower order one.
Note that here by tensor activations we mean not just that each activation is expressed as a multidimensional array of numbers (as the word is usually used in the neural networks literature), but also that it transforms in a speciﬁc way under permutations, which is a more stringent criterion.
The parameters of our covariant comp-nets are the entries of the mixing matrix that prescribe how these channels communicate with each other at each node.
Our experiments show that this new architecture can beat scalar message passing neural networks on several standard datasets.
L EARNING GRAPHS
Graph learning encompasses a broad range of problems where the inputs are graphs and the outputs are class labels (classiﬁcation), real valued quantities (regression) or more general, possibly combinatorial, objects.
In the standard supervised learning setting this means that the training set consists of m input/output pairs { ( G 1 , y 1 ) , ( G 2 , y 2 ) , . . . , ( G m , y m ) } , where each G i is a graph and y i is the corresponding label, and the goal is to learn a function h : G → y that will successfully predict the labels of further graphs that were not in the training set.
By way of ﬁxing our notation, in the following we assume the each graph G is a pair ( V, E ) , where V is the vertex set of G and E ⊆
V × V is its edge set.
For simplicity, we assume that
V = { 1 , 2 , . . . , n } .
We also assume that G has no self-loops ( ( i, i ) (cid:54)∈ E for any i ∈ V ) and that
G is symmetric, i.e., ( i, j ) ∈
E ⇒ ( j, i ) ∈ E 1 .
We will, however, allow each edge ( i, j ) to have a corresponding weight w i,j , and each vertex i to have a corresponding feature vector (vertex label) l i ∈ R d .
The latter, in particular, is important in many scientiﬁc applications, where l i might encode, for example, what type of atom occupies a particular site in a molecule, or the identity of a protein in a biochemical interaction network.
All the topological information about G can be summarized in an adjacency matrix A ∈ R n × n , where A i,j = w i,j if i and j are connected by an edge, and otherwise A i,j = 0 .
When dealing with labeled graphs, we also have to provide ( l 1 , . . . , l n ) to fully specify G .
One of the most fascinating aspects of graphs, but also what makes graph learning challenging, is that they involve structure at multiple different scales.
In the case when G is the graph of a protein, for example, an ideal graph learning algorithm would represent G in a manner that simultaneously captures structure at the level of individual atoms, functional groups, interactions between functional groups, subunits of the protein, and the protein’s overall shape.
The other major requirement for graph learning algorithms relates to the fact that the usual ways to store and present graphs to learning algorithms have a critical spurious symmetry:
If we were to permute the vertices of G by any permutation σ : { 1 , 2 , . . . , n } → { 1 , 2 , . . . , n } (in other words, rename vertex 1 as σ (1) , vertex 2 as σ (2) , etc.) , then the adjacency matrix would change to
[Equation]
and simultaneously the vertex labels would change to ( l (cid:48) 1 , . . . , l (cid:48) n ) , where l (cid:48) i =
l σ − 1 ( i ) .
However,
G (cid:48) =
( A (cid:48) , l (cid:48) 1 , . . . , l (cid:48) n ) would still represent exactly the same graph as G =
( A, l 1 , . . . , l n ) .
In partic-
) in or is presented to the algorithm must not make a difference to that it returns must its
Most learning algorithms for combinatorial objects hinge on some sort of ﬁxed or learned internal representation of data, called the feature map , which, in our case we denote φ ( G ) .
The set of all n ! possible permutations of { 1 , 2 , . . . , n } forms a group called the symmetric group of order n , denoted S n .
The permutation invariance criterion can then be formulated as follows (Figure 1).
Deﬁnition 1.
Let A be a graph learning algorithm that uses a feature map G (cid:55)→ φ ( G ) .
We say that the feature map φ (and consequently the algorithm A ) is permutation invariant if, given any n ∈ N , any n vertex labeled graph G =
( A, l 1 , . . . , l n ) , and any permutation σ ∈ S n , letting
G (cid:48)
=
( A (cid:48) , l (cid:48) 1 , . . . , l (cid:48) n ) , where A (cid:48)
i,j =
A σ − 1 ( i ) ,σ − 1 ( j ) and l (cid:48) i = l σ − 1 ( i ) , we have that φ ( G ) =
φ ( G (cid:48) ) .
Capturing multiscale structure and respecting permutation invariance are the two the key constraints around which most of the graph learning literature revolves.
In kernel based learning, for example, invariant kernels have been constructed by counting random walks (G¨artner, 2002), matching eigenvalues of the graph Laplacian (Vishwanathan et al., 2010) and using algebraic ideas (Kondor & Borgwardt, 2008).
C OMPOSITIONAL NETWORKS
Many recent graph learning papers, whether or not they make this explicit, employ a compositional approach to modeling graphs, building up the representation of G from representations of subgraphs.
At a conceptual level, this is similar to part-based modeling, which has a long history in machine learning (Fischler & Elschlager, 1973; Ohta et al., 1978;
Tu et al., 2005; Felzenszwalb & Huttenlocher, 2005; Zhu & Mumford, 2006; Felzenszwalb et al., 2010).
In this section we introduce a general, abstract architecture called compositional networks (comp-nets) for representing complex objects as a combination of their parts, and show that several exisiting graph neural networks can be seen as special cases of this framework.
Deﬁnition 2.
Let G be a compound object with n elementary parts (atoms)
E = { e 1 , . . e n } .
A composition scheme for G is a directed acyclic graph (DAG)
M in which each node n i is associated with some subset P i of E (these subsets are called the parts of G ) in such a way that
If n i is a leaf node, then P i contains a single atom e ξ ( i ) 2 . 2.
M has a unique root node n r , which corresponds to the entire set { e 1 , . . .
, e n } .
For any two nodes n i and n j , if n i is a descendant of n j , then P i ⊂ P j .
We deﬁne a compositional network as a composition scheme in which each node n i also carries a feature vector f i that provides a representation of the corresponding part (Figure 2).
When we want n{ e 1 } n{ e 2 } n{ e 3 } n{ e 4 } n 5 { e 3 ,e 4 } n{ e 1 ,e 4 } n 7 { e 2 ,e 3 } n 8 { e 2 ,e 3 ,e 4 } nn 10 { e 1 ,e 2 ,e 4 } n r { e 1 ,e 2 ,e 3 ,e 4 } fnfnfnfnfnfnfnf 8 nf 9 nf 10 nf r n r
[Figure] (cid:48) [Figure] to emphasize the connection to more classical neural architectures, we will refer to n i as the i ’th neuron , P i as its receptive ﬁeld 3 , and f i as its activation .
Deﬁnition 3.
Let G be a compound object in which each atom e i carries a label l i , and M a composition scheme for G .
The corresponding compositional network N is a DAG with the same structure as M in which each node n i also has an associated feature vector f i such that
If n i is a leaf node, then f i = l ξ ( i ) .
If n i is a non-leaf node, and its children are n c 1 , . . . , n c k , then f i =
Φ( f c 1 , f c 2 , . .
, f c k ) for some aggregation function Φ . (Note: in general, Φ can also depend on the relationships between the
subparts
, but for now, to keep the discussion as simple as possible, we ignore this
possibility The representation φ ( G ) afforded by the comp-net is given by the feature vector f r of the root.
Note that while, for the sake of concreteness, we call the f i ’s “feature vectors”, there is no reason a priori why they need to be vectors rather than some other type of mathematical object.
In fact, in the second half of the paper we make a point of treating the f i ’s as tensors, because that is what will make it the easiest to describe the speciﬁc way that they transform with respect to permutations.
In compositional networks for graphs, the atoms will usually be the vertices, and the P i parts will correspond to clusters of nodes or neighborhoods of given radii.
Comp-nets are particularly attractive in this domain because they can combine information from the graph at different scales.
The comp-net formalism also suggests a natural way to satisfy the permutation invariance criterion of
Deﬁnition 1.
Deﬁnition 4.
Let M be the composition scheme of an object G with n atoms and M (cid:48) the composition scheme of another object that is equivalent in structure to G , except that its atoms have been permuted by some permutation σ ∈ S n ( e (cid:48) i =
e σ − 1 ( i ) and (cid:96) (cid:48) i = (cid:96) σ − 1 ( i ) ).
We say that M (more precisely, the algorithm generating M ) is permutation invariant if there is a bijection ψ : M → M (cid:48) taking each n a ∈ M to some n (cid:48) b ∈ M (cid:48) such that if P a = { e i 1 , . . . , e i k } , then P (cid:48) b = { e (cid:48) σ ( i 1 ) , . .
, e (cid:48) σ ( i k ) } .
Proposition 1.
Let φ ( G ) be the output of a comp-net based on a composition scheme M .
Assume 1.
M is permutation invariant in the sense of Deﬁnition 4.
The aggregation function Φ( f c 1 , f c 2 , . . . , f c k ) used to compute the feature vector of each node from the feature vectors of its children is invariant to the permutations of its arguments.
Then the overall representation φ ( G ) is invariant to permutations of the atoms.
In particular, if G is a graph and the atoms are its vertices, then φ is a permutation invariant graph representation.
3. M ESSAGE PASSING NEURAL NETWORKS AS A SPECIAL CASE OF COMP - NETS
Graph learning is not the only domain where invariance and multiscale structure are important: the most commonly cited reasons for the success of convolutional neural networks (CNNs) in image tasks is their ability to address exactly these two criteria in the vision context.
Furthermore, each neuron n i in a CNN aggregates information from a small set of neurons from the previous layer, therefore its receptive ﬁeld, corresponding to P i , is the union of the receptive ﬁelds of its “children”, so we have a hierarchical structure very similar to that described in the previous section.
In this sense,
CNNs are a speciﬁc kind of compositional network, where the atoms are pixels.
This connection has inspired several authors to frame graph learning as a generalization of convolutional nets to the graph domain (Bruna et al., 2014;
Henaff et al., 2015; Duvenaud et al., 2015; Defferrard et al., 2016; Kipf & Welling, 2017).
While in mathematics convolution has a fairly speciﬁc meaning that is side-stepped by this analogy, the CNN analogy does suggest that a natural way to deﬁne the Φ aggregation functions is to let Φ( f c 1 , f c 2 , . . . , f c k ) be a linear function of f c 1 , f c 2 , . . . , f c k followed by a pointwise nonlinearity, such as a ReLU operation.
To deﬁne a comp-net for graphs we also need to specify the composition scheme M .
Many algorithms deﬁne M in layers, where each layer (except the last) has one node for each vertex of G :
M 1.
In layer (cid:96) =
each node n 0 i represents the single vertex P {+
+}
i = { i } .
M 2.
In layers (cid:96) = 1 , 2 , . . .
, L , node n (cid:96)i is connected to all nodes from the previous level that are neighbors of i in G , i.e., the children of n (cid:96)i are
[Equation]
where N ( i ) denotes the set of neighbors of i in G .
Therefore, P (cid:96)i =
[Table]
M 3.
In layer L +1 we have a single node n r that represents the entire graph and collects information from all nodes at level L .
Since this construction only depends on topological information about G , the resulting composition scheme is guaranteed to be permutation invariant in the sense of Deﬁnition 4.
A further important consequence of this way of deﬁning M is that the resulting comp-net can be equivalently interpreted as label propagation algorithm , where in each round (cid:96) = 1 , 2 , . . .
, L , each vertex aggregates information from its neighbors and then updates its own label.
Many authors choose to describe graph neural networks exclusively in terms of label propagation, without mentioning the compositional aspect of the model.
Gilmer et al.
(2017) call this general approach message passing neural networks , and point out that a range of different graph learning architectures are special cases of it.
More broadly, the classic Weisfeiler–Lehman test of isomorphism also follows the same logic (Weisfeiler & Lehman, 1968; Read & Corneil, 1977;
Cai et al., 1992), and so does the related Weisfeiler–Lehman kernel, arguably the most successful kernel-based approach to graph learning (Shervashidze et al., 2011).
Note also that in label propagation or message passing algorithms there is a clear notion of the source domain of vertex i at round (cid:96) , as the set of vertices that can inﬂuence f (cid:96) i , and this corresponds exactly to the receptive ﬁeld P (cid:96) i of “neuron” n (cid:96) i in the comp-net picture.
The following proposition is immediate from the form of Algorithm 1 and reassures us that message passing neural networks, as special cases of comp-nets, do indeed produce permutation invariant representations of graphs.
Proposition 2.
Any label propagation scheme in which the aggregation function Φ is invariant to the permutations of its arguments is invariant to permutations in the sense of Deﬁnition 1.
In the next section we argue that invariant message passing networks are limited in their representation power, however, and describe a generalization via comp-nets that overcomes some of these limitations.
C OVARIANT COMPOSITIONAL NETWORKS
One of the messages of the present paper is that invariant message passing algorithms, of the form described in the previous section, are not the most general possible compositional models for producing permutation invariant representations of graphs (or of compound objects, in general).
Once again, an analogy with image recognition is helpful.
Classical CNNs face two types of basic image transformations: translations and rotations.
With respect to translations (barring pooling, edge effects and other complications), CNNs behave in a quasi-invariant way, in the sense that if the input image is translated by any integer amount ( t x , t y ) , the activations in each layer (cid:96) = 1 , 2 , . . . L translate the same way: the activation of any neuron n (cid:96)i,j is simply transferred to neuron n (cid:96)i + t 1 ,j + t 2 , i.e., f (cid:48)
(cid:96)i + t 1 ,j + t 2 = f (cid:96)i,j .
This is the simplest manifestation of a well studied property of CNNs called equivariance (Cohen & Welling, 2016; Worrall et al., 2017).
With respect to rotations, however, the situation is more complicated: if we rotate the input image by, e.g., 90 degrees, not only will the part of the image that fell in the receptive ﬁeld of a particular neuron n (cid:96)i,j move to the receptive ﬁeld of a different neuron n (cid:96)j, − i , but the orientation of the receptive ﬁeld will also change (Figure 4).
Consequently, features which were, for example, previously picked up by horizontal ﬁlters will now be picked up by vertical ﬁlters.
Therefore, in general, f (cid:48) (cid:96)j, − i (cid:54) = f (cid:96)i,j .
It can be shown that one cannot construct a CNN for images that behaves in a quasi-invariant way with respect to both translations and rotations unless every ﬁlter is directionless.
It is, however, possible to construct a CNN in which the activations transform in a predictable and reversible way, in particular, f (cid:48) (cid:96) j, − i =
R ( f (cid:96) i,j ) for some ﬁxed invertible function R .
This phenomenon is called steerability , and has a signiﬁcant literature in both classical signal processing (Freeman &
Adelson, 1991; Simoncelli et al., 1992; Perona, 1995; Teo & Hel-Or, 1998; Manduchi et al., 1998) and the neural networks ﬁeld (Cohen & Welling, 2017).
The situation in compositional networks is similar.
The comp-net and message passing architectures that we have examined so far, by virtue of the aggregation function being symmetric in its arguments, are all quasi-invariant (with respect to permutations) in the following sense.
Deﬁnition 5.
Let G be a compound object of n parts and G (cid:48) an equivalent object in which the atoms have been permuted by some permutation σ .
Let N be a comp-net for G based on an invariant composition scheme, and N (cid:48) be the corresponding network for G (cid:48) .
We say that N is quasi-invariant if for any n i ∈ N , letting n (cid:48)
j be the corresponding node in N (cid:48) , f i = f (cid:48)
j for any σ ∈ S n
Quasi-invariance in comp-nets is equivalent to the assertion that the activation f i at any given node must only depend on P i
= {
e j 1 , . . . , e j k } as a set , and not on the internal ordering of the atoms e j 1 , . . . , e j k making up the receptive ﬁeld.
At ﬁrst sight this seems desirable, since it is exactly what we expect from the overall representation φ ( G ) .
On closer examination, however, we realize that this property is potentially problematic, since it means that n i has lost all information about which vertex in its receptive ﬁeld has contributed what to the aggregate information f i .
In the CNN analogy, we can say that we have lost information about the orientation of the receptive ﬁeld.
In particular, if, further upstream, f i is combined with some other feature vector f j from a node with an overlapping receptive ﬁeld, the aggregation process has no way of taking into account which parts of the information in f i and f j come from shared vertices and which parts do not (Figure 5).
The solution is to upgrade the P i receptive ﬁelds to be ordered sets , and explicitly establish how f i co-varies with the internal ordering of the receptive ﬁelds.
To emphasize that henceforth the P i sets are ordered, we will use parentheses rather than braces to denote their content.
Deﬁnition 6.
Let G , G (cid:48) , N and N (cid:48) be as in Deﬁnition 5.
Let n i be any node of N and n j the corresponding node of N (cid:48) .
Assume that P i =
( e p 1 , . . . , e p m ) while P (cid:48)
j
= ( e q 1 , . . , e q m ) , and let π ∈ S m be the permutation that aligns the orderings of the two receptive ﬁelds, i.e., for which e q π ( a ) =
e p a .
We say that N is covariant to permutations if for any π , there is a corresponding function R π such that f (cid:48) j = R π ( f i ) .
4. F IRST ORDER COVARIANT COMP - NETS
The form of covariance prescribed by Deﬁnition 6 is very general.
To make it more speciﬁc, in line with the classical literature on steerable representations, we make the assumption that the { f (cid:55)→ R π ( f ) } π ∈ S m maps are linear , and by abuse of notation, from now on simply treat them as matrices (with R π ( f )
= R π f ).
The linearity assumption automatically implies that { R π } π ∈ S m is a representation of S m in the group theoretic sense of the word (for the deﬁnition of group representations, see the Appendix) 4 .
Proposition 3.
If for any π ∈ S m , the f (cid:55)→ R π ( f ) map appearing in Deﬁnition 6 is linear, then the corresponding { R π } π ∈ S m matrices form a representation of S m .
The representation theory of symmetric groups is a rich subject that goes beyond the scope of the present paper (Sagan, 2001).
However, there is one particular representation of S m that is likely familiar even to non-algebraists, the so-called deﬁning representation , given by the P π ∈ R n × n permutation matrices [ P [Equation] ] i,
[Equation]
if π
[Equation]
j )
= i 0 otherwise . It is easy to verify that P π 2 π 1 = P π 2 P π 1 for any π 1 ,π 2 ∈ S m , so { P π } π ∈ S m is indeed a representation of S m .
If the transformation rules of the f i activations in a given comp-net are dictated by this representation, then each f i must necessarily be a |P i | dimensional vector, and intuitively each component of f i carries information related to one speciﬁc atom in the receptive ﬁeld, or the interaction of that speciﬁc atom with all the others.
We call this case ﬁrst order permutation covariance .
Deﬁnition 7.
We say that n i is a ﬁrst order covariant node in a comp-net if under the permutation of its receptive ﬁeld P i by any π ∈ S |P i | , its activation trasforms as f i (cid:55)→ P
π f i .
4. S ECOND ORDER COVARIANT COMP - NETS
It is easy to verify that given any representation ( R g ) g ∈ G of a group G , the matrices ( R g ⊗ R g ) g ∈ G also furnish a representation of G .
Thus, one step up in the hierarchy from P π –covariant comp-nets are P π ⊗ P π –covariant comp-nets, where the f i feature vectors are now |P
i | 2 dimensional vectors that transform under permutations of the internal ordering by π as f i (cid:55)→ ( P π ⊗ P π ) f i .
If we reshape f i into a matrix F i ∈ R
|P
i |×|P i | , then the action
[Equation]
is equivalent to P π ⊗ P π acting on f i .
In the following, we will prefer this more intuitive matrix view, since it clearly expresses that feature vectors that transform this way express relationships between the different constituents of the receptive ﬁeld.
Note, in particular, that if we deﬁne A ↓ P i as the restriction of the adjacency matrix to P i (i.e., if P i =
( e p 1 , . . . , e p m ) then [ A ↓ P i ] a,b =
A p a ,p b ), then A ↓
P i transforms exactly as F i does in the equation above.
Deﬁnition 8.
We say that n i is a second order covariant node in a comp-net if under the permutation of its receptive ﬁeld P i by any π ∈ S |P i | , its activation transforms as F i (cid:55)→ P π F i P (cid:62) π .
4. T HIRD AND HIGHER ORDER COVARIANT COMP - NETS
Taking the pattern further lets us consider third, fourth, and general, k ’th order nodes in our compnet, in which the activations are k ’th order tensors, transforming under permutations as
F i (cid:55)→ F (cid:48) i where
[Equation]
In the more compact, so called Einstein notation 5 ,
[Equation]
In general, we will call any quantity which transforms according to this equation a k’th order Ptensor .
Note that this notion of tensors is distinct from the common usage of the term in neural networks, and more similar to how the word is used in Physics, because it not only implies that F i is a quanity representable by an m × m × . . . × m array of numbers, but also that F i transforms in a speciﬁc way.
Since scalars, vectors and matrices can be considered as 0 th , 1 st and 2 nd order tensors, respectively, the following deﬁnition covers Deﬁnitions 5, 7 and 8 as special cases (with quasi-invariance being equivalent to zeroth order equivariance).
To unify notation and terminology, regardless of the dimensionality, in the following we will always talk about feature tensors rather than feature vectors , and denote the activations with F i rather than f i , as we did in the ﬁrst half of the paper.
Deﬁnition 9.
We say that n i is a k’th order covariant node in a comp-net if the corresponding activation F i is a k ’th order P –tensor, i.e., it transforms under permutations of P i according to (1), or the activation is a sequence of c separate P –tensors F (1) i , . . . , F ( c ) i corresponding to c distinct channels.
T ENSOR AGGREGATION RULES
The previous sections prescribed how activations must transform in comp-nets of different orders, but did not explain how this can be assured, and what it entails for the Φ aggregation functions.
Fortunately, tensor arithmetic provides a compact framework for deriving the general form of these operations.
Recall the four basic operations that can be applied to tensors 6 :
The tensor product of A ∈ T k with B ∈ T p yields a tensor C
=
A ⊗ B ∈ T p + k where
[-where C i 1 ;i 2 ;:::;i k + p = A i 1 ;i 2 ;:::;i k B i k +
 i k +2 ;:::;i k + p : 2.-][Equation]
The elementwise product of A ∈ T k with B ∈ T p along dimensions ( a 1 , a 2 , . .
. , a p ) yields a tensor C =
A (cid:12) ( a 1 ,...,a p ) B ∈ T k where
[Equation]
The projection (summation) of A ∈ T k along dimensions { a 1 , a 2 , . . . , a p } yields a tensor C =
A ↓ a 1 ,...,a p ∈ T k − p with
[Equation]
where we assume that i a 1 , . . , i a p have been removed from amongst the indices of C . 4.
The contraction of A ∈ T k along the pair of dimensions { a, b } (assuming a < b ) yields a k −order tensor
[Equation]
where again we assume that i a and i b have been removed from amongst the indices of C .
Using
Einstein notation this can be written much more compactly as
[Equation]
where δ i a ,i b is the diagonal tensor with δ i,j = 1 if i = j and 0 otherwise.
In a somewhat unorthodox fashion, we also generalize contractions to (combinations of) larger sets of indices {{ a 11 , . . , a 1 p 1 } , { a 21 , . . . , a 2 p 2 } , . . . , { a q 1 , . .
. , a qp q }} as the ( k − (cid:80) j p j ) order tensor
[Equation]
Note that this subsumes projections, since it allows us to write A ↓ a 1 ,...,a p in the slightly unusual looking form
[Equation]
The following proposition shows that, remarkably, all of the above operations (as well as taking linear conbinations) preserve the way that P –tensors behave under permutations and thus they can be freely “mixed and matched” within Φ .
Proposition 4.
Assume that A and B are k ’th and p ’th order P –tensors, respectively.
Then
A ⊗ B is a k + p ’th order P –tensor. 2. A (cid:12) ( a 1 ,...,a p ) B is a k ’th order P –tensor. 3.
A ↓ a 1 ,...,a p is a k − p ’th order P –tensor. 4.
A i 1 ,i 2 ,...,i k δ a 11 ,...,a 1 p 1 . . δ a q 1 ,...,a qpq is a k − (cid:80) j p j ’th order P –tensor.
In addition, if A 1 , . . . , A u are k ’th order P –tensors and α 1 , . . . , α u are scalars, then (cid:80)
j α j A j is a k ’th order P –tensor.
The more challenging part of constructing the aggregation scheme for comp-nets is establishing how to relate P –tensors at different nodes.
The following two propositions answer this question.
Proposition 5.
Assume that node n a is a descendant of node n b in a comp-net N , P a
=
( e p 1 , . . . , e p m ) and P b = ( e q 1 , . . . , e q m (cid:48) ) are the corresponding ordered receptive ﬁelds (note that this implies that, as sets, P a ⊆ P b ), and χ a → b ∈ R m × m (cid:48) is an indicator matrix deﬁned
[Equation]
Assume that F is a k ’th order P –tensor with respect to permutations of ( e p 1 , . . , e p m ) .
Then, dropping the a → b superscript for clarity,
[Equation]
is a k ’th order P –tensor with respect to permutations of ( e q 1 , . . , e q m (cid:48) ) .
Equation 2 tells us that when node n b aggregates P –tensors from its children, it ﬁrst has to “promote” them to being P –tensors with respect to the contents of its own receptive ﬁeld by contracting along each of their dimensions with the appropriate χ a → b matrix.
This is a critical element in comp-nets to guarantee covariance.
Proposition 6.
Let n c 1 , . . . , n c s be the children of n t in a message passing type comp-net with corresponding k ’th order tensor activations F c 1 , . . , F c s . Let [ (cid:101)
[Equation]
[Equation] [Equation] [Equation] χ
[Equation]
t ]
[Equation]
i 2 . .
[Equation]
χ
[Equation]
→
[Equation]
[Equation]
k
[Equation]
k [ F
[Equation]
[Equation]
[Equation]
k be the promotions of these activations to P –tensors of n t .
Assume that P t =
( e p 1 , . . . , e p m ) .
Now let F be a k +1 ’th order object in which the j ’th slice is (cid:101) F p j if n p j is one of the children of n t , i.e., F i
[Equation]
...,i
[Equation]
[Equation]
=
[Equation]
[Equation]
k , and zero otherwise.
Then F is a k +1 ’th order P –tensor of n t .
Finally, as already mentioned, the restriction of the adjacency matrix to P i is a second order P – tensor, which gives an easy way of explicitly adding topological information to the activation.
Proposition 7.
If F i is a k ’th order P –tensor at node n i , and A ↓ P i is the restriction of the adjacency matrix to P i as deﬁned in Section 4.2, then F ⊗ A ↓ P i is a k +2 ’th order P –tensor.
5. T HE GENERAL AGGREGATION FUNCTION AND ITS SPECIAL CASES
Combining all the above results, assuming that node n t has children n c 1 , . . . , n c s , we arrive at the following general algorithm for the aggregation rule Φ t :
Collect all the k ’th order activations F c 1 , . . , F c s of the children.
Promote each activation to (cid:101) F c 1 , . . , (cid:101) F c s (Proposition 5).
Stack (cid:101) F c 1 , . . . , (cid:101) F c s together into a k +1 order tensor T (Proposition 6).
Optionally form the tensor product of T with A ↓ P t to get a k +3 order tensor H (otherwise just set H = T ) (Proposition 7).
Contract H along some number of combinations of dimensions to get s separate lower order tensors Q 1 , . . , Q s (Proposition 4).
Mix Q 1 , . . , Q s with a matrix W ∈ R s (cid:48) × s and apply a nonlinearity Υ to get the ﬁnal activation of the neuron, which consists of the s (cid:48) output tensors
[Equation]
where the b i scalars are bias terms, and 1 is the |P t | × . . .
× |P t | dimensional all ones tensor.
A few remarks are in order about this general scheme:
Since (cid:101) F c 1 , . . . , (cid:101) F c s are stacked into a larger tensor and then possibly also multiplied by A ↓ P t , the general tendency would be for the tensor order to increase at every node, and the corresponding storage requirements to increase exponentially.
The purpose of the contractions in Stepis to counteract this tendency, and pull the order of the tensors back to some small number, typically 1 , 2 or 3 . 2.
However, since contractions can be done in many different ways, the number of channels will increase.
When the number of input channels is small, this is reasonable, since otherwise the number of learnable weights in the algorithm would be too small.
However, if unchecked, this can also become problematic.
Fortunately, mixing the channels by W on Step 6 gives an opportunity to stabilize the number of channels at some value s (cid:48) .
In the pseudocode above, for simplicity, the number of input channels is one and the number of output channels is s (cid:48) .
More realistically, the inputs would also have multiple channels (say, s 0 ) which would be propagated through the algorithm independently up to the mixing stage, making W an s (cid:48) × s × s 0 dimension tensor (not in the P –tensor sense!).
The conventional part of the entire algorithm is Step 6, and the only learnable parameters are the entries of the W matrix (tensor) and the b i bias terms.
These parameters are shared by all nodes in the network and learned in the usual way, by stochastic gradient descent.
Our scheme could be elaborated further while maintaining permutation covariance by, for example taking the tensor product of T with itself, or by introducing A ↓ P t in a different way.
However, the way that (cid:101) F c 1 , . . , (cid:101) F c s and A ↓ P t are combined by tensor products is already much more general and expressive than conventional message passing networks.
Our framework admits many design choices, including the choice of the order odf the activations, the choice of contractions, and c (cid:48) .
However, the overall structure of Steps 1–5 is fully dictated by the covariance constraint on the network.
The ﬁnal output of the network φ ( G ) =
F r must be permutation invariant.
That means that the root node n r must produce a tuple of zeroth order tensors (scalars) ( F (1) r , . . . , F ( c ) r ) .
This is similar to how many other graph representation algorithms compute φ ( G ) by summing the activations at level L or creating histogram features.
We consider a few special cases to explain how tensor aggregation relates to more conventional message passing rules.
5.1. Z EROTH ORDER TENSOR AGGREGATION
Constraining both the input tensors F c 1 , . . , F c s and the outputs to be zeroth order tensors, i.e., scalars, and foregoing multiplication by A ↓ P t greatly simpliﬁes the form of Φ .
In this case there is no need for promotions, and T is just the vector ( F (cid:96)c 1 , . . . , F (cid:96)c s ) .
There is only one way to contract a vector into a scalar, and that is to sum its elements.
Therefore, in this case, the entire aggregation algorithm reduces to the simple formula
[Equation]
For a neural network this is too simplistic.
However, it’s interesting to note that the Weisfeiler–
Lehmann isomorphism test essentially builds on just this formula, with a speciﬁc choice of Υ (Read & Corneil, 1977).
If we allow more channels in the inputs and the outputs, W becomes a matrix, and we recover the simplest form of neural message passing algorithms (Duvenaud et al., 2015).
5.1. F IRST ORDER TENSOR AGGREGATION
In ﬁrst order tensor aggregation, assuming that |P i | = m , (cid:101) F c 1 , . . . , (cid:101) F c s are m dimensional column vectors, and T is an m × m matrix consisting of (cid:101) F c 1 , . . . , (cid:101) F c s stacked columnwise.
There are two ways of contracting (in our generalized sense) a matrix into a vector: by summing over its rows, or summing over its columns.
The second of these choices leads us back to summing over all contributions from the children, while the ﬁrst is more interesting because it corresponds to summing (cid:101) F c 1 , . . .
, (cid:101) F c s as vectors individually.
In summary, we get an aggregation function that transforms a single input channel to two output channels of the form
[Equation]
where 1 denotes the m dimensional all ones vector.
Thus, in this layer W ∈ R 2 × 2 .
Unless constrained by c (cid:48) , in each subsequent layer the number of channels doubles further and these channels can all mix with each other, so W (2) ∈ R 4 × 4 , W ( 3) ∈ R 8 × 8 , and so on.
5.1. S ECOND ORDER TENSOR AGGREGATION WITHOUT THE ADJACENCY MATRIX
In second order tensor aggregation, T is a third order P –tensor, which can be contracted back to second order in three different ways, by projecting it along each of its dimensions.
Therefore the outputs will be the three matrices
[Equation]
and the weight matrix is W ∈ R 3 × 3 .
5.1. S ECOND ORDER TENSOR AGGREGATION WITH THE ADJACENCY MATRIX
The ﬁrst nontrivial tensor contraction case occurs when (cid:101) F c 1 , . . . , (cid:101) F c s are second order tensors, and we multiply with A ↓ P t , since in that case T is 5th order, and can be contracted down to second order in a total of 50 different ways:
The “1+1+1” case contracts T in the form T i 1 ,i 2 ,i 3 ,i 4 ,i 5 δ i {+a 
+}
δ i a {+
+}
δ i a 3 , i.e., it projects T down along 3 of its 5 dimensions.
This alone can be done in (cid:0) 53 (cid:1) =
10 different ways2.
The “1+2” case contracts T in the form T i 1 ,i 2 ,i 3 ,i 4 ,i 5 δ i {+a 
+}
δ i a 2 ,i a 3 , i e., it projects T along one dimension, and contracts it along two others.
This can be done in 3 (cid:0) 53 (cid:1) = 30 ways.
The “3” case is a single 3 -fold contraction T i 1 ,i 2 ,i 3 ,i 4 {+,i 
 δ+} i a 1 ,i a 2 ,i a 3 , which again can be done in (cid:0) 53 (cid:1)
= 10 different ways. The tensor T i 1 ,i 2 ,i 3 ,i 4 ,i 5 will be symmetric with respect to two sets of indices, following the structure of the promotion tensors and the adjacency matrix. Including these symmetries, the number of contractions is 18 including: ﬁve “1+1+1”, ten “1+2”, and three “3”.
[Figure]
E XPERIMENTS
We compared the second order variant (CCN 2D) of our CCNs framework (Section 4.2) to several standard graph learning algorithms on three types of datasets that involve learning the properties of molecules from their structure:
The Harvard Clean Energy Project (Hachmann et al., 2011), consisting of 2.3 million organic compounds that are candidates for use in solar cells.
The regression target in this case is Power Conversion Efﬁciency (PCE).
Due to time constraints, instead of using the entire dataset, the experiments were ran on a random subset of 50,000 molecules.
QM9 , which is a dataset of all 133k organic molecules with up to nine heavy atoms (C,O,N and F) out of the GDB-17 universe of molecules.
Each molecule has 13 target properties to predict.
The dataset does contain spatial information relating to the atomic conﬁgurations but we only used the chemical graph and atom node labels.
For our experiments we normalized each target variable to have mean 0 and standard deviation 1.
We report both MAE and RMSE for all normalized learning targets.
Graph kernels datasets , speciﬁcally (a) MUTAG, which is a dataset of 188 mutagenic aromatic and heteroaromatic compounds (Debnat et al., 1991); (b) PTC, which consists of 344 chemical compounds that have been tested for positive or negative toxicity in lab rats (Toivonen et al., 2003); (c) NCI1 and NCI109, which have 4110 and 4127 compounds respectively, each screened for activity against small cell lung cancer and ovarian cancer lines (Wale et al., 2008).
In the case of HCEP, we compared CCN to lasso, ridge regression, random forests, gradient boosted trees, optimal assignment Wesifeiler–Lehman graph kernel (Kriege et al., 2016) (WL), neural graph ﬁngerprints (Duvenaud et al., 2015), and the “patchy-SAN” convolutional type algorithm from (Niepert et al., 2016) (referred to as PSCN).
For the ﬁrst four of these baseline methods, we created simple feature vectors from each molecule: the number of bonds of each type (i.e. number of
H–H bonds, number of C–O bonds, etc) and the number of atoms of each type.
Molecular graph ﬁngerprints uses atom labels of each vertex as base features.
For ridge regression and lasso, we cross validated over λ .
For random forests and gradient boosted trees, we used 400 trees, and cross validated over max depth, minimum samples for a leaf, minimum samples to split a node, and learning rate (for GBT).
For neural graph ﬁngerprints, we used 2 layers and a hidden layer size of 10.
In PSCN, we used a patch size of 10 with two convolutional layers and a dense layer on top as described in their paper.
For the graph kernels datasets, we compare against graph kernel results as reported in (Kondor &
Pan, 2016) (which computed kernel matrices using the Weisfeiler–Lehman, Weisfeiler–edge, shortest paths, graphlets and multiscale Laplacian graph kernels and used a C-SVM on top), Neural graph ﬁngerprints (with 2 levels and a hidden size of 10) and PSCN.
For QM9, we compared against the
Weisfeiler–Lehman graph kernel (with C-SVM on top), neural graph ﬁngerprints, and PSCN.
The settings for NGF and PSCN are as described for HCEP.
For our own method, second order CCN, we initialized the base features of each vertex with computed histogram alignment features, inspired by (Kriege et al., 2016), of depth up to 10.
Each vertex receives a base label l i = concat 10 j =1 H j ( i ) where H j ( i ) ∈ R d (with d being the total number of distinct discrete node labels) is the vector of relative frequencies of each label for the set of vertices at distance equal to j from vertex i .
We use exactly 18 unique contractions deﬁned in 5.1.4 that result in additional channels.
We used up to three levels and the intermediate number of channels increases 18 time at each level.
To avoid exponentially growing channels, we applied learnable weight matrices to compress the channels into a ﬁxed number of channels.
In each experiment we used 80% of the dataset for training, 10% for validation, and evaluated on the remaining 10% test set.
For the kernel datasets we performed the experiments on 10 separate training/validation/test stratiﬁed splits and averaged the resulting classiﬁcation accuracies.
We used Adam optimization method (Kingma & Ba, 2015).
Our initial learning rate was set to 0.001 after experimenting on a held out set.
The learning rate decayed linearly after each step towards a minimum of 10 − 6 .
6. G RAPH F LOW D EEP L EARNING F RAMEWORK
We developed our custom Deep Learning framework in C++/CUDA named GraphFlow that supports symbolic/automatic differentiation, dynamic computation graphs, specialized tensor operations, and computational acceleration with GPU.
Our method, Covariant Compositional Networks, and other graph neural networks such as Neural Graph Fingerprints (Duvenaud et al., 2015), PSCN (Niepert et al., 2016) and Gated Graph Neural Networks (Li et al., 2016) are implemented based on the GraphFlow framework.
Our source code can be found at https://github.com/HyTruongSon/
GraphFlow .
One challenge of the implementation of Covariant Compositional Networks is that the high-order tensors (for example, in ﬁgure 6, we have a 5th order tensor after the tensor product operation) cannot be stored explicitly in the memory. Our solution is to propose a virtual indexing system in such a way that we never compute the whole sparse high-order tensor at once, but only compute its elements when given the indices. Basically, we always work with a virtual tensor , and that allows us to implement our tensor reduction/contraction operations efﬁciently with GPU. 6. D ISCUSSION
On the subsampled HCEP dataset, CCN outperforms all other methods by a very large margin.
For the graph kernels datasets, SVM with the Weisfeiler–Lehman kernels achieve the highest accuracy on NCI1 and NCI109, while CCN wins on MUTAG and PTC.
Perhaps this poor performance is to be expected, since the datasets are small and neural network approaches usually require tens of thousands of training examples at minimum to be effective.
Indeed, neural graph ﬁngerprints and
PSCN also perform poorly compared to the Weisfeiler–Lehman kernels.
In the QM9 experiments, CCN beats the three other algorithms in both mean absolute error and root mean squared error.
It should be noted that (Gilmer et al., 2017) obtained stronger results on
QM9, but we cannot properly compare our results with theirs because our experiments only use
[Table] [Table]
Niepert et al.)
[Table]
the adjacency matrices and atom labels of each node, while theirs includes comprehensive chemical features that better inform the target quantum properties.
C ONCLUSIONS
We have presented a general framework called covariant compositional networks (CCNs) for constructing covariant graph neural networks, which encompasses other message passing approaches as special cases
[Table]
but takes a more general and principled approach to ensuring covariance with respect to permutations.
Experimental results on several benchmark datasets show that CCNs can outperform other state-of-the-art algorithms
[Table]
[Table]
[Table]
A M ATHEMATICAL B ACKGROUND
Groups.
A group is a set G endowed with an operation G × G → G (usually denoted multiplicatively) obeying the following axioms:
G1.
for any u, v ∈ G , uv ∈ G (closure); G2.
for any u, v, w ∈ G , u ( vw ) =
( uv ) w (associativity); G3.
there is a unique e ∈ G , called the identity of G , such that eu = ue = u for any u ∈ G ; G4.
for any u ∈ G , there is a corresponding element u − 1 ∈ G called the inverse of u , such that uu − 1 = u − 1 u = e .
We do not require that the group operation be commutative, i.e., in general, uv (cid:54) = vu .
Groups can be ﬁnite or inﬁnite, countable or uncountable, compact or non-compact.
While most of the results in this paper would generalize to any compact group, the keep the exposition as simple as possible, throughout we assume that G is ﬁnite or countably inﬁnite.
As usual, | G | will denote the size (cardinality) of G , sometimes also called the order of the group.
Representations.
A (ﬁnite dimensional) representation of a group G over a ﬁeld F is a matrixvalued function R : G → F d ρ × d ρ such that R ( x ) R ( y )
=
R ( xy ) for any x, y ∈ G .
We generally assume that F = C , however in the special case when G is the symmetric group S n we can restrict ourselves to only considering real-valued representations, i.e., F = R .
B P ROOFS
Proof of Proposition 1.
Let G and G (cid:48) be two compound objects, where G (cid:48) is equivalent to G up to a permutation σ ∈ S n of the atoms.
For any node n a of G we let n (cid:48) a be the corresponding node of G (cid:48) , and let f a and f (cid:48) a be their activations.
We prove that f a = f (cid:48) a for every node in G by using induction on the distance of n a from its farthest descendant that is a leaf, which we call its height and denote h ( a ) .
For h ( a ) = 0 , the statment is clearly true, since f a = f (cid:48) a = (cid:96) ξ ( a ) .
Now assume that it is true for all nodes with height up to h ∗ .
For any node n a with h ( a ) =
h ∗ +1 , f a = Φ( f c 1 , f c 2 , . . . , f c k ) , where each of the children c 1 , . . . , c k are of height at most h ∗ , therefore
[Equation]
Thus, f a = f (cid:48) a for every node in G .
The proposition follows by φ ( G )
=
f r = f (cid:48) r = φ ( G (cid:48) ) .
[Equation]
Proof of Proposition 3.
Let G , G (cid:48) , N and N (cid:48) be as in Deﬁnition 5.
As in Deﬁnition 6, for each node (neuron) n i in N there is a node n (cid:48) j in N (cid:48) such that their receptive ﬁelds are equivalent up to permutation.
That is, if |P i | = m , then |P (cid:48)
j | = m , and there is a permutation π ∈ S m , such that if
P i =
( e p 1 , . . . , e p m ) and P (cid:48) j
= ( e q 1 , . . , e q m ) , then e q π ( a ) =
e p a .
By covariance, then f (cid:48) j = R π ( f i ) .
Now let G (cid:48)(cid:48) be a third equivalent object, and N (cid:48)(cid:48) the corresponding comp-net.
N (cid:48)(cid:48) must also have a node, n (cid:48)(cid:48) k , that corresponds to n i and n (cid:48) j .
In particular, letting its receptive ﬁeld be
P (cid:48)(cid:48) k = ( e r 1 , . . . , e r m ) , there is a permutation σ ∈ S m for which e r σ ( b ) =
e q b .
Therefore, f (cid:48)(cid:48) k
= R σ ( f (cid:48) j ) .
At the same time, n (cid:48)(cid:48) k is also in correspondence with n i .
In particular, letting τ = σπ (which corresponds to ﬁrst applying the permutation π , then applying σ ), e r τ ( a ) =
e p a , and therefore f (cid:48)(cid:48) k = R τ ( f i ) .
Hence, the { R π } maps must satisfy
[Equation]
for any f i .
More succinctly, R σπ =
R σ ◦ R π for any π, σ ∈ S m .
In the case that the { R π } maps are linear and represented by matrices, this reduces to R σπ =
R σ R π , which is equivalent to saying that they form a group representation of S m .
[Equation]
Proof of Proposition 4.
Under the action of a permutation π ∈ S m , A and B transform as
[Equation]
Case 1.
Let C
=
A ⊗ B .
Under (3) and (4), C transforms into
[Equation]
therefore, C is a k + p ’th order P –tensor.
Case 2.
Let C =
A (cid:12) ( a 1 ,...,a p ) B .
Under (3) and (4), C transforms as
[Equation]
Note that each of the [ P π ] i aj i (cid:48) aj factors in this expression repeats one of the earlier appearing [ P π ] i 1 i (cid:48) 1 , . . .
, [ P π ] i k i (cid:48) k factors, but since P π only has zero and one entries [ P π ] 2 a,b
=
[ P π ] a,b , so these factors can be dropped.
Thus, C is a k ’th order P –tensor.
Case 3.
Let C =
A ↓ a 1 ,...,a p and b 1 , . . . , b k − p be the indices (in increasing order) that are not amongst { a 1 , . . . , a p } .
Under (3), C becomes
[Equation]
Thus, C is a k − p ’th order P –tensor.
Case 4.
Follows directly from 3.
Case 5.
Finally, if A 1 , ..., A u are k ’th order P –tensors and C = (cid:80)
j α j A j then
[Equation]
so C is a k ’th order P –tensor.
[Equation]
Proof of Proposition 5.
Under the action of a permutation π ∈ S m (cid:48) on P b , χ (dropping the a → b superscipt) transforms to χ (cid:48) , where χ (cid:48)
i,j = χ π − 1 ( i ) ,j .
However, this can also be written as
[Equation]
Therefore, (cid:101) F i 1 ,...,i k transforms to
[Equation]
so (cid:101) F is a P –tensor.
[Equation]
Proof of Proposition 6.
By Proposition 5, under the action of any permutation π , each of the (cid:101) F p
j slices of F transforms as
[Equation]
At the same time, π also permutes the slices amongst each other according to
[Equation]
Therefore
[Equation]
so F is a k +1 ’th order P –tensor.
[Equation]
Proof of Proposition 7.
Under any permutation π ∈ S m of P i , A ↓ P (cid:48) i transforms to A ↓ P (cid:48) i , where [ A ↓ P (cid:48)
i ] π ( a ) ,π ( b ) =
[ A ↓ P i ] a,b .
Therefore, A ↓ P i is a second order P –tensor.
By the ﬁrst case of
Proposition 4, F ⊗ A ↓ P i is then a k +2 ’th order P –tensor.
[Equation]