O N THE E XPRESSIVE P OWER OF O VERLAPPING A RCHITECTURES OF D EEP L EARNING
Or { or.sharir,shashua } @cs.huji.ac.il
A BSTRACT
Expressive efﬁciency refers to the relation between two architectures A and B, whereby any function realized by B could be replicated by A, but there exists functions realized by A, which cannot be replicated by B unless its size grows signiﬁcantly larger.
For example, it is known that deep networks are exponentially efﬁcient with respect to shallow networks, in the sense that a shallow network must grow exponentially large in order to approximate the functions represented by a deep network of polynomial size.
In this work, we extend the study of expressive efﬁciency to the attribute of network connectivity and in particular to the effect of ”overlaps” in the convolutional process, i.e., when the stride of the convolution is smaller than its ﬁlter size (receptive ﬁeld).
To theoretically analyze this aspect of network’s design, we focus on a well-established surrogate for ConvNets called Convolutional Arithmetic Circuits (ConvACs), and then demonstrate empirically that our results hold for standard ConvNets as well.
Speciﬁcally, our analysis shows that having overlapping local receptive ﬁelds, and more broadly denser connectivity, results in an exponential increase in the expressive capacity of neural networks.
Moreover, while denser connectivity can increase the expressive capacity, we show that the most common types of modern architectures already exhibit exponential increase in expressivity, without relying on fully-connected layers.
I NTRODUCTION
One of the most fundamental attributes of deep networks, and the reason for driving its empirical success, is the “Depth Efﬁciency” result which states that deeper models are exponentially more expressive than shallower models of similar size.
Formal studies of Depth Efﬁciency include the early work on boolean or thresholded circuits (Sipser, 1983;
Yao, 1989; H ˚ astad and Goldmann, 1991; Hajnal et al., 1993), and the more recent studies covering the types of networks used in practice (Pascanu et al., 2013; Mont ´ ufar et al., 2014; Eldan and Shamir, 2016; Cohen et al., 2016a;
Cohen and Shashua, 2016; Telgarsky, 2016; Safran and Shamir, 2016; Raghu et al., 2016; Poole et al., 2016).
What makes the Depth Efﬁciency attribute so desirable, is that it brings exponential increase in expressive power through merely a polynomial change in the model, i.e. the addition of more layers.
Nevertheless, depth is merely one among many architectural attributes that deﬁne modern networks.
The deep networks used in practice consist of architectural features deﬁned by various schemes of connectivity, convolution ﬁlter deﬁned by size and stride, pooling geometry and activation functions.
Whether or not those relate to expressive efﬁciency, as depth has proven to be, remains an open question.
In order to study the effect of network design on expressive efﬁciency we should ﬁrst deﬁne ”efﬁciency” in broader terms.
Given two network architectures A and B , we say that architecture A is expressively efﬁcient with respect to architecture B , if the following two conditions hold: (i) any function h realized by B of size r B can be realized (or approximated) by A with size r A ∈ O ( r B ) ; (ii) there exist a function h realized by A with size r A , that cannot be realized (or approximated) by B , unless r B ∈ Ω( f ( r A )) for some super-linear function f .
The exact deﬁnition of the sizes r A and r B depends on the measurement we care about, e.g. the number of parameters, or the number of “neurons”.
The nature of the function f in condition (ii) determines the type of efﬁciency taking place – if f is exponential then architecture A is said to be exponentially efﬁcient with respect to architecture B , and if f is polynomial so is the expressive efﬁciency.
Additionally, we say A is completely efﬁcient with respect to B , if condition (ii) holds not just for some speciﬁc functions (realizable by A ), but for all functions other than a negligible set.
In this paper we study the efﬁciency associated with the architectural attribute of convolutions, namely the size of convolutional ﬁlters (receptive ﬁelds) and more importantly its proportion to their stride.
We say that a network architecture is of the non-overlapping type when the size of the local receptive ﬁeld in each layer is equal to the stride.
In that case, the sets of pixels participating in the computation of each two neurons in the same layer are completely separated.
When the stride is smaller than the receptive ﬁeld we say that the network architecture is of the overlapping type.
In the latter case, the overlapping degree is determined by the total receptive ﬁeld and stride projected back to the input layer – the implication being that for the overlapping architecture the total receptive ﬁeld and stride can grow much faster than with the non-overlapping case.
As several studies have shown, non-overlapping convolutional networks do have some theoretical merits.
Namely, non-overlapping networks are universal (Cohen et al., 2016a; Cohen and Shashua, 2016), i.e. they can approximate any function given sufﬁcient resources, and in terms of optimization, under some conditions they actually possess better convergence guaranties than overlapping networks.
Despite the above, there are only few instances of strictly non-overlapping networks used in practice (e.g. Sharir et al.
van den Oord et al.
(2016)), which raises the question of why are non-overlapping architectures so uncommon?
Additionally, when examining the kinds of architectures typically used in recent years, which employ a mixture of both overlapping and nonoverlapping layers, there is a trend of using ever smaller receptive ﬁelds, as well as non-overlapping layers having an ever increasing role (Lin et al., 2014;
Springenberg et al., 2015; Szegedy et al., 2015).
Hence, the most common networks used practice, though not strictly non-overlapping, are increasingly approaching the non-overlapping regime, which raises the question of why having just slightly overlapping architectures seems sufﬁcient for most tasks?
In the following sections, we will shed some light on these questions by analyzing the role of overlaps through a surrogate class of convolutional networks called Convolutional Arithmetic Circuits (ConvACs)
(Cohen et al., 2016a) – instead of non-linear activations and average/max pooling layers, they employ linear activations and product pooling.
ConvACs, as a theoretical framework to study ConvNets, have been the focused of several works, showing, amongst other things, that many of the results proven on this class are typically transferable to standard ConvNets as well (Cohen and Shashua, 2016; 2017).
Though prior works on ConvACs have only considered non-overlapping architectures, we suggest a natural extension to the overlapping case that we call Overlapping ConvACs.
In our analysis, which builds on the known relation between ConvACs and tensor decompositions, we prove that overlapping architectures are in fact completely and exponentially more efﬁcient than non-overlapping ones, and that their expressive capacity is directly related to their overlapping degree .
Moreover, we prove that having even a limited amount of overlapping is sufﬁcient for attaining this exponential separation.
To further ground our theoretical results, we demonstrate our ﬁndings through experiments with standard ConvNets on the CIFAR10 image classiﬁcation dataset.
O VERLAPPING C ONVOLUTIONAL A RITHMETIC C IRCUITS
In this section, we introduce a class of convolutional networks referred to as Overlapping Convolutional Arithmetic Circuits, or Overlapping ConvACs for short.
This class shares the same architectural features as standard ConvNets, including some that have previously been overlooked by similar attempts to model ConvNets through ConvACs, namely, having any number of layers and unrestricted receptive ﬁelds and strides, which are crucial for studying overlapping architectures.
For simplicity, we will describe this model only for the case of inputs with two spatial dimensions, e.g. color images, and limiting the convolutional ﬁlters to the shape of a square.
previous layer
GC layer
R
R
H (in) x ( R ) x (1) x ( R 2 ) ··· ... ... ...
H (in)
D (in)
H (out)
H ( out ) =
H ( in ) S D (out) y
= g ( W (1) x (1) + b (1) ,...,W ( R 2 ) x ( R 2 ) +
b ( R 2 ) )
We begin by presenting a broad deﬁnition of a Generalized Convolutional (GC) layer as a fusion of a 1 × 1 linear operation with a pooling function – this view of convolutional layers is motivated by the all-convolutional architecture (Springenberg et al., 2015), which replaces all pooling layers with convolutions with stride
[Figure]
[Figure]
[Figure] h y ( x 1 ,..., x N ) greater than 1.
The input to a GC layer is a 3order tensor (multi-dimensional array), having width and height equal to H ( in ) ∈ N and depth
D ( in ) ∈ N , also referred to as channels, e.g. the input could be a 2D image with RGB color channels.
Similarly, the output of the layer has width and height equal to H ( out ) ∈ N and D ( out )
∈
N channels, where H ( out ) =
H ( in ) S for S ∈ N that is referred to as the stride , and has the role of a sub-sampling operation.
Each spatial location ( i, j ) at the output of the layer corresponds to a 2D window slice of the input tensor of size R × R × D ( in ) , extended through all the input channels, whose top-left corner is located exactly at ( i · S, j · S ) , where
R ∈ N is referred to as its local receptive ﬁeld , or ﬁlter size.
For simplicity, the parts of window slices extending beyond the boundaries have zero value.
Let y ∈ R D
( ( out ) be a vector representing the channels at some location of the output, and similarly, let x (1) , . . .
, x ( R 2 ) ∈
R D ( in ) be the set of vectors representing the slice, where each vector represents the channels at its respective location inside the R × R window, then the operation of a GC layer is deﬁned as follows:
[Equation]
, where W (1) , . .
, W ( R 2 ) ∈ R D ( out )
× D ( in ) and b (1) , . . .
, b ( R 2 ) ∈
R D ( out ) are referred to as the weights and biases of the layer, respectively, and g : R D ( out ) × · · · × R D ( out ) → R D ( out ) is some point-wise pooling function.
See ﬁg. 1 for an illustration of the operation a GC layer performs.
With the above deﬁnitions, a GC network is simply a sequence of L GC layers, where for l ∈ [ L ] ≡ { 1 , . . .
, L } , the l ’th layer is speciﬁed by a local receptive ﬁeld R ( l ) , a stride S ( i ) , D ( l ) output channels, parameters θ ( l ) , and a pooling function g ( l ) .
For classiﬁcation tasks, the output of the last layer of the network typically has 1 × 1 spatial dimensions, i.e. a vector, where each output channel
y ∈
[ Y ] ≡ [ D ( L ) ] represents the score function of the y ’th class, denoted by h y , and inference is perform by y ∗ =
arg max y h
y
( X ) .
Oftentimes, it is common to consider the output of the very ﬁrst layer of a network as a low-level feature representation of the input, which is motivated by the observation that these learned features are typically shared across different tasks and datasets over the same domain (e.g. edge and Gabor ﬁlters for natural images).
Hence, we treat this layer as a separate ﬁxed “zeroth” convolutional layer referred to as the representation layer, where the operation of the layer can be depicted as applying a set of ﬁxed functions { f d : R s → R } Md =1 to the window slices denoted by x 1 , . . .
, x N ∈ R s , i.e. the entries of the output tensor of this layer are given by { f d ( x i ) } d ∈ [ M ] ,i ∈
[ N ] .
With these notations, the output of a GC network can be viewed as a function h y ( x 1 , . . .
, x N ) .
The entire GC network is illustrated in ﬁg. 2.
Given a non-linear point-wise activation function σ ( · ) (e.g. ReLU), then setting all pooling functions to average pooling followed by the activation, i.e. g ( x (1) , . . .
, x ( R 2 ) )
c
= σ (cid:16)(cid:80)
R 2 i =1 x ( i ) c (cid:17) for c ∈
[ D ( out ) ] , give rise to the common all-convolutional network with σ ( · ) activations, which served as the initial motivation for our formulation.
Alternatively, choosing instead a product pooling function, i.e. g ( x (1) , . .
, x ( R 2 ) )
c = (cid:81)
R 2 i =1 x ( i ) c for c ∈
[ D ( out ) ] , results in an Arithmetic Circuit, i.e. a circuit containing just product and sum operations, hence it is referred to as a Convolutional
Arithmetic Circuit, or ConvAC.
It is important to emphasize that ConvACs, as originally introduced by Cohen et al.
(2016a), are typically described in a very different manner, through the language of tensor decompositions (see app.
A for background).
Since vanilla ConvACs can be seen as an alternating sequence of 1 × 1 convolutions and non-overlapping product pooling layers, then the two formulations coincide when all GC layers are non-overlapping, i.e. for all l ∈ [ L ] , R ( l ) =
S ( l ) .
If, however, some of the layers are overlapping, i.e. there exists l ∈
[ L ] such that R ( l ) > S ( l ) , then our formulation through GC layers diverges, and give rise to what we call Overlapping ConvACs .
Given that our model is an extension of the ConvACs framework, it inherits many of its desirable attributes.
First, it shares most of the same traits as modern ConvNets, i.e. locality, sharing and pooling.
Second, it can be shown to form a universal hypotheses space (Cohen et al., 2016a).
Third, its underlying operations lend themselves to mathematical analysis based on measure theory and tensor analysis (Cohen et al., 2016a).
Forth, through the concept of generalized tensor decompositions (Cohen and Shashua, 2016), many of the theoretical results proven on ConvACs could be transferred to standard ConvNets with ReLU activations.
Finally, from an empirical perspective, they tend to work well in many practical settings, e.g. for optimal classiﬁcation with missing data (Sharir et al., 2016), and for compressed networks (Cohen et al., 2016b).
While we have just established that the non-overlapping GC Network with a product pooling function is equivalent to vanilla ConvACs, one might wonder if using overlapping layers instead could diminish what these overlapping networks can represent.
We show that not only is it not the case, but prove the more general claim that a network of a given architecture can realize exactly the same functions as networks using smaller local receptive ﬁelds, which includes the non-overlapping case.
Proposition 1.
Let A and B be two GC Networks with a product pooling function.
If the architecture of B can be derived from A through the removal of layers with 1 × 1 stride, or by decreasing the local receptive ﬁeld of some of its layers, then for any choice of parameters for B , there exists a matching set of parameters for A , such that the function realized by B is exactly equivalent to A .
Speciﬁcally,
A can realize any non-overlapping network with the same order of strides (excluding 1 × 1 strides).
Proof sketch.
This follows from two simple claims: (i) a GC layer can produce an output equivalent to that of a GC layer with a smaller local receptive ﬁeld, by “zeroing” its weights beyond the smaller local receptive ﬁeld; and (ii) GC layers with 1 × 1 receptive ﬁelds can be set such that their output is equal to their input, i.e. realize the identity function.
With these claims, the local receptive ﬁelds of
A can be effectively shrank to match the local receptive ﬁelds of B , and any additional layers of A with stride 1 × 1 could be set such that they are realizing the identity mapping, effectively “removing” them from A .
See app.
C.2 for a complete proof.
Proposition 1 essentially means that overlapping architectures are just as expressive as nonoverlapping ones of similar structure, i.e. same order of non-unit strides.
As we recall, this satisﬁes the ﬁrst condition of the efﬁciency property introduced in sec.
1, and does so regardless if we measure the size of a network as the number of parameters, or the number of “neurons” 1 .
In the following section we will cover the preliminaries required to show that overlapping networks actually lead to an increase in expressive capacity, which under some settings results in an exponential gain, proving that the second condition of expressive efﬁciency holds as well.
A NALYZING E XPRESSIVE E FFICIENCY T HROUGH G RID T ENSORS
In this section we describe our methods for analyzing the expressive efﬁciency of overlapping ConvACs that lay the foundation for stating our theorems.
A minimal background on tensor analysis required to follow our work can be found in sec.
3.1, followed by presenting our methods in sec.
3. P RELIMINARIES
In this sub-section we cover the minimal background on tensors analysis required to understand our analysis.
A tensor A ∈ R M 1 ⊗···⊗ M N of order N and dimension M i in each mode i ∈
[ N ] ≡ { 1 , . . .
, N } , is a multi-dimensional array with entries
A d 1 ,...,d N ∈ R for all i ∈
[ N ] and d i ∈
[ M i ] .
For simplicity, henceforth we assume that all dimensions are equal, i.e. M
[Equation]
One of the central concepts in tensor analysis is that of tensor matricization , i.e. rearranging its entries to the shape of a matrix.
Let P ·∪ Q =
[ N ] be a disjoint partition of its indices, such that
P = { p 1 , . .
, p | P | } with p 1 < . . .
< p | P | , and Q = { q 1 , . .
, q | Q | } with q 1 < . . .
< q | Q | .
The matricization of A with respect to the partition P · ∪ Q , denoted by (cid:74) A (cid:75) P,Q , is the M | P |
-by- M | Q | matrix holding the entries of
A , such that for all i ∈ [ N ] and d i ∈ [ M ] the entry
A d 1 ,...,d N is placed in row index 1 + (cid:80) | P | t =1 ( d p t − 1)
M | P |− t and column index 1 + (cid:80) | Q | t =1 ( d q t − 1) M | Q |− t .
Lastly, the tensors we study in this article originate by examining the values of some given function at a set of predeﬁned points and arranging them in a tensor referred to as the grid tensor of the function.
Formally, let f : R s × . . .
× R s
→ R be a function, and let { x (1) , . . .
, x ( M ) ∈ R s } be a set of vectors called template vectors , then the grid tensor of f is denoted by A ( f ) ∈ R M ⊗ ... ⊗ M and deﬁned by
A ( f ) d 1 ,...,d N = f
( x ( d 1 ) , . .
, x ( d N ) ) for all d 1 , . . .
, d N ∈ [ M ] .
3. B OUNDING THE S IZE OF N ETWORKS VIA G RID T ENSORS
We begin with a discussion on how to have a well-deﬁned measure of efﬁciency.
We wish to compare the efﬁciency of non-overlapping ConvACs to overlapping ConvACs, for a ﬁxed set of M representation functions (see sec.
2 for deﬁnitions).
While all functions realizable by non-overlapping
ConvACs with shared representation functions lay in the same function subspace (see Cohen et al.
(2016a)), this is not the case for overlapping ConvACs, which can realize additional functions outside the sub-space induced by non-overlapping ConvACs.
We cannot therefore compare both architectures directly, and need to compare them through an auxiliary objective.
Following the work of Cohen and Shashua (2016), we instead compare architectures through the concept of grid tensors, and speciﬁcally, the grid tensor deﬁned by the output of a ConvAC, i.e. the tensor A ( h ) for h ( x 1 , . . .
, x N ) .
Unlike with the ill-deﬁned nature of directly comparing the functions of realized by ConvACs, Cohen and Shashua (2016) proved that assuming the ﬁxed representation functions are linearly independent, then there exists template vectors x (1) , . . .
, x ( M ) , for which any nonoverlapping ConvAC architecture could represent all possible grid tensors over these templates, given sufﬁcient number of channels at each layer.
More speciﬁcally, if F ij = f i ( x ( j ) )
, then these template vector are chosen such that F is non-singular.
Thus, once we ﬁx a set of linearly independent representation functions, we can compare different ConvACs, whether overlapping or not, on the minimal size required for them to induce the same grid tensor, while knowing such a ﬁnite number always exists.
One straightforward direction for separating between the expressive efﬁciency of two network architectures A and B is by examining the ranks of their respective matricized grid tensors.
Speciﬁcally,
Let A ( h ( A ) ) and A ( h ( B ) ) denote the grid tensors of A and B, respectively, and let ( P, Q ) be a partition of [ N ] , then we wish to ﬁnd an upper-bound on the rank of (cid:74) A ( h ( A ) ) (cid:75) P,Q as a function of its size on one hand, while showing on the other hand that rank (cid:0) (cid:74)
A ( h ( B ) ) (cid:75) P,Q (cid:1) can be signiﬁcantly greater.
One beneﬁt of studying efﬁciency through a matrix rank is that not only we attain separation bounds for exact realization, but also immediately gain access to approximation bounds by examining the singular values of the matricized grid tensors.
This brings us to the following lemma, which connects upper-bounds that were previously found for non-overlapping ConvACs (Cohen and
Shashua, 2017), with the grid tensors induced by them (see app.
C.1 for proof):
Lemma 1.
Let h y ( x 1 , . . .
, x N ) be a score function of a non-overlapping ConvAC with a ﬁxed set of
M linearly independent and continuous representation functions, and L GC layers.
Let ( P, Q ) be a partition dividing the spatial dimensions of the output of the representation layer into two equal parts, either along the horizontal or vertical axis, referred to as the “left-right” and “top-bottom” partitions, respectively.
Then, for any template vectors such that F is non-singular and for any choice of the parameters of the network, it holds that rank ( (cid:74)
A ( h y ) (cid:75) P,Q ) ≤ D ( L − 1) .
Lemma 1 essentially means that it is sufﬁcient to show that overlapping ConvACs can attain ranks super-polynomial in their size to prove they are exponentially efﬁcient with respect to nonoverlapping ConvACs.
In the next section we analyze how the overlapping degree is related to the rank, and under what cases it leads to an exponentially large rank.
T HE E XPRESSIVE E FFICIENCY OF O VERLAPPING A RCHITECTURES
In this section we analyze the expressive efﬁciency of overlapping architectures.
We begin by deﬁning our measures of the overlapping degree that will used in our claims, followed by presenting our main results in sec.
For the sake of brevity, an additional set of results, in light of the recent work by Cohen and Shashua (2017) on “Pooling Geometry”, is deferred to app.
B.
4. T HE O VERLAPPING D EGREE OF A N ETWORK
[Figure]
To analyze the efﬁciency of overlapping architectures, we will ﬁrst formulate more rigorously the measurement of the overlapping degree of a given architecture.
As mentioned in sec.
1, we do so by deﬁning the concepts of the total receptive ﬁeld and total stride of a given layer l ∈
[ L ] , denoted by T ( l ) R and T ( l ) S , respectively.
Both measurements could simply be thought of as projecting the accumulated local receptive ﬁelds (or strides) to the the ﬁrst layer, as illustrated in ﬁg. 3, which represent a type of global statistics of the architecture.
However, note that proposition 1 entails that a given architecture could have a smaller effective total receptive ﬁeld, for some settings of its parameters.
This leads us to deﬁne the α -minimal total receptive ﬁeld, for any α ∈ R + , as the smallest effective total receptive ﬁeld still larger than α , which we denote by T ( l,α ) R .
The exact deﬁnitions of the above concepts are formulated as follows:
[Equation]
where we omitted the arguments of T ( l − 1) S and T ( k − 1) S for the sake of visual compactness.
Notice that for non-overlapping networks the total receptive ﬁeld always equals the total stride, and that only at the end of the network, after the spatial dimension collapses to 1 × 1 , does the the total receptive ﬁeld grow to encompass the entire size of the representation layer.
For overlapping networks this is not the case, and the total receptive ﬁeld could grow much faster.
Intuitively, this means that values in regions of the input layer that are far apart would be combined by non-overlapping networks only near the last layers of such networks, and thus non-overlapping networks are effectively shallow in comparison to overlapping networks.
Base on this intuition, in the next section we analyze networks with respect to the point at which their total receptive ﬁeld is large enough.
4. M AIN R ESULTS
With all the preliminaries in place, we are ready to present our main result:
Theorem 1.
Assume a ConvAC with a ﬁxed representation layer having M output channels and both width and height equal to H , followed by L GC layers, where the l ’th layer has a local receptive ﬁeld R ( l ) , a stride S ( l ) , and D ( l ) output channels.
Let K ∈
[ L ] be a layer with a total receptive ﬁeld T ( K ) R ≡ T ( K ) R ( R (1) , S (1) , . . .
, R ( K ) , S ( K ) ) , such that T ( K ) R > H 2 .
Then, for any choice of parameters, except a null set (with respect to the Lebesgue measure), and for any template vectors such that F is non-singular, the following equality holds:
[Equation]
where ( P, Q ) is either the “left-right” or the “top-bottom” partitions and
D ≡ min { M, D ( K ) , 12 min 1 ≤ l ≤ K D ( l ) } .
Proof sketch.
Because the entries of the matricized grid tensors are polynomials in the parameters, then according to a lemma by Sharir et al.
(2016), if there is a single example that attains the above
[Figure]
lower-bound on the rank, then it occurs almost everywhere with respect to the Lebesgue measure on the Euclidean space of the parameters.
Given the last remark, the central part of our proof is simply the construction of such an example.
First we ﬁnd a set of parameters for the simpler case where the ﬁrst GC layer is greater than a quarter of the input, satisfying the conditions of the theorem.
The motivation behind the speciﬁc construction is the pairing of indices from each side of the partition, such that they are both in the same local receptive ﬁeld, and designing the ﬁlters such that the output of each local application of them deﬁnes a mostly diagonal matrix of rank D , with respect to these two indices.
The rest of the parameters are chosen such that the output of the entire network results in a product of the entries of these matrices.
Under matricization, this results in a matrix who is equivalent 2 to a Kronecker product of mostly diagonal matrices.
Thus, the matricization rank is equal to the product of the ranks of these matrices, which results in the exponential form of eq. 4.
Finally, we extend the above example to the general case, by realizing the operation of the ﬁrst layer of the above example through multiple layers with small local receptive ﬁelds.
See app.
C.1 for the deﬁnitions and lemmas we rely on, and see app.
C.3 for a complete proof.
Combined with Lemma 1, it results in the following corollary:
Corollary 1.
Under the same setting as theorem 1, and for all choices of parameters of an overlapping ConvAC, except a negligible set, any non-overlapping ConvAC that realizes (or approximates) the same grid tensor must be of size at least:
[Equation]
While the complexity of the generic lower-bound above might seem incomprehensible at ﬁrst, its generality gives us the tools to analyze practically any kind of feed-forward architecture.
As an example, we can analyze the lower bound for the well known GoogLeNet architecture (Szegedy et al., 2015), for which the lower bound equals 32 98 , making it clear that using a non-overlapping architecture for this case is infeasible.
Next, we will focus on speciﬁc cases for which we can derive more intelligible lower bounds.
According to theorem 1, the lower bound depends on the ﬁrst layer for which its total receptive ﬁeld is greater than a quarter of the input.
As mentioned in the previous section, for non-overlapping networks this only happens after the spatial dimension collapses to 1 × 1 , which entails that both the total receptive ﬁeld and total stride would be equal to the width H of the representation layer, and substituting this values in eq.
4 results simply in D – trivially meaning that to realize one nonoverlapping network by another non-overlapping network, the next to last layer must have at least half the channels of the target network.
On the other extreme, we can examine the case where the ﬁrst GC layer has a local receptive ﬁeld R greater than a quarter of its input, i.e. R > H / 2 .
Since the layers following the ﬁrst GC layer do not affect the lower bound in this case, it applies to any arbitrary sequence of layers as illustrated in ﬁg. 4.
For simplicity we will also assume that the stride S is less than H / 2 , and that H 2 is evenly divided
[Figure]
If B ≤ H / 5 +
and D ( l ) ≥ 2 M for all
≤ l < L , then the lower bound of theorem 1 for this network results in M (2 B −
[Figure]
by S .
In this case the H 2 -minimal receptive ﬁeld equals to H 2 + 1 , and thus the lower bound results in D H 2 2 S .
Consider the case of D = M and S = 1 , then a non-overlapping architecture that satisﬁes this lower bound is of the order of magnitude at which it could already represent any possible grid tensor.
This demonstrate our point from the introduction, that through a a polynomial change in the architecture, i.e. increasing the receptive ﬁeld, we get an exponential increase in expressivity.
Though the last example already demonstrates that a polynomially sized overlapping architecture could lead to an exponential separation, in practice, employing such large convolutions is very resource intensive.
The common best practice is to use multiple small local receptive ﬁelds of size
B × B , where the typical values are B = 3 or B = 5 , separated by a 2 × 2 “pooling” layers, i.e. layers with both stride and local receptive ﬁeld equal to 2 × 2 .
For simplicity, we assume that
H = 2 L for some L ∈ N .
See ﬁg. 5 for an illustration of such a network.
Analyzing the above network with theorem 1 results in the following proposition:
Proposition 2. Consider a network comprising a sequence of GC blocks, each block begins with a layer whose local receptive ﬁeld is B × B and its stride 1 × 1 , followed by a layer with local receptive ﬁeld 2 × 2 and stride 2 × 2 , where the output channels of all layers are at least 2 M , and the spatial dimension of the representation layer is H × H for H =2 L .
Then, the lower bound describe by eq.for the above network is greater than or equal to:
[Equation]
whose limits are lim B →∞ τ ( B, H )
=
M H 2 2 and lim
H →∞ τ ( B, H ) =
M (2 B − 1)2.
Finally, assuming B ≤ H 5 + 1 , then τ ( B, H ) ≥ M (2 B − 1)2.
Proof sketch.
We ﬁrst ﬁnd a closed-form expression for the total receptive ﬁeld and stride of each of the B × B layers in the given network.
We then show that for layers whose total receptive ﬁeld is greater than H 2 , its α -minimal total receptive ﬁeld, for α = H 2 , is equal to H 2 +1 .
We then use the above to ﬁnd the ﬁrst layer who satisﬁes the conditions of theorem 1, and then use our closed-forms expressions to simplify the general lower bound for this case.
See app.
C.4 for a complete proof.
In particular, for the typical values of M = 64 , B = 5 , and H ≥ 20 , the lower bound is at least 64 20 , which demonstrates that even having a small amount of overlapping already leads to an exponential separation from the non-overlapping case.
When B grows in size, this bound approaches the earlier result we have shown for large local receptive ﬁelds encompassing more than a quarter of the image.
When H grows in size, the lower bound is dominated strictly by the local receptive ﬁelds.
Also notice that based on proposition 2, we could also derive a respective lower bound for a network following VGG style architecture (Simonyan and Zisserman, 2014), where instead of a single convolutional layer before every “pooling” layer, we have K layers, each with a local receptive ﬁeld of C × C .
Under this case, it is trivial to show that the bound from propositionholds for B = K · ( C − 1) + 1 , and under the typical values of C = 3 and K =
it once again results in a lower bound of at least 64 20 .
[Figure]
E XPERIMENTS
In this section we show that the theoretical results of sec.
4.2 indeed hold in practice.
In other words, there exists tasks that require the highly expressive power of overlapping architectures, on which non-overlapping architectures would have to grow by an exponential factor to achieve the same level of performance.
We demonstrate this phenomenon on standard ConvNets with ReLU activations that follow the same architecture that was outlined in proposition 2, while varying the number of channels and the size of the receptive ﬁeld of the B × B “conv” layers.
The only change we made, was to replace the 2 × 2 -“pooling” layers of the convolutional type, with the standard 2 × 2max-pooling layers, and using the same number of channels across all layers.
This was done for the purpose of having all the learned parameters located only at the (possibly) overlapping layers.
More speciﬁcally, the network has 5 blocks, each starting with a B × B convolution with C channels, stride 1 × 1 , and ReLU activation, and then followed by 2 × 2 max-pooling layer.
After the ﬁfth “conv-pool”, there is a ﬁnal dense layer with 10 outputs and softmax activations.
We train each of these networks for classiﬁcation over the CIFAR-10 dataset, with two types of data augmentation schemes: (i) spatial augmentations, i.e. randomly translating (up to 3 pixels in each direction) and horizontally ﬂipping each image, and (ii) color augmentations following Dosovitskiy et al.
(2014), i.e. randomly adding a constant shift (at most ± 0 . 3 ) to the hue, saturation, and luminance, for each attribute separately, and in addition randomly sampling a multiplier (in the range [0 . 5 , 1 . 5] ) just to the saturation and luminance.
Though typically data augmentation is only used for the purpose of regularization, we employ it for the sole purpose of raising the hardness of the regular CIFAR-10 dataset, as even small networks can already overﬁt and effectively memorize its small dataset.
We separately test both the spatial and color augmentation schemes to emphasize that our empirical results cannot be explained simply by spatial-invariance type arguments.
Finally, the training itself is carried out for 300 epochs with ADAM (Kingma and Ba, 2015) using its standard hyper-parameters, at which point the loss of the considered networks have stopped decreasing.
We report the training accuracy over the augmented dataset in ﬁg. 6, where for each value of the receptive ﬁeld B , we plot its respective training accuracies for variable number of channels C .
The source code for reproducing the above experiments and plots can be found at https://github.com/HUJI-Deep/OverlapsAndExpressiveness .
It is quite apparent that the greater B is chosen, the less channels are required to achieve the same accuracy.
Moreover, for the non-overlapping case of B =1 , more than 2048 channels are required to reach the same performance of networks with B> 2 and just 64 channels under the spatial aug- mentations – which means effectively exponentially more channels were required.
Even more so, under the color augmentations, we were not able to train non-overlapping networks to reach even the smallest overlapping network ( B = 2 and C = 16 ).
In terms of total number of parameters, there is a clear separation between the overlapping and the non-overlapping types, and we once again see more than an order of magnitude increase in the number of parameters between an overlapping and non-overlapping architectures that achieve similar training accuracy.
As a somewhat surprising result, though based only on our limited experiments, it appears that for the same number of parameters, all overlapping networks attain about the same training accuracy, suggesting perhaps that having the smallest amount of overlapping already attain all the beneﬁts overlapping provides, and that increasing it further does not affect the performance in terms of expressivity.
As ﬁnal remark, we also wish to acknowledge the limitations of drawing conclusions strictly from empirical experiments, as there could be alternative explanations to these observations, e.g. the effects overlapping has on the optimization process.
Nevertheless, our theoretical results suggests this is less likely the case.
D ISCUSSION
The common belief amongst deep learning researchers has been that depth is one of the key factors in the success of deep networks – a belief formalized through the depth efﬁciency conjecture.
Nevertheless, depth is one of many attributes specifying the architecture of deep networks, and each could potentially be just as important.
In this paper, we studied the effect overlapping receptive ﬁelds have on the expressivity of the network, and found that having them, and more broadly denser connectivity, results in an exponential gain in the expressivity that is orthogonal to the depth.
Our analysis sheds light on many trends and practices in contemporary design of neural networks.
Previous studies have shown that non-overlapping architectures are already universal (Cohen et al., 2016a), and even have certain advantages in terms of optimization (Brutzkus and Globerson, 2017), and yet, real-world usage of non-overlapping networks is scarce.
Though there could be multiple factors involved, our results clearly suggest that the main culprit is that non-overlapping networks are signiﬁcantly handicapped in terms of expressivity compared to overlapping ones, explaining why the former are so rarely used.
Additionally, when examining the networks that are commonly used in practice, where the majority of the layers are of the convolutional type with very small receptive ﬁeld, and only few if any fully-connected layers (Simonyan and Zisserman, 2014; Springenberg et al., 2015; He et al., 2016), we ﬁnd that though they are obviously overlapping, their overlapping degree is rather low.
We showed that while denser connectivity can increase the expressive capacity, even in the most common types of modern architectures already exhibit exponential increase in expressivity, without relying on fully-connected layers.
This could partly explain that somewhat surprising observation, as it is probable that such networks are sufﬁciently expressive for most practical needs simply because they are already in the exponential regime of expressivity.
Indeed, our experiments seems to suggests the same, in which we saw that further increases in the overlapping degree beyond the most limited overlapping case seems to have insigniﬁcant effects on performance – a conjecture not quite proven by our current work, but one we wish to investigate in the future.
There are relatively few other works which have studied the role of receptive ﬁelds in neural networks.
Several empirical works (Li and Perona, 2005; Coates et al., 2011; Krizhevsky et al., 2012) have demonstrated similar behavior, showing that the classiﬁcation accuracy of networks can sharply decline as the degree of overlaps is decreased, while also showing that gains from using very large local receptive ﬁelds are insigniﬁcant compared to the increase in computational resources.
Other works studying the receptive ﬁelds of neural networks have mainly focused on how to learn them from the data (Coates and Ng, 2011; Jia et al., 2012).
While our analysis has no direct implications to those speciﬁc works, it does lay the ground work for potentially guiding architecture design, through quantifying the expressivity of any given architecture.
Lastly, Luo et al.
(2016) studied the effective total receptive ﬁeld of different layers, a property of a similar nature to our total receptive ﬁeld, where they measure the the degree to which each input pixel is affecting the output of each activation.
They show that under common random initialization of the weights, the effective total receptive ﬁeld has a gaussian shape and is much smaller than the maximal total receptive ﬁeld.
They additionally demonstrate that during training the effective total receptive ﬁeld grows in size, and suggests that weights should be initialized such that the initial effective receptive ﬁeld is large.
Their results strengthen our theory, by showing that trained networks tend to maximize their effective receptive ﬁeld, taking full potential of their expressive capacity.
To conclude, we have shown both theoretically and empirically that overlapping architectures have an expressive advantage compared to non-overlapping ones.
Our theoretical analysis is grounded on the framework of ConvACs, which we extend to overlapping conﬁgurations.
Though are proofs are limited to this speciﬁc case, previous studies (Cohen and Shashua, 2016) have already shown that such results could be transferred to standard ConvNets as well, using most of the same mathematical machinery.
While adapting our analysis accordingly is left for future work, our experiments on standard ConvNets (see sec. 5) already suggest that the core of our results should hold in this case as well.
Finally, an interesting outcome of moving from non-overlapping architectures to overlapping ones is that the depth of a network is no longer capped at log 2 ( input size ) , as has been the case in the models investigated by Cohen et al.
(2016a) – a property we will examine in future works
A CKNOWLEDGMENTS
This work is supported by Intel grant ICRI-CI #9-2012-6133, by ISF Center grant 1790/12 and by the European Research Council (TheoryDL project)
A B ACKGROUND ON C ONVOLUTIONAL A RITHMETIC C IRCUITS
We base our analysis on the convolutional arithmetic circuit (ConvAC) architecture introduced by Cohen et al.
(2016a), which is illustrated by ﬁg. 7, and can be simply thought of as a regular ConvNet, but with linear activations and product pooling layers, instead of the more common non-linear activations (e.g. ReLU) and average/max pooling.
More speciﬁcally, each point in the input space of the network, denoted by
X = ( x 1 , . . .
, x N ) , is represented as an N -length sequence of s -dimensional vectors x 1 , . . .
, x N ∈ R s .
X is typically thought of as an image, where each x i corresponds to a local patches from that image.
The ﬁrst layer of the network is referred to as the representation layer, consisting of applying M representation functions f θ 1 , . . .
, f θ M : R s → R on each local patch x i , giving rise to M feature maps.
Under the common setting, where the representation functions are selected to be f θ d ( x ) = σ ( w Td x + b d ) for some point-wise activation σ ( · ) and parameterized by θ d = ( w d , b d ) ∈ R s × R , the representation layer reduces to the standard convolutional layer.
Other possibilities, e.g. gaussian functions with diagonal covariances, have also been considered in Cohen et al.
(2016a).
Following the representation layer, are hidden layers indexed by l = 0 , . .
, L − 1 , each begins with a 1 × 1 conv operator, which is just an r l − 1 × 1 × 1 convolutional layer with r l − 1 input chan- nels and r l output channels, with the sole exception that parameters of each kernel could be spatially unshared (known as locally-connected layer (Taigman et al., 2014)).
Following each conv layer is a spatial pooling, that takes products of non-overlapping two-dimensional windows covering the output of the previous layer, where for l =
L − 1 the pooling window is the size of the entire spatial dimension (i.e. global pooling), reducing its output’s shape to a r L − 1 × 1 × 1 , i.e. an r L − 1 -dimensional vector.
The ﬁnal L layer maps this vector with a dense linear layer into the Y network outputs, denoted by h y ( x 1 , . . .
, x N ) , representing score functions classifying each X to one of the classes through: y ∗ =
argmax y
h
y
( x 1 , . . .
, x N ) .
As shown in Cohen et al.
(2016a)
, these functions have the following form:
[Equation]
) where A y , called the coefﬁcients tensor , is a tensor of order N and dimension M in each mode, which for the sake of discussion can simply be seen as a multi-dimensional array, speciﬁed by N indices d 1 , . . .
, d N each ranging in { 1 , . . .
, M } , with entries given by polynomials in the network’s conv weights.
A byproduct of eq.is that for a ﬁxed set of M representation functions, all functions represented by ConvACs lay in the same subspace of functions.
B C OMPARISON TO P OOLING G EOMETRY
From theorem 1 we learn that overlaps give rise to networks which almost always cannot be efﬁciently im- plemented by non-overlapping ConvAC with standard pooling geometry.
However, as proven by Cohen and
Shashua (2017), a ConvAC that uses a different pooling geometry – i.e. the input to the pooling layers are not strictly contiguous windows from the previous layer – also cannot be efﬁciently implemented by the standard
ConvAC with standard pooling geometry.
This raises the question of whether overlapping operations are sim- ply equivalent to a ConvAC with a different pooling geometry and nothing more.
We answer this question in two parts.
First, a ConvAC with a different pooling geometry might be able to implement some function more efﬁciently than ConvAC with standard pooling geometry, however, the reverse is also true, that a ConvAC with standard pooling can implement some functions more efﬁciently than ConvAC with alternative pooling.
In con- trast, a ConvAC that uses overlaps is still capable to implement efﬁciently any function that a non-overlapping
ConvAC with standard pooling can.
Second, we can also show that some overlapping architectures are expo- nentially efﬁcient than any non-overlapping ConvAC regardless of its pooling geometry.
This is accomplished by ﬁrst extending lemma 1 to this case:
Lemma 2.
Under the same conditions as lemma 1, if for all partitions P ·∪ Q such that | P | = | Q | = N / 2 it holds that rank ( (cid:74)
A ( h y ) (cid:75) P,Q ) ≥ T , then any non-overlapping ConvAC regardless of its pooling geometry must have at least T channels in its next to last layer to induce the same grid tensor.
Next, in theorem 2 below show that some overlapping architectures can induce grid tensors whose matricized rank is exponential for any equal partition of its indices, proving they are indeed exponentially more efﬁcient:
Theorem 2.
Under the same settings as theorem 1, consider a GC network whose representation layer is followed by a GC layer with local receptive ﬁeld H × H , stride 1 × 1 , and D ≥ M output channels, whose parameters are “unshared”, i.e. unique to each spatial location in the output of the layer as opposed to shared across them, followed by ( L − 1) arbitrary GC layers, whose ﬁnal output is a scalar.
For any choice of the parameters, except a null set (with respect to the Lebesgue measure) and for any template vectors such that F is non-singular, then the matricized rank of the induced grid tensor is equal to M H 22 , for any equal partition of the indices.
The exact same result holds if the parameters of the ﬁrst GC layers are “shared” and D ≥ M · H 2 .
Proof sketch.
We follow the same steps of our proof of theorem 1, however, we do not construct just one speciﬁc overlapping network that attains a rank of D ≥ M · H 2 , for all possible matricizations of the induced grid tensor.
Instead, we construct a separate network for each possible matricization.
This proves that with respect to the Lebesgue measure over the network’s parameters space, separately for each pooling geometry, the set of parameters for which the lower bound does not hold is of measure zero.
Since a ﬁnite union of zero measured sets is also of measure zero, then the lower bound with respect to all possible pooling geometries holds almost everywhere, which concludes the proof sketch.
See app.
C.5 for a complete proof.
It is important to note that though the above theorem shows that pooling geometry on its own is less expressive than overlapping networks with standard pooling, it does not mean that pooling geometry is irrelevant.
Specifically, we do not yet know the effect of combining both overlaps and alternative pooling geometries together.
Additionally, many times sufﬁcient expressivity is not the main obstacle for solving a speciﬁc task, and the inductive bias induced by a carefully chosen pooling geometry could help reduce overﬁtting.
C D EFERRED P ROOFS
In this section we present our proofs for the theorems and claims stated in the body of the article.
C. P RELIMINARIES
In this section we lay out the preliminaries required to understand the proofs in the following sections.
We begin with a limited introduction to tensor analysis, followed by quoting a few relevant known results relating tensors to ConvACs.
We begin with basic deﬁnitions and operations relating to tensors.
Let A ∈ R M [Equation] M N be a tensor of order
N and dimension M i in each mode i ∈
[ N ] (where [ N ] = { 1 , . .
. , N } ), i.e. A d 1 ,...,d N ∈ R for all i ∈
[ N ] and d i ∈
[ M i ] .
For tensors A (1) and A (2) of orders N (1) and N (2) , and dimensions M (1) i 1 and M (2) i 2 in each of the modes
i 1 ∈
[ N (1) ]
and i 2 ∈
[ N (2) ] , respectively, we deﬁne their tensor product
A (1) ⊗ A (2) as the order
N (1) + N (2) tensor, where
[Equation]
For a set of vectors v (1) ∈ R M 1 , . . .
, v ( N ) ∈ R M N , the N ordered tensor A = v (1) ⊗· · ·⊗ v ( N ) is called an elementary tensor , or rank-1 tensor .
More generally, any tensor can be represented as a linear combination of rank-1 tensors, i.e. A = (cid:80) Zz =1 v ( Z, 1) ⊗· · ·⊗ v ( Z, 1) , known as rank-1 decomposition , or CP decomposition, where the minimal Z for which this equality holds is knows as the tensor rank of A .
Given a set of matrices
F (1) ∈ R M (cid:48)
1 × M 1 , . .
, F ( N ) ∈ R M (cid:48)
N × M N , we denote by F = ( F (1) ⊗· · ·⊗ F ( N ) )
the linear transformation from
R M 1 ⊗···⊗ M N to R M (cid:48) 1 ⊗···⊗ M (cid:48) N , such that for any elementary tensor A , with notations as above, it holds that:
[Equation]
F ( A ) is deﬁned for a general tensor A through its rank-1 decomposition comprising elementary tensors and applying F on each of them, which can be shown to be equivalent to
[Equation]
A central concept in tensor analysis is that of tensor matricization .
Let P ·∪ Q =
[ N ] be a disjoint partition of its indices, such that P = { p 1 , . .
, p | P | } with p 1 < . . .
< p | P | , and Q = { q 1 , . .
, q | Q | } with q 1 < . . .
< q | Q | .
The matricization of A with respect to the partition P ·∪ Q , denoted by (cid:74) A (cid:75) P,Q , is the (cid:16)(cid:81) | P | t =1 M p t (cid:17) -by- (cid:16)(cid:81) | Q | t =1 M q t (cid:17) matrix holding the entries of A , such that for all i ∈ [ N ] and d i ∈
[ M i ] the entry
A d 1 ,...,d N is placed in row index 1 + (cid:80) | P | t =1 ( d p t − 1) (cid:81) | P | t (cid:48)
=
t +1 M p t (cid:48) and column index 1 + (cid:80) | Q | t =1 ( d q t − 1) (cid:81) | Q | t (cid:48) =
t +1 M q t (cid:48) .
Applying the matricization operator (cid:74) · (cid:75) P,Q on the tensor product operator results in the Kronecker Product , i.e. for an N -ordered tensor A , a K -ordered tensor B , and the partition P ·∪ Q =
[ N + K ] , it holds that
[Equation]
] where P − N and Q − N are simply the sets obtained by subtracting the number N from every element of P or Q , respectively.
In concrete terms, the Kronecker product for the matrices
A ∈ R M 1 × M 2 and B ∈ R N 1 × Nresults in the matrix A (cid:12) B ∈ R M 1 N 1 × M 2 N 2 holding A ij B kl in row index ( i − 1) N 1 + k and column index ( j − 1) N 2 + l .
An important property of the Kronecker product is that rank ( A (cid:12) B ) = rank ( A ) · rank ( B ) .
Typically, when wish to compute rank (
(cid:74) A (cid:75) P,Q ) , we will ﬁrst decompose it to a Kronecker product of matrices.
For a linear transform F , as deﬁned above in eq.6, and a partition P ·∪ Q , if F (1) , . . .
, F ( N ) are non-singular matrices, then F is invertible and the matrix rank of (cid:74) A (cid:75) P,Q equals to the matrix rank of (cid:74) F ( A ) (cid:75) P,Q (see proof in Hackbusch (2012)).
Finally, we deﬁne the concept of grid tensors : for a function f : R s × · · · × R s → R and a set of template vectors x (1) , . . .
, x ( M ) ∈ R s , the N -order grid tensor A ( f ) is deﬁned by ( A ( f ))
d 1 ,...,d N = f
( x ( d 1 ) , . .
, x ( d N ) ) .
In the context of ConvACs, circuits and the functions they can realize are typically examined through the matricization of the grid tensors they induce.
The following is a succinct summary of the relevant known results used in our proofs – for a more detailed discussion, see previous works (Cohen et al., 2016a; Cohen and Shashua, 2016; 2017).
Using the same notations from eq.
5 describing a general ConvAC, let A y be the coefﬁcients tensor of order N and dimension M in each mode, and let f θ 1 , . . .
, f θ M : R s → R be a set of M representation functions (see app.
A).
Under the above deﬁnitions, a non-overlapping ConvAC can be said to decompose the coefﬁcients tensor A y .
Different network architectures correspond to known tensor decompositions: shallow networks corresponds to rank-1 decompositions, and deep networks corresponds to Hierarchical Tucker decompositions.
In Cohen and Shashua (2017), it was found that the matrix rank of the matricization of the coefﬁcients tensors
A y could serve as a bound for the size of networks decomposing A y .
For the conventional non-overlapping
ConvAC and the contiguous “low-high” partition P = { 1 , . .
N / 2 } , Q = { N / 2 + 1 , . . .
, N } of [ N ] , the rank of the matricization (cid:74)
A y (cid:75) P,Q serves as a lower-bound on the number of channels of the next to last layer of any network which decomposes the coefﬁcients tensor A .
In the common case of square inputs, i.e. the input is of shape H × H and N = H 2 , it is more natural to represent indices by pairs ( j, i ) denoting the spatial location of each “patch” x ( j,i ) , where the ﬁrst argument denotes the vertical location and the second denotes the horizontal location.
Under such setting the equivalent “low-high” partitions are either the “left-right” partition, i.e. P = { ( j, i ) | j ≤ H 2 } , Q = { ( j, i ) | j > H 2 } , or the “top-bottom” partition, i.e. P = { ( j, i ) | i ≤ H 2 } , Q = { ( j, i ) | i > H 2 } .
More generally, when considering networks using other pooling geometries , i.e. not strictly contiguous pooling windows, then for each pooling geometry there exists a corresponding partition P ·∪ Q such that rank ( (cid:74)
A y (cid:75) P,Q ) serves as its respective lower-bound.
Though the results in Cohen and Shashua (2017) are strictly based on the matricization rank of the coefﬁ- cients tensors, they can be transferred to the matricization rank of grid tensors as well.
Grid tensors were ﬁrst considered for analyzing ConvACs in Cohen and Shashua (2016).
For a set of M template vectors x (1) , . . .
, x ( M ) ∈
R s , we deﬁne the matrix F ∈ R M × M by F ij = f
θ j
( x ( i ) ) .
With the above notations in place, we can write the the grid tensor A ( h y ) for the function
h y ( x 1 , . . .
, x N ) as:
[Equation]
[Equation] If the representation functions are linearly independent and continuous, then we can choose the template vec- tors such that F is non-singular (see Cohen and Shashua (2016)), which according to the previous discussion on tensor matricization, means that for any partition P ·∪ Q and any coefﬁcients tensor A y , it holds that rank ( (cid:74)
A y (cid:75) P,Q ) =
rank ( (cid:74) A ( h y ) (cid:75) P,Q ) .
Thus, any lower bound on the matricization rank of the grid tensor translates to a lower bound on the matricization rank of the coefﬁcients tensors, which in turn serves as lower bound on the size of non-overlapping ConvACs.
The above discussion leads to the proof of lemma 1 and lemma 2 that were previously stated:
Proof of lemma 1 and lemma 2.
For the proofs of the base results with respect to the coefﬁcients tensor, see
Cohen and Shashua (2017).
To prove it is possible to choose the template vectors such that F is non-singular, see Cohen and Shashua (2016).
To prove that if F is non-singular, then the grid tensor and the coefﬁcients tensor have the same matricization rank, see lemma 5.6 in Hackbusch (2012).
We additionally quote the following lemma regarding the prevalence of the maximal matrix rank for matrices whose entries are polynomial functions:
Lemma 3.
Let M, N, K ∈ N , 1 ≤ r ≤ min { M, N } and a polynomial mapping A : R K → R M × N , i.e. for every i ∈
[ M ] and j ∈ [ N ] it holds that A ij : R k
→ R is a polynomial function.
If there exists a point x ∈ R K such that rank ( A ( x )) ≥ r , then the set { x ∈ R K | rank A ( x ) < r } has zero measure (with respect to the Lebesgue measure over R K ).
Proof.
See Sharir et al.
Finally, we simplify the notations of the GC layer with product pooling function for the beneﬁt of following our proofs below.
We will represent the parameters of the l ’th GC layer by { ( w ( l,c ) ∈ R D ( l − 1) × R ( l ) × R ( l ) , b ( l,c ) ∈
R R ( l ) × R ( l ) ) } c ∈ [ D ( l ) ] , where w ( l,c ) represents the weights and b ( l,c ) the biases.
Let X ∈ R H ( in ) × H ( in ) × D ( in ) be the input to the layer and Y ∈ R D ( out )
× H ( out ) × H ( out ) be the output, then the following equality holds:
[Equation]
The above treats the common case where the parameters are shared across all spatial locations, but sometimes we wish to consider the “unshared” case, in which there is are different weights and biases for each location, which we denote by { ( w ( l,c,u,v ) ∈
R D ( l − 1)
× R ( l ) × R ( l ) , b ( l,c,u,v ) ∈
R R
[Equation]
With the above deﬁnitions and lemmas in place, we are ready to prove the propositions and theorems from the body of the article.
C. P ROOF OF P ROPOSITION
Proposition 1 is a direct corollary of the following two claims:
Claim 1.
Let f : R D ( in ) × H ( in ) × H ( in ) → R D ( out ) × H ( out ) × H ( out ) be a function realized by a single GC layer with R × R local receptive ﬁeld, S × S stride, and D ( out ) output channels, that is parameterized by { ( w ( c ) , b ( c ) ) } D ( out ) c
= .
For all ˜ R ≥ R , a GC layer with ˜ R × ˜ R local receptive ﬁeld, S × S stride, and C output channels, parameterized by { ( ˜ w ( c ) , ˜ b ( c ) ) } D ( out ) c
= , could also realize f .
The same is true for the unshared case of both layers.
Proof.
The claim is trivially satisﬁed by setting ˜ w ( c ) such that it is equal to w ( c ) in all matching coordinates, while using zeros for all other coordinates.
Similarly, we set ˜ b ( c ) to be equal to b ( c ) in all matching coordinates, while using ones for all other coordinates.
Claim 2.
Let f : R D ( in ) × H ( in ) × H ( in ) → R D ( out ) × H ( out ) × H ( out ) be a function realized by a GC layer with
R × R local receptive ﬁeld and 1 × 1 stride, parameterized by { ( w ( c ) , b ( c ) ) } D ( out ) c
= .
Then there exists an as- signment to ( w , b ) such that f is the identity function f
( X ) =
X .
The same is true for the unshared case of both layers.
Proof.
From claim 1 it is sufﬁcient to show the above holds for R = 1 .
Indeed, setting
[Equation]
and b ( c ) ≡ 0 satisﬁes the claim.
C. P ROOF OF T HEOREM
We wish to show that for all choices of parameters, except a null set (with respect to Lebesgue measure) the grid tensor induced by the given GC network has rank satisfying eq. 4.
Since the entries of the matricized grid tensor are polynomial function of its parameters, then according to lemma 3, it is sufﬁcient to ﬁnd a single example that achieves this bound.
Hence, our proof is simply the construction of such an example.
Recall that the template vectors must hold that for the matrix F , deﬁned by F ij = f j ( x ( i ) ) , where { f j } Mj = are the representation matrices, is a non-singular matrix.
We additionally assume in the following claims that the output of the representation layer is of width and height equal to H ∈ N , where H is an even number – the claims and proofs can however be easily adapted to the more general case.
Assume a ConvAC as described in the theorem, with representation layer deﬁned according to above, followed by L GC layers, where the l ’th layer has a local receptive ﬁeld of R ( l ) , a stride of S ( l ) , and D ( l ) output channels.
We ﬁrst construct our example, that achieves the desired matricization rank, for the simpler case where the ﬁrst layer following the representation layer has a local receptive ﬁeld large enough, i.e. when it is larger than
H 2 .
Recall that for the ﬁrst layer the total receptive ﬁeld is equal to its local receptive ﬁeld.
In the context of theorem 1, this ﬁrst layer satisﬁes the conditions necessary to produce the lower bound given in the theorem.
The speciﬁc construction is presented in the following claim, which relies on utilizing the large local receptive ﬁeld to match each spatial location in the left side of the input with one from the right side, such that for each such pair, the respective output of the ﬁrst layer will represent a mostly diagonal matrix.
We then set the rest of the parameters such that the output of the entire network is deﬁned by a tensor product of mostly diagonal matrices.
Since the matricization rank of the tensor product of matrices is equal to the product of the individual ranks, it results in an exponential form of the rank as is given in the theorem.
Claim 3.
Assume a ConvAC as deﬁned above, ending with a single scalar output.
For all l ∈ [ L ] , the parameters of the l -th GC layer are denoted by { ( w ( l,c ) ∈ R D ( l − 1)
× R ( l ) × R ( l ) , b ( l,c ) ∈ R R ( l ) × R ( l ) ) } c ∈
[ D ( l ) ] .
Let h ( x 1 , . . .
, x N ) be the function realized the output of network.
Additionally deﬁne R ≡ R (1) , S ≡ S (1) and
D ≡ min { D (1) , M } .
If R > H 2 , and the weights w (1 ,c ) and biases b (1 ,c ) of the ﬁrst GC layer layer are set to:
[Equation]
and (
[Equation]
, i ) ∈ { (1 , 1) , ( [Equation] , τ ) }
Otherwise
[Equation]
= 
 β c ≤ D and ( j, i ) ∈ { (1 , 1) , ( ρ, τ ) } c ≤ D and ( j, i ) (cid:54)∈ { (1 , 1) , ( ρ, τ ) }
Otherwise where β = 2 αD , then there exists an assignment to α and the parameters of the other GC layers such that rank ( (cid:74)
A ( h ) (cid:75) P,Q ) =
D (cid:98) H − RS
+
(cid:99) · (cid:100) HS (cid:101) , where P ·∪ Q is either the “left-right” or “top-bottom” partition, and ( ρ, τ ) equals to (1 , R ) or ( R, 1) , respectively.
Proof.
The proof for either the “left-right” or “top-bottom” partition is completely symmetric, thus it is enough to prove the claim for the “left-right” case, where ( ρ, τ ) =
(1 , R ) .
We wish to compute the entry
A ( h ) d (1 , 1) ,...,d ( H,H ) of the induced grid tensor for arbitrary indices d (1 , 1) , . . .
, d ( H,H ) .
Let O ∈ R M
×
H
×
H be the 3-order tensor output of the representation layer, where O m,j,i = F d ( j,i ) ,m for the aforementioned indices and for all 1 ≤ i, j ≤ H and m ∈
[ M ] .
We begin by setting the parameters of all layers following the ﬁrst GC layer, such that they are equal to com- puting the sum along the channels axis of the output of the ﬁrst GC layer, followed by a global product of all of the resulting sums.
To achieve this, we can ﬁrst assume w.l.o.g. that these layers are non-overlapping through proposition 1.
We then set the parameters of the second GC layer to w (2 ,c )
= 1 and b (2 ,c ) ≡ 0 , i.e. all ones and all zeros, respectively, which is equivalent to taking the sum along the channels axis for each spatial location, followed by taking the products over non-overlapping local receptive ﬁelds of size R (2) .
For the other layers, we simply set them as to take just the output of the ﬁrst channel of the output of the preceding layer, which is equal to setting their parameters to w ( l,c ) dji =
[Equation]
0 d
[Equation]
= 1 and b ( l,c ) ≡ 0 .
Setting the parameters as described above results in:
[Equation]
where we extended O with zero-padding for the cases where uS + j > H or vS + i > H , as mentioned in sec. 2.
Next, we go through the technical process of reducing eq.
8 to a product of matrices.
Substituting the values of w (1 ,c ) dji and b (1 ,c ) ji with those deﬁned in the claim, and computing the value of g ( u, v, c, j, i ) results in:
[Equation]
from which we derive:
[Equation]
where ( A ( h ))
d (
[Equation]
H
[Equation]
( u, v ) .
At this point we branch into two cases.
If S divides R − 1 , then for all u, v ∈ N such that vS + R ≤ H and uS < H , the above expression for f ( u, v ) and f ( u, v + R − 1 S ) depends only on the indices d ( uS +1 ,vS +1) and d ( uS +1 ,vS + R ) , while these two indices affect only the aforementioned expressions.
By denoting
A ( u,v ) d ( uS +1 ,vS +1) ,d ( uS +1 ,vS + R ) =
f ( u, v ) · f ( u, v + R − 1 S )
, we can write it as:
[Equation]
where i, j ∈ [ M ] stand for the possible values of d ( uS +1 ,vS +1) and d ( uS +1 ,vS + R ) , respectively.
Substituting β
=
αD , as stated in the claim, and setting α = (cid:0) D 2 (cid:1) 1 / 4 , results in:
A ( u,v ) ij =
  1 [ i = j ] i, j ≤ D
D i ≤ D and j > D D i > D and j ≤ D
D i, j > D which means rank (cid:16) A ( u,v ) (cid:17) =
D .
Since ( A ( h ))
d (1 , 1) ,...,d ( H,H ) equals to the product (cid:81) 0 ≤ uS<H 0 ≤ vS ≤ H − R A ( u,v ) d ( uS +1 ,vS +1) ,d ( uS +1 ,vS + R ) , then (cid:74)
A ( h ) (cid:75) P,Q equals to the Kronecker product of the matrices in { A ( u,v ) |
≤ uS < H, 0 ≤ vS ≤ H − R } , up to permutation of its rows and columns, which do not affect its matrix rank.
Thus, the matricization rank of A ( h ) satisﬁes:
[Equation]
which proves the claim for this case.
If S does not divide R − 1 , then for all u, v ∈ N , such that vS + R ≤ H and uS < H , it holds that f ( u, v ) depends only on the indices d ( uS +1 ,vS +1) and d ( uS +1 ,vS + R ) , and they affect only f ( u, v ) .
Additionally, for all u, v ∈ N , such that H < vS + R , vS < H and uS < H , it holds that f ( u, v ) depends only on the index d ( uS +1 ,vS +1) , and this index affects only f ( u, v ) .
Let us denote A ( u,v ) d ( uS +1 ,vS +1) ,d ( uS +1 ,vS + R ) =
f ( u, v ) for vS
+ R ≤ H :
A ( u,v ) ij
[Equation]
which by setting β = 2 αD and α = 1 results in:
[Equation]
which means rank (cid:16) A ( u,v ) (cid:17)
= D .
For vS + R > H , and we can deﬁne the vector a ( u,v ) d ( uS +1 ,vS +1) =
f ( u, v ) , which by using the same values of α and β results in:
[Equation]
By viewing a ( u,v ) as either a column or row vector, depending on whether d ( uS +1 ,vS +1) ∈ P or d ( uS +1 ,vS +1) ∈ Q , respectively, it holds that (cid:74) A ( h ) (cid:75) P,Q equals to the Kronecker product of the matrices in { A ( u,v ) } 0 ≤ uS<H 0 ≤ vS ≤ H − R ∪ { a ( u,v ) } 0 ≤ uS<H H − R<vS<H , up to permutations of its rows and columns, which do not affect the rank.
Since a ( u,v ) (cid:54) = 0 then rank (cid:16) a ( u,v ) (cid:17) = 1 , which means the matricization rank of A ( h ) once again holds:
[Equation]
In the preceding claim we have describe an example for the case where the total receptive of the ﬁrst GC layer is already large enough for satisfying the conditions of the theorem.
In the following claim we extend this result for the general case.
This is accomplished by showing that a network comprised of just L GC layers with local receptive ﬁelds { R ( l ) × R ( l ) } l ∈ [ L ] , strides { S ( l ) × S ( l ) } l ∈
[ L ] , and output channels { D ( l ) } l ∈
[ L ] , can effectively compute the same output as the ﬁrst GC layer from claim 3, for all inputs 3 .
Recall that the layer from claim 3 performs an identical transformation on each M × 1 × 1 patch from its input, followed by taking the point-wise product of far-away pairs of transformed patches.
Thus, the motivation behind the speciﬁc construction we use, is to use the ﬁrst of the L layers to perform this transformation, while using half of its output channels for storing the transformed patch from the same location, and the other half for storing a transformed patch, but from a location farthest to the right, constrained by its local receptive ﬁeld.
This is equal to having one set of transformed patches sitting still, while another “shifted” set of transformed patches.
The other layers simply pass the the ﬁrst half of the channels as is, using an identity operation as deﬁned in claim 2, while continuously shifting the other half of the channels more and more to the left, bringing faraway patches closer together.
Finally, at the last layer we take both halves and multiple them together.
Claim 4.
Assume Φ is a ConvAC comprised of just L GC layers as described above, where the output of the network is not limited to a scalar value.
Assume the total stride of the L -th GC layer is greater than H / 2 , and let T ( L ) S and T ( L,α ) R be the total stride and the α -minimal total receptive ﬁeld, respectively, for α = (cid:98) H / 2 (cid:99)
+ 1 .
Let Ψ be a ConvAC comprised of a single GC layer with local receptive ﬁeld R ≡ T ( L,α ) R , stride S ≡ T ( L ) S , output channels D ≡ min { 1 2 min 1 ≤ l<L D ( l ) , D ( L ) } , where its weights and biases are set to the following:
[Equation]
j, i ) ∈ { (1 , 1) , ( ρ, τ ) }
Otherwise b ( c ) ji = (cid:40) β c ( j, i ) ∈ { (1 , 1) , ( ρ, τ ) }
Otherwise for β ∈ R D , A ∈ R M × D and ( ρ, τ ) ∈ { (1 , R ) , ( R, 1) } .
Then, there exists a set of weights to the layers of Φ such that for every input X , the output of Φ is equivalent to the output of Ψ for channels ≤ D , and zero otherwise.
Proof.
The two possible cases for ( ρ, τ ) are completely symmetric, thus it is enough to prove the claim just for ( ρ, τ ) =
(1 , R ) .
Additionally, we can assume w.l.o.g. that ∀ l, R ( l ) > 1 , by setting any 1 × 1 layer to act as pass-through according to claim 2, and also assume that the α -minimal total receptive ﬁeld is exactly equal to the total receptive ﬁeld of the L -th layer, by applying claim 1 to realize an equivalent network with smaller windows.
Finally, the case for L = 1 is trivial, and thus we assume L > 1 .
Let us set the parameters (cid:110) w ( l,k ) , b ( l,k ) (cid:111) of the layers of Φ as follows: w ( l,k ) dji =
  − A d,k ( l = 1) and (1 ≤ k ≤ D ) and ( j, i ) = (1 , 1) − A d,
k
− D (
l
= 1) and ( D < k ≤ 2 D ) and ( j, i = (
R[-( l ) ) 
-]
( l ) 1 [ d = k ] (1 < l < L ) and (1 ≤ k ≤ D ) and ( j, i ) = (1 , 1) 1 [ d = k ] (1 < l < L ) and ( D < k ≤ 2 D ) and ( j, i ) =
(1 , R ( l ) )
[ d = k ] ( l = L ) and (1 ≤ k ≤ D ) and ( j, i ) = [ d = M + k ] ( l = L ) and (1 ≤ k ≤ D ) and ( j, i ) =
(1 , R ( l ) )
Otherwise b ( l,k ) ji =     β k ( l = 1) and (1 ≤ k ≤ D ) and ( j, i ) =
(1 , 1) β k − D ( l = 1) and ( D < k ≤ 2 D ) and ( j, i ) =
(1 , R ( l ) ) (1 < l < L ) and (1 ≤ k ≤ D ) and ( j, i ) =
(1 , 1) (1 < l < L ) and ( D < k ≤ 2 D ) and ( j, i ) =
(1 , R ( l ) )
( l = L ) and (1 ≤ k ≤ D ) and ( j, i ) =
(1 , 1) ( l = L ) and (1 ≤ k ≤ D ) and ( j, i ) =
(1 , R ( l ) )
Otherwise
It is left to prove the above satisﬁes the claim.
Let O ( l ) ∈ R D ( l ) × H ( l ) × H ( l ) be the output of the l -th layer, for l ∈ [0 , . . .
, L ] , where H ( l ) is the width and height of the output.
We additionally assume that for indices beyond the bounds of [ D ( l ) ] ×
[ H ( l ) ] × [ H ( l ) ] the value of O ( l ) is zero, i.e. we assume zero padding when applying the convolutional operation of the GC layer.
We extend the deﬁnition for l = 0 , by setting D (0) ≡ M and H (0) ≡ H , where we identify O (0) with the input to the network Φ .
Given the above, the output of the ﬁrst layer for k ∈ [ D (1) ] and 0 ≤ u, v < H (1) , is as follows:
[Equation]
Otherwise
We will show by induction that for 1 < l < L , k ∈ [ D ( l ) ] and 0 ≤ u, v < H ( l ) the output of the l -th layer
O ( l ) k,u +1 ,v +1 always equals to:
[Equation]
Otherwise where η ( l ) =
(cid:81) li =2 S ( i ) and ξ ( l ) =
R ( l ) · η ( l − 1) + (cid:80) l − 1 k =
( R ( k ) − S ( l ) ) ·
η ( k − 1) .
It is trivial to verify that for l = 2 it indeed holds, since:
[Equation]
Otherwise where η (2) =
S (2) and ξ (2) = R (2) .
Assume the claim holds up to l − 1 , and we will show it also holds for l :
O ( l )
[Equation]
,u
[Equation]
[Equation]
+
[Equation]
Otherwise
Induction Hypothesis
[Equation]
Otherwise
[Equation]
Otherwise
Were we used the fact that η ( l ) = S ( l ) η ( l − 1) and ξ ( l ) =
R ( l ) η ( l − 1) + ξ ( l − 1) − η ( l − 1) .
Finally, we show that O ( L ) k,u +1 ,v +1 for k ≤ D and 0 ≤ u, v < H ( L ) equals to the output of the single GC layer speciﬁed in the claim:
O ( L ) k,u +1 ,v +1 =
R ( L ) (cid:89)
[Equation]

 which is indeed equal to the single GC layer.
For k > D , both the bias and the weights for the last layer are zero, and thus O ( L ) k,u +1 ,v +1 = 1 .
Finally, with the above two claims set in place, we can prove our main theorem:
Proof.
( of theorem 1 ) Using claim 4 we can realize the networks from claim 3, for which the matricization rank for either partition equals to:
[Table]
Since for any matricization [ A (Ψ)] P,Q the entries of the matricization are polynomial functions with respect to the parameters of the network, then, according to lemma 3, the set of parameters of Φ , that does not attain the above rank, has zero measure.
Since the union of zero measured sets is also of measure zero, then all parameters except a set of zero measure attain this matricization rank for both partitions at once, concluding the proof.
C. P ROOF OF P ROPOSITION
Following theorem 1, to compute the lower bound for the network described in proposition 2, we need to ﬁnd the ﬁrst layer for which its total receptive ﬁeld is greater than H / 2 , and then estimate its total stride and its α -minimal total receptive ﬁeld, for α = (cid:98) H / 2 (cid:99) .
In the following claims we analyze the above properties of the given network:
Claim 5.
The total stride and total receptive ﬁeld of the l -th B × B layer in the given network, i.e. the (2 l − 1) -th
GC layer after the representation layer, are given by the following equations:
[Equation]
Proof.
From eq.
1 it immediately follows that T (2 l − 1) S ( S [Equation]
, S (2 l − 1) )
=
2 l − 1 .
From eq. 2, the 2 × 2 stride 2 layers do not contribute to the receptive ﬁeld as R (2 l ) − S (2 l ) = 0 , which results in the following equation:
[Equation]
Claim 6.
The α -minimal total receptive ﬁeld for the l -th B × B layer in the given network, for α ∈ N and 2 l − 1 ≤ α < 2 l − 1 , always equals ( α + 1) .
Proof.
From eq. 3, the following holds:
[Equation]
Notice that the right term in the equation resembles a binary representation.
If we limit t 2 i − 1 to the set { 1 , 2 } , this term can represent any number in the set { 0 , . .
, 2 l − 1 − 1 } , and by choosing t 2 l − 1 = 1 , the complete term can represent any number in the set { 2 l − 1 , . .
, 2 l − 1 } , and speciﬁcally, for 2 l − 1 ≤ α < 2 l − 1 , there exists an assignment for t 2 i − 1 ∈ { 1 , 2 } for i ∈
[ l − 1] such that this terms equal ( α +1) , and thus T (2 l − 1 ,α ) R = α +1 .
With the above general properties for the given network, we can simplify the expression for the lower bound given in theorem 1:
Claim 7.
If the l -th
B ×
B layer in the given network satisﬁes
T (2 l − 1) R
( R (1) , S (1) , . . .
, R (2 l − 1) , S (2 l − 1) )
> H / 2 , then the lower bound given in theorem 1 equals to
M 2 2 L − 2 l +
Proof.
From the description of the network and the previous claims it holds that D = M , H = 2 L ,
T (2 l − 1) S = 2 l − 1 , and T (2 l − 1 , (cid:98) H / 2 (cid:99)
[Equation]
=
[Equation]
L − 1 + 1 .
Substituting all the above in eq.
4 results in:
[Equation]
With all of the above claims in place, we our ready to prove proposition 2:
Proof.
( of proposition 2 )
From claim 5, we can infer which is the ﬁrst B × B layer such that its receptive ﬁeld is greater than H / 2 :
[Equation]
Combining the above with claim 7, results in:
[Equation]
The limits and the special case for B ≤ H 5 + 1 are both direct corollaries of the above expressions.
C. P ROOF OF THEOREM
We begin by proving an analogue of claim 3, where we show that for any given matricization of the grid tensor A ( h ) , induced by the overlapping network realizing the function h , the matricization rank is exponential.
The motivation behind the construction, for when parameters are “unshared”, is that we can utilized the fact that there are separate sets of kernels, with local receptive ﬁelds the size of the input, for each spatial location.
Thus, each kernel can “connect” the index (of the grid tensor) matching its spatial location, with almost any other index, and speciﬁcally such that the two indices come from different sets of the matricization I ·∪ J of
A ( h ) .
For the “shared” case, we simply use polynomially more output channels to simulate the “unshared” case.
Claim 8.
For an arbitrary even partition ( I, J ) of { (1 , 1) , . . .
, ( H, H ) } , such that | I | = | J | =
H 2 2 , there exists an assignment to the parameters of the network given in theorem 2, for either the “unshared” or “shared” settings, such that rank ( (cid:74)
A ( h ) (cid:75) )
=
M H 22 .
Proof.
Let ( I, J ) be an arbitrary even partition of { (1 , 1) , . . .
, ( H, H ) } , such that | I | = | J | =
H 2 2 , I =
{ i 1 , . .
, i | I | } , and J = { j 1 , . . .
, j | J | } , where for k ∈ [ H 2 2 ] it holds that i k < i k +1 and j k < j k +1 (using lexical ordering), and we assume w.l.o.g. that i 1 = (1 , 1) .
We deﬁne the set { ( q k , p k ) } H 22 k =1 such that q k = i k and p k
=
j k if i k < j k , and otherwise q k = j k and p k
= i k .
We prove the t“unshared” case ﬁrst, where the parameters of the ﬁrst GC layers are given by { ( w ( c,u,v ) , b ( c,u,v ) )
} D,H,Hc =1 ,u =1 ,v =1 , for which we choose the following assignment: w ( c,u,v ) m,j,i = (cid:40) ( F − 1 ) m,c c ≤ M and ∃ k ∈ (cid:104) H 2 2 (cid:105) , q k = ( u, v ) and ( j, i ) ∈ { (1 , 1) , p k − q k + (1 , 1) }
Otherwise b ( c,u,v ) j,i = (cid:40)c ≤ M and ∃ k ∈ (cid:104) H 2 2 (cid:105) , q k = ( u, v ) and ( j, i ) ∈ { (1 , 1) , p k − q k + (1 , 1) } 1 [ c ≤ M ]
Otherwise where for u, v ∈ [ H ] such that q k = ( u, v ) it holds that p k − q k + (1 , 1) ∈ { (1 , 1) , . . .
, ( H, H ) } because q k ≤ p k .
Similar to the proof of claim 3, we set all layers following the ﬁrst GC layer such that the following equality holds:
[Equation]
Which under the assignment to parameters we chose earlier, it results in:
[Equation]
which means (cid:74) A ( h ) (cid:75)
( I,J ) equals to the Kronecker product of H 2 2 M × M -identity matrices, up to permutations of its rows and columns which do not affect its matrix rank.
Thus, rank ( (cid:74)
A ( h ) (cid:75) )
=
M H 22 .
For the “shared” setting, we denote the parameters of the ﬁrst GC layer by { ( w ( d ) , b ( d ) ) } D c =1 , and set them as: w ( d ) m,j,i = 
 ( F − 1 ) m,c (cid:0) ∃ c ∈ [ M ] ∃ u, v ∈
[ H ] , d = cH 2 + uH + v (cid:1) and (cid:16)
∃ k ∈ (cid:104) H 2 2 (cid:105) , q k = ( u, v ) and ( j, i ) ∈ { (1 , 1) , p k − q k + (1 ,
1) } (cid:17)
Otherwise b ( d ) j,i = 
 (cid:0) ∃ c ∈ [ M ] ∃ u, v ∈
[ H ] , d = cH 2 + uH + v (cid:1) and (cid:16)
∃ k ∈ (cid:104) H 2 2 (cid:105) , q k = ( u, v ) and ( j, i ) ∈ { (1 , 1) , p k − q k + (1 , 1) } (cid:17)
[ d ≤ MH 2 ]
Otherwise
the parameters of the other layers are set as in the “unshared” case, and the proof follows similarly.
In the preceding claim, we have found a separate example for each matricization, such that the matricization rank is exponential.
In the following proof of the theorem 2, we leverage basic properties from measure theory to show that almost everywhere the induced grid tensor has an exponential matricization rank, under every possible even matricization – without explicitly constructing such an example.
Proof.
( of theorem 2 )
For any even partition ( I, J ) of { (1 , 1) , . . .
, ( H, H ) } , according to claim 8 there exist parameters for which rank ( (cid:74)
A ( h ) (cid:75) )
( I,J ) =
M H 22 , and thus according to lemma 3 the set of parameters for which rank ( (cid:74) A ( h ) (cid:75) ) <
M H 22 is of measure zero.
Since the ﬁnite union of sets of measure zero is also of measure zero, then almost everywhere (with respect to the Lebesgue measure) the parameters results in networks such that for all even partitions rank ( (cid:74)
A ( h ) (cid:75) )
( I,J ) =
M H 22 .