T HE R OLE OF M INIMAL C OMPLEXITY F UNCTIONS IN U NSUPERVISED L EARNING OF S EMANTIC M APPINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
We discuss the feasibility of the following learning problem: given unmatched samples from two domains and nothing else, learn a mapping between the two, which preserves semantics.
Due to the lack of paired samples and without any deﬁnition of the semantic information, the problem might seem ill-posed.
Speciﬁcally, in typical cases, it seems possible to build inﬁnitely many alternative mappings from every target mapping.
This apparent ambiguity stands in sharp contrast to the recent empirical success in solving this problem.
We identify the abstract notion of aligning two domains in a semantic way with concrete terms of minimal relative complexity.
A theoretical framework for measuring the complexity of compositions of functions is developed in order to show that it is reasonable to expect the minimal complexity mapping to be unique.
The measured complexity used is directly related to the depth of the neural networks being learned and a semantically aligned mapping could then be captured simply by learning using architectures that are not much bigger than the minimal architecture.
Various predictions are made based on the hypothesis that semantic alignment can be captured by the minimal mapping.
These are veriﬁed extensively.
In addition, a new mapping algorithm is proposed and shown to lead to better mapping results.
I NTRODUCTION
Multiple recent reports (Xia et al., 2016;
Kim et al., 2017; Zhu et al., 2017; Yi et al., 2017) convincingly demonstrated that one can learn to map between two domains that are each speciﬁed merely by a set of unlabeled examples.
For example, given a set of unlabeled images of horses, and a set of unlabeled images of zebras, CycleGAN (Zhu et al., 2017) creates the analog zebra image for a new image of a horse and vice versa.
These recent methods employ two types of constraints.
First, when mapping from one domain to another, the output has to be indistinguishable from the samples of the new domain.
This is enforced using GANs (Goodfellow et al., 2014) and is applied at the distribution level: the mapping of horse images to the zebra domain should create images that are indistinguishable from the training images of zebras and vice versa.
The second type of constraint enforces that for every single sample, transforming it to the other domain and back (by a composition of the mappings in the two directions) results in the original sample.
This is enforced for each training sample from either domain: every training image of a horse (zebra), which is mapped to a zebra (horse) image and then back to the source domain, should be as similar as possible to the original input image.
In another example, taken from DiscoGAN (Kim et al., 2017), a function is learned to map a handbag to a shoe of a similar style.
One may wonder why striped bags are not mapped, for example, to shoes with a checkerboard pattern.
If every striped pattern in either domain is mapped to a checkerboard pattern in the other and vice-versa, then both the distribution constraints and the circularity constraints might hold.
The former could hold since both striped and checkerboard patterned objects would be generated.
Circularity could hold since, for example, a striped object would be mapped to a checkerboard object in the other domain and then back to the original striped object.
One may claim that the distribution of striped bags is similar to those of striped shoes and that the distribution of checkerboard patterns is also the same in both domains.
In this case, the alignment follows from ﬁtting the shapes of the distributions.
This explanation is unlikely, since no effort is being made to create handbags and shoes that have the same distributions of these properties, as well as many other properties.
Our work is dedicated to the alternative hypothesis that the target mapping is implicitly deﬁned by being approximated by the lowest-complexity mapping that has a low discrepancy between the mapped samples and the target distribution, i.e., the property that even a good discriminator cannot distinguish between the generated samples and the target ones.
In Sec.
2 we explore the inherent ambiguity of cross domain mapping.
In Sec. 3, we present the hypothesis and two veriﬁable predictions, as well as a new unsupervised mapping algorithm.
In Sec. 4, we show that the number of minimal complexity mappings is expected to be small.
Sec.
5 veriﬁes the various predictions.
Some context to our work, including classical ideas such as Occam’s Razor, MDL, and Kolmogorov complexity are discussed in Sec. 6.
T HE U NSUPERVISED A LIGNMENT P ROBLEM
The learning algorithm is provided with only two unlabeled datasets: one includes i.i.d samples from the ﬁrst distribution and the second includes i.i.d samples from the other distribution (all notations are listed in Appendix B, Tab. 5).
x i ∈
X A for i = 1 . . .
m where x i i.i.d ∼ D A and X A denotes the space of domain A
= ( X A , D A ) x j ∈ X B for j = 1 . . .
n where x j i.i.d ∼ D B and X B denotes the space of domain B = ( X B , D B )
To semantically tie the two distributions together, a generative view can be taken.
This view is well aligned with the success of GAN-based image generation, e.g., (Radford et al., 2015), in mapping random input vectors into realistic-looking images.
Let z ∈
X be a random vector that is distributed according to the distribution D Z and which we employ to denote the semantic essence of samples in
X A and X B .
We denote D
A =
y A ◦ D Z and D B = y B ◦ D Z , where the functions
y A : X → X A and y B : X → X B , and f ◦ D denotes the distribution of f ( x ) , where x ∼ D .
Following the circularity-based methods (Xia et al., 2016; Kim et al., 2017; Zhu et al., 2017; Yi et al., 2017), we assume that both y A and y B are invertible.
The assumption of invertibility is further justiﬁed by the recent success of supervised pre-image computation methods (Dosovitskiy & Brox, 2016).
In unsupervised learning, given training samples, one may be expected to be able to recover the underlying properties of the generated samples, even with very weak supervision (Chen et al., 2016).
However, if the target function between domains A and B is not invertible, because for each member of A there are a few possible members of B (or vice versa), we can add a stochastic component to A that is responsible for choosing which member in B to take, given a member of A .
For example, if A is a space of handbag images and B is a space of shoes, such that for every handbag, there are a few analogous shoes, then a stochastic variable can be added such that given a handbad, one shoe is selected among the different analog shoes.
We denote by y AB = y B ◦ y − 1 A , the function that maps the ﬁrst domain to the second domain.
It is semantic in the sense that it goes through the shared semantic space X .
The goal of the learner is to ﬁt a function h ∈ H , for some hypothesis class H that is closest to y AB ,
[Equation]
where R D [ f 1 , f 2 ] = E x ∼ D (cid:96) ( f 1 ( x ) , f 2 ( x )) , for a loss function (cid:96) : R × R
→
R and a distribution D .
It is not clear that such ﬁtting is possible without further information.
Assume, for example, that there is a natural order on the samples in X B .
A mapping that transforms an input sample x ∈ X A to the sample that is next in order to y AB ( x ) , could be just as feasible.
More generally, one can permute the samples in X A by some function Π that replaces each sample with another sample that has a similar likelihood (see Def. 1 below) and learn h that satisﬁes h = y AB ◦ Π .
We call this difﬁculty “the alignment problem” and our work is dedicated to understanding the plausibility of learning despite this problem.
In multiple recent contributions (Xia et al., 2016; Kim et al., 2017; Zhu et al., 2017; Yi et al., 2017) circularity is employed.
Circularity requires the recovery of both y AB and y BA =
y A ◦ y − 1 B simultaneously.
Namely, functions h and h (cid:48) are learned jointly by minimizing the risk:
[Equation]
where disc C ( D 1 , D 2 )
= sup c 1 ,c 2 ∈C | R D 1 [ c 1 , c 2 ] − R D 2 [ c 1 , c 2 ] | denotes the discrepancy between distributions D 1 and D 2 that is implemented with a GAN (Ganin et al., 2016).
The ﬁrst term in Eq.
3 ensures that the samples generated by mapping domain
A to domain B follow the distribution of samples in domain B .
The second term is the analog term for the mapping in the other direction.
The last two terms ensure that mapping a sample from one domain to the second and back, results in the original sample.
While the circularity constraints, expressed as the last two terms in Eq.
3, are elegant and do not require additional supervision, for every invertible permutation Π of the samples in domain B (not to be confused with a permutation of the vector elements of the representation of samples in B )
we have
[Equation]
Therefore, every circularity preserving pair h and h (cid:48) gives rise to many possible solutions of the form ˜ h
= h ◦ Π and ˜ h (cid:48)
=
Π − 1 ◦ h (cid:48) .
If Π happens to satisfy D B ( x ) ≈ D B (Π( x )) , then the discrepancy terms in Eq.
3 also remain largely unchanged.
Circularity by itself cannot, therefore, explain the recent success of unsupervised mapping.
T HE S IMPLICITY H YPOTHESIS
Despite the availability of a large number of alternative hypotheses h (cid:48) that satisfy the constraints of
Eq. 3, the methods of Xia et al.
; Kim et al.
; Zhu et al.
; Yi et al.
(2017) enjoy empirical success, Why?
Our hypothesis is that the lowest complexity small discrepancy mapping approximates the alignment of the target semantic function.
We further hypothesize that when performing research in unsupervised mapping, goldilock architectures are selected.
These architectures are complex enough to allow small discrepancies but not complex enough to support mappings that are not minimal in complexity.
By doing so, one of the minimal-complexity low-discrepancy mappings is learned.
3. A N I LLUSTRATIVE E XAMPLE
In order to illustrate our hypothesis, we present a very simple toy example, depicted in Fig.
Consider the domain A of uniformly distributed points ( x 1 , x 2 ) (cid:62) ∈ R 2 , where 0 ≤ x 1 < 1 and x 2 = 0 .
Let B be a similar domain, except x 2 = 2 .
We are interested in learning the mapping y 2 DAB (( x 1 , 0 . 5) (cid:62) )
=
( x 1 , 2) (cid:62) .
We note that there are inﬁnitely many mappings from domain A to B that satisfy the constraints of Eq. 3.
However, when we learn the mapping using a neural network with one hidden layer of size 2, and
Leaky ReLU activations 1 (Maas et al., 2013), y 2 DAB is one of only two options.
In this case h ( x ) =
σ a ( Wx + b ) , for W ∈ R 2 × 2 , b ∈ R 2 and where σ a is applied per coordinate.
The only admissible solutions are of the form W b = (cid:18)− 2 b 1 0− 2 b(cid:19) or W (cid:48)
b = (cid:18)
[Equation]
(cid:19) and b = ( b 1 , b 2 ) (cid:62) , which are identical, for every b , to y 2 DAB or to an alternative y 2 D (cid:48) AB (( x 1 , 0 . 5) (cid:62) )
= (1 − x 1 , 2) (cid:62) .
Exactly the same situation holds for any pair of line segments in R d + .
Therefore, by restricting the hypothesis space of h , we eliminate all alternative solutions, except two.
These two are exactly the two mappings that would commonly be considered “more semantic” than any other mapping, and can be expressed as the simplest possible mapping through a shared one dimensional space.
While this is an extreme example, we believe that the principle is general since limiting the complexity of the admissible solutions eliminates the solutions that are derived from y AB by permuting the samples in the space X A , because such mixing requires added complexity.
3. A C OMPLEXITY M EASURE FOR F UNCTIONS
In this work, we focus on functions of the form
[Equation]
here, W 1 , ..., W n +1 are invertible linear transformations from R M to itself.
In addition, σ is a non-linear element-wise activation function.
We will mainly focus on σ that is Leaky ReLU with parameter 0 < a (cid:54) = 1 .
In addition, for any function f , we deﬁne the complexity of f , denoted by
C ( f ) as the minimal number n such that there are invertible linear transformations W 1 , ..., W n + that satisfy f = F [ W n +1 , ..., W 1 ] .
Our function complexity framework, therefore, measures the complexity of a function as the depth of a neural network which implements it, or the shallowest network, if there are multiple such networks.
In other words, we use the number of layers of a network as a proxy for the Kolmogorov complexity of functions, using layers in lieu of the primitives of the universal Turing machines, which is natural for studying functions that can be computed by feedforward neural networks.
Note that capacity is typically controlled by means of norm regularization, which is optimized during training.
Here, the architecture is bounded to a certain number of layers.
This measure of complexity is intuitive and provides a clear and stable stratiﬁcation of functions.
Norm capacity (for norms larger than zero) are not effective in comparing functions of different architectures.
In Sec. 5, we demonstrate that the L1 and L2 norms of the desired mapping are within the range of norms that are obtained when employing bigger or smaller architectures.
Other ways to deﬁne the complexity of functions, such as the VC-dimension (Vapnik & Chervonenkis, 1971b) and
Rademacher complexity (Bartlett & Mendelson, 2003), are not suitable for measuring the complexity of individual functions, since their natural application is in measuring the capacity of classes of functions.
3. C ONSEQUENCES OF THE S IMPLICITY H YPOTHESIS
The simplicity hypothesis leads to concrete predictions, which are veriﬁed in Sec. 5.
The ﬁrst one states that in contrast to the current common wisdom, one can learn a semantically aligned mapping between two spaces without any matching samples and even without circularity.
Prediction 1.
When learning with a small enough network in an unsupervised way a mapping between domains that share common characteristics, the GAN constraint in the target domain is sufﬁcient to obtain a semantically aligned mapping.
The strongest clue that helps identify the alignment of the semantic mapping from the other mappings is the suitable complexity of the network that is learned.
A network with a complexity that is too low cannot replicate the target distribution, when taking inputs in the source domain (high discrepancy).
A network that has a complexity that is too high, would not learn the minimal complexity mapping, since it could be distracted by other alignment solutions.
We believe that the success of the recent methods results from selecting the architecture used in an appropriate way.
For example, DiscoGAN (Kim et al., 2017) employs either eight or ten layers, depending on the dataset.
We make the following prediction:
Prediction 2.
When learning in an unsupervised way a mapping between domains, the complexity of the network needs to be carefully adjusted.
This prediction is also surprising, since in supervised learning, extra depth is not as detrimental, if at all.
As far as we know, this is the ﬁrst time that this clear distinction between supervised and unsupervised learning is made 2 .
3. A LIGNMENT WITH N ON - MINIMAL A RCHITECTURES
If the simplicity hypothesis is correct, then in order to capture the target alignment, one would need to learn with the minimal complexity architecture that supports a small discrepancy.
However, deeper architectures can lead to even smaller discrepancies and to better outcomes.
In order to enjoy both the alignment provided by our hypothesis and the improved output quality, we propose to ﬁnd a function h of a non-minimal complexity k 2 that minimizes the following objective function
[Equation]
where k 1 is the minimal complexity for mapping with low discrepancy between domain A and domain B .
In other words, we suggest to ﬁnd a function h that is both a high complexity mapping from domain A to B and is close to a function of low complexity that has low discrepancy.
There are alternative ways to implement an algorithm that minimizes the objective function presented in Eq. 6.
Assuming, based on this equation, that for h that minimizes the objective function, the corresponding g ∗ = arg inf g s.t C ( g )=
k 1 R D
A [ h, g ] has a (relatively) small discrepancy, leads to a two-step algorithm.
The algorithm ﬁrst ﬁnds a function g that has small complexity and small discrepancy and then ﬁnds h of a larger complexity that is close to g .
This is implemented in Alg. 1.
Note that in the ﬁrst step, k 1 is being estimated, for example, by gradually increasing its value, until g with a discrepancy lower than a threshold (cid:15) 0 is found.
We suggest to use a liberal threshold, since the goal of the network g is to provide alignment and not the lowest possible discrepancy.
C OUNTING M INIMAL C OMPLEXITY M APPINGS
Recall, from Sec. 2, that disc is the discrepancy distance, which is based on the optimal discriminator.
Also discussed were the functions Π , that switches between members in the domain B that have similar probabilities.
These are deﬁned using the discrepancy distance as follows (simpliﬁed version; the deﬁnitions and results of this section are stated more broadly in Appendix A):
Unlabeled training sets S
.i.d
D
Deﬁnition 1 (Density preserving mapping) .
Let X =
( X , D X ) be a domain.
A (cid:15)
-density preserving mapping over X (or an (cid:15) 0 -DPM for short) is a function Π such that
[Equation]
We denote the set of all (cid:15) 0 -DPMs of complexity k by
[Equation]
Below, we deﬁne a similarity relation between functions that reﬂects whether the two are similar.
In this way, we are able to bound the number of different (non-similar) minimal complexity mappings by the number of different DPMs.
Deﬁnition 2.
Let D be a distribution.
We denote f D ∼ (cid:15)
g , if C ( f ) =
C ( g ) and there are minimal decompositions: f = F [ W n +1 , ..., W 1 ] and g = F [ V n +1 , ..., V 1 ] such that: ∀ i ∈ [ n + 1] : disc ( F [ W i , ..., W 1 ] ◦ D, F [ V i , ..., V 1 ]
[Equation]
Put differently, two functions of the same complexity have this relation, if for every step of their processing, the activations of the matching functions are similar.
The deﬁned relation is reﬂexive and symmetric, but not transitive.
Therefore, there are many different ways to partition the space of functions into disjoint subsets such that in each subset, any two functions have the closeness property.
We count the number of functions as the minimal number of subsets required in order to cover the entire space.
This quantity is denoted by N ( U , ∼ U ) where U is the set and ∼ U is the closeness relation.
The formal presentation is in Def. 9, which slightly generalizes the notion of covering numbers (Anthony & Bartlett, 2009).
Informally, the following theorem states that the number of minimal low-discrepancy mappings is upper bounded by both the number of DPMs of size 2 C (cid:15)
A,B over D A and over D B .
This result is useful, since DPMs are expected to be rare in real-world domains.
When imagining mapping a space to itself, in a way that preserves the distribution, one ﬁrst considers symmetries.
Near-perfect symmetries are rare in natural domains, and when these occur, e.g., (Kim et al., 2017), they form wellunderstood ambiguities.
Another option that can be considered is that of replacing speciﬁc samples in domain B with other samples of the same probability.
However, these very local discontinuous mappings are of very high complexity, since this complexity is required for reducing the modeling error for discontinuous functions.
One can also consider replacing larger sub-domains with other sub-domains such that the distribution is preserved.
This could be possible, for example, if the distribution within the sub-domains is almost uniform (unlikely), or if it is estimated inaccurately due to the limitations of the training set.
We, therefore, make the following prediction.
Prediction 3.
The number of DPMs of low complexity is small.
Given two domains A and B , there is a certain complexity C (cid:15) 0 A,B , which is the minimal complexity of the networks needed in order to achieve discrepancy smaller than (cid:15) 0 for mapping the distribution D A to the distribution D B .
The set of minimal complexity mappings, i.e., mappings of complexity C (cid:15)
A,B that achieve (cid:15) 0 discrepancy is denoted by H (cid:15)
( A, B ) :
=
(cid:110) h (cid:12)(cid:12) C ( h ) ≤ C (cid:15)
A,B and disc ( h ◦ D A , D B ) ≤ (cid:15)
(cid:111) .
The following theorem shows that the covering number of this set is similar to the covering number of the DPMs.
Therefore, if predictionabove holds, the number of minimal low-discrepancy mappings is small.
Theorem 1 (Informal) .
Let σ be a Leaky ReLU with parameter 0 < a (cid:54) = 1 and assume identiﬁability.
Let (cid:15) 0 , (cid:15) 1 and (cid:15) 2 < (cid:15) 1 be three positive constants and A =
( X A , D A ) and B = ( X B , D B ) are two domains.
Then,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proof.
See Appendix D.
The theorem assumes identiﬁability.
In the context of neural networks, the general question of uniqueness up to invariants, also known as identiﬁability, is an open question.
Several authors have made progress in this area for different neural network architectures.
The most notable work has been done by Fefferman & Markel (1993) that proves identiﬁability for σ = tanh .
Furthermore, the representation is unique up to some invariants.
Other works (Williamson & Helmke, 1995;
F. Albertini & Maillot, 1993; Kurková & Kainen, 2014; Sussmann, 1992) prove such uniqueness for neural networks with only one hidden layer and various activation functions.
Similarly, in Lem.
3 in the Appendix, we show that identiﬁability holds for Leaky ReLU networks with one hidden layer.
E XPERIMENTS
The ﬁrst group of experiments is dedicated to test the validity of the three predictions made, in order to give further support to the simplicity hypothesis.
Next, we evaluate the success of the proposed algorithm in comparison to the DiscoGAN method of Kim et al.
We chose to experiment with the DiscoGAN architecture since it focuses on semantic tasks that contain a lesser component of texture or style transfer.
The CycleGAN architecture of Zhu et al.
(2017) inherits much from the style transfer architecture of Pix2Pix Isola et al.
(2017), and the discrepancy term is based on a patch-based analysis, which introduces local constraints that could mask the added freedom introduced by adding layers.
In addition, the U-net architecture of Ronneberger et al.
(2015) used by Isola et al.
(2017) deviates from the connectivity pattern of our model.
Experiments in this architecture and with the architecture of DualGAN (Yi et al., 2017), which focuses on tasks similar to CycleGAN, and shares many of the architectural choices, including U-nets and the use of patches, are left for future work.
5. E MPIRICAL V ALIDATION OF T HE P REDICTIONS
Prediction 1 states that since the unsupervised mapping methods are aimed at learning minimal complexity low discrepancy functions, GANs are sufﬁcient.
In the literature (Zhu et al., 2017; Kim et al., 2017), learning a mapping h :
X A → X B , based only on the GAN constraint on B , is presented as a failing baseline.
In (Yi et al., 2017), among many non-semantic mappings obtained by the GAN baseline, one can ﬁnd images of GANs that are successful.
However, this goes unnoticed.
In order to validate the prediction that a purely GAN based solution is viable, we conducted a series of experiments using the DiscoGAN architecture and GAN loss only.
We consider image domains A and B , where X A = X B = R 3 × 64 × 64 .
In DiscoGAN, the generator is build of: (i) an encoder consisting of convolutional layers with 4 ×ﬁlters followed by Leaky ReLU activation units and (ii) a decoder consisting of deconvolutional layers with 4 × 4 ﬁlters followed by a ReLU activation units.
Sigmoid is used for the output layer.
Between four to ﬁve convolutional/deconvolutional layers are used, depending on the domains used in A and B (we match the published code architecture per dataset).
The discriminator is similar to the encoder, but has an additional convolutional layer as the ﬁrst layer and a sigmoid output unit.
The ﬁrst set of experiments considers the CelebA face dataset.
Transformations are learned between the subset of images labeled as male and those labeled as female, as well as from blond to black hair and eyeglasses to no eyeglasses.
The results are shown in Fig.
2, 3, and 4, (resp.).
It is evident that the output image is highly related to the input images.
In the case of mapping handbags to shoes, as seen in Fig. 5, the GAN does not provide a meaningful solution.
However, in the case of edges to shoes and vice versa (Fig. 6), the GAN solution is successful.
In Prediction 2, we predict that the selection of the right number of layers is crucial in unsupervised learning.
Using fewer layers than needed, will not support the modeling of the target alignment between the domains.
In contrast, adding superﬂuous layers would mean that more and more alternative mappings obscure the target transformation.
In (Kim et al., 2017), 8 or 10 layers are employed (counting both convolution and deconvolution) depending on the experiment.
In our experiment, we vary the number of layers and inspect the inﬂuence on the results.
These experiments were done on the CelebA gender conversion task, where 8 layers are employed in the experiments of (Kim et al., 2017).
Using the public implementation and adding and removing layers, we obtain the results in Fig.
Note that since the encoder and the decoder parts of the learned network are symmetrical, the number of layers is always even.
As can be seen, changing the number of layers has a dramatic effect on the results.
The best results are obtained at 6 or 8 layers with 6 having the best alignment and 8 having better discrepancy.
The results degrade quickly, as one deviates from the optimal value.
Using fewer layers, the GAN fails to produce images of the desired class.
Adding layers, the semantic alignment is lost, just as expected.
Note that Kim et al.
(2017) have preferred low discrepancy over alignment in their choice.
In other words, the selected architecture of size k = 8 presents acceptable images at the price of lower alignment compared to an architecture of size k − 2 .
This is probably a result of ambiguity that is already present at the size k architecture.
On the other hand, the smaller architecture of size k −does not produce images of extremely low discrepancy, and there is no architecture that beneﬁts both, an extremely low discrepancy and high alignment.
This is observed for example in Fig.
7 where females are translated to males.
For 4 layers the discrepancy is too low and the mapping fails to produce images of males.
For 6 layers, the discrepancy is relatively low and the alignment is at its highest.
For 8 layers, the discrepancy is at its lowest value, nevertheless, the alignment is worse.
While our discrete notion of complexity seems to be highly related to the quality of the results, the norm of the weights do not seem to point to a clear architecture, as shown in Tab. 2(a).
Since the table compares the norms of architectures of different sizes, we also approximated the functions using networks of a ﬁxed depth k = 18 and then measured the norm.
These results are presented in
Tab. 2(b).
In both cases, the optimal depth, which is 6 or 8, does not appear to have a be an optimum in any of the measurements.
Prediction 3 states that there are only a handful of DPMs, except for the identity function.
In order to verify it, we trained a DiscoGAN from a distribution A to itself with an added loss of the form − (cid:80) x ∈
A | x − h ( x ) | .
In our experiment, testing network complexities from 2 to 12, we could not ﬁnd a DPM, see Fig.
15 and Tab. 3.
For lower complexities, the identity was learned despite the added loss.
For higher complexities, the network learned the identity while changing the background color.
For even higher complexities, other mapping emerged.
However, these mappings did not satisfy the circularity constraint, and are unlikely to be DPMs.
5. R ESULTS FOR A LG .
The goal of Alg. 1 is to ﬁnd a well-aligned solution with higher complexity than the minimal solution and potentially smaller discrepancy.
It has two stages.
In the ﬁrst one, k 1 , which is the minimal complexity that leads to a low discrepancy, is identiﬁed.
This follows a set of experiments that are similar to the one that is captured, for example, by Fig.
To demonstrate robustness, we select a single value of k 1 across all experiments.
Speciﬁcally, we use k 1 = 6 , which, as discussed above, typically leads to a low (but not very low) discrepancy, while the alignment is still unambiguous.
Once g is trained, we proceed to the next step of optimizing a second network of complexity k 2 .
Note that while the ﬁrst function ( g ) uses the complete DiscoGAN architecture, the second network ( h ) only employs a one-directional mapping, since alignment is obtained by g .
Figs.
20– 28 depict the obtained results, for a varying number of layers.
First, the result obtained by the DiscoGAN method with k 1 is displayed.
The results of applying Alg.
1 are then displayed for a varying k 2 .
As can be seen, our algorithm leads to more sophisticated mappings.
Kim et al.
(2017) have noted that their solutions are, at many times, related to texture or style transfer and, for example, geometric transformations are not well captured.
The new method is able to better capture such complex transformations.
Consider the case of mapping male to female in Fig.
19, ﬁrst row.
A man with a beard is mapped to a female image.
While for g the beard is still somewhat present, it is not so for h with k 2 > k 1 .
On the female to male mappings in Fig. 20 it is evident in most mappings that g produces a more blurred image, while h is more coherent for k 2 > k 1 .
Another example is in the blond to black hair mapping in Fig.
In the 5th row, the style transfer nature of g is evident, since it maps a red object behind the head together with the whole blond hair, producing an unrealistic black hair.
h of complexity k 2 = 8 is able to separate that object from the hair, and in k 2 > 8 it produces realistic looking black hair.
This kind of transformation requires more than a simple style transfer.
On the edges to shoes and edges to handbags mappings of Fig. 25 and Fig. 27, while the general structure is clearly present, it is signiﬁcantly sharpened by mapping h with k 2 > k 1 .
For the face datasets, we also employ face descriptors in order to learn whether the mapping is semantic.
Namely, we can check if the identity is preserved post mapping by comparing the VGG face descriptors of Parkhi et al. (2015).
One can assume that two images that match will have many similar features and so the VGG representation will be similar.
The cosine similarities are used, as is commonly done.
In addition, we train a linear classiﬁer in the space of the VGG face descriptors in order to distinguish between Male/Female, Eyeglasses/No-eyeglasses, and Blond/Black.
This way, we can check, beyond discrepancy, that the mapping indeed transforms between the domains.
The training samples in domains A and B are used to train this classiﬁer, which is then applied to a set of test images before and after mapping, measuring the accuracy.
The higher the accuracy, the better the separation.
Tab. 4 presents the results for both the k 1 layers network g , alternative networks g of higher complexity (shown as baseline only), and the network h trained using Alg. 1.
We expect the alignment of g to be best at complexity k 1 , and worse due to the loss of discrepancy for alternative network g with complexity k > k 1 .
We expect this loss of alignment to be resolved for networks h trained with
Alg. 1.
In the experiments of black to blond hair and blond to black hair mappings, we note that h with k 2 = 8 has the best descriptor similarity, and very good separation accuracy and discrepancy.
Higher values of k 2 are best in terms of separation accuracy and discrepancy, but lose somewhat in descriptor similarity.
A similar situation occurs for male to female and female to male mappings and in eyeglasses to non-eyeglasses, where k 2 = 8 results in the best similarity score and higher values of k 2 result in better separation accuracy and discrepancy.
It is interesting to note, that the distance between g and h is also minimal for k 2 = 8 .
Perhaps, with more effective optimization, higher complexities could also maintain similarity, while delivering lower discrepancies.
D ISCUSSION
Our stratiﬁed complexity model is related to structural risk minimization by Vapnik & Chervonenkis (1971a), which employs a hierarchy of nested subsets of hypothesis classes in order of increasing complexity.
In our stratiﬁcation, which is based on the number of layers, the complexity classes are not necessarily nested.
We point to a key difference between supervised learning and unsupervised learning.
While in the former, deeper networks, which can learn even random labels, work well (Zhang et al., 2017), unsupervised learning requires a careful control of the network capacity.
This realization, which echoes the application of MDL for model selection in unsupervised learning (Zemel, 1994), was overshadowed by the overgeneralized belief that deeper networks lead to higher accuracy.
The limitations of unsupervised based learning that are due to symmetry, are also a part of our model.
For example, the mapping of cars in one pose to cars in the mirrored pose that sometimes happens in (Kim et al., 2017), is similar in nature to the mapping of x to 1 − x in the simple example given in
Sec. 3.1.
Such symmetries occur when we can divide y AB into two functions y AB =
y 2 ◦ y 1 such that a function W is a linear mapping and also a DPM of y 1 ◦ D A and, therefore, D B ≈ y 2 ◦
W ◦ y 1 .
While we focus on unsupervised learning, the emergence of semantics when learning with a restricted capacity is widely applicable, such as with autoencoders, transfer learning, semi-supervised learning and elsewhere.
As an extreme example, Sutskever et al.
present empirical evidence that a meaningful mapper can be learned, even from very few examples, if the network trained is kept small.
C ONCLUSION
The recent success in mapping between two domains in an unsupervised way and without any existing knowledge, other than network hyperparameters, is nothing less than extraordinary and has far reaching consequences.
As far as we know, nothing in the existing machine learning or cognitive science literature suggests that this would be possible.
We provide an intuitive deﬁnition of function complexity and employ it in order to identify minimal complexity mappings, which we conjecture play a pivotal role in this success.
If our hypothesis is correct, simply by training networks that are not too complex, the target mapping stands out from all other alternative mappings.
Our analysis leads directly to a new unsupervised cross domain mapping algorithm that is able to avoid the ambiguity of such mapping, yet enjoy the expressiveness of deep neural networks.
The experiments demonstrate that the analogies become richer in details and more complex, while maintaining the alignment.
We show that the number of low-discrepancy mappings that are of low-complexity is expected to be small.
Our main proof is based on the assumption of identiﬁability, which constitutes an open question.
We hope that there would be a renewed interest in this problem, which has been open for decades for networks with more than a single hidden layer and is unexplored for modern activation functions.
A A G ENERALIZED AND F ORMAL S TATEMENT OF THE R ESULTS
For brevity, we have not presented our results in the most general way.
For example, in Def. 1, we did not bound the complexity of the discriminators.
For the same reason, some of our terms were described and not yet formally deﬁned.
A. A C OMPLEXITY M EASURE FOR F UNCTIONS
In order to model the composition of neural networks, we deﬁne a complexity measurement that assigns a value based on the number of simple functions that make up a complex function.
Deﬁnition 3 (Stratiﬁed complexity model (SCM)) .
A stratiﬁed complexity model N := SCM [ C ] is a hypothesis class of functions p : R M → R M speciﬁed by a set of functions C .
Every function p in N has an appropriate decomposition:
• N =
(cid:83) ∞ n =0 C n (where, C n = { p n ◦ ... ◦ p 1 | p 1 , ..., p n ∈ C} and C
= { Id } ).
• Every function in C is invertible.
Informally, a SCM partitions a set of invertible functions into disjoint complexity classes ,
[Equation]
When considering simple functions p i that are layers in a neural network, each complexity class contains the functions that are implemented by networks of n hidden layers.
In addition, we denote the complexity of a function p :
C (
[Equation] [Equation]
[Equation]
If the complexity of a function p equals n , then any appropriate decomposition p = p n ◦ ... ◦ p
[Equation]
will be called a minimal decomposition of p .
According to this measurement, the complexity of a function p is determined by the minimal number of primitive functions required in order to represent it.
In this work, we focus our attention on SCMs that represent the architectures of fully connected neural networks with layers of a ﬁxed size, i.e.,
Deﬁnition 4 (NN-SCM) .
A NN-SCM is a SCM N = SCM [ C ] that satisﬁes the following conditions:
• C =
(cid:110) W 2 ◦ σ ◦ W 1 (cid:12)(cid:12)(cid:12) W 1 , W 2 ∈ R M × M and W 1 , W 2 are invertible (cid:111) .
Here, W 1 , W 2 denote both linear transformations and the associated matrix forms.
• σ is a non-linear element-wise activation function.
For brevity, we denote N := SCM [ σ ] to refer to a NN-SCM with the activation function σ .
The NN-SCM with the Leaky ReLU activation function is of a particular interest, since (Kim et al., 2017; Zhu et al., 2017) employ it as the main activation function (plain ReLUs and tanh are also used).
In the NN-SCM framework, to specify the function obtained by a decomposition
W n ◦ σ ◦ W n
[Equation]
◦ σ ◦ ... ◦ σ ◦ W 1 we simply write:
[Equation]
It is useful to characterize the effect of inversion on the complexity of functions, since, for example, we consider both h (cid:48)
=
Π ◦ h and h
= Π − 1 ◦ h (cid:48) .
The following lemma states that, in the case of
NN-SCM with σ that is the Leaky ReLU, the complexity of the inverse function is the same as that of the original function.
Lemma 1.
Let N = SCM
[ σ ] be a NN-SCM with σ that is the Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Then, for any u ∈ N , C ( u − 1 ) =
C ( u ) .
Proof.
First, we denote C (cid:48) ( p ) the minimal number n such that there are invertible linear mappings
W 1 , ..., W n +1 such that p = F [ W n +1 , ..., W 1 ] (if p = Id then C (cid:48) ( p ) = 0 ).
This complexity measure is similar to the complexity measure C .
For a function p such that C ( p ) (cid:54) = 0 , we have,
C ( p ) =
C (cid:48) ( p ) .
Nevertheless, for p such that C ( p ) = 0 , it is not necessarily true that C (cid:48) ( p ) = 0 .
For example, if p (cid:54) =
Id is an invertible linear mapping, we have, C ( p )
= 0 and C (cid:48) ( p )
= 2 .
Let p =
F [ W
[Equation] [Equation]
] =
[Equation]
2 ◦ σ ◦ W 1 be any function such that C ( p )
=
We consider that:
[Equation]
Therefore,
[Equation] [Equation]
In particular, C (cid:48)
( p − 1 ) ≤ 1 .
If C (cid:48)
[Equation] [Equation]
, then, Id = − W − 1◦ σ ◦ − W − 1 2 /a and, therefore, σ is a linear mapping - in contradiction.
Thus, C (cid:48) ( p − 1 ) = 1 .
Next, we would like to show that for any u ∈ N , C (cid:48)
( u − 1 ) =
C (cid:48) ( u ) .
Let u such that C (cid:48) ( u ) = 0 .
Then, u = u − 1 = Id and therefore, C (cid:48)
( u − 1 )
= 0 .
Let u = F [ W n +1 , ..., W 1 ] be a function such that
C (cid:48) ( u ) = n > 0 .
Then,
[Equation] [Equation]
In particular,
[Equation]
or,
[Equation]
Therefore, by Lem. 7,
[Equation]
On the other hand, if v = u − 1 , n = C (cid:48) ( u ) =
C (cid:48) ( v − 1 ) ≤ C (cid:48) ( v ) =
C (cid:48) ( u − 1 ) ≤ n and C (cid:48) ( u − 1 ) =
C (cid:48) ( u ) .
Finally, we would like to show that for every u ∈ N , we have: C ( u − 1 ) =
C ( u ) .
If C ( u ) = 0 , then, by Lem.
11, C ( u − 1 ) = 0 .
On the other hand, if C ( u ) (cid:54) = 0 , then, by Lem. 11, C ( u − 1 ) (cid:54) = 0 and by the above: C ( u ) =
C
[Equation]
= C (cid:48)
( u − 1 )
=
C ( u ) .
A. M INIMAL C OMPLEXITY M APPINGS
Based on our simplicity hypothesis, we present a deﬁnition of a minimal complexity mapping that is both intuitive and well-deﬁned in concrete complexity terms.
Given two distributions D A and D B , a minimal complexity mapping f : X A → X B between domains A and B is a mapping that has minimal complexity among the functions h :
[Equation]
Consider, again, the example of a line segment in R M (Sec. 3.1) and the semantic space of the interval, [0 , 1] ⊂ R .
The two linear mappings, which map either segment ends to 0 and the other toare minimal, when using f that are ReLU based neural networks.
Other mappings to this segment are possible, simply by permuting points on the segment in R M .
However, these alternative mappings have higher complexity, since the two mappings above are the only ones with the minimal possible complexity.
In order to measure the distance between h ◦ D A and D B , we use the discrepancy distance, disc D .
In this work, we focus on classes of discriminators D of the form D m := { u | C ( u ) ≤ m } for some m ∈ N .
In addition, for simplicity, we will write disc m :=
disc D m .
Deﬁnition 5 (Minimal complexity mappings) .
Let N =
SCM
[ C ] .
Let A =
( X A , D A ) and B = ( X B , D B ) be two domains.
We deﬁne the ( m, (cid:15) 0 ) -minimal
complexity between A and B as:
[Equation]
The set of ( m, (cid:15) 0 ) -minimal complexity mappings between A and B is:
[Equation]
We note that for any ﬁxed (cid:15) 0 > 0 , the sequence { C m,(cid:15) 0 A,B } ∞ m =0 is monotonically increasing as m tends to inﬁnity.
In addition, we assume that for every two distributions of interest, D I and D J , and an error rate (cid:15) 0 > 0 , there is a function h of ﬁnite complexity such that disc ∞ ( h ◦ D I , D J ) ≤ (cid:15) 0 .
Therefore, the sequence { C m,(cid:15)
A,B } ∞ m =0 is upper bounded by C ( h ) for all m ∈ N ∪ { 0 } .
In particular, there is a minimal value m 0 >
such that C m,(cid:15)
A,B = C m 0 ,(cid:15)
A,B for all m ≥ m 0 .
We denote: E (cid:15)
A,B := m 0 and
C (cid:15)
A,B := C m 0 ,(cid:15)
A,B .
For simplicity, sometimes we will assume that m = ∞ .
In this case, we will write
H (cid:15)
( A, B ) :
= H (cid:15)
( A, B ; ∞ ) .
A. I DENTIFIABILITY
Every neural network implementation gives rise to many alternative implementations by performing simple operations, such as permuting the units of any hidden layer, and then permuting back as part of the linear mapping in the next layer.
Therefore, it is ﬁrst required to identify and address the set of transformations that could be inconsequential to the function which the network computes.
Deﬁnition 6 (Invariant set) .
Let N = SCM [ σ ] be a NN-SCM.
The invariant set Invariant ( N ) is the set of all τ : R M → R M that satisfy the following conditions:
• τ : R M → R M is an invertible linear transformation.
• σ ◦ τ = τ ◦ σ .
Functions in Invariant ( N ) are called invariants or invariant functions.
For example, for neural networks with the tanh activation function, the set of invariant functions contains the linear transformations that take vectors, permute them and multiply each coordinate by ± 1 .
Formally, each τ =
[ (cid:15) 1 · e π (1) , ..., (cid:15) M · e π ( M ) ] where e i is the i ’th standard basis vector, π is a permutation over [ M ] and (cid:15) i ∈ {± 1 } (Fefferman & Markel, 1993).
In the following lemma, we characterize the set of all invariant functions for σ that is Leaky ReLU with parameter 0 < a (cid:54) = 1 .
Lemma 2.
Let N = SCM [ σ ] with σ be Leaky ReLU with parameter 0 < a (cid:54) = 1 .
Then,
Invariant ( N ) =
(cid:110)
τ ∈ R M × M (cid:12)(cid:12)(cid:12) τ
=
[ c 1 · e π (1) , ..., c M · e π ( M ) ] , where ∀ i ∈ [ M ] : c i > 0 and π ∈ Sym M (cid:111) (20)
Here, e i denotes the i ’th standard basis vector in R M and Sym M is the set of permutations of [ M ] .
Proof.
Let τ be an invertible linear mapping satisfying σ ◦ τ = τ ◦ σ .
We consider that for all i ∈
[ M ] and vector x ; σ ( (cid:104) τ i , x (cid:105) )
= (cid:104)
τ i , σ ( x ) (cid:105) , where τ i is the i ’th row of τ and τ i,j is the ( i, j ) entry of τ .
For x = e j , we have:
[Equation]
i,
[Equation]
i,j ) (21)
For x = − e j , we have:
[Equation]
If τ i,j < 0 , then the ﬁrst equation leads to contradiction.
Otherwise, the equations are both satisﬁed.
Finally, for x = e j − e k , we have:
[Equation]
[Equation]
If τ i,j − τ i,k = 0 , then, τ i,j − aτ i,k =
and since a (cid:54) = 1 , 0 , we have, τ i,j =
τ i,k = 0 .
If τ i,j − τ i,k ≥ 0 , then, τ i,j − τ i,k
= τ i,j − aτ i,k that gives τ i,k = 0 .
If τ i,j − τ i,k ≤ 0 , then, a ( τ i,j − τ i,k ) =
τ i,j − aτ i,k that yields τ
i,j = 0 .
Therefore, for each i ∈
[ M ] there is at most one entry
τ i,j that is not 0 .
If for all j ∈ [ M ] , τ i,j = 0 , then the mapping τ is not invertible, in contradiction.
Therefore, for each i ∈
[ M ] there is exactly one entry
τ i,j > 0 (it is non-negative as shown above).
Finally, if there are i 1 (cid:54) =
isuch that τ i 1 ,j , τ i 2 ,j (cid:54) =
then the matrix is invertible.
Therefore, τ is a member of the set deﬁned in
Eq. 20.
In addition, it is easy to see that every member of the noted set satisﬁes the conditions of the invariant set.
Thus, we obtain the desired equation.
Our analysis is made much simpler, if every function has one invariant representation up to a sequence of manipulations using invariant functions that do not change the essence of the processing at each layer.
Assumption 1 (Identiﬁability) .
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Then, every function p ∈ N is identiﬁable (with respect to Invariant ( N ) ), i.e., for any two minimal decompositions, p = F [ W n +1 , ..., W 1 ]
=
F [ V n +1 , ..., V 1 ] , there are invariants τ 1 , ..., τ n ∈ Invariant ( N ) such that:
[Equation]
Uniqueness up to invariants, also known as identiﬁability, forms an open question.
Fefferman &
Markel (1993) proved identiﬁability for the tanh activation function.
Other works (Williamson &
Helmke, 1995; F. Albertini & Maillot, 1993; Kurková & Kainen, 2014; Sussmann, 1992) prove such uniqueness for neural networks with only one hidden layer and various classical activation functions.
In the following lemma, we show that identiﬁability holds for Leaky ReLU networks with only one hidden layer.
Lemma 3.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Any function p such that C ( p ) = 1 is identiﬁable, i.e, if p = F [ W 2 , W 1 ] =
F [ V 2 , V 1 ] , then, W 1 = τ ◦ V 1 and
W 2 =
V 2 ◦ τ − 1 for some τ ∈ Invariant ( N ) .
Proof.
An alternative representation of the equation is:
[Equation]
We would like to prove that if σ ◦ U
= V ◦
σ then V = U .
We have:
[Equation]
In particular, if v i is the i ’th row of V (similarly u i ) and x = e j :
[Equation]
where v i,j is the ( i, j ) entry of V (similarly u i,j ).
Similarly, for x = − e j :
[Equation]
If u i,j is negative, we obtain: au i,j = v i,j (the ﬁrst equation) and − u i,j = − av i,j (the second equation) that yields a = 1 in contradiction.
Therefore, u i,j ≥ 0 and u i,j = v
i,j (the second equation).
We conclude that W 1 ◦ V − 1= W − 1◦ V 2 := τ .
Finally, since ( σ ◦ W 1 ◦ V − 1 1 ) =
( W − 1◦ V 2 ◦ σ ) we have σ ◦ τ = τ ◦ σ
and τ is invertible linear mapping.
Differently said, W 1 =
τ ◦ V 1 and
W 2 =
V 2 ◦ τ − 1 such that τ ∈ Invariant ( N ) .
As far as we know, there are no other results continuing the identiﬁability line of work for activation functions such as Leaky ReLU.
Uniqueness, which is stronger than identiﬁability, since it means that even multiple representations with different number of layers do not exist, does not hold for these activation functions.
To see this, note that for every M × M invertible linear mapping W , the following holds:
[Equation]
29) where σ is the Leaky ReLU activation function with parameter a .
We conjecture that for networks with Leaky ReLU activations identiﬁability holds, or at least for networks with a ﬁxed number of neurons per layer.
In addition to identiﬁability, we make the following assumption, which states that almost all mappings are non-degenerate.
Assumption 2.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Assume that the set of ( W 1 , ..., W n +1 ) ∈ R M × M × m such that C ( F [ W n +1 , ..., W 1 ])
=
n is dense in R M × M × m .
A. C OUNTING M INIMAL C OMPLEXITY M APPINGS
In the unsupervised alignment problem, the algorithms are provided with only two unmatched datasets of samples from the domains A and B and the task is to learn a well-aligned function between them.
Since we hypothesize that the alignment of the target mapping is typically captured by the lowest complexity low-discrepancy mapping, we develop the machinery needed in order to show that such mappings are rare.
Recall that disc m is the discrepancy distance for discriminators of complexity up to m .
In Sec. 2, we have discussed the functions Π which replaces between members in the domain B that have similar probabilities.
Formally, these are deﬁned using the discrepancy distance.
Deﬁnition 7 (Density preserving mapping) .
Let N =
SCM
[ C ] and X =
( X , D X ) a domain.
A ( m, (cid:15) 0 ) -density preserving mapping over X (or an ( m, (cid:15) 0 ) -DPM for short) is a function Π such that
[Equation]
We denote the set of all ( m, (cid:15) 0 ) -DPMs of complexity k by DPM (cid:15)
( X
[Equation]
m
[Equation]
k ) :
= (cid:8) Π (cid:12)(cid:12) disc m (Π
[Equation]
D
[Equation]
and
[Equation]
We would like to bound the number of mappings that are both low-discrepancy and low-complexity by the number of DPMs.
We consider that there are inﬁnitely many DPMs.
For example, if we slightly perturb the weights of a minimal representation of a DPM, Π , we obtain a new DPM.
Therefore, we deﬁne a similarity relation between functions that reﬂects whether the two are similar.
In this way, we are able to bound the number of different (non-similar) minimal-complexity mappings by the number of different DPMs.
Deﬁnition 8 (Closeness between pairs of distributions or functions) .
Let N = SCM [ σ ] .
• We denote D 1 ∼ m,(cid:15)
D 2 ⇐⇒ disc m
( D 1 , D 2 ) ≤ (cid:15) 0 .
• We denote f D ∼ m,(cid:15)
g , if C ( f ) =
C ( g ) =: n and there are minimal decompositions: f = F [ W n +1 , ..., W 1 ] and g = F [ V n +1 , ..., V 1 ] such that ∀ i ∈
[ n +1] : F [ W i , ..., W 1 ] ◦ D ∼ m,(cid:15)F [ V i , ..., V 1 ] ◦ D .
The deﬁned relation is reﬂexive and symmetric, but not transitive.
Therefore, there are many different ways to partition the space of functions into disjoint subsets such that in each subset, any two functions are similar.
We count the number of functions up to the similarity as the minimal number of subsets required in order to cover the entire space.
This idea is presented in Def. 9, which slightly generalizes the notion of covering numbers (Anthony & Bartlett, 2009).
Deﬁnition 9 (Covering number) .
Let ( U , ∼ U ) be a set and a reﬂexive and symmetric relation.
A covering of (
U , ∼ U ) , is a tuple ( U , ≡ U ) such that: ≡ U is an equivalence relation and u 1 ≡ U u 2 = ⇒
u 1 ∼ U u 2 .
The covering number of ( U , ∼ U ) , denoted by N ( U , ∼ U ) , is:
[Equation]
U / ≡ U (cid:12)(cid:12) s.t: the minimum is taken over ( U , ≡ U ) that is a covering of ( U , ∼ U ) (31)
Here, U / ≡ U is the quotient set of U by ≡ U .
Thm. 1 below states that the number of low discrepancy mappings of complexity C (cid:15)
A,B is upper bounded by the number of DPMs of size 2 C (cid:15)
A,B .
By prediction 3, the number of such DPMs is small.
The theorem employs the following weak assumption.
In Lem.
19, we prove that this assumption holds for the case of a continuous risk if the discriminators have bounded weights.
Assumption 3.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
For every m > 0 (possibly ∞ ) and n > 0 , the function disc m ( F [ W n , ..., W 1 ] ◦ D 1 , D 2 ) is continuous as a function of the weights of W 1 , ..., W n ∈ R M × M .
Theorem 1.
Let N = SCM
[ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54) = 1 .
Assume Assumptions 1, 2 and 3.
Let (cid:15) 0 , (cid:15) 1 and (cid:15) 2 be three constants such that (cid:15) 0 < (cid:15) 1 / 4 and (cid:15) 2 < (cid:15)− 4 (cid:15) 0 be three positive constants and A =
( X A , D A ) and B = ( X B , D B ) are two domains.
Then,
[Equation]
Proof.
See Sec.
D.
B S UMMARY OF N OTATION
Tab. 5 lists the symbols used in our work.
[Table]
The sample spaces of A and B (
[Table] [Table]
over
[Table]
Two domains; Speciﬁed by ( X A , D A ) and ( X B , D B ) (resp.)
[Table]
Functions from the feature space to the domains, y A : X → X A and
[Table]
a feature
[Table]
The risk function R D [ f 1 , f 2 ] =
E x ∼ D (cid:96) (
[Table]
1 ( x ) , f 2 ( x )) where (cid:96) is a loss function and D is a distribution
[Table]
The discrepancy between two distributions D 1 and D 2 , i.e,
[Table]
A non-
[Table]
-wise activation function
[Table]
A class of functions; in most cases C = { W 2 ◦ σ ◦ W 1 | W 1 , W 2 ∈ R M × M are invertible linear transformation }
[Equation]
A SCM speciﬁed by a class of functions C (see Def. 3)
N = SCM [ σ ]
A NN-SCM speciﬁed by the activation function σ (see Def. 4)
C ( p )
The complexity of a function p (see Eqs
[Table] [Table]
The invariant set of N (
[Table] [Table]
An invariant function (
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The ( m, (cid:15) 0 ) -minimal
complexity between A and B (see Def. 5)
[Table]
[Equation]
[Table]
C
[Table] [Table]
= max m
[Table]
C m
[Table]
and
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The set of ( (cid:15) 0 , m ) -minimal complexity mappings between A and B (see
Def. 5)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
N ( U
[Equation]
∼
[Equation]
The covering number of U with respect to relation ∼ U on U (see Def.9) X : ← x x is assigned to X
C L EMMAS
In this section, we prove various lemmas that are used in the proof of Thm. 1.
In Sec.
C.1 we present the assumptions taken in various lemmas in the appendix.
In Sec.
C.2 we prove useful inequalities involving the discrepancy distance.
Sec.
C.3 provides lemmas concerning the deﬁned complexity measure and invariant functions.
The lemmas in Sec.
C.4 concern the properties of inverse functions.
C. A SSUMPTIONS
We list the assumptions employed in our proofs.
Assumptions 1 and 2 were already presented and are heavily used.
Assumptions 3 and its relaxation 4 are mild assumptions that were taken for convenience.
Assumption 1 (Identiﬁability) .
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Then, every function p ∈ N is identiﬁable (with respect to Invariant ( N ) ), i.e., for any two minimal decompositions, p = F [ W n +1 , ..., W 1 ]
=
F [ V n +1 , ..., V 1 ] , there are invariants τ 1 , ..., τ n ∈ Invariant ( N ) such that:
[Equation]
Assumption 2.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Assume that the set of ( W 1 , ..., W n +1 ) ∈ R M × M × m such that C ( F [ W n +1 , ..., W 1 ])
=
n is dense in R M × M × m .
Assumption 3.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
For every m > 0 (possibly ∞ ) and n > 0 , the function disc m ( F [ W n , ..., W 1 ] ◦ D 1 , D 2 ) is continuous as a function of the weights of W 1 , ..., W n ∈ R M × M .
In the case that the norm of the discriminator is bounded, Lem 19, it follows from the following assumption, which is well-justiﬁed, (cf.
Shalev-Shwartz & Ben-David (2014), page 162, Eq.14.13).
Assumption 4.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
For all m > 0 , the function R D [ F [ V m , ..., V 1 ] , F [ W m , ..., W 1 ]] is continuous as a function of
V m , ..., V 1 , W m , ..., W 1 .
C. P ROPERTIES OF D ISCREPANCIES
Lemma 4.
Let D 1 and D 2 be two classes of functions and D 1 , D 2 two distributions.
Assume that
D 1 ◦ { p } ⊂ D 2 , then,
[Equation]
In particular, if m ≥ k + C ( p ) , then,
[Equation] [Equation]
Proof.
By the deﬁnition of discrepancy:
[Equation] [Equation]
Since D 1 ◦ { p } ⊂ D 2 we have:
[Equation]
The second inequality is a special case for D 1 = D k and D
= D m .
Lemma 5.
Let A
=
( X 1 , D 1 ) and B =
( X 2 , D 2 ) be two domains and D Z a distribution.
Assume that m ≥ k + C ( p ) .
Then,
[Equation]
Let y 1 , y 2 and y =
y 2 ◦ y − 1 1 be three functions and m ≥ k + C
( y 2 ) .
Then, disc k
y ◦ D 1 , D 2 ) ≤
[Equation]
m
[Equation]
D
Z
, y
[Equation] [Equation]
D
[Equation]
Let h be any function and m ≥ k + C
( h − 1 ) .
Then,
[Equation]
Proof.
Follows from Lem.
4, since m ≥ k + C ( p ) , we have:
[Equation]
Therefore, by the triangle inequality,
[Equation]
We use Lem.
4 with p : ← y 2 , D 1 : ← D k , and D 2 : ← D m and D k ◦ { y 2 }
[Equation]
D
[Equation]
Therefore
[Equation]
by the triangle inequality,
[Equation]
Follows
immediately from Lem.
4 for
C. P ROPERTIES OF THE C OMPLEXITY M EASURE AND I NVARIANTS
Lemma 6.
Let N = SCM
[ C ] .
In addition, let u, v be any two functions.
Then,
[Equation]
Proof.
We begin with the case C ( v ) = 0 .
In this case, C ( u ◦ v ) =
C ( u ) =
C ( u ) + C ( v ) .
By deﬁnition, C ( v ) = 0 implies that C ( v − 1 )
=
and C ( u ◦ v ) =
C ( u ) − C ( v − 1 ) .
Finally,
C ( u ) − C ( v − 1 ) =
C ( u ) =
C ( u ◦ v ) .
The case C ( u ) = 0 is analogous.
Next, we assume that
C ( u ) = n > 0 and C ( v ) = m > 0 .
Let u =
u n ◦ ... ◦
u 1 and v = v m ◦ ... ◦ v 1 be minimal decompositions of u and v (resp.).
Therefore, we can represent, u ◦ v = u n ◦ ... ◦
u 1 ◦ v m ◦ ... ◦ v 1 .
In particular, C ( u ◦ v ) ≤ n + m =
C ( u ) + C ( v ) .
The lower bound follows immediately from the upper bound:
[Equation]
[Equation]
By similar considerations, we also have: C ( v ) − C ( u −
[Equation]
≤ C ( u ◦ v ) .
For a given function u ∈ N = SCM [ C ] , we deﬁne,
[Equation] [Equation]
Lemma 7.
Let N =
SCM
[ C ] .
In addition, let u, v be any two functions.
Then,
[Equation] [Equation]
Proof.
We begin by proving the upper bound.
We assume C (cid:48) ( u ) = n and C (cid:48) ( v ) = m .
Let u = u n ◦ ... ◦
u 1 and v = v m ◦ ... ◦ v 1 be minimal decompositions of u and v (resp.).
Therefore, we can represent, u
[Equation]
In particular,
[Equation]
The lower bound follows immediately from the upper bound:
[Equation]
[Equation]
By similar considerations, C (cid:48) ( v
[Equation]
−
[Equation]
u
[Equation]
Lemma 8.
Invariant ( N ) is closed under inverse and composition, i.e,
[Equation]
And,
[Equation]
Proof.
Inverse: Let
τ ∈
Invariant ( N ) .
Then, by deﬁnition, τ is an invertible linear mapping and τ ◦ σ = σ ◦ τ .
In particular, τ − 1 is also an invertible linear mapping and τ − 1 ◦ σ = σ ◦ τ − 1 .
Thus, τ − 1 ∈ Invariant ( N ) .
Composition: Let τ 1 , τ 2 ∈ Invariant ( N ) .
Then, τ i is an invertible linear mapping and τ i ◦ σ = σ ◦
τ
i for i = 1 , 2 .
In particular, τ◦ τ 2 is also an invertible linear mapping and τ 1 ◦ τ 2 ◦ σ = τ 1 ◦ σ ◦ τ 2 =
σ ◦ τ 1 ◦ τ 2 .
Thus, τ 1 ◦ τ 2 ∈ Invariant ( N ) .
Lemma 9.
Let N = SCM [ σ ] with σ that is Leaky ReLU with 0 < a (cid:54) = 1 .
Assume that p obeys identiﬁability, i.e., that Assumption 1 holds.
Then, for any two minimal decompositions p = F [ W n +1 , ..., W 1 ] =
F [ V n +1 , ..., V 1 ] , we have:
[Equation]
Proof.
We prove that F [ W i , ..., W 1 ] ◦ F
[ V i , ..., V 1 ] − 1 ∈ Invariant ( N ) .
If i = n + 1 , then,
F [ W i , ..., W 1 ] ◦ F
[ V i , ..., V 1 ]
[Equation]
Id ∈ Invariant ( N ) .
Otherwise, by minimal identiﬁability,
[Equation]
In addition,
[Equation]
Since each for all k ∈ [ i ] , τ k commutes with σ , we have,
[Equation]
and
[Equation]
By similar considerations, F [ W n +1 , ..., W i ] ◦ F [ V n +1 , ..., V i ] − 1 ∈ Invariant ( N ) .
Lemma 10.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Then, every invertible linear mapping W is a member of C 0 .
Proof.
Let p ∈ C n .
Then, p = F [ W n +1 , ..., W 1 ] , for invertible linear mappings W 1 , ..., W n +1 .
In particular, W ◦ p = F [ W · W n +1 , ..., W 1 ] ∈ C n , p ◦ W = F [ W n +1 , ..., W 1 · W ] ∈ C n and similarly, W − 1 ◦ p, p ◦ W − 1 ∈ C n .
Therefore, W ∈ C 0 .
Lemma 11.
C 0 is closed under inverse and composition, i.e,
[Equation] [Equation]
and,
[Equation] [Equation]
Proof.
Inverse:
By deﬁnition, u ∈ C 0 iff for all n ∈ N and q ∈ C n , we have: u ◦ q, q ◦ u, u − 1 ◦ q, q ◦ u − 1 ∈ C n iff u − 1 ∈ C 0 .
Decomposition: Let f ∈ C n .
Then, g = u 1 ◦ f ∈ C n and u 2 ◦ u 1 ◦ f = u 2 ◦ g ∈ C n .
Similarly, u − 1 1 ◦ u − 1 2 ◦ f, f ◦ u − 1 1 ◦ u − 1 2 , f ◦ u 2 ◦ u 1 ∈ C n .
C. P ROPERTIES OF I NVERSES
Lemma 12.
Let N = SCM [ σ ] where σ is the Leaky ReLU activation function, with parameter 0 < a (cid:54)
= 1 .
Let f = F [ W n +1 , ..., W 1 ] be a minimal decomposition.
Then, for all i ∈ [ n ] , we have:
[Equation]
Proof.
We prove this statement by induction on i from i = n backwards to i = 1 .
Case i = n : Then, F [ W − 1 i +1 /a, ..., − W − 1 n +1 /a ]
= F [ − W − 1 n +1 /a ]
[Equation]
1 n +1 /a .
In addition,
[Equation]
Induction hypothesis: We assume that:
[Equation]
Case i − 1 : We consider that by the induction hypothesis:
[Equation]
Finally, we conclude that:
[Equation]
Lemma 13.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Then, for all (cid:15) 0 > 0 , we have C (cid:15) 0 B,A = C (cid:15)
A,B .
Proof.
Let k ≥ (cid:110) E (cid:15)
A,B , E (cid:15) 0 B,A (cid:111) and m ≥ k + C (cid:15) 0 B,A .
We take y ∈ H (cid:15)
( A, B ; m ) .
Then, C ( y ) =
C (cid:15)
A,B .
In addition, disc m ( y ◦ D A , D B ) ≤ (cid:15) 0 .
By the third part of Lem.
5, for h : ← y , we have:
[Equation]
[Equation]
In particular, C (cid:15) 0 B,A ≤ C ( y − 1 ) .
In addition, by Lem. 1, C ( y − 1 ) =
C ( y ) .
Therefore, C (cid:15) 0 B,A ≤ C (cid:15)
A,B .
By symmetric arguments (switching between A and B ) we also have the opposite side and thus,
[Equation]
D P ROOF OF T HM . 1 AND I TS G ENERALIZATION T HM .D. C OVERING N UMBERS
Deﬁnition 10 (Set embedding) .
Let ( U , ∼ U ) and ( V , ∼ V ) be two tuples of sets and symmetric and reﬂexive relations on them (resp.).
A function G :
U → V is an embedding of ( U , ∼ U ) in ( V , ∼ V ) and we denote (
U , ∼ U ) (cid:22) ( V , ∼ V ) if:
[Equation]
Lemma 14.
Let ( U , ∼ U ) and ( V , ∼ V ) be two tuples of sets and reﬂexive and symmetric relations on them (resp.).
If (
U , ∼ U ) (cid:22) ( V , ∼ V ) then N ( U , ∼ U ) ≤ N ( V , ∼ V ) .
Proof.
Assume that ( U , ∼ U ) (cid:22) ( V , ∼ V ) .
Then, by deﬁnition, there is an embedding function G :
U → V such that:
[Equation]
Let ( V , ≡ V ) be a covering of ( V , ∼ V ) .
We deﬁne a covering ( U , ≡ U ) of ( U , ∼ U ) as follows:
[Equation]
Part 1: We would like to prove that ( U , ≡ U ) is a covering of ( U , ∼ U ) .
It is easy to see that ≡ U is an equivalence relation since ≡ V is an equivalence relation.
Next, we would like to prove that u 1 ≡ U u 2 = ⇒
u
1 ∼ U u 2 .
By the deﬁnition of ≡ U :
[Equation]
In addition, since ( V , ≡ V ) is a covering of ( V , ∼ V ) :
[Equation]
Finally, since G is an embedding:
[Equation]
We conclude:
[Equation]
Therefore, ( U , ≡ U ) is indeed a covering of ( U , ∼ U ) .
Part 2: We would like to prove that |U / ≡ U | ≤ |V / ≡ V | .
Let u 1 , u 2 ∈ U such that u 1 (cid:54)≡ U u 2 .
Then, by deﬁnition of ≡ U we have: G ( u 1 ) (cid:54)≡ V G ( u 2 ) .
Therefore, if we take u 1 , ..., u n ∈ U representations of n different equivalence classes in (
U , ≡ U ) then, G ( u 1 ) , ..., G ( u n ) ∈
V are n representations of n different equivalence classes in (
V , ≡ V ) .
In particular, |U / ≡ U | ≤ |V / ≡ V | .
Therefore, the covering number of (
U , ∼ U ) is at most the covering number of ( V , ∼ V ) .
Lemma 15.
Let ( U , ≡ 1 ) and ( U , ≡ 2 ) be two coverings of ( U , ∼ U ) .
Then, ( U 2 , ≡ 1 × ≡ 2 ) is a covering of (
U 2 , ∼ 2 U ) .
Where U 2 =
U × U and the relation ∼ 2 U is deﬁned as follows:
[Equation]
and ≡ 1 × ≡ 2 is deﬁned as:
[Equation] [Equation]
Proof.
We have to prove that ≡ 1 × ≡ 2 is an equivalence relation and that ( u 1 , u 2 ) ≡ 1 × ≡ 2 ( v 1 , v 2 ) =
⇒ ( u 1 , u 2 ) ∼ 2 U ( v 1 , v 2 ) .
Reﬂexivity:
[Equation]
The RHS is true since ≡ 1 and ≡ 2 are reﬂexive relations.
Symmetry:
[Equation]
Since ≡ 1 and ≡ 2 are symmetric, we have:
[Equation]
In addition, ( v 1 , v
[Equation] [Equation]
and v
[Equation]
Therefore,
[Equation]
Transitivity: follows from similar arguments.
Covering:
[Equation]
Since ( U , ≡ i ) is a covering of ( U , ∼ U ) , for i
= 1 , 2 , we have: u 1 ≡ 1 v 1 and u 2 ≡ 2 v 2 = ⇒
u
1 ∼ U v
[Equation]
and u 2 ∼ U v
[Equation]
By the deﬁnition of ∼ 2 U we have:
[Equation]
Therefore,
[Equation] [Equation]
Lemma 16.
Let ( U , ∼ U ) be a tuple of a set and a reﬂexive and symmetric relation on it (resp.).
Then,
[Equation]
Proof.
Let ≡ U be an equivalence relation such that ( U , ≡ U ) is a covering of ( U , ∼ U ) .
By Lem.
U 2 , ≡ 2 U ) is a covering of ( U 2 , ∼ 2 U ) .
In addition,
[Equation]
Thus, for every covering ( U , ≡ U ) of ( U , ∼ U ) , there is a covering of ( U 2 , ∼ 2 U ) of size |U / ≡ U | 2 .
In particular, N (
U 2 , ∼
[Equation]
≤ N ( U , ∼
[Equation] [Equation]
Lemma 17.
Let ( U , ∼ U ) be a tuple of a set and a reﬂexive and symmetric relation on it (resp.).
Then,
[Equation] [Equation]
Proof.
We deﬁne an embedding from ( U , ∼ U ) to ( U 2 , ∼ 2 U ) as follows F ( u ) =
( u, u ) .
This is an embedding, because, F ( u )
[Equation]
Lemma 18.
Let ( U , ∼ U ) and ( V , ∼ V ) be two tuples of sets and reﬂexive and symmetric relations on them (resp.).
Assume that U ⊂ V and ∼ U :
=
( ∼ V ) (cid:12)(cid:12) U , i.e, ∀ u, v ∈ U
[Equation] [Equation]
Then,
[Equation]
Proof.
Let ( V , ≡ V ) be a covering of ( V , ∼ V ) .
Then, it is easy to see that ( U , ≡ U ) is a covering of ( U , ∼ U ) , where ≡ U :=
( ≡ V ) (cid:12)(cid:12) U .
In addition, we have: |U / ≡ U | ≤ |V / ≡ V | .
Thus, for every covering of (
V , ∼ V ) , we can ﬁnd a smaller covering for ( U , ∼ U ) .
In particular, N ( U , ∼ U ) ≤ N (
V , ∼ V ) .
D
P ERTURBATIONS AND D ISCREPANCY
Thm.
1 employs assumption 3.
In Lem.
19 we prove that this assumption holds for the case of a continuous risk (assumption 4) if the discriminators have bounded weights.
Assumption 3.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
For every m > 0 (possibly ∞ ) and n > 0 , the function disc m ( F [ W n , ..., W 1 ] ◦ D 1 , D 2 ) is continuous as a function of the weights of W
[Equation]
Assumption 4.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
For all m > 0 , the function R D [ F [ V m , ..., V 1 ] , F [ W m , ..., W 1 ]] is continuous as a function of
V m , ..., V 1 , W m , ..., W 1 .
Lemma 19.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 and assume
Assumption 4 for D : ← D 1 .
Let disc m,E :=
disc C m,E for
[Equation] [Equation]
Then, for all m > 0 , n > 0 and E > 0 , the function disc m,E ( F [ W n , ..., W 1 ] ◦ D 1 , D 2 ) is continuous as a function of W n , ..., W 1 .
Proof.
Let W n , ..., W 1 and W kn , ..., W k 1 be any invertible matrices in R M × M such that for all i ∈ [ n ] ,
W ki
→ W i .
We denote G E = (cid:8)
W ∈ R M
× M (cid:12)(cid:12) ||
W || ≤ E (cid:9) .
By the triangle inequality, disc m,E ( D 1 ,
[Equation] [Equation]
Similarly,
[Equation]
therefore
[Equation]
|
[Equation]
In particular,
[Equation]
[Table]
91) Assume by contradiction that the last expression does not converge to 0 .
Therefore
[Table]
there is a sequence ( V k 1 , ..., V km , U k 1 , ..., U km ) such that V k 1 , .., V km , U k 1 , ..., U km ∈ G E and
Q
[Table]
= (cid:12)(cid:12)(cid:12)
[Equation]
[Equation] [Equation]
In particular, there is some (cid:15) > 0 and an increasing sequence { k j } ∞ j =1 ⊂ N such that Q k j > (cid:15) for all j ∈ N .
With no loss of generality, we can assume that k j =
j
(otherwise, we replace the original sequence with the new one).
Since ( V k j 1 , ..., V k j m , U k j 1 , ..., U k j m ) ∈ G 2 mE and G 2 mE is compact in R M × M × 2 m , by the Bolzano-Weierstrass theorem, it has a converging subsequence.
With no loss of generality, we can assume that ( V k j 1 , ..., V k j m , U k j 1 , ..., U k j m ) converges (otherwise, we replace it with a converging sub-sequence):
[Equation]
In particular,
[Equation]
By Assumption
[Equation]
the function R D 1 [ F [ X m + n , ...
[Equation]
X 1 ]
[Equation]
F [ Y m + n , ...
[Equation]
Y 1 ]] is continuous.
Therefore,
[Equation]
and,
[Equation]
Therefore, by the triangle inequality,
[Equation]
in contradiction.
Thus, we conclude that:
[Equation]
Lemma 20.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
In addition, let f = F [ W n +1 , ..., W 1 ] and g = F [ V n +1 , ..., V 1 ] be two minimal decompositions.
Assume Assumptions 2 and 3.
Then, there are functions ¯ f = F [ ¯ W n +1 , ..., ¯ W 1 ] and ¯ g = F [ ¯ V n +1 , ..., ¯ V 1 ] such that:
• C ( ¯ f ◦ g ) =
2 n .
• ∀ j ∈ [ n + 1] : disc m ( F [ ¯ W j , ..., ¯ W 1 ] ◦ D, F [ W j , ..., W 1 ] ◦ D ) ≤ (cid:15) .
• ∀ j ∈ [ n + 1] : disc m
( F [ ¯ V j , ..., ¯ V 1 ] ◦ D, F [ V j , ..., V 1 ] ◦ D ) ≤ (cid:15) .
Proof.
We consider that f ◦ g = F [ W n +1 , ..., W 2 , W 1 · V n +1 , V n , ..., V 1 ] .
By Assumption 2, for each δ > 0 , there are ¯ f = F [ ¯ W n +1 , ..., ¯ W 1 ] and ¯ g = F [ ¯ V n +1 , ..., ¯ V 1 ] such that C ( ¯ f ◦ ¯ g ) =
2 n and for all j ∈ [ n + 1] :
|| ¯ W j − W j || ,
||
¯ V j − V j || ≤ δ .
By Assumption.
3, for each (cid:15) > 0 , there is a small enough δ > 0 such that: ¯ f = F [ ¯ W n +1 , ..., ¯ W 1 ] and ¯ g = F [ ¯ V n +1 , ..., ¯ V 1 ] such that for all j ∈ [ n + 1] : || ¯ W j − W j || , ||
¯ V j − V j || ≤ δ , we have:
• ∀ j ∈ [ n + 1] : disc m ( F [ ¯ W j , ..., ¯ W 1 ] ◦ D, F [ W j , ..., W 1 ] ◦ D ) ≤ (cid:15) .
• ∀ j ∈ [ n + 1] : disc m
( F [ ¯ V j , ..., ¯ V 1 ] ◦ D, F [ V j , ..., V 1 ] ◦ D ) ≤ (cid:15) .
In particular, for any (cid:15) > 0 , there are functions ¯ f = F [ ¯ W n +1 , ..., ¯ W 1 ] and ¯ g = F [ ¯ V n +1 , ..., ¯ V 1 ] with the desired properties.
Lemma 21.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Assume
Assumption 1.
Let f D ∼ m,(cid:15) g .
Then, for every minimal decomposition f = F [ W (cid:48) n +1 , ..., W (cid:48) 1 ] there is a minimal decomposition g = F [ V (cid:48) n +1 , ..., V (cid:48) 1 ] such that:
[Equation]
Proof.
Since f
D ∼ m,(cid:15) g there are minimal decompositions f = F [ W n +1 , ..., W 1 ] and g =
F [ V n +1 , ..., V 1 ] such that:
[Equation] [Equation]
By Assumption 1, W (cid:48) 1 = τ 1 ◦ W 1 , for all i = 2 , ..., n : W (cid:48) i
= τ i ◦ W
i ◦ τ − 1 i − 1 and W (cid:48) n +1 =
W n +1 ◦ τ − 1 n .
Therefore, we deﬁne a minimal decomposition for g as follows: g = F [ V (cid:48) n +1 , ..., V (cid:48) 1 ] such that
V (cid:48)
= τ 1 ◦ V 1 , for all i
= 2 , ..., n : V (cid:48)
i =
τ i ◦ V
i ◦ τ − 1 i − 1 and V (cid:48) n +1 =
V n +1 ◦ τ − 1 n .
This is a minimal decomposition of g , since each invariant function is an invertible linear mapping and commutes with σ .
By Lem.
9 we have:
[Equation]
Therefore, by Lem. 4, since C ( τ i ) = 0 , we have:
[Equation]
Alternatively,
[Equation]
Since F [ W (cid:48) n +1 , ..., W (cid:48) 1 ]
= f = F [ W n +1 , ..., W 1 ] and F [ V (cid:48) n +1 , ..., V (cid:48) 1 ]
= g = F [ V n +1 , ..., V 1 ] we also have F
[ W
[Equation]
Lemma 22.
Let N = SCM [ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
We have:
[Equation]
Proof.
Assume by contradiction that ¯ f
[Equation]
¯ f (cid:48) .
Then, there are decompositions ¯ f =
F [ ¯ W n +1 , ..., ¯ W 1 ] and ¯ f (cid:48) =
F [ ¯ W (cid:48) n +1 , ..., ¯ W (cid:48) 1 ] such that:
[Equation] [Equation]
F [
[Equation] [Equation]
By Lem.
[Equation]
since ¯ f D A ∼ k,(cid:15) f and ¯ f (cid:48) D A ∼ k,(cid:15) f (cid:48)
[Equation]
there are minimal decompositions f = F [ W n +1 , ..., W 1 ] and f (cid:48)
=
F [ W (cid:48) n +1 , ..., W (cid:48) 1 ] such that:
[Equation]
Since f D A (cid:54)∼ k,(cid:15) 1 f (cid:48)
[Equation]
there is an index i ∈ [ n + 1] such that:
[Equation]
Therefore
[Equation]
by the triangle inequality
[Equation]
we arrive to a contradiction:
[Equation]
Lemma 23.
Let N = SCM [ σ ] with σ that is a Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Let
A =
( X A , D A ) and B = ( X B , D B ) are two domains.
We have:
[Equation]
Proof.
We would like to show that the function G ( h )
=
h − 1 is an embedding of (cid:18)
H
[Equation]
[Equation] [Equation]
∼
[Equation]
(cid:19) into
[Equation]
First, we consider that if h ∈ H (cid:15)
( A, B ) , then,
[Equation]
≤
[Equation]
(110) and by Lem.
1 and Lem.
13, C ( G ( h ))
=
C ( h − 1 ) =
C ( h ) =
C (cid:15)
A,B = C (cid:15)
B,A .
Therefore, G ( h ) ∈
H (cid:15)
( B, A ) .
Next, we would like to prove that for all h 1 , h 2 ∈ H (cid:15)
( A, B ) : G ( h 1 ) D B ∼ (cid:15) 1 G ( h 2 ) =
⇒ h
1 D A ∼ (cid:15) 1 +2 (cid:15)
h 2 .
Let h 1 , h 2 ∈ H (cid:15)
( A, B ) such that G ( h 1 ) D B ∼ (cid:15) 1 G ( h 2 ) .
Then, there are minimal decompositions
G ( h 1 ) =
F [ W n +1 , ..., W 1 ] and G ( h 2 ) =
F [ V n +1 , ..., V 1 ] such that:
[Equation]
i
[Equation]
[ n ] :
[Equation] [Equation]
i , ...,
[Equation]
and:
[Equation]
We consider that by Lem.
1, G ( h 1 ) =
F [ − W − 1 1 , W − 1 2 /a, ..., W − 1 n /a, − W − 1 n +1 /a ] and G ( h 2 ) =
F [ − V − 1 1 , V − 1 2 /a, ..., V − 1 n /a, − V − 1 n +1 /a ] are minimal decompositions.
In addition, by Lem.
12, we have:
[Equation]
By the ﬁrst item of Lem. 5, for D 1 := D B , D 2 :=
h 1 ◦ D A , D 3 :=
F [ V − 1 i +1 /a, ...
V − 1 n /a, − V − 1 n +1 /a ] ◦
D B and p :=
F [ W − 1 i +1 /
[Equation]
Similarly (by the ﬁrst item of Lem. 5)
[Equation]
we have:
[Equation]
≤ disc ( F [ W i , ..., W 1 ] ◦ D A , F
[ V i , ...,
[Equation] [Equation]
Therefore, we conclude that h 1 D A ∼ (cid:15) 1 +2 (cid:15)
h 2 .
D. P ROOF OF T HM .
Theorem 2.
Let N = SCM [ σ ] with σ that is a Leaky ReLU with parameter 0 < a (cid:54)
= 1 .
Assume
Assumptions 1, 2 and 3.
Let (cid:15) 0 , (cid:15) 1 and (cid:15) 2 such that (cid:15) 0 < (cid:15) 1 / 2 and (cid:15) 2 < (cid:15) 1 − 2 (cid:15) 0 be three positive constants and
A =
( X A , D A ) and B = ( X B , D B ) are two domains.
Assume that m ≥ k +2 C (cid:15)
A,B +2 .
Then,
N
[Equation]
Proof.
Let (cid:15) be any positive constant such that: (cid:15) < min { ( (cid:15) 1 − 2 (cid:15) 0 − (cid:15) 2 ) / 4 , (cid:15) 2 / 2 } .
For such (cid:15) , we have 2 (cid:15) 0 ≤ (cid:15) 1 − 4 (cid:15) and (cid:15) 2 ≤ (cid:15) 1 − 2 (cid:15) 0 − 4 (cid:15) .
In addition, let t :
= k + C (cid:15)
A,B + 1 .
We would like to ﬁnd an embedding mapping:
[Equation] [Equation]
Part 1:
In this part, we show how to construct G .
Let ( f, g ) ∈
( H (cid:15)
( A, B ; m )) 2 .
We denote: f = F [ W n +1 , ..., W 1 ] and g = F [ V n +1 , ..., V 1 ] minimal decompositions of f and g (resp.).
By
Lem.
20, there are functions ¯ f = F [ ¯ W n +1 , ..., ¯ W 1 ] and ¯ g = F [ ¯ V n +1 , ..., ¯ V 1 ] such that:
• C ( ¯ f ◦ ¯ g − 1 ) =
2 n .
• ∀ j ∈ [ n + 1] : disc m ( F [ ¯ W j , ..., ¯ W 1 ] ◦ D A , F [ W j , ..., W 1 ] ◦ D A ) ≤ (cid:15) .
• ∀ j ∈ [ n + 1] : disc m
( F [ ¯ V j , ..., ¯ V 1 ] ◦ D A , F
[ V j , ..., V 1 ] ◦ D A ) ≤ (cid:15) .
We deﬁne G ( f, g ) =
¯ f ◦ ¯ g − 1 .
Part 2:
In this part, we show that:
[Equation]
By Part 1 , C ( ¯ f ◦ ¯ g − 1 ) =
2 n = 2 C (cid:15) 0 A,B .
In addition, by the ﬁrst item of Lem. 5, for D 1 : ← ¯ g − 1 ◦ D B ,
D 2 : ← D A , D 3 : ← D B , p : ← ¯ f , t ≥ k + C (cid:15)
A,B we have:
[Equation]
Since f ∈ H (cid:15)
( A, B ; m ) : disc t ( ¯ f ◦ D A , D
[Equation]
) ≤ disc m ( f
[Equation] [Equation]
disc
[Equation]
( ¯ f
[Equation]
(cid:15)
[Equation]
(cid:15) (119)
In addition, by the third item of Lem. 5, for h : ← ¯ g and m ≥ t + C (cid:15) 0 A,B ≥ t + C
(¯ g − 1 ) , we have:
[Equation] [Equation]
A
[Equation] [Equation]
[Equation]
Finally, disc k
( ¯ f ◦ ¯ g − 1 ◦ D B , D B ) ≤ 2 (cid:15)
+ 2 (cid:15)
and we conclude that:
[Equation]
Part 3:
In this part, we show that G is an embedding.
It requires showing that
[Equation]
Assume by contradiction that G ( f, g ) D B ∼ m,(cid:15) 2 G ( f (cid:48) , g (cid:48) ) and that ( f, g ) D A (cid:54)∼ k,(cid:15)
( f (cid:48) , g (cid:48) ) .
Then, we have f D A
[Equation]
or g
D A (cid:54)∼
[Equation]
We denote G ( f, g ) =
¯ f ◦ ¯ g − 1 and G ( f (cid:48) , g (cid:48) )
=
¯ f (cid:48) ◦
(¯ g (cid:48) )
− 1 (see Part 1 ).
Assume that f D A
[Equation]
: By Lem.
[Equation]
¯ f
[Equation]
A
[Equation] [Equation]
[Equation] [Equation]
¯ f
[Equation]
In particular, for every two decompositions:
[Equation]
and
[Equation] [Equation]
[Equation]
there is an index i ∈ [ n + 1] such that:
disc k
[Equation]
The option i = n + 1 is not a possibility, since:
[Equation]
By the ﬁrst item of Lem. 5, for D 1 : ← D A , D 2 : ← ¯ g − 1 ◦ D B , D 3 : ← F [ ¯ W (cid:48)
i , ..., ¯ W (cid:48) 1 ] ◦ D A , p : ← F [ ¯ W i , ..., ¯ W 1 ] and t ≥ k + C (cid:15)
A,B ≥ k + C ( F [ ¯ W i , ..., ¯ W 1 ]) , we have:
[Equation]
Again, by the ﬁrst item of Lem. 5, for D 1 : ← D A , D 2 : ← ( g (cid:48) ) − 1 ◦ D B , D 3 : ← F [ ¯ W i , ..., ¯ W 1 ] ◦ g − 1 ◦ D B , p : ← F [ ¯ W (cid:48)
i , ..., ¯ W (cid:48) 1 ] and m ≥ t + C (cid:15)
A,B ≥ t + C
( F [ ¯ W (cid:48)
i , ..., ¯ W (cid:48) 1 ]) , we have:
[Equation]
t
[Equation]
F [
[Equation] [Equation]
Therefore, we conclude that:
[Equation]
Alternatively, for any minimal decompositions
[Equation]
) − 1 there are right partial functions F [ ¯
[Equation]
[Equation] [Equation]
and F [ ¯
[Equation] [Equation]
.., ¯ W
[Equation]
◦
[Equation]
) − 1 such that:
[Equation]
in contradiction to F ( f, g ) D B ∼ m,
[Equation]
2 F
[Equation]
f
[Equation]
, g
[Equation]
Assume that g D A (cid:54)∼ k,(cid:15) 1 g (cid:48) :
By Lem.
22, ¯ g
[Equation] [Equation]
(cid:15) ¯ g (cid:48) .
Let ¯ g
[Equation] [Equation]
¯
[Equation]
and (
[Equation]
(cid:48)
[Equation]
131) be any two minimal decompositions of ¯ g − 1 and (¯ g (cid:48) )
− 1 (resp.).
Then, by Lem. 12, there are minimal decompositions ¯ g = F [ ¯ V n +1 , ..., ¯ V 1 ] and ¯ g (cid:48)
=
F [ ¯ V (cid:48) n +1 , ..., ¯ V (cid:48) 1 ] such that:
[Equation] [Equation]
¯
[Equation] [Equation]
F [¯ V j , ..., ¯ V 1 ] ◦ D
[Equation]
and: F [(¯ V (cid:48)
[Equation]
+
[Equation]
− 1 /a, ..., (¯ V (cid:48) n ) − 1 /a, − (¯
[Equation]
a ] ◦ ¯ g
[Equation]
, ..., ¯
[Equation]
Since ¯ g
D A (cid:54)∼ k,(cid:15) 1 − 2 (cid:15) ¯ g (cid:48) , there is an index i ∈ [ n + 1] such that:
[Equation]
The case i = n + 1 is not a possibility, similarly to Eq. 126.
Therefore, there is i ∈ [ n ] such that Eq.
133 holds.
In addition,
[Equation]
By Lem.
4, for p : ← − 1 /a · σ of complexity 1 we have:
[Equation]
In addition
[Equation]
by Lem.
[Equation]
for D
[Equation]
A,
[Equation]
≥ ( k + 1) + C ( F [ ¯ V − 1 i +1 /a, ..., ¯ V − 1 n /a, − ¯ V
[Equation]
1 n +1 /a ]) , we have:
[Equation]
[Table]
we
[Table]
[Equation]
Finally,
[Equation] [Equation]
In particular
[Equation]
Alternatively, for any minimal decompositions
[Equation]
there are right partial functions
[Equation] [Equation]
such that Eq.
139 holds, in contradiction to F ( f, g ) D B ∼ m,(cid:15) 2 F ( f (cid:48)
[Equation]
g
[Equation]
Part 3: Finally, by Lem. 17 and Lem. 14,
[Equation]
Alternatively, for all (cid:15) 0 , (cid:15) 1 , (cid:15) 2 , (cid:15) such that (cid:15) < min { ( (cid:15) 1 − 2 (cid:15) 0 − (cid:15) 2 ) / 4 , (cid:15) 2 /
[Equation]
[Equation]
In particular, we can replace (cid:15) with (cid:15)/ 2 in the inequality.
By Lem.
18, the function q (cid:15) =
N (cid:18)
[Equation]
is monotonically decreasing as (cid:15) tends to 0 and is lower bounded by N
[Equation]
Therefore, by the monotone convergence theorem, the limit lim (cid:15) → 0 q
[Equation]
exists and upper bounds N
[Equation]
Theorem 1.
Let N = SCM
[ σ ] with σ that is Leaky ReLU with parameter 0 < a (cid:54) = 1 .
Assume Assumptions 1, 2 and 3.
Let (cid:15) 0 , (cid:15) 1 and (cid:15) 2 be three constants such that (cid:15) 0 < (cid:15) 1 / 4 and (cid:15) 2 < (cid:15)−
[Equation]
0 be three positive constants and A =
( X A , D A ) and B = ( X B , D B ) are two domains.
Then,
[Equation]
Proof.
By Lem. 2, with m = k = ∞ , we have:
[Equation]
Similarly,
[Equation]
By Lem.
N
[Equation]
Since the limits in the RHS of Eqs.
144 and 145 are limits of positive integers, we have:
[Equation]
[Table]

[Equation] [Table]