N ONPARAMETRIC L EARNING OF T WO -L AYER R E LU R ESIDUAL U NITS
Anonymous authors
Paper under double-blind review
A BSTRACT
We describe an algorithm that learns two-layer residual units using rectiﬁed linear unit (ReLU) activation: suppose the input x is from a distribution with support space R d and the ground-truth generative model is a residual unit of this type, given by y
= B ∗ (cid:104)
( A ∗ x ) + + x (cid:105) , where ground-truth network parameters
A ∗ ∈ R
d × d represent a nonnegative full-rank matrix and B ∗ ∈ R m × d is full-rank with m ≥ d and for c ∈ R d , [ c + ] i = max { 0 , c i } .
We design layer-wise objectives as functionals whose analytic minimizers express the exact ground-truth network in terms of its parameters and nonlinearities.
Following this objective landscape, learning residual units from ﬁnite samples can be formulated using convex optimization of a nonparametric function: for each layer, we ﬁrst formulate the corresponding empirical risk minimization (ERM) as a positive semi-deﬁnite quadratic program (QP), then we show the solution space of the QP can be equivalently determined by a set of linear inequalities, which can then be efﬁciently solved by linear programming (LP).
We further prove the statistical strong consistency of our algorithm, and demonstrate its robustness and sample efﬁciency through experimental results.
I NTRODUCTION
Neural networks have achieved remarkable success in various ﬁelds such as computer vision (LeCun et al., 1998; Krizhevsky et al., 2012;
He et al., 2016a) and natural language processing (Kim, 2014;
Sutskever et al., 2014).
This success is largely due to the strong expressive power of neural networks (Bengio & Delalleau, 2011), where nonlinear activation units, such as rectiﬁed linear units (ReLU) (Nair & Hinton, 2010) and hyperbolic tangents ( tanh ) play a vital role to ensure the large learning capacity of the networks (Maas et al., 2013).
However, the nonlinearity of neural networks makes them signiﬁcantly more difﬁcult to train than linear models (Livni et al., 2014).
Therefore, with the development of neural network applications, ﬁnding efﬁcient algorithms with provable properties to train such nontrivial neural networks has become an important and a relatively new goal.
A
ReLU
B
Residual networks, or ResNets (He et al., 2016a), are a class of deep neural networks that adopt skip connections to feed values between nonadjacent layers, where skipped layers may contain nonlinearities in between.
Without loss of expressivity, ResNets avoid the vanishing gradient problem by directly passing gradient information from previous layers to current layers where otherwise gradients might vanish without skipping.
In practice,
ResNets have shown strong learning efﬁciency in several tasks, e.g. achieving at least 93% test accuracy on CIFAR-10 classiﬁcation, lowering single-crop error to 20 .
1% on the 1000 -class
ImageNet dataset (Russakovsky et al., 2015; He et al., 2016b).
Common ResNets are often aggregated by many repeated shallow networks, where each network acts as a minimal unit with this kind of skip propagation, named a residual unit (He et al., 2016b).
Given the ﬂexibility and simplicity of residual units, much theoretical work has been devoted to study them and develop training algorithms for them in a way that sidesteps from the standard backpropagation regime and provides guarantees on the quality of estimation (see subsection 1.1).
In this paper, we propose algorithms that can learn a general class of single-skip twolayer residual units with ReLU activation as shown on the right by the equation:
y = B (cid:104)
( A x ) + + x (cid:105) , where for a scalar c , c + = max { 0 , c } (for a vector, this maximization is applied coordinate-wise), x
Compared to previous work (Ge et al., 2018; 2017; Zhang et al., 2018; Wu et al., 2019; Tian, 2017;
Du et al., 2017; Brutzkus & Globerson, 2017; Soltanolkotabi, 2017; Li & Yuan, 2017; Zhong et al., 2017), the introduction of residual connections simpliﬁes the recovery of the network parameters by removing the permutation and scaling invariance.
However, the naive mean square error minimization used in estimating the parameters remains nonconvex.
Unlike most previous work, we do not assume a speciﬁc input distribution nor that the distribution is symmetric (Ge et al., 2018; Du & Goel, 2018).
We show that under regularity conditions on the weights of the residual unit, the problem of learning the unit can be formulated through quadratic programming (QP).
We use nonparametric estimation (Guntuboyina et al., 2018) to estimate the ReLU function values in the networks.
We further rewrite our constructed quadratic programs to linear programs (LPs).
The LP formulation is simpler to optimize and has the same solution space as the QP for the network parameters.
1. R ELATED W ORK
Provable learning of neural networks has become an important topic of research.
Arora et al.
(2014) recover a multi-layer generative network with sparse connections and Livni et al.
(2014) study the learning of multi-layer neural networks with polynomial activation.
Goel et al.
(2018) learns a one-layer convolution network with a perceptron-like rule.
They prove the correctness of an iterative algorithm for exact recovery of the target network.
Rabusseau et al.
(2019) describe a spectral algorithm for two-layer linear networks.
Recent work has connected optimization and two-layer network learning (Ergen & Pilanci, 2021; Sahiner et al., 2020; Ergen & Pilanci, 2020; Pilanci &
Ergen, 2020) and has showed how to optimize networks layer by layer (Belilovsky et al., 2019).
For learning a one-layer ReLU network, Wu et al.
(2019) optimize the norm and direction of neural network weight vectors separately, and Zhang et al.
(2018) use gradient descent with a speciﬁc initialization.
For learning a two-layer ReLU network, Ge et al.
(2017) redesign the optimization landscape such that it is more amenable to theoretical analysis and Ge et al.
(2018) use a momentbased method for estimating a neural network Janzamin et al.
Many others have studied ReLU networks in various settings (Tian, 2017; Du et al., 2017; Brutzkus & Globerson, 2017; Soltanolkotabi, 2017; Li & Yuan, 2017; Zhong et al., 2017; Goel & Klivans, 2017).
The study of ReLU networks with two hidden layers has also been gaining attention (Goel & Klivans, 2017; Allen-Zhu et al., 2019), with a focus on PAC learning (Valiant, 1984).
In relation to our work,
Allen-Zhu & Li (2019) examined the PAC-learnable function of a speciﬁc three-layer neural network with residual connections.
Their work differs from ours in two aspects.
First, their learnable functions include a smaller (in comparison to the student network) three-layer residual network.
Second, the assumptions they make on their three-layer model are rather different than ours.
In relation to nonparametric estimation, Guntuboyina et al.
(2018) treat the ﬁnal output of a shaperestricted regressor as a parameter, placing some restrictions on the type of function that can be estimated (such as convexity).
They provide solutions for estimation with isotonic regression (Brunk, 1955;
Ayer et al., 1955; van Eeden, 1956), convex regression (Seijo et al., 2011), shape-restricted additive models (Meyer, 2013; Chen & Samworth, 2016) and shape-restricted single index models (Kakade et al., 2011; Kuchibhotla et al., 2017).
1. M AIN R ESULTS
We design quadratic objective functionals with a linear bounded feasible domain that take network parameters and functions as variables to estimate the ground-truth network parameters and nonlinearities.
The values of the objectives are moments over the input distribution.
Thm.
1.1 summarizes the landscapes of the objectives.
Theorem 1.1 (objective landscape, informal) .
Suppose a ground-truth residual unit has nondegenerate weights in both layers and nonnegative weights in layer 1.
Then there exist quadratic functionals deﬁned in linear-constrained domains whose minimizers a ) are unique, and are the exact ground-truth, or b ) are not unique, but can be adjusted to the exact ground-truth.
In practice, the exact moments over unobserved variables are not available.
We can only construct the empirical risk minimization (ERM) of the moment-valued objectives by generated samples.
With functions as variables in moment-valued objectives being optimized nonparametrically, the empirical objectives become quadratic functions with linear constraints (QP).
We further show the convexity of our QP which guarantees its solution in polynomial time w.r.t.
sample size and dimension.
With the solution to the QP available, the strong consistency of our network learner is guaranteed:
Theorem 1.2 (strong consistency, informal) .
Suppose samples are generated by a ground-truth residual unit that has nondegenerate weights in both layers and nonnegative weights in layer 1.
Then there exists an algorithm that learns a network a.s. −−→ the exact ground-truth as sample size grows.
Roadmap: Assume A and B are student network weights of layer 1 and 2.
We ﬁrst give a warm-up vanilla linear regression approach only knowing A is in R d × d and B is in R m × d (section 3).
We then move to the details of our nonparametric learning for layer 2 (section 4), and similarly, layer 1 (section 5).
We formalize Thm. 1.1, showing how nonparametric learning allows us to select the values of A and B from reduced spaces that are seeded by A ∗ and B ∗ , under which LR is faster than vanilla LR.
In section 6, we describe the strong consistency of our methods for respective layers, formalizing Thm.
1.2 using the continuous mapping theorem (Mann & Wald, 1943).
P RELIMINARIES
We describe the notations used in this paper, introduce the model and its underlying assumptions, and state conditions which simplify the problem but which can be removed without loss of learnability.
Notation The ReLU residual units we use take a vector x ∈ R d as input and return a vector y ∈ R m .
We use A ∗ ∈ R d × d and B ∗ ∈ R m ×
d to denote the ground-truth network parameters for layer 1 and 2, respectively.
We use circumﬂex to denote predicted terms (e.g. an empirical objective function ˆ f , estimated layer 1 weights ˆ A ).
We use n to denote the number of i.i.d.
samples available for the training algorithm, { x ( i ) , y ( i ) } to denote the i -th sample drawn.
For an integer k , We deﬁne [ k ] to be { 1 , 2 , . . . , k } , and e ( j ) as the standard basis vector with a 1 at position j .
All scalar-based operators are element wise in the case of vectors or matrices unless speciﬁed otherwise.
We use x and y to refer to the input and output vectors, respectively, as random variables.
Linear Regression: Linear regression (LR) in this paper refers to the problem of estimating L for unbiased model y
= Lx .
In this case, estimation is done by minimizing the empirical risk ˆ R ( L ) = 12 n (cid:80) i ∈ [ n ] (cid:13)(cid:13)
Lx ( i ) − y
( i ) (cid:13)(cid:13) 2 – using linear least squares (LLS).
Its solution has a closed form which we use as given.
We refer the reader to Hamilton (1994) for more information.
Models, Assumptions and Conditions Following previous work (Livni et al., 2014), we assume a given neural network structure speciﬁes a hypothesis class that contains all the networks conforming to this structure.
Learning such a class means using training samples to ﬁnd a set of weights such that the neural networks predictions generalize well to unseen samples, where both the training and the unseen samples are drawn from an unobserved ground-truth distribution.
In this paper, the hypothesis class is given by ReLU residual units and we assume it has sufﬁcient expressive power to ﬁt the ground-truth model.
More speciﬁcally, we discuss the realizable case of learning, in which the ground-truth model is set to be a residual unit taken from the hypothesis class with the form:
[Equation]
1) and is used to draw samples for learning.
Unlike other multi-layer ReLU-afﬁne models which do not apply skip connections, we cannot permute the weight matrices of the residual unit and retain the same function because of skip-adding x
(it breaks symmetry).
This helps us circumvent issues of identiﬁability 1 , and allows us to precisely estimate the ground-truth weight matrices A ∗ and B ∗ .
Our general approach for residual unit layer 2 learns a scaled ground-truth weight matrix that also minimizes the layer 2 objective.
The existence of such scaled equivalence of our layer 2 approach comes from what is deﬁned below.
Deﬁnition 2.1 (component-wise scale transformation) .
A matrix A ∈ R d × d is said to be a scale transformation w.r.t.
the j -th component if ( A j, : ) (cid:62) =
A j,j · e ( j ) .
Additionally, estimation is more complex when the layer 2 weights B ∗ is nonsquare.
For simplicity of our algorithm presentation for layer 2, we stick to Cond.
2.1 in the following sections 2 .
Condition 2.1 (layer 2 objective minimizer unique) .
A ∗ is not a scale transformation w.r.t.
any components and B ∗ is a square matrix, i.e. m = d .
W ARM -U P : V ANILLA L INEAR R EGRESSION
Consider a ground-truth two-layer residual unit.
If we assume that the inputs only contain vectors with negative entries, i.e. x < 0 , the effect of the ReLU function in the residual unit then disappears because of the nonnegativity of A ∗ .
The residual unit turns into linear model y
=
B ∗ x .
Thus, direct
LR on samples with negative inputs can learn the exact ground-truth layer 2 parameter B ∗ with at least d samples, when formulating the LR as a solvable full-rank linear equation system.
On the contrary, if the inputs only contain vectors with positive entries, i.e. x > 0 , all the neurons in the hidden layer are then activated by the ReLU and the nonlinearity is eliminated.
The residual unit in this case turns into y
= B ∗ ( A ∗ + I d ) x .
Taking the value that left-multiplies x as a single weight matrix D ∗ , this is also a linear model.
Direct LR on at least d samples with positive inputs by the residual unit can learn the exact D ∗ .
Since we have the access to the exact B ∗ , solving for the exact
A ∗ can be accomplished through solving a full-rank linear equation system B ∗ ·
˜ A
= D ∗ , where the unique solution ˜
A =
A ∗ + I d .
While simple, this vanilla LR approach requires a large number of redundant samples, since sampled inputs usually have a small proportion of fully negative/positive vectors.
Taking random input vectors i.i.d.
with respect to each entry as an example, the probability of sampling a vector with all negative entries is p d − , where p − is the probability of sampling a negative vector entry, then the expected number of samples to get one negative vector is 1 /p d − .
Denoting p + similarly, 1 /p d + samples are expected for a positive vector.
Besides, each LR step in this approach requires d such samples respectively to make the linear equation system full-rank, which implies the expected sample size to be d -exponential d · (cid:0) 1 /p
d
−
+ 1 /p d + (cid:1) .
For other common random vectors like Gaussian samples, the proportions of fully negative/positive vectors in sampled inputs are also expected to decrease exponentially as d grows, as high-dimensional random vectors like Gaussian are essentially concentrated uniformly in a sphere (Johnstone, 2006).
Technical and experimental details about sample size expectations and the vanilla LR algorithm are further discussed in Appendix F.
N ONPARAMETRIC L EARNING : L AYER
We present how we learn a residual unit layer 2 under Cond.
2.1 (estimating B ∗ ):
We ﬁrst design an objective functional with the arguments being a matrix and a function.
The objective uses expectation of a loss over the true distribution generating the data and is uniquely minimized by [ B ∗ ] − 1 and a rectiﬁer function (ReLU).
We then formulate its ERM using nonparametric estimation as a standard convex QP, further simpliﬁed as an LP that has the same capability as the QP to learn layer 2.
4. O BJECTIVE D ESIGN AND L ANDSCAPE
Consider the formulation of a residual unit as in Eq.
It is possible to rewrite the model as equation:
C ∗ y =
( A ∗ x ) + + x , where the output of the hidden neuron with skip addition is on both sides of the equation, and C ∗ B ∗ =
I d .
We aim to estimate the inverse of B ∗ by matrix variable C and the nonlinearity x
(cid:55)→ ( A ∗ x ) + by a function variable h .
The objective is formulated as risk functional by the L 2 error between values respectively computed by C and h
[Equation]
where the estimator C ∈ R d × m , the domain of h is the nonnegative 3 continuous 4 R d → R d function space, written as C 0 ≥ 0 in shorthand.
This objective is quadratic because the forward mapping x
(cid:55)→ h ( x ) + x and the backward mapping y
(cid:55)→ Cy are both linear w.r.t.
C and h , and the two are linearly combined in a L 2 norm.
The objective in Eq. 2 is minimized by the ground-truth, i.e. C ∗ and x (cid:55)→ ( A ∗ x ) + , is one of its minimizers.
However, it is not simple to describe other variable values that minimize the objective if any exist.
We give that detail under Cond.
2.1, the minimizer of G 2 is unique, as the exact ground-truth in the given domain.
Theorem 4.1 (objective minimizer, layer 2) .
Deﬁne G 2 ( C , h ) as Eq. 2, where C ∈ R d × m , h ∈ C 0 ≥ 0 .
Then under Cond.
2.1, G 2 ( C , h ) reaches its zero minimum iff C
=
[ B ∗ ] − 1 and h : x (cid:55)→ ( A ∗ x ) + .
Technical details are in Appendix H, where we use Lem.
4.2 (in subsection 4.3) to prove a more general theorem which does not require Cond.
and is sufﬁcient for Thm. 4.1.
In the next subsection, we construct the ERM of G 2 and present our convex QP formulation.
4. ERM WITH N ONPARAMETRIC E STIMATION IS C ONVEX QP
Consider the second layer objective (Eq. 2) with nonnegative continuous function space as the domain of h .
We follow Vapnik (1992) and deﬁne its standard empirical risk functional:
[Equation]
The function variable h ∈ C 0 ≥ 0 can be optimized either parametrically or nonparametrically.
If we parameterize h , and show that the nonlinearity w.r.t.
its parameters would make ˆ G 2 lose its quadratic form.
Instead, we estimate h nonparametrically: for each sample input x ( i ) , we introduce variables ξ ( i ) that estimates mapped values by h .
This avoids introducing nonlinearity to the objective and keeps ˆ G 2 quadratic.
On the other hand, the domain of h , i.e. nonnegative continuous function space, turns into a set of linear inequalities as constraints when optimizing nonparametrically.
In this sense, learning the second layer of the residual unit can be formulated as the following QP:
[Equation]
where Ξ = { ξ ( i ) } ni =1 is the nonparametric estimator of x (cid:55)→ ( A ∗ x ) + .
Nonparametric Estimation Validation: A solution to the ERM with nonparametric estimation, i.e. the QP, is guaranteed to be sufﬁcient for minimizing the standard empirical risk functional (Eq. 3).
More speciﬁcally, assuming C and Ξ = { ξ ( i ) } ni =1 are a solution to layer 2 QP
(Eq. 4), it is clear that
C and h ∈ C 0 ≥
such that h ( x ( i ) ) =
ξ ( i ) minimize the empirical risk functional Eq. 3.
Conversely, a minimizer of the standard empirical risk Eq. 3, C and h , corresponds to a solution to layer 2 QP as we set ξ ( i )
=
h ( x ( i ) ) .
Therefore, minimizing ˆ G 2 and solving layer 2 QP are empirically equivalent.
Convexity: The convexity of the QP: Eq.
4 is also guaranteed.
First, constraints are linear.
Second, for each sample with index i ∈ [ n ] , the L 2 norm wraps linearity w.r.t.
C and ξ ( i ) .
Such formulation ensures the quadratic coefﬁcient matrix is positive semideﬁnite.
Thus, with the sum of convex functions still being convex, the QP objective (Eq. 4) becomes convex.
Even without the knowledge of how samples are generated, this QP would be a convex program.
Strict proofs are in Appendix I.
4. LP S IMPLIFICATION
Consider single-sample error written as g 2 ( C , h ; x , y )
= 12 (cid:107) h ( x ) + x − Cy (cid:107) 2 .
If there is a feasible C such that Cy − x ≥ 0 holds for all x ∈ R d , then C and h : x (cid:55)→ Cy − x always minimize g 2 as zero, and thereby minimize the layer 2 objective (Eq. 2).
Thus, we obtain a condition that is equivalent to G 2 reaching the minimum in Thm.
4.1 and avoids randomness (see Lem. 4.2).
Lemma 4.2.
G 2 ( C , h ) reaches its zero minimum iff Cy − x ≥ 0 holds for any x ∈ R d and its corresponding residual unit output y , and h : x
(cid:55)→ Cy − x .
The pointwise satisfaction of the inequality in Lem.
4.2 describes the solution space for the minimization of G 2 .
The sufﬁciency of satisfying this inequality to minimize G 2 is directly obtained by assigning h : x (cid:55)→ Cy − x where C complies with Cy − x ≥ 0 for all x in the support space R d .
The necessity of satisfying this inequality comes from its contraposition: If a C violates the inequality, there must be a non-null set of x that yield the violation due to the continuity of Cy − x .
In this sense, the resulting G 2 value becomes nonzero.
Empirically speaking, we cannot solve an inequality that holds w.r.t.
to every point in the support space if we only observe ﬁnite samples.
We can only estimate C by solving the inequality that holds w.r.t.
each sample.
Following this, we formulate such estimation as to ﬁnd a feasible point in the space deﬁned by a set of linear inequalities, each of which corresponds to a sample:
Cy ( i ) − x ( i ) ≥ 0 .
Each point in the feasibility deﬁned by the inequalities has a one-to-one correspondence in the solution space to layer 2 QP
(Eq. 4): C ↔ ( C , { Cy ( i ) − x
( i ) } i ∈ [ n ] ) .
The set of inequalities can be solved by a standard LP with a constant objective and with constraints that are inequalities, i.e.
[Equation]
With the one-to-one correspondences, LP: Eq. 5 and QP: Eq. 4 have equivalent solution spaces.
Moreover, LP: Eq. 5 is also a convex program.
Alg.
1 summarizes the layer 2 estimator: Simply solve the QP/LP 5 and return the inverse of ˆ C as the estimate of layer 2 weights and ˆ Ξ as x (cid:55)→ ( A ∗ x ) + estimate.
Regardless of time complexity, QP and LP in Alg.
1 work equivalently since their solution spaces are equivalent to each other.
Our nonparametric learning directly ﬁnds a unique layer 2 estimate under Cond.
In the general case without Cond. 2.1 (discussed in Appendix G), nonparametric learning essentially reduces the possible values of B from R m × d to a B ∗ scale-equivalent matrix space, where LR uses sampled data much more efﬁciently than vanilla LR on layer 2 (section 3).
N ONPARAMETRIC L EARNING : L AYER
With layer 2 learned, outputs by hidden neurons become observable.
The two-layer problem is thereby reduced to single-layer.
Consider a ground-truth single-layer model: h
=
( A ∗ x ) + .
To construct a learning objective for this model, we rewrite the model as a nonlinearity plus a linear mapping by A ∗ : h
= ( − A ∗ x ) + + A ∗ x , where on both sides of the equation is the output of layer 1, and the nonlinearity is x (cid:55)→ ( − A ∗ x ) + .
The objective of layer 1 is formulated as:
[Equation]
where A ∈ R d × d is of the layer 1 weights estimator, the domain of r is also C 0 ≥ 0 .
The minimizer A of the risk G 1 falls into a matrix space such that for any matrix A in the space, each row of A is a scaled-down version of the same row of A ∗ without changing the direction (Thm. 5.1).
Theorem 5.1 (objective minimizer space, layer 1) .
Deﬁne G
( A , r ) as Eq. 6, where A ∈ R d × d , r ∈ C 0 ≥ 0 .
Then G 1 ( A , r ) reaches its zero minimum iff for each j ∈ [ d ] , A j, :
= k
j A ∗ j, : where 0 ≤ k j ≤ 1 , and r : x (cid:55)→ ( A ∗ x ) + − diag( k ) · A ∗ x .
The scale equivalence in the solution space is derived from a ReLU inequality: ( x ) + ≥ kx
where 0 ≤ k ≤ 1 .
Let the j -th row of A be a scaled-down version of A ∗ , i.e. A j, : =
k
j A ∗ j, : where 0 ≤ k j ≤ 1 .
According to the inequality, we have (cid:0) A ∗ j, : x (cid:1)
+ ≥ k j A ∗ j, : x , which indicates ( A ∗ x ) + ≥ diag( k ) ·
A ∗ x .
Thus, r minimizing G 1 in Thm.
5.1 lies in feasibility C 0 ≥
when A = diag( k ) · A ∗ .
Due to the existence of scale equivalence, we must compute the scale factor to obtain the ground-truth weights A ∗ .
The scale factor k j is sufﬁciently obtainable with ( A j, : x ) + and (cid:0) A ∗ j, : x (cid:1)
+ observable:
Conditioned on nonnegative ReLU input, we have a linear model
A j, : x = k
j A ∗ j, : x where A j, : x and A ∗ j, : x are observed.
Thm.
5.2 summarizes layer 1 scale factor property, allowing us to correct a minimizer of G 1 to the ground-truth weights A ∗ by computing a scalar for each row.
Theorem 5.2 (scale factor, layer 1) .
Assume A is a minimizer of G 1 .
Then for any j ∈ [ d ] , ( A j, : x ) + / (cid:0) A ∗ j, : x (cid:1) + is always equal to the scale factor k j given that ( A j, : x ) + > 0 .
The elimination of randomness for G 1 follows the same pattern as layer 2.
If there is a feasible A such that ( A ∗ x ) + − Ax ≥ 0 holds for all x ∈ R d , such A is a solution to our objective by Eq. 6.
We can also have the following proposition that avoids randomness.
Lemma 5.3.
G 1 ( A , r ) reaches its zero minimum iff h − Ax ≥ 0 holds for any x ∈ R d and its corresponding hidden output h , and r : x (cid:55)→ h − Ax .
We now turn into empirical discussion.
Similar to layer 2, we formulate layer 1 QP by its empirical objective with nonparametric estimation and linear constraints representing the nonnegativity of r :
[Equation]
where Φ = { φ ( i ) } ni =1 is the function estimator of x (cid:55)→ ( A ∗ x ) + − diag( k ) · A ∗ x where k refers to the scaling equivalence.
Similarly, the solution space of QP: Eq.
7 can be represented by a set of linear inequalities as constraints of the following efﬁciently solvable LP
[Equation]
Alg. 2 describes how a layer 1 is learned: First, a G 1 minimizer estimate ˆ A is obtained by solving the QP/LP.
Then for the j -th row, the scale factor k j is estimated by running LR on h ( i ) j and ˆ A j, : x ( i ) s.t.
h ( i ) j > 0 to correct ˆ A , as A j, : x = k
j h j is an unbiased linear model given that h j > 0 .
By nonparamatric learning, the value space of A is reduced from R d × d to { diag( k ) ·
A ∗ | 0 ≤ k j ≤ 1 } , where LR uses sampled data much more efﬁciently than vanilla LR layer 1 (section 3).
F ULL A LGORITHM AND A NALYSIS
The full algorithm concatenates Alg.
1 and 2 in a layerwise fashion, with observations of input/output by the ground-truth network: a ) Estimates layer 2 and nonlinearity:
x (cid:55)→ ( A ∗ x ) + by Alg. 1.
b ) Estimates layer 1 by running Alg.
2 on input samples and the nonlinearity estimate.
It has provable guarantees.
For empirical analysis, we use ˆ C n to denote the estimation of C ∗ from n random samples.
Similar notations are applied to other estimations.
First of all, our methods to solve respective layers,
Alg. 1 and 2, are strongly consistent if any convex QPs/LPs involved can be solved exactly.
Lemma 6.1 (layer 2 strong consistency) .
Under Cond.
2.1, ˆ C n a.s. −−→
C ∗ and ˆ B n a.s. −−→ B ∗ , n → ∞ .
For ˆ C n : In Appendix J, we prove its more general a.s. convergence without satisfying Cond.
2.1, where the solution space to layer 2 objective is a non-compact continuous set where all the elements are scaling equivalences.
We use Hausdorff distance (Rockafellar & Wets, 2009) as metric and prove that the empirical solution space a.s. converges to the theoretical solution space.
Then the more general a.s. convergence holds with the strong consistency of layer 2 scale factor estimator where we use LR to estimate the scale factors.
For ˆ B n : Using the continuous mapping theorem (Mann & Wald, 1943), we directly propagate the strong consistency of C ∗ estimator to its inverse B ∗ ’s estimator.
According to full algorithm description, layer 1 estimation uses ˆ C n y ( i ) − x
( i ) as the outputs where i ∈ [ n ] .
Thus, the strong consistency of the hidden neuron estimator is also guaranteed by the continuous mapping theorem.
Following the proof sketch of the ˆ C n a.s. convergence, we obtain the strong consistency of layer 1 estimator (See Lem. 6.2).
Lemma 6.2 (layer 1 strong consistency) .
ˆ A n a.s. −−→
A ∗ , n → ∞ .
By the continuous mapping theorem, the strong consistency of the full algorithm (Thm. 6.3, formal version of Thm. 1.2), which is commonly deﬁned by a loss function that is continuous on network weights, is implied by the strong consistency of network weights estimators (Lem. 6.1 and 6.2).
See
Appendix J for proofs of strong consistency discussions in this section.
Theorem 6.3 (strong consistency, formal) .
Deﬁne L as L 2 output loss.
L ( ˆ A n , ˆ B n ) a.s. −−→ 0 , n → ∞ .
E XPERIMENTS
We provide experimental analysis to demonstrate the effectiveness and robustness of our approach in comparison to stochastic gradient descent (SGD) on L 2 output loss: L ( A , B ) =
12 E x (cid:107) ˆ y − y (cid:107) 2 , where we parameterize the output prediction by ˆ y =
B (cid:104)
( A x ) + + x (cid:105) .
Our proposed methods outperform SGD in terms of sample efﬁciency and robustness to different network weights and noise strengths, which indicates a poor optimization landscape of L 2 output loss for ReLU residual units.
Setup : The ground-truth weights are generated through i.i.d.
folded standard Gaussian 6 and standard
Gaussian for layer 1 and 2 respectively, i.e. A ∗ i.i.d.
∼ |N| (0 , 1) , B ∗ i.i.d.
∼ N (0 , 1) .
The input distribution is set to be an i.i.d.
zero mean Gaussian-uniform equal mixture N ( − 0 .
1 , 1) – U ( − 0 .
SGD is conducted on mini-batch empirical losses of L ( A , B ) with batch size 32 for 256 epochs in each learning trial.
We apply time-based learning rate decay η = η 0 / (1 + γ · T ) with initial rate η 0 = 10 − 3 and decay rate γ = 10 − 5 , where T is the epoch number.
The above hyperparameters are tuned to outperform other hyperparameters in learning ReLU residual units in terms of output errors.
Evaluation : We use relative errors to measure the accuracy of our vector/matrix estimates: For a network with weights A and B and its teacher network with weights A ∗ and B ∗ , a ) layer 1 error refers to (cid:107) A −
A ∗ (cid:107) / (cid:107) A ∗ (cid:107) , similar to layer 2.
b ) output error refers to ˆ E [ (cid:107)
ˆ y − y (cid:107) / (cid:107) y (cid:107) ] by test data.
Due to the equivalence between the solution spaces of our QP and LP without label noise, we choose LP in noiseless experiments, referred to as “ours”.
In addition, to reduce variance, the results of learning the same ground-truths are computed as means across 16 trials.
Sample Efﬁciency: Consider Fig.
It depicts the variation in the prediction errors (warmer color, larger error) as a function of d (input dimension) and the number of samples the learning algorithm is using.
We compare SGD against our algorithm.
We observe that our approach to the estimating the neural network is more sample efﬁcient.
For SGD, the estimation is relatively easy with only up to 10 dimensions.
As expected, once the dimension grows, the sample size required for the same level of error as our method is larger.
Still, overall, our method is capable of learning robustly with small sample sizes and more efﬁciently than SGD even for larger sample sizes.
Network Weight Robustness: This experiment aims to verify whether our method can learn a broader class of residual units.
In Tab. 1, our method shows a light-tailed distribution with nearly zero means and standard deviations for layers 1 and 2, and output errors across various ground-truth networks, whereas SGD is less robust in the same context.
Our method shows strong robustness to network weight changes, indicating its applicability across the whole hypothesis class.
[Figure]
[Figure]
Noise Robustness: Fig.
2 conﬁrms the robustness of our methods when output noise exists.
Samples are generated by a ground-truth residual unit with output noise being i.i.d.
zero-mean Gaussian in different strengths (i
[Figure]
e. standard deviations).
We try both QP and LP because in noisy setting the two approaches are not equivalent w.r.t.
the solution space 7 .
First, SGD always gives larger errors than our methods, even though it is hardly affected by tuning the noise strength.
For QP/LP, all the errors for layer 1, 2 and output grow almost linearly as noise strength increases, indicating that both QP/LP learn the optima robustly when output noise is present, where QP slightly outperforms LP.
C ONCLUSION
In this paper, we address the problem of learning a general class of two-layer residual units and propose an algorithm based on landscape design and convex optimization: We demonstrate ﬁrstly that minimizers of our objective functionals can express the exact ground-truth network.
Then, we show that the corresponding ERM with nonparametric function estimation can be solved using convex
QP/LP, which indicates polynomial-time solvability w.r.t.
sample size and dimension.
Moreover, our algorithms that are used to estimate both layers as well as the whole networks are strongly consistent , with very weak conditions on input distributions.
Our work opens the door to a variety of open problems to explore.
We provide a strong consistency result without assuming a particular input distribution.
It would be interesting to explore the sample complexity of our algorithm when stronger assumptions are placed on input distribution.
Another extension could be to dispose of the limitations on ground-truth weights, e.g. m ≥ d , non-degeneration, nonnegativity of A ∗ .
In addition, given that our algorithm solves ReLU by using nonparametric estimation through convex optimization, this might provide an inspiration to solving learning problems with other nonlinearities using nonparametric methods that improve or change the optimization landscape.
R EPRODUCIBILITY S TATEMENT
The full set of assumptions of all theoretical results in this paper are stated - section 2, Appendix F and G. The complete proofs of all theoretical results are all included - Appendix F, G, H, I and J.
Code in MATLAB is zipped as supplementary ﬁle.
The numerical simulation details, including data sizes and splits, hyperparameters and how they were chosen, clear deﬁnitions of the speciﬁc measures or statistics used to report results, are discussed in detail - the Setup and Evaluation paragraphs in section 7.
Results with central tendency (e g. mean) and variation (e g. stddev) are reportedTab. 1.
The use of external computing infrastructure, CVX , is clariﬁed and the consent obtaining and licensing of CVX and SDPT3 solver used in this work exactly follows the original author instructions: http://cvxr.com/cvx/doc/citing.html - footnote 5.
A PPENDICES
We include an outline of the appendices:
• Appendix A:
An empirical analysis of the running time of our algorithm and an analysis of the effect of condition number on the estimation accuracy.
• Appendix B: A brief discussion of our learning algorithms in a noisy context, referring to the LP slack variable technique that is used in our experiments with noise in the main paper.
Also, this section provides an insight to potential future direction of this work.
• Appendix C: A detailed explanation of the computational complexity and the methods through which the QPs/LPs in this paper are solved.
• Appendix D: A preliminarily proposed general optimization formula for the extension of learning multi-layer ResNets, to show the scalability of our algorithm.
• Appendix E: A justiﬁcation of how our QP/LP approaches get rid of the exponentiality that vanilla LR approach (section 3) has.
• Appendix F: A formal result and an empirical validation of the example given in the main paper in the context of the vanilla linear regression method (section 3).
• Appendix G: A generalization of layer 2 learning to the case where
Cond.
2.1 is not satisﬁed.
• Appendix H: Proofs regarding the minimizers of the objective functions we use.
• Appendix I: Proofs that justify the convexity of our QPs.
• Appendix J: Proofs that show our estimation algorithm is strongly consistent.
A R UNNING T IME AND S ENSITIVITY TO C ONDITION N UMBER
Following the setup in section 7, we report additional experiments testing the running time of our algorithm in comparison to SGD and the feasibility of estimating the inverse of B ∗ when ill-conditioned. SGD-GT 1 SGD-GT 2 SGD-GT SGD-GT 4 SGD-GT
LP-GT 1 LP-GT 2 LP-GT
LP-GT 4 LP-GT      12 - -0.
SGD-GT 1 SGD-GT 2 SGD-GT
SGD-GT 4 SGD-GT
LP-GT 1 LP-GT 2 LP-GT
LP-GT 4 LP-GT
[Figure]
A. R UNNING T IME E FFICIENCY VERSUS SGD
In Fig.
we compare the running time of our algorithm against the running time of SGD for ground-truth networks sampled at random.
The running time of our algorithm is signiﬁcantly lower, with an error level which is also signiﬁcantly lower.
Furthermore, this is demonstrated across input dimensions, where the difference between the running time of SGD until convergence and our algorithm’s running time increases as d increases.
In addition, the running time of our algorithm is ﬁxed, and does not depend on the ground-truth network, while the running time of SGD varies based on the network sampled.
A. L AYER 2 W EIGHTS C ONDITION N UMBER R OBUSTNESS
Our algorithm depends on a matrix-inversion step to obtain the estimated parameters.
This affects the estimation of B ∗ .
Due to this, we further investigate the effect of the condition number of B ∗ in the ground-truth parameters on the accuracy of estimation.
For this experiment, to obtain ground-truth network parameters with different condition number, we follow the procedure of Ge et al.
(2018), and multiply a diagonal matrix with exponentially-dropping values on the diagonal ( λ − i for the i -th element) by two random orthonormal matrices, U and V (cid:62) , on the left and on the right respectively:
B = U diag (cid:0) λ − 1 , . . .
, λ − d (cid:1) V (cid:62) .
In Fig.
4, we compare the effect of the condition number of B ∗ on SGD and on our algorithm.
SGD turns out to be quite sensitive to the condition number, especially for lower dimensions.
Our algorithm, on the other hand, is almost not affected by the condition number, with a ﬁnal estimation error consistently close to 0 , recovering the matrix B ∗ .
B D ISCUSSION : N OISY M ODEL
Here, we discuss our learning methods in the noisy case.
We introduce output noise to our model, namely
[Equation]
where the label noise z ∈ R d is an i.i.d.
random vector with respect to each component, satisfying E [ z ] = 0 .
In addition z and x are statistically independent.
Taking layer 2 as an example, our original objective functional does not reach zero by substituting the ground-truth: G 2 ( C ∗ , x (cid:55)→ ( A ∗ x ) + ) =
E z [ (cid:107) C ∗ z (cid:107) 2 ] =
σ 2 Tr C ∗ C ∗ (cid:62) where σ is the noise strength.
However, if layer 2 is well conditioned, the ground-truth will still assign a value close to zero to the objective.
In this sense, with G 2 ’s continuity, the ground-truth can approximately minimize G 2 , which validates our QP approach in terms of learning noisy residual units.
Our original LP (Eq. 5) fails to give feasible solutions due to possible violation of the inequality
C ∗ y − x ≥ 0 , since C ∗
y − x =
( A ∗ x ) + + C ∗ z is not necessarily an entrywise nonnegative vector because the term C ∗ z might have negative entries.
So we introduce slack variables ζ ( i ) to soften the constraints:
[Equation]
For a sample ( x ( i ) , y ( i ) ) with noise z ( i ) , if (cid:0) A ∗ x ( i ) (cid:1) + + C ∗ z ( i ) < 0 , then its L 1 norm would be added to the objective.
This method remedies violations to the inequality C ∗ y − x ≥ 0 .
With access to a sufﬁciently large sample and with the “stability” assumption, our solution ˆ C would be close to
C ∗ since large deviations seldom rise and their penalties are diluted in the objective.
C D ISCUSSION : S OLVING C ONVEX P ROGRAMS
The theoretical foundation of solving convex QP and LP has been driven to maturity in terms of computational complexity (Kozlov et al., 1980; Murty, 1980) and convergence analysis (Jarre, 1990;
Zhang et al., 1992).
The time complexity for a convex QP/LP is analyzed in terms of the number of scalar variables N and the number of bits L in the input (Ye & Tse, 1989).
For example, under
Cond.
2.1, N = d 2 + nd for the QP because the matrix variables have d 2 scalars and the function estimators have nd scalars.
Similarly we have N = d 2 for the noiseless LP and N = d 2 + nd for the noisy LP.
It is guaranteed that primal-dual interior point methods can solve the convex QP/LP in a polynomial number of iterations O ( √ NL ) , where each iteration costs at worst O ( N 2 . 5 ) arithmetic operations from the Cholesky decomposition for the needed matrix inversion (Monteiro & Adler, 1989; Karmarkar, 1984).
This indicates that our QPs/LPs are guaranteed to be solvable in O ( N 3 L ) arithmetic operations, which is generally an O (poly( n, m, d, L )) complexity.
To solve convex QPs/LPs in this paper, we use CVX , a commonly used package for specifying and solving convex programs (Grant & Boyd, 2014; 2008).
For both QPs and LPs, CVX calls a solver,
SDPT3 (Toh et al., 1999), which is speciﬁed for semideﬁnite-quadratic-linear programming and applies interior-point methods with the computational complexity mentioned above.
Experimentally,
SDPT3 indeed speciﬁes and solves our programs fast and makes our numerical results robust and stable.
D D ISCUSSION : E XTENSION TO M ULTI -L AYER
In this appendix, we propose a generalized optimization formula of nonparametric learning of multilayer residual networks to show that our algorithm is scalable.
Assume we have an L -layer network of the form
[Equation]
We can scale the objective to use nonparameter variables { ξ ( i ) l } l ∈
[ L − 1] to estimate every ReLU activation ( W ∗ l r l ) + in this network.
Generally, for the l -th ReLU we can have the outer layer objective
[Equation]
where hat denotes estimated values and ˆ h ( i ) L − 1 =
y ( i ) , and its corresponding inner layer objective
[Equation]
We can apply techniques, e.g., tune the combinations of those objectives with respect to layers or whether to have a hat or not, or even get them weighted, to keep its convex quadratic form and at the same time improve its optimization landscape.
Speciﬁcally when L = 2 , this becomes the two-layer
ResNet learning discussed in the main paper.
E S AMPLE E FFICIENCY : V ANILLA LR VERSUS O URS
In this appendix, we describe an example to justify how our core QP/LP approaches eliminate the exponential sample efﬁciency compared to the vanilla LR approach.
In layer 2 learning, C ’s optimization is to ﬁnd a feasible point in the space determined by n inequalities each of which corresponds to a sample.
Taking the j -th row of the inequality, i.e. C j, : y ≥ x j .
Every time we get a new sample x ( i ) , y ( i ) in,
• The inequality C j, : y ( i ) ≥ x
( i )
j eliminates the solution space of C j, : by one of the spaces divided by plane C j, : y ( i ) =
x ( i ) j .
This property guarantees fast convergence speed at early phase since C j, : ’s feasibility starts from R d . •
In fact, when ReLU does not activate A ∗ x ( i ) at the j -th row, i.e. A ∗ x ( i ) ≤ 0 , the theoretical solution of C j, : that C j, : B ∗ = (0 , . .
. , 0) (a 1 at index j ) directly lies on the divisive plane C j, : y ( i ) =
x ( i ) j :
[Equation]
This property remarkably speeds up further constraints on the solution space of C j, : to the correct estimate and is with high probability, since it directly depends on the sign of A ∗ x ( i ) .
With the vanilla LR approach mentioned in section 3 all the dimensions need to have the correct sign at the same time.
This is the reason for the exponential complexity of this approach.
Our main algorithm only requires one dimension to get the correct sign for a single sample, which avoids the exponential sample size expectation.
The convergence speed is determined by the actual probability of each dimension not getting activated by ReLU, and such probability is determined by speciﬁed input distribution.
F V ANILLA L INEAR R EGRESSION : M ORE D ETAILS
Alg. 3 gives the full list of steps of learning two-layer residual units by LR, where we ﬁrst split the drawn samples into two halves for the respective two key steps, then for both halves we ﬁlter negative and positive vectors, respectively.
By running LR on both ﬁltered sets we obtain an estimation of the ground-truth network.
In the following sections, we formalize this intuition and describe the exponential sample complexity of this approach under entry-wise i.i.d.
setting as an example of the inefﬁciency of this method.
Theorem F.1 (exponential sample complexity, vanilla LR) .
Assume the input vectors are i.i.d.
with respect to each component.
Then
Alg.
3 learns a neural network from a ground-truth residual unit with at least O (cid:0) d · 2 d +1 (cid:1) expected number of samples.
Proof.
For each j ∈ [ d ] , let P j be the marginal probability of x j being positive, i.e. P j = P ( x j > 0) > 0 .
Thus, the probability that a sample is positive P ( x > 0) = (cid:81)
j ∈ [ d ] P j .
Then the expected number of sampling trials to obtain d positive samples is d/ (cid:81) j ∈
[ d ] P j .
Similarly, the expected number of sampling trials to obtain d negative samples is d/ (cid:81) j ∈
[ d ] [1 − P j − P ( x j = 0)] .
Let n be a random natural number s.t.
with n samples the network has learned.
The expected number of samples that guarantees successful learning is the sum of the two expectations
[Equation]
Thus E [ n ] ≥ O
( d · 2 d +1 ) .
With an exponential sample complexity, the time complexity of the LR approach is thereby also exponential because ﬁltering through all the samples costs time with the same complexity as the number of samples, even though the LR itself only costs polynomial time in the number of samples.
To summarize, this vanilla LR approach learns exact ground-truth residual units but with exponential complexity in terms of both computational cost and sample size.
While this approach to learning a residual unit such as ours is simple and intuitive, we aim to improve on this approach, making full use of all samples available.
F. E XPERIMENTS : S AMPLE E FFICIENCY
We present the results of the vanilla LR in learning the residual units.
As discussed in section 3, LR requires full-rank linear systems parameterized by samples to learn the exact ground-truth parameters. [Table]
[Table] However, with degenerate linear systems, LR is completely incapable of learning the parameters.
Therefore, there exists a hard threshold for the number of samples required by LR that makes the linear equation system full-rank.
With such a dichotomous constraint on LR learning, we take the learning success rate among 1000 trials as the metric to evaluate the performance of LR with different sample sizes and number of dimensions.
Tab. 2 shows the learning success rates with zero-mean Gaussian inputs.
For each ﬁxed number of dimensions, it appears there is a hard threshold that switches the learnability of LR.
The exponential sample complexity is also reﬂected there as the number of dimensions grows linearly.
Tab. 3 shows the rates with input mean non-zero, where an overall decline in success rates happens.
This observation is explainable because the bottleneck of LR learning is the lower value found between the probability of sampling a positive and a negative vector.
A positive mean reduces the probability of the latter, and thereby increases the sample size required.
G G ENERALIZATION OF L AYER 2 L EARNING
In this appendix, we discuss how layer 2 is learned without satisfying Cond. 2.1.
It is a fact that
A ∗ being a scale transformation w.r.t.
some components causes scaling equivalence to our layerobjective functional minimizers just as in layer 1.
To be more speciﬁc, we give a general version of Thm. 4.1 that handles the case without Cond. [-2.
-]2.1 and describes the scaling equivalence of Gminimizers.
See Thm.
G.1 for a formal description.
Theorem G.1 (objective minimizer space, layer 2, general) .
Let G 2 be 1 2 E x (cid:104) (cid:107)
h
( x ) + x − C y (cid:107) 2 (cid:105) as a functional, where C ∈ R d × m , h ∈ C 0 ≥ 0 .
Then G 2 ( C , h ) reaches its zero minimum iff
CB ∗ =
diag( k ) where for each j ∈ [ d ] ,
a ) 1 1+ A ∗ j,j ≤ k j ≤ 1 , if A ∗ is a scale transformation w.r.t.
the j -th component.
b ) k j
= 1 , if A ∗ is not a scale transformation w.r.t.
the j -th component.
As in layer 1, the scaling equivalence in layer 2 can also be obtained by assigning A ∗ as a scale transformation w.r.t.
some component and using the properties of the ReLU nonlinearity.
See subsection H.1 for a detailed explanation of Thm.
G.1.
To obtain the scale factors k and correct a scale-equivalent G 2 minimizer C to the ground-truth, we observe linear models parameterized by the scale factors, where for each j ∈ [ d ] , [ C y ] j = k j x j given that x j < 0 .
See Thm.
G.2 and its proof for justiﬁcations of the linear models that are used to compute k .
Theorem G.2 (scale factor, layer 2, general) .
Assume C is a minimizer of G 1 in the context of
Thm. G.1.
Then for any j ∈ [ d ] , the following three propositions are equivalent:
a) A ∗ is a scale transformation w.r.t.
the j -th component.
b) [ C y ]
j / x j is a constant c n j given that x j < 0 .
c) [ C y ] j / x j is a constant c p j given that x j > 0 .
Additionally, if one of the above three propositions is true, then k j = c n j .
Proof.
For j ∈ [ d ] , we ﬁrst prove a) =
⇒ b), c): From (cid:0)
A ∗ j, : (cid:1) (cid:62) =
A ∗ j,j · e ( j ) we have [ C y
j
[Equation]
Then we prove ¬ a) =
⇒ ¬ b), ¬ c): ¬ a) =
⇒ ∃
j (cid:48) ∈
[ d ] and j (cid:48) (cid:54) =
j s.t.
A ∗ j,j (cid:48) (cid:54) = 0 .
Recall
[Equation]
The value of x j (cid:48) affects the value of [ C y ] j / x j because A ∗ j,j (cid:48) (cid:54) = 0 .
In fact, from supp p ( x ) =
R d we have supp p ( x j (cid:48)
| x j ) =
R , indicating that the value of [ C y ] j / x j can never be kept as a constant when given both x j < 0 and x j > 0 because x j (cid:48) can be any real number.
With the exact derivations above, we are able to obtain a left inverse of B ∗ , namely C , that satisﬁes
CB ∗ =
I d .
Consider Eq.
1 left multiplied by B ∗ C
B ∗ C y = y .
Eq.
12 is also a noiseless unbiased linear model where C y and y are observable, and thereby B ∗ is computable due to the easy solvability of its linearity.
Alg. 5 describes how a residual unit second layer is learned without satisfying Cond.
We ﬁrst solve the QP/LP and obtain a scaling equivalence to an estimated left inverse of B ∗ and the function estimate, namely ˆ C and ˆ Ξ .
Then we compute the scale factor estimate ˆ k by running Alg.
4, where for each component index j ∈
[ d ] , we ﬁrst use a tolerance parameter as a threshold to determine whether ground-truth layer 1 is a scale transformation, and if so, we run LR to estimate the model [ [Equation]
, otherwise the scale factor is directly assigned by 1 .
Upon correcting ˆ C and ˆ Ξ by ˆ k , we obtain a layer 2 estimate ˆ B by running LR to estimate the linear model Eq.
The strong consistency of our results in this appendix is justiﬁed in Appendix J.
H E XACT D ERIVATION OF O BJECTIVE F UNCTIONAL M INIMIZERS H. L AYER
Lem.
4.2 is proved as follows.
Correct ˆ Ξ to the function estimation of ( A ∗ x ) + .} 9: return ˆ B , ˆ Ξ .
Proof.
“ ⇐ = ”: Since h ( x ) =
Cy − x ≥ 0 , random vector h ( x ) + x − C y is always a zero vector, which implies G 2 ( C , h ) = 0 .
Hence “ ⇐ = ” holds.
“ = ⇒ ”:
Since r.v.
(cid:107) h
( x ) + x − C y (cid:107) 2 ≥
we have
[Equation]
where λ is the Lebesgue measure on R d .
Proof by contradiction: Assume that ∃ x (cid:48) ∈ R d , ∃ i ∈
[ d ] , s.t.
( Cy (cid:48) − x (cid:48) )
i < 0 , where y (cid:48) is the corresponding network output.
Let f ( x ) =
( Cy − x ) i which is continuous on R d .
Therefore for (cid:15) =
− f ( x (cid:48) )
> 0 , ∃ δ > 0 , ∀ x ∈ B ( x ; δ ) i.e. (cid:107) x − x (cid:48) (cid:107) <
{+δ , | f ( x ) − f ( x (cid:48) ) | < (cid:15) = ⇒ 
+}
f ( x (cid:48) ) < f
( x ) < 0 =
⇒ [ x − Cy ] i >= ⇒ [ h ( x ) + x − Cy ] i > 0 = ⇒ (cid:107) h ( x ) + x − Cy (cid:107) 2 > 0 .
Since λ ( B ( x ; δ ))
=
π d/Γ ( d/ 2 + 1) δ d > 0 and B ( x ; δ ) is a subset of the measured set in Eq.
13, we have a contradiction.
Thus Cy − x ≥ 0 holds in a pointwise manner in R d , indicating (cid:8) x ∈ R d | h ( x ) (cid:54) =
Cy − x (cid:9) must be a null set.
With h ’s continuity, h must be x (cid:55)→ Cy − x .
Therefore “ = ⇒ ” holds.
Lemma H.1.
Cy − x ≥ 0 holds for any x ∈ R d and its corresponding output y only if CB ∗ is a diagonal matrix.
Proof.
Let D = CB ∗ .
We rewrite Cy − x ≥ 0 as
[Equation]
For further use, we substitute x by − x and the resulting inequality D (cid:104) ( − [Equation] ∗ x ) + − x [Equation]
x ≥still holds.
Added by Eq.
14 we have
[Equation]
Proof by contradiction: Assume D is not diagonal, then ∃ i (cid:54) =
j , such that D i,j (cid:54) = 0 .
Consider the following two cases: a) D i,j > 0 .
We take the i -th row of Eq.
14 as follows
[Equation]
[Equation]
Let x − j = 0 and x j < 0 .
Then (cid:80) dk
=1 D i,k (cid:20)(cid:16)
A ∗ k, : · x (cid:17) + + x k (cid:21) =
D i,j · x j < 0 = ⇒ ⊥ .
b) D i,j < 0 .
We take the i -th row of Eq. 15 as follows
[Equation]
≥
[Equation]
Let x =
( A ∗ ) − 1 v , where v = e ( j ) .
Then (cid:80) dk
=1 D i,k (cid:12)(cid:12)(cid:12) A ∗ k, : · x (cid:12)(cid:12)(cid:12) =
D i,j < 0 =
⇒ ⊥ .
With Lem.
4.2 and Lem.
H.1, we prove Thm.
G.1 as follows.
Proof.
With Lem.
4.2, we only need to prove ∀ x ∈ R
[Equation]
“ ⇐ = ”: We have
[Equation]
For the i -th row of Eq.
18, consider the following two cases:
a) A ∗ i, : · x = A i,i x i , i.e. A ∗ is a scale transformation w.r.t.
the i -th row.
With
[Equation]
+ A ∗
i,i
[Equation]
k i ≤
we have
[Equation]
For x
i
≥
, [ Cy
− x ] i = (cid:0) k
i A ∗ i,i + k
i − 1 (cid:1) x i ≥ 0 . For x i <
, [ Cy −
x i =
( k i − 1) x i ≥ 0 .
b) A ∗ i, : · x (cid:54) =
A i,i x i .
With k i
=
we have
[Equation]
Hence “ ⇐ = ” holds.
“ = ⇒ ”: Let D = CB ∗ .
With Lem.
H.1, D is diagonal.
Consider the following two cases:
a) A ∗ i, : · x = A i,i x i .
The i -th
inequality can be written as
[Equation]
Proof by contradiction: we need to ﬁnd x ∈ R d which contradicts with Eq. 19 in the following three cases:
a) If D i,i ≤ 0 , then, x i > 0 = ⇒ D
i,i (cid:0)
A ∗ i,i x i (cid:1)
+
+ ( D i,i − 1) x i < 0 =
⇒ ⊥ .
b) If D i,i > 1 , then, x i < 0 =
⇒ ⊥ .
c)
If 0 < D i, [Equation] <
[Equation]
, then ∃ a >
[Equation]
s [Equation] D
[Equation]
Letting x i > 0 , we have [Equation] Hence [Equation]
b) A
[Equation]
i, : · x (cid:54) = A i,i x i , i
[Equation]
=
i [Equation]
s.t.
A ∗ i,j > 0 .
The i -th inequality can be written as
[Equation]
Proof by contradiction: we need to ﬁnd x ∈ R d which contradicts with Eq. 20 in the following three cases:
a) If D i,i ≤ 0 , then, x i > 0 = ⇒ D
i,i (cid:0)
A ∗ i, : x (cid:1) +
+ ( D i,i − 1) x i < 0 =
⇒ ⊥ .
b) If D i,i > 1 , then, x i <
∧ x − i ≤ 0 = ⇒ ⊥ .
c)
If 0 < D i,i < 1 , then, x i >
∧ x j ≤ − A ∗ i,i
A ∗ i,j x i ∧ x k ≤ 0 = ⇒ ⊥ where k (cid:54) = i, j .
Hence D i,i = 1 .
Hence D = CB ∗ =
diag( k ) , and thereby “ = ⇒ ” holds.
H. L AYER
The proof of Lem.
5.3 is similar to that of Lem.
4.2 because the two lemmas follow the same idea, which is to link objective functional minimization with always-hold inequalities.
With Lem.
5.3 , we prove Thm. 5.1 as follows.
Proof.
With Lem.
5.3, we only need to prove ∀ x ∈ R d , ( A ∗ x ) + −
Ax ≥⇐⇒ ∀ i ∈
[ d ] , A i, : = k i
A ∗ i, : , where 0 ≤ k i ≤ 1 .
This is equivalent to what it is for a single row, i.e. ∀ x ∈
R d , (cid:16) a ∗(cid:62) x (cid:17) + − a (cid:62) x ≥
⇐⇒ a =
k a ∗ , where 0 ≤ k ≤ 1 .
“ ⇐ = ”: The case where a ∗ = 0 is clear.
If a ∗ is not a zero vector and a = k a ∗ , we have
a) If a ∗(cid:62) x ≥ 0 , (cid:16) a ∗(cid:62) x (cid:17) + − a (cid:62) x = a ∗(cid:62) x − k a ∗(cid:62) x
= (1 − k ) a ∗(cid:62) x ≥ 0 .
b) If a ∗(cid:62) x < 0 , (cid:16) a ∗(cid:62) x (cid:17) + − a (cid:62) x = − k a ∗(cid:62) x ≥ 0 .
Hence “ ⇐ = ” holds.
“ = ⇒ ”: If a ∗ = 0 , a must be a zero vector, otherwise let x = a , then − a (cid:62) x < 0 = ⇒ ⊥ .
If a ∗ is not a zero vector, consider two cases below:
a) If a (cid:54) =
k a ∗ where k ≥ 0 .
Let x = (cid:107) a ∗ (cid:107) (cid:107) a (cid:107) · a − a ∗ , then x is not a zero vector
and
[Equation]
where θ denotes the angle between a and a ∗
b
If a = k a ∗ where k > 1 , then let x = a ∗ we have
[Equation]
Hence a = k a ∗ where 0 ≤ k ≤ 1 if a ∗ is not zero, and thereby “ = ⇒ ” holds.
I QP C ONVEXITY
The LPs in the main paper are trivially convex.
So in this appendix, we only justify the convexity of our QPs:
We ﬁrst prove the convexity of single-sample objectives, then the convexity of the empirical objectives with nonparametric estimation, i.e. ˆ G NPEand ˆ G NPEis obtained by the convexity of convex function summations.
Lemma I.1.
Suppose f ( u ) = 12 (cid:107) T u − b (cid:107) 2 where u is a real matrix.
Then f is convex w.r.t.
u .
Lem.
I.1 is easily obtained since the Hessian f (cid:48)(cid:48)
( T ) =
T (cid:62) T is positive semideﬁnite.
In the following, we demonstrate and justify the convexity of the QPs of both layers by rewriting their single-sample objectives into the formulation of f and summing them without loss of convexity.
Theorem I.2.
QP:
Eq. 4, and QP: Eq.
7 are convex optimization.
Proof.
First of all, constraints of both QPs are trivially linear and convex.
Thus, we only need to justify the convexity of the two empirical objectives, ˆ G NPEand ˆ G NPE(see Eq.
7 and 4).
Consider the single-sample version of ˆ G NPE 1 , namely g NPE( A , φ ; x , h )
= 12 (cid:107) φ
+ Ax − h (cid:107) 2 ,
which, in the formulation of f , can be rewritten with
[Equation]
21) which guarantees the convexity of g NPEw.r.t.
A and φ by Lem.
I.1.
For g NPE( C , ξ [Equation] x , y ) = 12 (cid:107) ξ + x − Cy (cid:107) 2 , we have
T =   − y (cid:62) − y (cid:62) . . .
− y (cid:62)   ,
[Equation]
which guarantees the convexity of g NPEw.r.t.
C and ξ by Lem.
I.1.
Now we consider the summation.
Taking layer 1 as an example, by deﬁnition we have
[Equation]
For each i ∈ [ n ] , equivalently, we take Φ as a variable instead of φ ( i ) in g NPE, but with only φ ( i ) ∈
Φ determining the value of g NPE.
In this sense, g NPEis convex w.r.t.
A and Φ for each i ∈ [ n ] .
Thus, the sum ˆ G NPEis convex w.r.t.
A and Φ .
Similarly, ˆ G NPEis convex w.r.t.
C and Ξ .
J S TRONG C ONSISTENCY
In this appendix, we justify the strong consistency of our estimators for the residual unit layer 1/ learning and the whole network.
According to Thm.
G.1 and Thm. 5.1, the solutions to our objective functionals are continuous sets.
In addition, there are theoretical intermediate results that are also represented as continuous sets, e.g. possible left-inverse matrices for B ∗ in the results for layer 2.
Thus, to analyze the consistency of our learning algorithm, we deﬁne distances between sets, so that the convergence of sets can be well deﬁned.
Further point convergence results, i.e. layer 1/2 estimator strong consistency, are based on the set convergence we deﬁne.
Deﬁnition J.1 (deviation) .
Let A ⊆ M and B ⊆
M be two non-empty sets from a metric space ( M , d ) .
The deviation of set A from the set B , denoted by D ( A , B ) , is
[Equation]
where sup and inf represent supremum and inﬁmum, respectively.
Deﬁnition J.2 (Hausdorff distance) .
Let A ⊆ M and B ⊆
M be two non-empty sets from a metric space ( M , d ) .
The Hausdorff distance between A and B , denoted by D H ( A , B ) , is
[Equation]
[Equation]
Remark.
In the following, we use the Frobenius norm to deﬁne the distance between matrices, i.e. d ( X [Equation]
Y [Equation] = [Equation]
J. L AYER
In this subsection, we prove the strong consistency of the layer 2 estimator.
J.1. O BJECTIVE M INIMIZER S PACE E STIMATOR
For simplicity of notation, we use ˆ S n to denote our layer 2 QP/LP solution space by n random samples 8 as a random set
[Equation]
26) and S ∗ to denote the value space of C in Lem.
4.2 which minimizes layer 2 objective functional
[Equation] , ∀ x
[Equation]
R d and its corresponding output y } .
[Equation]
For further use, we name ˆ P n as the set of n sampled inputs which deﬁne ˆ S n , i.e. ˆ P n :=
{ x ( i ) } i ∈ [ n ] where each x ( i ) is the same random variable in Eq.
By the deﬁnitions above, we describe the strong consistency of our QP/LP as Lem.
J.1.
Lemma J.1 (QP/LP strong consistency, layer 2) .
D H ( ˆ S n , S ∗ ) a.s. −−→ 0 as n → ∞ .
Proof.
First we prove that D H (ˆ S n , S ∗ ) p −→ 0 as n → ∞ .
Recall Thm.
G.1, C y − x ≥ 0 only if
CB ∗ =
diag( k ) .
We inherit the notation as deﬁning D = CB ∗ .
Theorem G.1 is based on Lem.
H.1, and we prove them by raising points that show contradiction, i.e. violate the inequality that holds in a pointwise fashion:
a)
In the proof of Lem.
H.1, we use d points: − e ( i ) , for i ∈
[ d ] , to make (cid:80)
[Equation] =1 D i, [Equation] (cid:20)(cid:16)
A ∗ k [Equation] · x (cid:17) + + x k (cid:21) <
x i , so that D i,j ( i (cid:54) =
j ) cannot be positive; and another d points: ( A ∗ ) − 1 e ( i ) , to show (cid:80) dk
=1 D i,k (cid:12)(cid:12)(cid:12) A ∗ k, : · x (cid:12)(cid:12)(cid:12) < 0 , so that D i,j ( i (cid:54) =
j ) cannot be negative.
For each − e ( i ) we use here, since the violations follow strict inequalities, we know there exists a neighborhood of − e ( i ) , N i = N ( − e ( i ) ) , such that ∀ z ∈ N i , (cid:80) dk
=1 D i,k (cid:20)(cid:16)
A ∗ k, : · z (cid:17) + + z k (cid:21) <
z i .
We can similarly ﬁnd such neighborhood of each ( A ∗ ) − 1 e ( i ) that the strict inequality holds within the neighborhood respectively.
We index them as N d +1 to N 2 d . b)
In the proof of Thm.
G.1, we further construct d points: for each
i ∈
[ d ] , we take a point x such that x i >
∧ x j ≤ − A ∗ i,i
A ∗ i,j x i ∧ x k ≤ 0 , where k (cid:54) =
i, j .
This counterexample shows [ Cy − x ] i < 0 , and eliminates the possibility of 0 < D i,i < 1 when A ∗ i, : is not a scale transformation.
We can similarly ﬁnd neighborhood of each point and index them as N 2 d + to N 3 d .
Note that we omit some cases in the proof of Thm.
G.1, because the ﬁrst 2 d points are sufﬁcient to use in those cases to show contradiction.
In the sampling procedure, if we sample at least one point in each neighborhood N i , Thm.
G. assures the solution we get ˆ C n would lie in the true optimal set S ∗ .
The probability that the sampling procedure “omits” any of the neighborhoods is
P (cid:16)
[Equation]
or [Equation] [Equation] Since the measure on each neighborhood P ( N i ) = (cid:82) x ∈ N i p ( x ) > 0 ,
P
[Equation]
or ˆ P
[Equation]
N
[Equation]
or . . or ˆ P n
[Equation]
→ ∞ .
Here we obtain D H (ˆ S n , S ∗ )
p −→ 0 as n → ∞ .
Now we take the inﬁnite sum over the both sides of
Eq.
[Equation]
By the Borel-Cantelli lemma (Borel, 1909), D H (ˆ S n , S ∗ ) a.s. −−→ 0 as n → ∞ .
J.1.
S CALE F ACTOR E STIMATOR
To avoid ambiguity, we use n sf to denote the number of samples used in Alg. 4.
The samples pairs are { ( x ( i ) , y ( i ) ) } n sf i =1 .
Without loss of generality, the following discussion focuses on some ﬁxed index j ∈ [ d ] .
In Alg. 4, we plug in our estimator ˆ C n and use LR to estimate k j given that x ( i ) j <
[Equation]
We ﬁrst give the strong consistency of layer 2 scale factor estimator for as n sf → ∞ , as described in
Lem.
J.2.
Lemma J.2 (scale factor estimator strong consistency, layer 2) .
Suppose A ∗ is a scale transformation w.r.t.
the j -th component, and k n sf ( ˆ C n ) is the n sf -sample estimator of k j via LR: Eq. 30 given that x ( i ) j < 0 .
Deﬁne sets
[Equation]
Then lim n
[Equation]
Proof.
Following our notation, Thm. G.1 and Thm.
G.2 ensure that if A ∗ is a scale transformation w.r.t.
the j -th component, for any C belonging to the true optimal set S ∗ , k n sf ( C ) ∈ (cid:104)
[Equation]
And the “iff” statement strengthens that U ∗ =
{ k n sf ( C ) : C ∈ S ∗ } for any n sf ∈ Z + .
Note that since
S ∗ ⊂ ˆ S n , we have U ∗ ⊂
U n sf ,n .
We only need to prove D ( U n sf ,n , U ∗ )
a.s. −−→ 0 as n → ∞ .
∀ ˆ C n ∈ ˆ S n and ∀ C ∈ S ∗ ,
[Equation]
[Table]
[Equation]
Then
[Equation]
which implies
[Equation]
Take the n sf → ∞ limit over both sides of Eq. 34.
With the strong law of large numbers 9 we have
[Equation]
[Equation] [Equation] Since D (ˆ S n , S ∗ ) a.s. −−→ 0 as n → ∞ , we have D ( U n sf ,n , U ∗ )
a.s. −−→ 0 as n → ∞ then n sf → ∞ .
J.1. L AYER 2 W EIGHTS E STIMATOR
In Alg. 5, we solve B via LR.
Let ˆ z ( i ) =
diag − 1 (ˆ k ) · ˆ C n y ( i ) ∈ R d , where ˆ k is obtained through
Alg. 4 with input ˆ C n .
Assume we are using sample size of n w to do the LR.
The optimization problem is
[Equation]
Now we present the strong consistency of layer 2 estimator, as described in Thm.
J.3.
Theorem J.3 (strong consistency, layer 2) .
Suppose ˆ B n sf is the solution to Eq.
Then ˆ B n w a.s. −−→ B ∗ as n, n sf , n w → ∞ .
Proof.
Let β denote vec B (ﬂattening B into a vector), then B ˆ
z ( i ) =
(cid:0)
ˆ z ( i ) (cid:1) (cid:62) ⊗ I m .
Here the operation ⊗ denotes the Kronecker product.
Then we can deﬁne an equivalent optimization problem
[Equation]
[Equation]
Take the derivatives of β [Equation] we obtain
[Equation]
Then the optimal solution ˆ β n w of this optimization can be written in closed form:
[Equation]
We inherit the notation from the last two subsections.
By Lem.
J.1, d ( ˆ C n , S ∗ ) a.s. −−→ 0 as n → ∞ .
Thus ∀ ε > 0 , ∃ N such that ∀ n ≥ N , d ( ˆ C n , S ∗ ) ≤ ε w.p. 1 , i.e. ∃ C n ∈ S ∗ s.t.
d ( ˆ C n , C n ) ≤ ε w.p. 1 .
Then by Lem.
J.2, ∃ K > 0 , for ε that is small enough, ∃ N sf such that ∀ n sf ≥ N sf , we have (cid:12)(cid:12)(cid:12) k n sf ( ˆ C n ) − k n sf ( C n ) (cid:12)(cid:12)(cid:12) ≤
Kε w.p. 1 .
For simplicity, we omit the under-script n sf of k n sf in the following discussion.
In fact,
[Equation]
[Equation]
[Table]
[Table]
It follows that (cid:13)(cid:13)(cid:13)
ˆ z ( i ) (cid:0)
ˆ z ( i ) (cid:1) (cid:62) − z
( i ) (cid:0)
z
( i ) (cid:1) (cid:62) (cid:13)(cid:13)(cid:13) F is also bounded by O ( ε ) .
With similar techniques we can prove
[Equation]
F ≤
[Equation]
(40) and (cid:13)(cid:13)(cid:13)
[Equation]
Denote (cid:104)(cid:80) i ∈ [ n w ] z
( i ) (cid:0)
z
( i ) (cid:1) (cid:62) (cid:105) as P and (cid:80) i ∈
[ n w ] z ( i ) as Q .
Substitute z ( i ) with ˆ z ( i ) in the above expression we have ˆ P and ˆ Q .
Hence,
[Equation]
[Table]
I
[Equation]
] y
[Equation]
[Equation]
In the ﬁrst part, by triangle inequality,
[Equation]
F (42) So we only need to prove (cid:13)(cid:13)(cid:13)(cid:13)(cid:104) ˆ P ⊗ I m (cid:105) − 1 − [ P ⊗ I m ] − 1 (cid:13)(cid:13)(cid:13)(cid:13)
F ≤ O ( ε ) to claim (cid:13)(cid:13)(cid:13) ˆ β n w − β ∗ (cid:13)(cid:13)(cid:13) F ≤ O ( ε ) .
Denote ˆ P − P = ∆ P .
From Eq.
39, we know every entry of ∆ P ⊗ I m can be bounded by O ( ε ) .
By simple calculation we have
[Equation]
Then we have
[Equation]
[Equation]
J. L AYER
In this subsection, we justify the strong consistency of layer 1 objective functional minimizer estimator in detail, i.e. the layer 1 QP/LP solution space.
We will omit the detailed proof of Alg.
2 line 4 tostrong consistency since it is similar to the proof of Lem.
J.2.
Additionally, we also omit the strong consistency of the x (cid:55)→ ( A ∗ x ) + function estimator because it can be directly obtained by Lem.
J. and J.2 and the continuous mapping theorem.
We use a new optimization problem equivalent to the optimization of G 1 .
Before that, we ﬁrst deﬁne the equivalence between two optimization problems as follows.
Deﬁnition J.3.
Let opt1 and opt2 be two optimization problems, with f 1 , f 2 as the respective objective functions.
Then opt1 and opt2 are said to be equivalent if given a feasible solution to opt1 , namely x 1 , a feasible solution to opt2 is uniquely corresponded, namely x 2 , such that f 1 ( x 1 ) =
f 2 ( x 2 ) , and vice versa.
The new optimization problem and its equivalence to the optimization of G 1 is described in Lem.
J.4.
Lemma J.4.
The optimization of G 1 (Eq. 6) is equivalent to
[Equation]
Proof.
To see this, suppose A 1 is one optimal solution to Eq.
45, then we can construct r 1 ( x ) =
( A 1 x − h )
+
so that G 1 ( A 1 , r 1 ) =
f ( A 1 ) and the optimality implies
[Equation]
On the other hand, suppose ( A 2 , r 2 ) is an optimum of G 1 .
Let r 3 ( x )
=
( h − A 2 x ) + , then ∀ x ∈ R d
and h be the corresponding hidden output, if [ h − A 2 x ] j ≥ 0 , then [ r 3 ( x ) + A 2 x − h ] j = 0 , otherwise [ r 3 ( x ) + A 2 x − h ] 2 j
=
[ A 2 x − h ] 2 ≤ [ h 2 ( x ) + A 2 x − h ] 2 j since h 2 is nonnegative.
So that we know that
[Equation]
From the optimality, we further have
[Equation]
From the simple calculation above, we can see that one optimal solution to Eq.
45 has a one-to-one correspondence to an optimal solution to G 1 .
Similarly, the empirical version of the two problems are equivalent, which indicates their consistency in the empirical estimation being equivalent.
In the following, we justify the strong consistency of empirical Eq. 45 instead of G
[Equation]
[Equation]
Denote T ∗ := { diag( k ) ·
A ∗ | 0 ≤ k j ≤ 1 , j ∈
[ d ] } as the true optimal solution set, and ˆ T n as the optimal solution set corresponding to the n -sample problem.
In the following, we justify four conditions in a row that hold for f to derive the strong consistency of its optimal solution estimator.
Lemma J.5.
Let ˆ T n (cid:48) be the layer 1 QP/LP solution space by n (cid:48) samples.
Then there exists a compact set C determined by A ∗ , namely C ( A ∗ ) , s.t.
ˆ T n (cid:48)
⊂ C ( A ∗ ) w.p. 1 as n → ∞ .
Proof.
∀ l ∈ [ d ] , let a ∗ be the l -th row of A ∗ , and ˆ a l be the l -th row of ˆ A n (cid:48) .
We’d
like ﬁrst to prove that the set ˆ T ln (cid:48)
= { a l : a l is the l -th row of A , where A ∈ ˆ T n (cid:48) } (47) is compact w.p.1.
Suppose n (cid:48) >
d , and among the n (cid:48) samples, we classify them into two folds.
To avoid ambiguity, let u ( i ) be the points such that ( a ∗ ) (cid:62) u ( i ) > 0 , i ∈
[ q ] ; and v ( j ) be the points such that ( a ∗ ) (cid:62) v ( j ) < 0 , j ∈ [ n − q ] .
From the analysis of Thm.
5.1, we have ( a ∗ ) (cid:62) u ( i ) ≥ ˆ a (cid:62) u ( i ) , ∀ i ∈ [ q ] and ˆ a (cid:62) v ( j ) ≤ 0 , ∀ j ∈ [ n − q ] .
It follows that we can rewrite ˆ T l n (cid:48) as a polyhedron
[Equation]
We are going to show the polyhedron ˆ T ln (cid:48) is bounded by contradiction.
If it is not bounded, then ∃ d ∈ R d , d (cid:54) = 0 and ˜ a ∈ ˆ T ln (cid:48) , such that ∀ λ > 0 , ˜ a + λ d ∈ ˆ T ln (cid:48) .
Then
[Equation]
˜ [Equation] 49) similarly,
[Equation]
From the deﬁnition, λ can be arbitrarily big, then d (cid:62) u ( i ) ≤ 0 , ∀ i ∈ [ q ] , and d (cid:62) v ( j ) ≤ 0 , ∀ j ∈ [ n − q ] .
Since we know span { u ( i ) } = R d w.p. 1 , then there ∃ some i ∗ such that d (cid:62) u ( i ∗ ) < 0 , w.p. 1 .
(Otherwise if d (cid:62) u ( i ) =
for all i ∈ [ q ] , then either span { u ( i ) } (cid:54) =
R d or d = 0 .)
Under our assumption that, ˆ T ln (cid:48) is not bounded, we know the following system (w.r.t x ) has a feasible solution w.p. [Equation]
[Equation]
where every row of U and V is (cid:0) u ( i ) (cid:1) (cid:62) and (cid:0) v ( j ) (cid:1) (cid:62) respectively.
By Farkas’ lemma (Farkas, 1902), the system
[Equation]
is not feasible (w.p. 1 ).
We claim that u ( i ∗ ) lies in the conic hull of − v ( j ) ’s w.p. 1 .
So that the second system actually has a feasible solution and thus it raises the contradiction.
Denote the conic hull as
[Equation]

[Equation] Now suppose u ( i ∗ ) / ∈ H , by the supporting hyperplane theorem (Luenberger, 1997), ∃ b ∈ R d , b (cid:54) = 0 , such that b (cid:62) u
( i ∗ ) ≤ b (cid:62) t for ∀ t ∈ H .
Then by deﬁnition,
[Equation]
Denote the hyperplane J = { t : b (cid:62) t ≤ b (cid:62) u
( i ∗ ) } , then
[Equation]
[Equation]
[Equation]
[Equation]
Since we have in this case a geometric sequence, we know its inﬁnite sum is bounded.
By BorelCantelli lemma (Borel, 1909), we conclude that u ( i ∗ ) ∈ H w.p. 1 .
Then system II is feasible w.p. 1 .
So that ˆ T ln (cid:48) is compact w.p. 1 .
Now we prove that there exists a compact set C
( A ∗ ) , s.t.
ˆ T n (cid:48)
⊂ C ( A ∗ ) w.p. 1 as n → ∞ .
Similarly, we focus on the analysis of one row.
As discussed above, ˆ T ln (cid:48) is compact w.p. 1 .
Let ˆ W 1 n (cid:48) be the set of all u ( i ) sampled in estimating ˆ T n (cid:48) , and ˆ W 2 n (cid:48) be the set of all v ( j ) sampled.
Now for another set ˆ T n (cid:48)(cid:48) , similarly deﬁne sample point sets ˆ W 1 n (cid:48)(cid:48) and ˆ W 2 n (cid:48)(cid:48) .
We claim that ∀ u ( i ) ∈
ˆ W 1 n (cid:48) , u ( i ) lies in the conic hull of ˆ W 1 n (cid:48)(cid:48) .
Actually, this part of the proof is very similar to the way we prove u ( i ∗ ) ∈ H w.p. 1 , so we will omit it here.
∀ a ∈ ˆ T n (cid:48)(cid:48) , let x (1) and x (2) be two different points in ˆ W 1 n (cid:48)(cid:48) .
Then a (cid:62) ( λ 1 x (1) + λ 2 x (2) ) ≤ ( a ∗ ) (cid:62) ( λ 1 x (1) + λ 2 x (2) ) for ∀ λ 1 ≥ 0 and λ 2 ≥ 0 .
This simple calculation reveals a (cid:62) u ( i ) ≤ ( a ∗ ) (cid:62) u ( i ) for ∀ u ( i ) ∈
ˆ W 1 n (cid:48) (from the claim we made).
This implies that ˆ T n (cid:48)(cid:48) ⊂
C ( A ∗ ) w.p. 1 as n → ∞ , too.
Lemma J.6.
The minimizer space of f ( A ) , i.e. T ∗ = { diag( k ) · A ∗ | 0 ≤ k j ≤ 1 , j ∈
[ d ] } , is contained in C ( A ∗ ) .
Lemma J.7.
f ( A ) is ﬁnite valued and continuous on C ( A ∗ ) .
Lem.
J.6 and J.7 are easily obtained by the formulation of f (see Eq. 45) and Lem.
J.5.
Lemma J.8 (uniform a.s. convergence) .
ˆ f n ( A ) a.s. −−→
f ( A ) as n → ∞ , uniformly in A ∈ C
( A ∗ ) .
Proof.
Name single-sample objective g ( x , A ) =
12 (cid:13)(cid:13)(cid:13) ( Ax − h )
+ (cid:13)(cid:13)(cid:13) 2 .
The uniform a.s. convergence is guaranteed by the uniform law of large numbers (Jennrich, 1969):
a) By Lem.
J.5, C
( A ∗ ) is a compact set.
b) g is continuous w.r
t. A by its formulation and measurable over x at each A ∈ C ( A ∗ ) .
c)
In fact,
[Equation]
Since A ∈ C ( A ∗ ) is in a compact set,
[Equation]
Thus the dominating function exists 10 .
By Lem.
J.5, J.6, J.7 and J.8, all of the conditions are satisﬁed in (Shapiro et al., 2014, Thm. 5.3).
Thus, we have the strong consistency of layer 1 objective optima estimator as described in Lem.
J.9.
Lemma J.9 (QP/LP strong consistency, layer 1) .
D H (ˆ T n (cid:48) , T ∗ )
a.s. −−→ 0 as n → ∞ .
Similar to Lem.
J.2, we have the strong consistency of the layer 1 scale factor estimator as described in Lem.
J.10.
Lemma J.10 (scale factor estimator strong consistency, layer 1) .
Let k n (cid:48) sf ( ˆ C n ) be the n (cid:48) sf -sample estimator of k j via LR: Alg.
2 line 5.
given that h ( i ) j > 0 .
Deﬁne sets
[Equation]
Then lim n
[Equation]
Remark.
In case k j = 0 , suppose the algorithm ﬁnds a solution over a continuous distribution with [0 , 1] as support and the probability that it ﬁnds a solution with scale factor 0 is 0 .
With Thm. 5.2 and the continuous mapping theorem, the strong consistency of layer 1 estimation is guaranteed.
Theorem J.11 (strong consistency, layer 1) .
Suppose ˆ A n (cid:48) is scaled by ˆ k n (cid:48) sf .
Then ˆ A n (cid:48)
a.s. −−→
A ∗ as n (cid:48) , n (cid:48) sf → ∞ .
By Thm.
J.3 and J.11, Thm.
6.3 is guaranteed by the continuous mapping theorem.