L EARNING D ISTRIBUTIONS G ENERATED BY S INGLEL AYER R E LU N ETWORKS IN THE P RESENCE OF A RBITRARY O UTLIERS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider a set of data samples such that a constant fraction of the samples are arbitrary outliers and the rest are the output samples of a single-layer neural network (NN) with rectiﬁed linear unit (ReLU) activation.
The goal of this paper is to estimate the parameters (weight matrix and bias vector) of the NN assuming the bias vector to be non-negative.
Our proposed method is a two-step algorithm.
We ﬁrst estimate the norms of the rows of the weight matrix and the bias vector using the gradient descent algorithm.
Here, we also incorporate either the median or the trimmed mean based ﬁlters to mitigate the effect of the arbitrary outliers.
Next, we estimate the angles between any two row vectors of the weight matrix.
Combining the estimates of the norms and the angles, we obtain the ﬁnal estimate of the weight matrix.
Our main contribution is the sample complexity for robust estimation rather than the algorithm itself.
Here, we prove that O ( 1 (cid:15)p 4 log dδ ) samples are sufﬁcient for our algorithm to estimate the NN parameters within an error of (cid:15) with probability 1 − δ when the probability of a sample being uncorrupted is p and the problem dimension is d .
Our theoretical and simulation results provide insights on how the estimation of the NN parameters depends on the probability of a sample being uncorrupted, the number of samples, and the problem dimension.
I NTRODUCTION
A fundamental challenge in machine learning and statistics is to estimate a high dimensional distribution given a set of observed data samples from the distribution.
One solution technique is the deep generative method that models the unknown distribution as the output distribution of a neural network (NN) when the input of the NN is drawn from a known distribution like standard Gaussian which is motivated from the case of image generation by generative adversarial networks (GANs) (Goodfellow et al.
Arjovsky et al.
; Radford et al.
; Kingma & Welling (2013);
Van Oord et al.
Then, the unknown distribution is estimated by learning the NN parameters from the data samples.
Several approaches like GANs (Goodfellow et al.
Arjovsky et al.
(2017); Radford et al.
(2015)), variational autoencoders (Kingma & Welling (2013)), and autoregressive models (Van Oord et al.
(2016)) have been proposed to train the NN.
However, they do not provide theoretical guarantees for the parameter learning.
So, we address the open problem of computing the sample complexity for the parameter estimation of a single-layer NN with rectiﬁed linear unit (ReLU) activation using a corrupted dataset in the unsupervised learning framework.
Here, the corrupted dataset refers to the model wherein the output samples consists of a fraction of arbitrary outliers introduced by an adversary (Byzantine).
Motivated by the deep generative models, we consider the unsupervised learning framework, i.e., we assume the knowledge of the output samples but, the corresponding inputs are unknown and drawn from the standard Gaussian distribution.
We start with a brief review of the related literature.
The NN parameter estimation has been considered under both supervised and unsupervised learning frameworks which rely on the stochastic gradient descent (SGD)-based algorithms (Goel et al.
Allen-Zhu et al.
; Chen et al.
Oymak (2019); Mazumdar & Rawat (2018); Wu et al. ; Lei et al. (2020)); or the gradient descent (GD)-based approach (Cao & Gu (2019);
Du et al.
A major shortcoming of these works is that they assume that the available samples are not corrupted by noise or outliers.
Several works in the literature have addressed the problem of corrupted samples in the context of supervised learning.
The existing works have considered the learning of the parameters of a NN with ReLU activation using both SGD-based (Bakshi et al.
; Goel et al.
; Mukherjee &
Muthukumar (2020)) and GD-based algorithms(Zhang et al.
Frei et al.
; Vempala &
Wilmes (2019)).
However, to the best of our knowledge, the estimation of a NN in the presence of noise or outliers in the unsupervised learning framework has not been studied in the literature.
Our setting of unsupervised parameter estimation is also related to the area of robust statistics as described below.
We recall that the goal of our paper is to estimate the parameters of a NN using corrupted output samples assuming that the input distribution is Gaussian.
Mathematically, this problem is equivalent to estimating the parameters of a truncated Gaussian distribution (see Section 2 for details).
The area of robust statistics also deals with the estimation of high dimensional distributions like Gaussian, Gaussian product, and Gaussian mixture distributions where a constant fraction of the samples were corrupted by noise (Huber (2004);
Hampel et al.
Lai et al.
; Diakonikolas et al.
; Diakonikolas & Kane (2019); Kane (2021)).
However, in this paper, we consider parameter estimation from truncated Gaussian samples in the adversarial setting which has not been studied in the literature.
We present our major contributions as the following.
• Parameter Estimation (Sections 2 and 3):
We estimate the parameters (weight matrix and bias vector) of a single-layer NN with ReLU in the unsupervised learning framework where each output sample can potentially be an arbitrary outlier with a ﬁxed probability.
We propose an estimation algorithm that has two steps: (i) we estimate the row norms of the weight matrix and the bias vector using GD along with either the median or trimmed mean-based ﬁlters to mitigate the effect of arbitrary outliers, and (ii) we estimate the angles between any two row vectors of the weight matrix using a simple geometric result (Williamson & Shmoys, 2011, Lemma 6.7).
• Theoretical guarantees (Section 4): We show that the proposed algorithm requires O ( 1 (cid:15)p 4 log dδ ) samples to estimate the network parameters within an error of (cid:15) with probability 1 − δ when the probability of a sample being uncorrupted is p .
Here, (cid:15) > 0 represents the estimation error.
• Empirical validation (Section 5): We evaluate the performance of our algorithm empirically in terms of its relation to the probability of a sample being uncorrupted, number of samples, and dimension (see Figs. 1 and 2).
We see that our proposed ﬁltering schemes ensure robustness to the arbitrary outliers.
Also, the performance of our algorithm improves with the increase in the probability of a sample being uncorrupted and the number of samples, as expected.
Further, we note that the estimation error increases slowly as the dimension grows for a ﬁxed number of samples in the presence of arbitrary outliers.
These observations from the empirical results are consistent with our theoretical results.
To summarize, in this paper, we estimate the parameters of a single-layer NN with ReLU activation in the presence of arbitrary outliers.
The results obtained in this paper provide insights to generalize to a multi-layer NN with various activation functions and obtain generalization bounds.
L EARNING A N EURAL N ETWORK M ODEL USING C ORRUPTED O UTPUT S AMPLES
We consider a single-layer ReLU NN.
Let the weight matrix of the NN be denoted by W ∈ R d × k and the bias vector by b ∈ R d .
The input to the NN is denoted by the latent variable z ∈ R k .
We assume that the variable z is drawn from the standard Gaussian distribution.
The Gaussianity assumption is for mathematical tractability, and it is motivated by a popular generative approach to estimate a high-dimensional distribution from observed samples in the case of image generation by
GANs Goodfellow et al.
; Radford et al.
Arjovsky et al.
Thus, the output of the network is the random vector x ∈ R d given by [Equation] where N ( 0 , I k ) denotes the Gaussian distribution with mean 0 and covariance matrix I k which is the identity matrix of size k × k .
Let the distribution of the random vector x be denoted by D ( W , b ) .
Then, our goal is to estimate the unknown parameters W and b of the distribution D ( W , b ) using the knowledge of n output samples x 1 , x 2 , . . .
, x n .
Here, we assume that a fraction of the samples are arbitrary outliers.
The distribution of the observed samples are modeled using the Huber’s pcontamination model 1 deﬁned as the following.
Deﬁnition 2.1 (Huber’s p -Contamination Model Huber (1964)) .
The observed samples are said to be following Huber’s contamination model if any given sample is drawn from the true distribution
D and an arbitrary distribution G with probability p , and 1 − p , respectively.
In other words, the observed samples are drawn from the mixture distribution, D p ( W , b ) given by
[Equation]
Note that if b has large negative values, then most of the output samples are zeros due to the ReLU operation.
Further, if W is a zero matrix, then any negative coordinate of b cannot be identiﬁed as it is reset to zero after the ReLU operation. Further, in Wu et al.
(2019), the authors have shown that if b ∈ R d then exponentially large number of samples are required to estimate the bias.
This holds true in our case as in the presence of arbitrary outliers if b has large negative values, then we would require larger number of samples to estimate the bias vector b compared to that in Wu et al.
Hence, we also assume b to be non-negative.
Our estimation problem is challenging due to two reasons: 1) ReLU operation is not invertible, and therefore, estimation of W and b utilizing maximum likelihood of the probability density function of the random vector x is intractable; 2) the distribution of the corrupted samples is unknown.
We tackle these issues using a new formulation combining the GD algorithm and a ﬁltering technique.
Note that the GD algorithm is similar to that in Wu et al.
(2019) wherein the authors use the SGD algorithm.
The proposed algorithm is presented in the next section.
E STIMATION OF P ARAMETERS OF S INGLE - LAYER NN
To design our estimation algorithm, we ﬁrst note that the weight matrix W may not be identiﬁable from the distribution D ( W , b ) .
This is because of the fact that for any matrices W 1 and W 2 if W 1 W T 1 = {+W 2 W T 2 then there exists a unitary matrix Q such that W 
 = W 1 Q .+}
As z ∼ N ( 0 , I k ) , we have Qz ∼ N ( 0 , I k ) . Hence, for any vector b , we have D ( W 1 , b ) =
D ( W 2 , b ) (Wu et al.
Since our goal is to learn the distribution, learning either W 1 or W 2 is sufﬁcient.
In short, we focus on the learnability of the underlying distribution and not the learnability of the NN parameters.
Therefore, our proposed algorithm estimates WW T ∈ R
d × k and b ∈ R d from the observed samples from D p ( W , b ) .
Next, we note that ( i, j ) -th entry (cid:0) WW T (cid:1) ( i, j ) of the symmetric matrix WW T is (cid:107) W ( i, : ) (cid:107) 2 (cid:107) W
( j, :) (cid:107) 2 cos( θ ij ) , where θ ij = θ ji is the angle between vectors W ( i, :) and W ( j, :) which are the i -th and j -th rows of matrix W , respectively.
Also, note that the (cid:96) p norm of a vector is given by (cid:107) x (cid:107) p
=
( (cid:80) i | x ( i ) | p ) 1 /p .
Thus, we can construct the matrix WW T using the row norms {(cid:107) W ( i, :) (cid:107) 2 } di =1 and angles { θ ij } di,j =1 .
Our estimation algorithm consists of two steps: 1) estimation of row norms of W and bias vector b ; and 2) estimation of angles between the row vectors of W .
To estimate (cid:107) W ( i, :) (cid:107) 2 ∈ R
and
b ( i ) ∈ R , for i ∈
[ d ] = { 1 , 2 , . . .
, d } , suppose x ∼ D ( W , b ) .
Its i -th coordinate can be written as
[Equation]
where W ( i, :) T z + b ( i ) ∼ N ( b ( i ) , (cid:107) W ( i, :) (cid:107) 22 ) .
Thus, to compute b ( i ) and (cid:107) W ( i, :) (cid:107) 22 , it is enough to consider the i -th entries of the observed samples.
Also, in (3), the ReLU operator sets x ( i ) to zero if W ( i, :) T z + b ( i ) is negative.
Consequently, we estimate b ( i ) and (cid:107) W ( i, :) (cid:107) 22 using the positive entries of the observed samples.
Hence, estimation of b ( i ) and (cid:107) W ( i, :) (cid:107) 2 is equivalent to estimating the parameters of a one-dimensional normal distribution using positive samples, i.e., the samples that belong to the set R > 0 :
= { x ( i ) ∈ R : x ( i ) > 0 } .
We use µ ∗ =
b ( i ) and σ ∗ 2 = (cid:107) W
( i, :) (cid:107) 2 to denote the parameters of this one-dimensional normal distribution.
The estimation of µ ∗ and σ ∗ 2 using corrupted samples is discussed next.
NN with outliers
Input: Samples x 1 , . . .
, x n ∈ R d 1 for i ∈
[ d ] do
X > 0 ← { x m , m ∈ [ n ] : x m ( i ) > 0 } ˆ v = ProjGD ( X > 0 )
ˆ Σ i,i ← 1 / ˆ v (1) ˆ b ( i ) ← max { 0 , ˆ v (2) /
ˆ v (1) } 6 for i < j ∈
[ d ] do
Compute ˆ θ ij using (11) ˆ Σ ( i, j ) ← (cid:113) ˆ Σ ( i, i ) ˆ Σ ( j, j ) cos(ˆ θ ij ) ˆ Σ ( j, i ) ← ˆ Σ ( i, j )
Output: ˆ Σ ∈ R d × d , ˆ b ∈ R d
Input: Samples X > 0 , Parameters T , γ t , and n b 1 Compute { x j
( i ) }
n b j =1 from X > 0 using (7) 2 ˜ g x ← ﬁlter (cid:16) { x j ( i ) } n b j =1 (cid:17) 3 v 0 ← [0 0] 4 for t = 1 , 2 , . .
, T do µ ← v t − 1 (2) / v t − 1 (1) σ 2 ← 1 / v t − 1 (1)
˜ g
z ← (cid:2) − (˘ σ 2 + ˘ µ 2 ) / 2 ˘ µ (cid:3) T using (5) and (6) ˜ g t ← ˜ g x + ˜ g
z ˜ v t ← v t − 1 − γ t − 1 ˜ g
t − v t ← P (˜ v t ) using (10)
Output: v T ∈ R
We determine the parameters of the univariate normal distribution using maximum likelihood estimation (Daskalakis et al.
given by arg min v (cid:96) ( v ) where (cid:96) ( v ) is the expected negative log-likelihood with respect to x ( i ) and v =
[1 /σ 2 ; µ/σ 2 ] ∈ R 2 .
This optimization problem can be solved using a learning algorithm like GD or SGD.
SGD utilizes only one sample for the gradient computation which introduces large variance due to the stochasticity and low reliability in the presence of arbitrary outliers.
Thus, we use GD that utilizes all the samples for the gradient.
To derive our algorithm based on GD, we compute the gradient (Daskalakis et al. , ∇ (cid:96) ( v ) =
E x , z [ g ( i )]
where
[Equation]
Here, x ( i ) ∼ N ( µ ∗ , σ ∗ 2 ; R > 0 ) and z ( i ) ∼ N ( µ, σ 2 ; R > 0 ) .
Also, µ ∗ and σ ∗ 2 are the parameters of the true distribution, and µ and σ 2 are functions of v .
In (4), E z [ g z ( i )] admits a closed form expression (Johnson et al.
, ˜ g
z =
E z [Equation] z ( i
[Equation] ] = (cid:2) −
[Equation]
+
[Equation]
˘
[Equation]
(cid:3) T . Here, we deﬁne
[Equation]
where φ ( · ) and Φ( · ) denote the probability density function and the cumulative distribution function of the standard normal distribution, respectively.
We also need E x
[ g x ( i )] to compute E x , z [ g ( i )] .
Since the value of E x [ g x ( i )] is not known, we use the sample mean to estimate E
x
[ g x ( i )] .
For this, all the positive samples are partitioned into n b batches.
Let S j be the j -th batch.
We then compute the vector x j ( i ) ∈ R 2 as follows:
[Equation]
We then combine the n b vectors { x j ( i ) } n b j =1 to estimate E x , z [ g ( i )] .
Recall that the observed samples are from the mixture distribution D p ( W , b ) .
Therefore, to compute the expectation with respect to the true distribution D ( W , b ) , a ﬁlter is applied on this set of n b vectors to mitigate the effect of the arbitrary outliers which is a common tool in the robust statistics literature and obtain E x [ g x ( i )] .
We consider two ﬁlters: median and trimmed mean.
Median: The median based ﬁlter possesses the following robustness property.
Typically, a median is the value separating the higher half from the lower half of a given set of points.
If more than half of a given set of points are in [ − M, M ] for some M > 0 , then their median must be in [ − M, M ] .
Thus, the median of { x j ( i ) } n b j =1 is the vector computed either from all the true samples or outliers whose magnitudes are comparable to the true samples.
Trimmed Mean: The trimmed mean removes the vectors among the set of n b vectors with relatively large and small values and computes the estimate of E
x [ g x ( i )] as the mean of the remaining vectors.
Here, we use the parameter β to indicate the number of vectors to be discarded.
This technique prunes the vectors x j ( i ) , j ∈ [ n b ] that are computed from the batches with outliers having relatively high or low magnitudes.
Therefore, we obtain the gradient estimate as
[Equation]
where ﬁlter is either median or trimmed mean, and ˘ µ and ˘ σ 2 are given in (5) and (6), respectively.
Having computed the gradient estimate, we next present the proposed GD algorithm.
In the t -th iteration, the algorithm performs the following three steps:
First step is Gradient Computation Step where the gradient ˜ g t ∈ R 2 is computed from the observed positive samples using (8).
Second step is Update Step where the gradient computed in the previous step is used to perform the GD update:
[Equation]
where γ t − 1 > 0 is the diminishing step size.
Last step is Projection Step where the objective function l ( v ) is a strongly convex function of v , if v belongs to a bounded region.
To control the strong-convexity of the objective function, we project the update vector v into the domain D r = (cid:8) v ∈ R 2 : 1 /r ≤ v (1) ≤ r, | v (2) | ≤ r (cid:9) .
Thus, the projection is
P
[Equation]
[Equation]
[Equation]
min
[Equation]
max
[Equation]
[Equation]
r [Equation] [Equation] The GD algorithm is run for T iterations where T is the algorithm parameter.
The overall algorithm is summarized in Algorithm 2.
This completes the ﬁrst step of our algorithm that uses the GD algorithm to obtain the estimate of the bias vector and the row norms of the weight matrix.
Finally, using the estimates ˆ b and ˆ Σ obtained using the GD algorithm, we estimate ˆ θ ij similar to Wu et al.
(2019) using (Williamson & Shmoys, 2011, Lemma 6.7).
Speciﬁcally, we have
[Equation]
where 1 ( · ) denotes the indicator function and ˆ b is the estimate of the bias vector computed in the previous step using the projected GD algorithm.
The overall algorithm is given in Algorithm 1 where ˆ Σ is the estimate of WW T .
Here, Steps 1estimate the row norms of W and b , and Steps 6-9 estimate the angles between any two rows of W .
Our algorithm is similar to that in Wu et al. except for the ﬁlter which is also a standard technique in robust statistics. However, our algorithm is different from the algorithm in Wu et al.
(2019) in the following aspects.
We do not rescale the unknown parameters (see Step 1 of Algorithm 2 in Wu et al.
We employ median or trimmed mean based ﬁlter to compute the full gradient and utilize the GD algorithm instead of SGD used in Wu et al.
We note that our main contribution is the analysis rather than the algorithm which is presented in the next section.
T HEORETICAL G UARANTEES FOR P ARAMETER E STIMATION
In this section, we provide our main result which characterizes the sample complexity for robust estimation of NN parameters. We address the general problem of learning the distribution generated by a NN in the presence of arbitrary outliers ( p ≤ 1 ). Hence, the proof techniques which have been devised for only the special case of p = 1 in Wu et al. (2019), are not applicable to our setup due to the presence of corrupted samples.
Therefore, we develop novel proof techniques for performing the parameter estimation in the presence of arbitrary outliers.
To arrive at the main result, we ﬁrst present the error bounds for the two steps of Algorithm 1, which are described in Sec. 3.
The following proposition provides the estimation error bounds for the ﬁrst step, i.e., the estimation of the bias vector b and the norms of the row vectors of WW T .
It shows that we can estimate b ( i ) within an additive error of Ξ (cid:107) W ( i, :) (cid:107) and Σ ( i, i ) within an additive error of Ξ (cid:107) W ( i, :) (cid:107) where Ξ decreases with both the number of samples and the total number of iterations T .
The proof relies on the fact that the objective function is η -strongly-convex and L -smooth which we ﬁrst prove.
The parameters η, L are functions of r .
Another critical step in the proof is the bound on the error between the true gradient and the estimated gradient from the output samples with arbitrary outliers obtained in terms of (cid:15) .
Also, Ψ (cid:15) is the probability of a sample being uncorrupted deviating from its mean and variance bounded in terms of (cid:15) as O ( √ (cid:15) ) .
Proposition 4.1.
Suppose we initialize the projected gradient step (Algorithm 2) such that (cid:107) v 0 − v ∗ (cid:107) =
V and choose the step size γ t = γ and the number of iterations as T .
Then, there exists γ > 0 such that for any δ, (cid:15) ∈ (0 , 1) , the output ( ˆ Σ ( i, i ) , ˆ b ( i )) of the projected GD step of our algorithm with median-based ﬁlter for any i ∈
[ d ] satisﬁes
[Equation]
|
[Equation]
with probability at least 1 − δ if the number of data samples n ≥
[Equation]
[Equation]
Here, the error bound is Ξ = V (cid:16) 1 − Lη + L (cid:17) T + 2 (cid:15)
( L + η ) Lη with η, L > 0 being constants that depend only on the parameter r .
Also, p is the probability of a sample being uncorrupted and Ψ (cid:15) ∈ [1 / 2 p, 1] is such that O ( (cid:15) 3 ) ≤ Ψ (cid:15) ≤ O ( √ (cid:15) ) with (cid:15)
>
O (cid:0) p − 1 / 3 (cid:1) where the order constants depend only on r .
The two terms in the error bound Ξ in Proposition 4.1 can be interpreted as the following.
The ﬁrst term (cid:16) 1 − Lη + L (cid:17) T V captures the convergence error due to the GD algorithm.
For any upper bound on the convergence error Ω > 0 , Algorithm 1 runs in time (log Ω − log V ) / log (cid:16) η η + L (cid:17) . This relation shows that smaller convergence errors require more iterations. The second term 2 (cid:15) ( L + η ) Lη captures the error due to the gradient computed using the observed samples. Two factors contribute to this error: 1) difference between the (true) sample mean and the true mean, and 2) the arbitrary outliers.
Also, the difference between the sample mean and true mean decreases with the number of samples.
Consequently, for smaller error (cid:15) , we need a large number of samples n = O ( 1 (cid:15)p 4 log dδ ) .
We also note that the lower bound on (cid:15) decreases with p .
The following proposition bounds the estimation error in the second step, i.e., the estimation of the angle between any two row vectors W ( i, :) and W ( j, :) (where i (cid:54) =
j ∈ [ d ] ).
The result shows that we can estimate θ ij within an additive error of Ξ .
Proposition 4.2.
Assume that (12) holds.
Then, for a ﬁxed pair of i (cid:54) =
j ∈ [ d ] and δ, (cid:15) ∈ (0 , 1) , the estimate ˆ θ ij of Algorithm 1 that uses a median-based ﬁlter satisﬁes
[Equation]
with probability at least 1 − δ if the number of data samples n ≥
[Equation]
p 2 ( p Ψ (cid:15) − 1 / 2) 2 log 1 δ .
Here, the error bound is Ξ = V (cid:16)
[Equation]
L
[Equation]
with η, L > 0 being constants that depend only on the parameter r .
Also, p is the probability of a sample being uncorrupted and Ψ (cid:15) ∈ [1 / 2 p, 1] is such that O ( (cid:15) 3 ) ≤ Ψ (cid:15) ≤ O ( √ (cid:15) ) with (cid:15)
>
O (cid:0) p − 1 / 3 (cid:1) where the order constants depend only on r .
Combining Propositions 4.1 and 4.2, we next present the sample complexity needed to achieve a small parameter estimation error.
Speciﬁcally, the diagonal entries, Σ ( i, i ) , for i
∈
[ d ] are obtained from Proposition 4.1.
Further, the off-diagonal entries of Σ are obtained by combining the Propositions 4.1 and 4.2 to obtain the ﬁnal result as follows:
Theorem 4.3.
Suppose we initialize the projected gradient step (Algorithm 2 with median-based ﬁlter) such that (cid:107) v 0 − v ∗ (cid:107) =
V and choose the step size γ t = γ and the number of iterations as T .
Then, there exists γ > 0 such that for δ, (cid:15) ∈ (0 , 1) , the output of Algorithm 1, ( ˆ Σ , ˆ b ) satisﬁes [Equation]
[Equation]
(15) with probability at least 1 − δ if the number of data samples n = O ( 1 (cid:15)p 4 log dδ ) .
Here, p is the probability of a sample being uncorrupted and Ψ (cid:15) ∈ [1 / 2 p, 1] is such that O ( (cid:15) 3 ) ≤ Ψ (cid:15) ≤ O ( √ (cid:15) ) with (cid:15)
>
O (cid:0) p − 1 / 3 (cid:1) where the order constants depend only on r .
[Table]
We note that in Propositions 4.1 and 4.2, and Theorem 4.3, η, L and the other order constants depend on the parameter r > 0 .
Recall that the parameter r is required for the convexity region D r
and we can choose r to be a large constant.
Thus, r is a relatively insensitive parameter which does not require any hand-tuning.
As the probability of a sample being uncorrupted p reduces, we observe that the error bound in (14) becomes larger as expected.
Also, as the probability of a sample being uncorrupted decreases, the number of samples n required for the error bound increases.
Moreover, the obtained error bounds increase with the problem dimension for a ﬁxed number of samples.
Our results are also comparable with the existing error bounds for ReLU parameter estimation without outliers from (Wu et al., 2019, Theorem 1).
Speciﬁcally, if there are no outliers, for any ζ, δ ∈ (0 , 1) , the number of samples n =
O ( 1 ζ 2 log dδ ) are sufﬁcient for the SGD algorithm in Wu et al.
(2019) to achieve the following error bounds with probability at least 1 − 2 δ , [Equation]
[Equation]
where ζ indicates the parameter estimation error.
Comparing the above bounds with (14) and (15), we observe that our bounds are larger due to the presence of arbitrary outliers.
In particular, Ξ consists of two error terms. The ﬁrst term V (cid:16) 1 − Lη + L (cid:17) T < 1 in our error bounds results from the convergence of the GD algorithm and is similar to ζ < 1 in (16).
However, the second error term 2 (cid:15) ( L + η )
Lη quantiﬁes the error due to the gradient estimated from the output samples with arbitrary outliers and is not present in (16).
Additionally, in (14), the extra term 3 π (2 − p ) further increases the bound compared to the error bound in (16).
Therefore, the error ζ in (16) can not be directly compared to the error in (14) and (15).
Further, our sample complexity depends on the probability of a sample being uncorrupted p and the second error term (cid:15) whereas the complexity in (Wu et al., 2019, Theorem 1) depends on the square of the convergence error term ζ in their sample complexity.
Also, when there is no outlier ( p = 1 ), the two sample complexities do not match.
S IMULATION R ESULTS
In this section, we provide numerical results to verify the performance of our algorithm.
In our setup, the columns of W are chosen as the left singular vectors of random matrices from the standard
Gaussian distribution.
For b , we use a random vector from the standard normal distribution whose negative values are replaced with zeros.
The mixture of samples are generated such that a sample comes from D ( W , b ) with probability p and from G = N (5 , 1) with probability 1 − p .
The hyperparameters in our proposed algorithm are set as r = 3 and γ t = 10 .
1 t .
We use the batch-splitting approach to compute the gradient.
This approach induces randomization.
As the output samples consist of a fraction of arbitrary outliers, the standard gradient descent algorithm without any ﬁlter would perform poorly as shown in our results.
Also, we choose the total number of GD and SGD iterations T to be |X > 0 | / 100 where |X > 0 | is the number of positive output samples.
From our experiments, we observe that the errors ﬁrst decrease with the number of iterations and then after a certain number of iterations, the errors ﬂatten.
Based on this observation, we chose the number of iterations (see Appendix A.4).
Note that we discard the zero entries and only consider only the set of positive samples X > 0 for the number of iterations as they do not convey any information about the row norms of W and bias vector b .
We compute two error metrics from the estimated parameters and the ground truth, (cid:107) ˆ Σ −
WW T (cid:107) F / (cid:107) W (cid:107) 2 F and (cid:107) ˆ b − b (cid:107) 2 / (cid:107) W (cid:107) F .
Also, we compare our algorithm with two other schemes: oracle scheme (estimation using the true samples only), and scheme without ﬁlter.
The results are shown in Figs.
1 and 2, and Table 1, and the observations from them are as follows.
[Figure]
[Figure]
Effect of ﬁlters: From Fig.
[Figure] we observe that the GD schemes perform better than the corresponding
SGD schemes.
Further, the performance is improved when we use median based ﬁlter along with
GD or SGD (for instance, as seen by comparing the performances of GD without Filter and GD with
Median in Fig. 1).
Thus, the ﬁlter ensures that the effect of the outliers is reduced
[Figure]
and the curves are closer to the Oracle GD scheme.
Finally, from Fig. 1, we also infer that the median-based approach performs slightly better than the trimmed mean-based approach.
Dependence on probability of a sample being uncorrupted p : The variation in the error estimation as a function of p is shown in Figs.
1a and 2a.
The performance of the proposed schemes improve with p , as expected.
This trend is because as p increases, the fraction of outliers in the observed samples decreases, which leads to better performance.
Note that the oracle schemes assume the knowledge of true samples.
Thus, their performance does not change with p .
On the contrary, the performance of the schemes without ﬁlters and SGD schemes with ﬁlter do not have a monotonic behavior with p .
This observation shows that these schemes are not able to handle the outliers effectively.
Further, all the schemes converge to the corresponding oracle schemes when p = 1 .
Dependence on number of samples n : Figs.
1b and 2b show how the estimation error of different schemes varies with n .
We observe that the parameter estimation errors computed using Oracle
SGD and Oracle GD schemes decrease as the number of samples increases.
This trend is because the oracle schemes consider only the true samples.
Also, the Oracle GD scheme performs better than the Oracle SGD scheme.
Further, the estimation errors computed using our proposed schemes with median and trimmed mean based ﬁlters perform better than GD without ﬁlter.
Hence, the ﬁlters mitigate the effect of arbitrary outliers and having more samples reduces the estimation errors.
Dependence on dimension d : In Figs.
1c and 2c, we plot the error in different schemes as a function of the dimension d .
We note that due to the presence of arbitrary outliers, the errors increase as the dimension increases for GD without ﬁlter.
Further, we observe a performance improvement using our proposed schemes with median and trimmed mean-based ﬁlters compared to the other benchmarking schemes, supporting our previous conclusions that the ﬁlters mitigate the effect of outliers.
However, we observe a slow increase in the parameter estimation errors using our proposed schemes with median and trimmed mean based ﬁlters as we increase the dimension for ﬁxed number of samples.
We observe that the errors when using the oracle schemes also increase slowly as the dimension is increased (Wu et al.
Comparison of runtimes: From Table 1, we observe that the SGD schemes run faster than their GD counterparts.
This is because the SGD utilizes only one sample for the gradient whereas GD utilizes all the samples for the gradient.
Further, the usage of median and trimmed mean based ﬁlters along with SGD or GD increases their computation times.
However, we reiterate that using the GD scheme with ﬁlters offers the best performance in terms of error (see Fig. 2).
Further, the runtimes of the trimmed mean-based schemes are signiﬁcantly higher than those of the median-based schemes.
To summarize, from our results, we conclude that GD schemes offer better performance than the
SGD schemes.
Also, we deduce that compared to the trimmed mean-based scheme, the medianbased ﬁlter is computationally more efﬁcient while offering similar or lower estimation error.
Moreover, the median-based scheme is parameter free (the trimmed mean depends on the parameter β ) and enjoys strong theoretical guarantees (see Theorem 4.3).
Overall, the median-based GD algorithm is the most effective approach to estimate the NN parameters in the presence of the outliers and performs slightly better than trimmed mean-based GD algorithm.
Further, we observe that GD schemes with ﬁlters perform better than Oracle SGD scheme.
C ONCLUSION
In this paper, we proposed an algorithm for estimation of the parameters of a single-layer ReLU neural network from the truncated Gaussian samples where each sample was assumed to be an arbitrary outlier with a ﬁxed probability.
Our only assumption was that the bias vector was nonnegative.
We derived a GD-based algorithm for parameter estimation and incorporated ﬁlters to handle the outliers.
We analyzed the sample complexity of the proposed algorithm in terms of the parameter estimation error.
The efﬁcacy of our approach was also demonstrated using numerical experiments.
Removing the non-negativity assumption on the bias vector is a promising direction for future work.
Also, extending our results to multi-layer neural networks is also an interesting problem to consider in the future.