L EARNING D ISTRIBUTIONS G ENERATED BY S INGLEL AYER R E LU N ETWORKS IN THE P RESENCE OF A RBITRARY O UTLIERS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider a set of data samples such that a constant fraction of the samples are arbitrary outliers and the rest are the output samples of a single-layer neural network (NN) with rectiﬁed linear unit (ReLU) activation.
The goal of this paper is to estimate the parameters (weight matrix and bias vector) of the NN assuming the bias vector to be non-negative.
Our proposed method is a two-step algorithm.
We ﬁrst estimate the norms of the rows of the weight matrix and the bias vector using the gradient descent algorithm.
Here, we also incorporate either the median or the trimmed mean based ﬁlters to mitigate the effect of the arbitrary outliers.
Next, we estimate the angles between any two row vectors of the weight matrix.
Combining the estimates of the norms and the angles, we obtain the ﬁnal estimate of the weight matrix.
Further, we prove that O ( 1 (cid:15)p 4 log dδ ) samples are sufﬁcient for our algorithm to estimate the NN parameters within an error of (cid:15) with probability 1 − δ when the probability of a sample being uncorrupted is p and the problem dimension is d .
Our theoretical and simulation results provide insights on how the estimation of the NN parameters depends on the probability of a sample being uncorrupted, the number of samples, and the problem dimension.
I NTRODUCTION
A fundamental challenge in machine learning and statistics is to estimate a high dimensional distribution given a set of observed data samples from the distribution.
One solution technique is the deep generative method that models the unknown distribution as the output distribution of a neural network (NN) when the input of the NN is drawn from a known distribution like standard Gaussian (Goodfellow et al.
Arjovsky et al.
; Radford et al.
; Kingma & Welling (2013);
Van Oord et al.
Then, the unknown distribution is estimated by learning the NN parameters from the data samples.
Several approaches like generative adversarial networks (Goodfellow et al.
(2014); Arjovsky et al.
; Radford et al.
(2015)), variational autoencoders (Kingma & Welling (2013)), and autoregressive models (Van Oord et al.
(2016)) have been proposed to train the NN.
However, they do not provide theoretical guarantees for the parameter learning.
So, we address the problem of computing the sample complexity for the parameter estimation of a single-layer NN with rectiﬁed linear unit (ReLU) activation using a corrupted dataset.
Here, the corrupted dataset refers to the model wherein every known output sample is an arbitrary outlier with a ﬁxed probability and this model is relevant to the datasets impacted by a Byzantine.
Motivated by the deep generative models, we consider the unsupervised learning framework, i.e., we assume the knowledge of the output samples but, the corresponding inputs are unknown and drawn from the standard Gaussian distribution.
We start with a brief review of the related literature in the next subsection.
1. R ELATED W ORKS
An important NN parameter estimation model is the supervised learning framework in which we estimate the parameters using noiseless output samples of a NN.
A large class of algorithms for supervised learning is available in the literature, which relies on the stochastic gradient descent
(SGD)-based algorithms (Goel et al.
Allen-Zhu et al.
; Chen et al.
Oymak (2019)) or the gradient descent (GD)-based approach (Cao & Gu (2019); Du et al.
Some studies have also considered the unsupervised learning framework wherein the output labels corresponding to the input samples of the NN are unavailable (Mazumdar & Rawat (2018)
; Wu et al.
(2019); Lei et al.
A major shortcoming of these works is that they assume that the available samples are not corrupted by noise or outliers.
Therefore, we address an unsupervised learning framework wherein the output samples are potentially outliers with a ﬁxed probability.
Several works in the literature have addressed the problem of corrupted samples in the context of supervised learning.
The existing works have considered the learning of the parameters of a NN with ReLU activation using both SGD-based (Bakshi et al.
; Goel et al.
; Mukherjee &
Muthukumar (2020)) and GD-based algorithms(Zhang et al.
Frei et al.
; Vempala &
Wilmes (2019)).
However, to the best of our knowledge, the estimation of a NN in the presence of noise or outliers in the unsupervised learning framework has not been studied in the literature.
Our setting of unsupervised parameter estimation is also related to the area of robust statistics as described below.
We recall that the goal of our paper is to estimate the parameters of a NN using corrupted output samples assuming that the input distribution is Gaussian.
Mathematically, this problem is equivalent to estimating the parameters of a truncated Gaussian distribution (see Section 2 for details).
The area of robust statistics also deals with the estimation of high dimensional distributions like Gaussian, Gaussian product, and Gaussian mixture distributions where a constant fraction of the samples were corrupted by noise (Huber (2004);
Hampel et al.
Lai et al.
; Diakonikolas et al.
; Diakonikolas & Kane (2019); Kane (2021)).
However, in this paper, we consider parameter estimation from truncated Gaussian samples in the adversial setting which has not been studied in the literature.
1. O UR C ONTRIBUTIONS
In this subsection, we present our major contributions as the following.
• Parameter Estimation (Sections 2 and 3):
We estimate the parameters (weight matrix and bias vector) of a single-layer NN with ReLU in the unsupervised learning framework where each output sample can potentially be an arbitrary outlier with a ﬁxed probability.
We propose an estimation algorithm that has two steps: (i) we estimate the row norms of the weight matrix and the bias vector using GD along with either the median or trimmed mean-based ﬁlters to mitigate the effect of arbitrary outliers, and (ii) we estimate the angles between any two row vectors of the weight matrix using a simple geometric result (Williamson & Shmoys, 2011, Lemma 6.7).
• Theoretical guarantees (Section 4): We show that the proposed algorithm requires O ( 1 (cid:15)p 4 log dδ ) samples to estimate the network parameters within an error of (cid:15) with probability 1 − δ when the probability of a sample being uncorrupted is p .
Here, (cid:15) > 0 represents the estimation error.
• Empirical validation (Section 5): We evaluate the performance of our algorithm empirically in terms of its relation to the probability of a sample being uncorrupted, number of samples, and dimension (see Figs. 1 and 2).
We see that our proposed ﬁltering schemes ensure robustness to the arbitrary outliers.
Also, the performance of our algorithm improves with the increase in the probability of a sample being uncorrupted and the number of samples, as expected.
Further, we note that the estimation error increases slowly as the dimension grows for a ﬁxed number of samples in the presence of arbitrary outliers.
These observations from the empirical results are consistent with our theoretical results.
To summarize, in this paper, we estimate the parameters of a single-layer NN with ReLU activation in the presence of arbitrary outliers.
The results obtained in this paper provide insights to generalize to a multi-layer NN with various activation functions and obtain generalization bounds.
L EARNING A N EURAL N ETWORK M ODEL USING C ORRUPTED O UTPUT S AMPLES
We consider a single-layer ReLU NN.
Let the weight matrix of the NN be denoted by W ∈ R d × k and the bias vector by b ∈ R d .
The input to the NN is denoted by the latent variable z ∈ R k .
We assume that the variable z is drawn from the standard Gaussian distribution.
Thus, the output of the network is the random vector (RVe) x ∈ R d given by x = ReLU ( Wz + b ) , where z ∼ N ( 0 , I k ) , (1) where N ( 0 , I k ) denotes the Gaussian distribution with mean 0 and covariance matrix I k which is the identity matrix of size k × k .
Let the distribution of the RVe x be denoted by D ( W , b ) .
Then, our goal is to estimate the unknown parameters W and b of the distribution D ( W , b ) using the knowledge of n output samples x 1 , x 2 , . . .
, x n .
Here, we assume that a fraction of the samples are arbitrary outliers.
The distribution of the observed samples are modeled using the Huber’s pcontamination model 1 deﬁned as the following.
Deﬁnition 2.1 (Huber’s p -Contamination Model Huber (1964)) .
The observed samples are said to be following Huber’s contamination model if any given sample is drawn from the true distribution
D and an arbitrary distribution G with probability p , and 1 − p , respectively.
In other words, the observed samples are drawn from the mixture distribution, D p ( W , b ) given by
[Equation]
Note that if b has large negative values, then most of the output samples are zeros due to the ReLU operation.
Further, if W is a zero matrix, then any negative coordinate of b cannot be identiﬁed as it is reset to zero after the ReLU operation.
Hence, we assume b to be non-negative.
Our estimation problem is challenging due to two reasons: 1) ReLU operation is not invertible, and therefore, estimation of W and b utilizing maximum likelihood of the probability density function of RVe x is intractable; 2) the distribution of the corrupted samples is unknown.
We tackle these issues using a new formulation combining the gradient decent algorithm and a ﬁltering technique.
The proposed algorithm is presented in the next section.
E STIMATION OF P ARAMETERS OF S INGLE - LAYER NN
To design our algorithm, we ﬁrst note that only WW T can be identiﬁed from D ( W , b ) .
This is because of the fact that for any vector b and any matrices W 1 and W 2 satisfying W 1 W T 1 =
W 2 W T 2 , we have D ( W 1 , b ) =
D ( W 2 , b ) (Wu et al.
Therefore, our proposed algorithm estimates
WW T ∈ R
d × k and b ∈ R d from the observed samples from D p ( W , b ) .
Next, we note that ( i, j ) - th entry (cid:0) WW T (cid:1) ( i, j ) of the symmetric matrix WW T is (cid:107) W ( i, :) (cid:107) 2 (cid:107) W
( j, :) (cid:107) 2 cos( θ ij ) , where θ ij = θ ji is the angle between vectors W ( i, :) and W ( j, :) which are the i -th and j -th rows of matrix
W , respectively.
Also, note that the (cid:96) p norm of a vector is given by (cid:107) x (cid:107) p
=
( (cid:80) i | x ( i ) | p ) 1 /p .
Thus, we can construct the matrix WW T using the row norms {(cid:107) W ( i, :) (cid:107) 2 } di =1 and angles { θ ij } di,j =1 .
Our estimation algorithm consists of two steps: 1) estimation of row norms of W and bias vector b ; and 2) estimation of angles between the row vectors of W .
We describe the two steps in the following subsections.
3. E STIMATION OF R OW N ORMS AND B IAS V ECTOR
In this subsection, we estimate (cid:107) W ( i, :) (cid:107) 2 ∈ R
and
b ( i ) ∈ R , for i ∈
[ d ] = { 1 , 2 , . . .
, d } .
Suppose x ∼ D
( W , b ) .
Its i -th coordinate can be written as
[Equation]
where W ( i, :) T z + b ( i ) ∼ N ( b ( i ) , (cid:107) W ( i, :) (cid:107) 22 ) .
Thus, to compute b ( i ) and (cid:107) W ( i, :) (cid:107) 22 , it is enough to consider the i -th entries of the observed samples.
Also, in (3), the ReLU operator sets x ( i ) to zero if W ( i, :) T z + b ( i ) is negative.
Consequently, we estimate b ( i ) and (cid:107) W ( i, :) (cid:107) 22 using the positive entries of the observed samples.
Hence, estimation of b ( i ) and (cid:107) W ( i, :) (cid:107) 2 is equivalent to estimating the parameters of a one-dimensional normal distribution using positive samples, i.e., the samples that belong to the set R > 0 :
= { x ( i ) ∈ R : x ( i ) > 0 } .
We use µ ∗ =
b ( i ) and σ ∗ 2 = (cid:107) W
( i, :) (cid:107) 2 to denote the parameters of this one-dimensional normal distribution.
The estimation of µ ∗ and σ ∗ 2 using corrupted samples is discussed next.
NN with outliers
Input: Samples x 1 , . . .
, x n ∈ R d 1 for i ∈
[ d ] do
X > 0 ← { x m , m ∈ [ n ] : x m ( i ) > 0 } ˆ v = ProjGD ( X > 0 )
ˆ Σ i,i ← 1 / ˆ v (1) ˆ b ( i ) ← max { 0 , ˆ v (2) /
ˆ v (1) } 6 for i < j ∈
[ d ] do
Compute ˆ θ ij using (12) ˆ Σ ( i, j ) ← (cid:113) ˆ Σ ( i, i ) ˆ Σ ( j, j ) cos(ˆ θ ij ) ˆ Σ ( j, i ) ← ˆ Σ ( i, j )
Output: ˆ Σ ∈ R d × d , ˆ b ∈ R d
Input: Samples X > 0 , Parameters T , γ t , and n b 1 Compute { x j
( i ) }
n b j =1 from X > 0 using (7) 2 ˜ g x ← ﬁlter (cid:16) { x j ( i ) } n b j =1 (cid:17) 3 v 0 ← [0 0] 4 for t = 1 , 2 , . .
, T do µ ← v t − 1 (2) / v t − 1 (1) σ 2 ← 1 / v t − 1 (1)
˜ g
z ← (cid:2) − (˘ σ 2 + ˘ µ 2 ) / 2 ˘ µ (cid:3) T using (5) and (6) ˜ g t ← ˜ g x + ˜ g
z ˜ v t ← v t − 1 − γ t − 1 ˜ g
t − v t ← P (˜ v t ) using (10)
Output: v T ∈ R
We determine the parameters of the univariate normal distribution using maximum likelihood estimation.
For this, we deﬁne (cid:96) ( µ, σ ; x ( i )) as the negative log-likelihood that x ( i ) ∼ N ( µ, σ 2 ; R > 0 ) is drawn from the truncated normal distribution derived from N ( µ, σ 2 ) by bounding the random variable (RV) to the set of nonnegative real numbers.
Since the maximum likelihood estimate minimizes the negative likelihood, the parameter estimates are given by arg min µ,σ (cid:96) ( µ, σ ; x ( i )) .
Further, from Daskalakis et al.
(2018), (cid:96) ( µ, σ ; x ( i )) is a convex function with respect to the reparameterization: v = [1 /σ 2 ; µ/σ 2 ] ∈ R 2 .
We use (cid:96) ( v ; x ( i )) to denote the negative log-likelihood after the reparameterization.
Thus, the estimate of the new parameter v ∈ R 2 is given by arg min v l ( v ) , where (cid:96) ( v ) =
E x ( i ) [ (cid:96) ( v ; x ( i ))] is the expected negative log-likelihood with respect to x ( i ) .
The above optimization problem can be solved using a learning algorithm like GD or SGD.
The SGD algorithm utilizes only one sample for the gradient computation which introduces large variance due to the stochasticity and low reliability in the presence of arbitrary outliers.
Therefore, we use the
GD algorithm that utilizes all the samples for the gradient.
To derive the GD algorithm, we compute the gradient, ∇ (cid:96) ( v ) =
E x , z [ g ( i )]
where
[Equation]
Here, x ( i ) ∼ N ( µ ∗ , σ ∗ 2 ; R > 0 ) and z ( i ) ∼ N ( µ, σ 2 ; R > 0 ) .
Also, µ ∗ and σ ∗ 2 are the parameters of the true distribution, and µ and σ 2 are functions of v .
In (4), E z [ g z ( i )] admits a closed form expression (Johnson et al.
, ˜ g
z =
E z [ g z ( i )]
= (cid:2) − (˘ σ 2 +
[Equation] [Equation]
T .
Here, we deﬁne
[Equation]
i
[Equation]
i
[Equation]
where φ ( · ) and Φ( · ) denote the probability density function and the cumulative distribution function of the standard normal distribution, respectively.
We also need E x
[ g x ( i )] to compute E x , z [ g ( i )] .
Since the value of E x [ g x ( i )] is not known, we use the sample mean to estimate E
x
[ g x ( i )] .
For this, all the positive samples are partitioned into n b batches.
Let S j be the j -th batch.
We then compute the vector x j ( i ) ∈ R 2 as follows:
[Equation]
We then combine the n b vectors { x j ( i ) } n b j =1 to estimate E x , z [ g ( i )] .
Recall that the observed samples are from the mixture distribution D p ( W , b ) .
Therefore, to compute the expectation with respect to the true distribution D ( W , b ) , a ﬁlter is applied on this set of n b vectors to mitigate the effect of the arbitrary outliers and obtain E x [ g x ( i )] .
We consider two ﬁlters: median and trimmed mean.
Median: The median based ﬁlter possesses the following robustness property.
Typically, a median is the value separating the higher half from the lower half of a given set of points.
If more than half of a given set of points are in [ − M, M ] for some M > 0 , then their median must be in [ − M, M ] .
Thus, the median of { x j ( i ) } n b j =1 is the vector computed either from all the true samples or outliers whose magnitudes are comparable to the true samples.
Trimmed Mean: The trimmed mean removes the vectors among the set of n b vectors with relatively large and small values and computes the estimate of E x
[ g x ( i )] as the mean of the remaining vectors.
Here, we use the parameter β to indicate the number of vectors to be discarded.
This technique prunes the vectors x j ( i ) , j ∈ [ n b ] that are computed from the batches with outliers having relatively high or low magnitudes.
Therefore, we obtain the gradient estimate as
[Equation]
where ﬁlter is either median or trimmed mean, and ˘ µ and ˘ σ 2 are given in (5) and (6), respectively.
Having computed the gradient estimate, we next present the proposed GD algorithm.
In the t -th iteration, the algorithm performs the following three steps:
Gradient Computation Step: In this step, the gradient ˜ g t ∈ R 2 is computed from the observed positive samples using (8). 2.
Update Step: The gradient computed in the previous step is used to perform the GD update:
[Equation]
where γ t − 1 > 0 is the diminishing step size.
Projection Step: The gradient of l ( v ) is a strongly convex function of v , if v belongs to a
bounded region.
To control the strong-convexity of the objective function, we project the update vector v into the domain D r = (cid:8) v ∈ R 2 : 1 /r ≤ v (1) ≤ r, | v (2) | ≤ r (cid:9) .
Thus, the projection is
[Equation]
The GD algorithm is run for T iterations where T is the algorithm parameter.
The overall algorithm is summarized in Algorithm 2.
In short, this subsection presented the ﬁrst step of our algorithm that uses the GD algorithm to obtain the estimate of the bias vector and the row norms of the weight matrix.
We next discuss the estimation of the angle between row vectors.
3. E STIMATION OF A NGLES B ETWEEN THE R OWS
To estimate θ ij , we utilize the fact that the angle between any two vectors can be estimated from their inner products with a random Gaussian vector (Williamson & Shmoys, 2011, Lemma 6.7), i.e., for any two non-zero vectors u , v ∈ R k , the following holds:
[Equation]
Also, for the true samples, we have x ( i ) =
ReLU ( W ( i, :) T z + b ( i ))
(cid:54) =
if b is non-negative and
W ( i, :) T z > 0 .
Thus, we estimate ˆ θ ij as given below:
[Equation]
where 1 ( · ) denotes the indicator function and ˆ b is the estimate of the bias vector computed in the previous step using the projected GD algorithm.
Combining the procedures in Sections 3.1 and 3.2, we obtain the overall estimation algorithm.
The algorithm is summarized in Algorithm 1 where ˆ Σ is the estimate of WW T .
Here, Steps 1-5 estimate the row norms of W and b , and Steps 6-9 estimate the angles between any two row vectors of W .
We note that our algorithm is different from the unsupervised learning algorithm in Wu et al.
in the following aspects.
We do not rescale the unknown parameters (see Step 1 of Algorithmin Wu et al.
We employ median or trimmed mean based ﬁlter to compute the full gradient and utilize the GD algorithm instead of SGD used in Wu et al.
T HEORETICAL G UARANTEES FOR P ARAMETER E STIMATION
In this section, we provide our main result which characterizes the sample complexity required to achieve a given small error in parameter estimation.
To arrive at the main result, we ﬁrst present the error bounds for the two steps of Algorithm 1, which are described in Secs.
3.1 and 3.2, respectively.
The following proposition provides the estimation error bounds for the ﬁrst step, i.e., the estimation of the bias vector b and the norms of the row vectors of WW T .
It shows that we can estimate b ( i ) within an additive error of Ξ (cid:107) W ( i, :) (cid:107) and Σ ( i, i ) within an additive error of Ξ (cid:107) W ( i, :) (cid:107) 2 where Ξ decreases with both the number of samples and the total number of iterations T .
Proposition 4.1.
Suppose we initialize the projected gradient step (Algorithm 2) such that (cid:107) v 0 − v ∗ (cid:107) =
V and choose the step size γ t = γ and the number of iterations as T .
Then, there exists γ >such that for any δ ∈ (0 , 1) , the output ( ˆ Σ ( i, i ) , ˆ b ( i )) of the projected GD step of our algorithm for any i ∈
[ d ] satisﬁes
[Equation] [Equation]
(14) with probability at least 1 − δ if the number of data samples n ≥ 1 p
[Equation]
Ψ
[Equation]
log 1 δ .
Here, the error bound is Ξ = V (cid:16)
[Equation]
with η, L > 0 being constants that depend only on the parameter r .
Also, p is the probability of a sample being uncorrupted and Ψ (cid:15) ∈ [1 / 2 p, 1] is such that O ( (cid:15) 3 ) ≤ Ψ (cid:15) ≤ O ( √ (cid:15) ) with (cid:15)
>
O (cid:0) p − 1 / 3 (cid:1) where the order constants depend only on r .
The following proposition bounds the estimation error in the second step, i.e., the estimation of the angle between any two row vectors W ( i, :) and W ( j, :) (where i (cid:54) =
j ∈ [ d ] ).
The result shows that we can estimate θ ij within an additive error of Ξ .
Proposition 4.2.
Assume that (13) and (14) hold.
Then, for a ﬁxed pair of i (cid:54) =
j ∈ [ d ] and δ ∈ (0 , 1) , the estimate ˆ θ ij of Algorithm 1 satisﬁes
[Equation]
(15) with probability at least 1 − δ if the number of data samples n ≥
[Equation]
2 log 1 δ .
Here, the error bound is Ξ = V (cid:16)
[Equation]
+ L (cid:17)
[Equation]
+
[Equation]
(cid:15) ( L + η ) Lη with η, L > 0 being constants that depend only on the parameter r .
Also, p is the probability of a sample being uncorrupted and Ψ (cid:15) ∈ [1 / 2 p, 1] is such that O ( (cid:15) 3 ) ≤ Ψ (cid:15) ≤ O ( √ (cid:15) ) with (cid:15)
>
O (cid:0) p − 1 / 3 (cid:1) where the order constants depend only on r .
Combining Propositions 4.1 and 4.2, we next present the sample complexity needed to achieve a small parameter estimation error.
Speciﬁcally, the diagonal entries, Σ ( i, i ) , for i
∈
[ d ] are obtained from Proposition 4.1.
Further, the off-diagonal entries of Σ are obtained by combining the Propositions 4.1 and 4.2 to obtain the ﬁnal result as follows:
Theorem 4.3.
Suppose we initialize the projected gradient step (Algorithm 2) such that (cid:107) v 0 − v ∗ (cid:107) =
V and choose the step size γ t = γ and the number of iterations as T .
Then, there exists γ > 0 such that for δ ∈ (0 , 1) , the output of Algorithm 1, ( ˆ Σ , ˆ b ) satisﬁes
[Equation]
with probability at least 1 − δ if the number of data samples n = O ( 1 (cid:15)p 4 log dδ ) .
Here, p is the probability of a sample being uncorrupted and Ψ (cid:15) ∈ [1 / 2 p, 1] is such that O ( (cid:15) 3 ) ≤ Ψ (cid:15) ≤ O ( √ (cid:15) ) with (cid:15)
>
O (cid:0) p − 1 / 3 (cid:1) where the order constants depend only on r .
We note that in Propositions 4.1 and 4.2, and Theorem 4.3, η, L and the other order constants depend on the parameter r > 0 .
Recall that the parameter r is required for the convexity region D r
and we can choose r to be a large constant.
Thus, r is a relatively insensitive parameter which does not require any hand-tuning.
The two terms in the error bound in Theorem 4.3 can be interpreted as the following.
The ﬁrst term (cid:16) 1 − Lη + L (cid:17) T V captures the convergence error due to the GD algorithm.
For any upper bound on the convergence error Ω > 0 , Algorithm 1 runs in time (log Ω − log V ) / log (cid:16) η η + L (cid:17) .
This relation shows that smaller convergence errors require more iterations.
The second term 2 (cid:15) ( L + η ) Lη captures the error due to the gradient computed using the observed samples.
Two factors contribute to this error: 1) difference between the (true) sample mean and the true mean, and 2) the arbitrary outliers.
Also, the difference between the sample mean and true mean decreases with the number of samples.
Consequently, for smaller error (cid:15) , we need a large number of samples n = O ( 1 (cid:15)p 4 log dδ ) .
We also note that the lower bound on (cid:15) decreases with p .
As the probability of a sample being uncorrupted p reduces, we observe that the error bound in (16) becomes larger as expected.
Also, as the probability of a sample being uncorrupted decreases, the number of samples n required for the error bound increases.
Moreover, the obtained error bounds increase with the problem dimension for a ﬁxed number of samples.
Our results are also comparable with the existing error bounds for ReLU parameter estimation without outliers from (Wu et al., 2019, Theorem 1).
Speciﬁcally, if there are no outliers, for any ζ, δ ∈ (0 , 1) , the number of samples n =
O ( 1 ζ 2 log dδ ) are sufﬁcient for the SGD algorithm in Wu et al.
(2019) to achieve the following error bounds with probability at least 1 − 2 δ ,
[Equation]
Comparing the above bounds with (16) and (17), we observe that our bounds are larger due to the presence of arbitrary outliers.
In particular, the ﬁrst term V (cid:16) 1 − Lη + L (cid:17) T < 1 in our error bounds is similar to ζ < 1 in (18).
However, the second error term 2 (cid:15) ( L + η ) Lη is not present in (18).
Additionally, in (16), the extra term 3 π (2 − p ) further increases the bound compared to the error bound in (18).
Further, our sample complexity depends on the probability of a sample being uncorrupted p and the second error term (cid:15) whereas the complexity in (Wu et al., 2019, Theorem 1) depends on the square of the convergence error term ζ in their sample complexity.
Also, when there is no outlier ( p = 1 ), the two sample complexities do not match because the two results deal with different algorithms (GD and SGD).
S IMULATION R ESULTS
In this section, we provide numerical results to verify the performance of our algorithm.
In our setup, the columns of W are chosen as the left singular vectors of random matrices from the standard
Gaussian distribution.
For b , we use a random vector from the standard normal distribution whose negative values are replaced with zeros.
The mixture of samples are generated such that a sample comes from D ( W , b ) with probability p and from G = N (5 , 1) with probability 1 − p .
The hyperparameters in our proposed algorithm are set as r = 3 and γ t = 10 .
1 t .
Also, we choose the total number of GD iterations T to be |X > 0 | / 100 where |X > 0 | is the number of positive output samples.
We compute two error metrics from the estimated parameters and the ground truth, (cid:107) ˆ Σ −
WW T (cid:107) F / (cid:107) W (cid:107) 2 F and (cid:107) ˆ b − b (cid:107) 2 / (cid:107) W (cid:107) F .
Also, we compare our algorithm with two other schemes: oracle scheme (estimation using the true samples only), and scheme without ﬁlter.
The results are shown in Figs.
1 and 2, and Table 1, and the observations from them are as follows.
Effect of ﬁlters: From Fig.
2, we observe that the GD schemes perform better than the corresponding
SGD schemes.
Further, the performance is improved when we use median based ﬁlter along with
GD or SGD (for instance, as seen by comparing the performances of GD without Filter and GD with
Median in Fig. 1).
Thus, the ﬁlter ensures that the effect of the outliers is reduced, and the curves are
[Figure] [Figure]
i n
[Figure]
closer to the Oracle GD scheme.
Finally, from Fig. 1, we also infer that the median-based approach performs slightly better than the trimmed mean-based approach
[Figure]
Dependence on probability of a sample being uncorrupted p : The variation in the error estimation as a function of p is shown in Figs.
1a and 2a.
The performance of the proposed schemes improve with p , as expected.
This trend is because as p increases, the fraction of outliers in the observed samples decreases, which leads to better performance.
Note that the oracle schemes assume the knowledge of true samples.
Thus, their performance does not change with p .
On the contrary, the performance of the schemes without ﬁlters and SGD schemes with ﬁlter do not have a monotonic behavior with
[Table]
p .
This observation shows that these schemes are not able to handle the outliers effectively.
Further, all the schemes converge to the corresponding oracle schemes when p = 1 .
Dependence on number of samples n : Figs.
1b and 2b show how the estimation error of different schemes varies with n .
We observe that the parameter estimation errors computed using Oracle
SGD and Oracle GD schemes decrease as the number of samples increases.
This trend is because the oracle schemes consider only the true samples.
Also, the Oracle GD scheme performs better than the Oracle SGD scheme.
Further, the estimation errors computed using our proposed schemes with median and trimmed mean based ﬁlters perform better than GD without ﬁlter.
Hence, the ﬁlters mitigate the effect of arbitrary outliers and having more samples reduces the estimation errors.
Dependence on dimension d : In Figs.
1c and 2c, we plot the error in different schemes as a function of the dimension d .
We note that due to the presence of arbitrary outliers, the errors increase as the dimension increases for GD without ﬁlter.
Further, we observe a performance improvement using our proposed schemes with median and trimmed mean-based ﬁlters compared to the other benchmarking schemes, supporting our previous conclusions that the ﬁlters mitigate the effect of outliers.
However, we observe a slow increase in the parameter estimation errors using our proposed schemes with median and trimmed mean based ﬁlters as we increase the dimension for ﬁxed number of samples.
We observe that the errors when using the oracle schemes also increase slowly as the dimension is increased (Wu et al.
Comparison of runtimes: From Table 1, we observe that the SGD schemes run faster than their GD counterparts.
This is because the SGD utilizes only one sample for the gradient whereas GD utilizes all the samples for the gradient.
Further, the usage of median and trimmed mean based ﬁlters along with SGD or GD increases their computation times.
However, we reiterate that using the GD scheme with ﬁlters offers the best performance in terms of error (see Fig. 2).
Further, the runtimes of the trimmed mean-based schemes are signiﬁcantly higher than those of the median-based schemes.
To summarize, from our results, we conclude that GD schemes offer better performance than the
SGD schemes.
Also, we deduce that compared to the trimmed mean-based scheme, the medianbased ﬁlter is computationally more efﬁcient while offering similar or lower estimation error.
Moreover, the median-based scheme is parameter free (the trimmed mean depends on the parameter β ) and enjoys strong theoretical guarantees (see Theorem 4.3).
Overall, the median-based GD algorithm is the most effective approach to estimate the NN parameters in the presence of the outliers and performs slightly better than trimmed mean-based GD algorithm.
Further, we observe that GD schemes with ﬁlters perform better than Oracle SGD scheme.
C ONCLUSION
In this paper, we proposed an algorithm for estimation of the parameters of a single-layer ReLU neural network from the truncated Gaussian samples where each sample was assumed to be an arbitrary outlier with a ﬁxed probability.
Our only assumption was that the bias vector was nonnegative.
We derived a GD-based algorithm for parameter estimation and incorporated ﬁlters to handle the outliers.
We analyzed the sample complexity of the proposed algorithm in terms of the parameter estimation error.
The efﬁcacy of our approach was also demonstrated using numerical experiments.
Removing the non-negativity assumption on the bias vector is a promising direction for future work.
Also, extending our results to multi-layer neural networks is also an interesting problem to consider in the future.
We provide our code for reference in the supplementary material along with a read me ﬁle for instructions.