Learning Distributions Generated by Single-Layer ReLU Networks in the Presence of Arbitrary Outliers
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider a set of data samples such that a fraction of the samples are arbitraryoutliers, and the rest are the output samples of a single-layer neural network withrectified linear unit (ReLU) activation.
Our goal is to estimate the parameters(weight matrix and bias vector) of the neural network, assuming the bias vector tobe non-negative.
We estimate the network parameters using the gradient descentalgorithm combined with either the median- or trimmed mean-based filters tomitigate the effect of the arbitrary outliers.
We then prove that ˜ O (cid:16) 1 p 2 + 1 ϵ 2 p (cid:17)samples and ˜ O (cid:16) d 2 p 2 + d 2 ϵ 2 p (cid:17) time are sufficient for our algorithm to estimate theneural network parameters within an error of ϵ when the outlier probability is 1 − p ,where 2 / 3 < p ≤ 1 and the problem dimension is d (with log factors being ignoredhere).
Our theoretical and simulation results provide insights into the trainingcomplexity of ReLU neural networks in terms of the probability of outliers andproblem dimension.
Introduction
Probability densities based on generative models are used in various applications such as imageinpainting, deblurring, and generating new images.
In particular, generative models representing thedistribution over natural images are used for image inpainting and to generate new images Van Oordet al.
A critical obstacle in generative modeling is building complex and expressive modelsthat are tractable and scalable.
Hence, a fundamental challenge in machine learning, as well asin statistics, is to estimate a high dimensional distribution using a set of observed data samplesgenerated by the distribution.
One solution technique is the deep generative method that modelsthe unknown distribution as the output distribution of a neural network when the input of the neuralnetwork is drawn from a known distribution (for example, standard Gaussian in image generation bygenerative adversarial networks (GANs))
(Goodfellow et al., 2014; Arjovsky et al., 2017; Radfordet al., 2015; Kingma, Welling, 2013; Van Oord et al., 2016).
Here, the unknown distribution isestimated by learning the neural network parameters from the data samples.
Several deep generativemodels like GANs (Goodfellow et al., 2014; Arjovsky et al., 2017; Radford et al., 2015), variationalautoencoders (Kingma, Welling, 2013), and autoregressive models (Van Oord et al., 2016) havebeen proposed.
However, the guarantees associated with such models are generally unknown.
So,we address the problem of computing the sample complexity for the parameter estimation of asingle-layer neural network with rectified linear unit (ReLU) activation using a corrupted dataset.
corrupted dataset to the model wherein the output samples consists of a fractionof arbitrary outliers introduced by an adversary (Byzantine).
Motivated by the deep generative models, we consider the unsupervised learning framework, i.e., we assume the knowledge of theoutput samples, but the corresponding inputs are unknown and drawn from the standard Gaussiandistribution.
We start with a brief review of the related literature.
The algorithms for neural network parameter estimation have been widely studied under both supervised and unsupervised learning frameworks.
Theestimation algorithms generally rely on the stochastic gradient descent (SGD)-based algorithm
(Goelet al., 2018; Allen-Zhu et al., 2019;
Chen et al., 2022; Oymak, 2019; Lei et al., 2020); or the gradientdescent (GD)-based approach (Cao, Gu, 2019; Du et al., 2019).
Further, some works have consideredthe sample complexity of a single-layer NN with ReLU activation in the unsupervised learningframework (Mazumdar, Rawat, 2018; Wu et al., 2019).
It is known that when the bias vector isassumed to be a random vector, the column space of the weight matrix can be recovered withinan error of O ( d ) when the output dimension is d (Mazumdar, Rawat, 2018).
Further, when thebias vector is assumed to be nonnegative, ˜ O
(1 /ϵ 2 ) samples and ˜ O
( d 2 /ϵ 2 ) iterations are sufficient toestimate the parameters of the ReLU network within an error of ϵ .
However, none of the above workshave considered the problem of corrupted samples.
To the best of our knowledge, the estimationof a neural network in the presence of noise or outliers has been addressed only in the context ofsupervised learning (Bakshi et al., 2019;
Goel et al., 2019; Mukherjee, Muthukumar, 2020; Zhanget al., 2019; Frei et al., 2020; Vempala, Wilmes, 2019).
Our paper focuses on a special case ofunsupervised learning with corrupted samples wherein we consider a single layer ReLU network.
Our neural network parameter estimation is also related to the area of robust statistics.
Mathematically,the problem of learning the distribution using a neural network is equivalent to estimating theparameters of a truncated Gaussian distribution (see Section 2 for details).
Robust statistics alsodeals with the estimation of high dimensional distributions like Gaussian, Gaussian product, and
Gaussian mixture distributions where a fraction of the samples are arbitrary outliers (Huber, 2004;
Hampel et al., 2011; Lai et al., 2016; Diakonikolas et al., 2019; Diakonikolas, Kane, 2019; Kane,2021).
However, parameter estimation from truncated Gaussian samples in the noisy setting has notbeen studied in the robust statistics literature, and we address this literature gap.
We present our major contributions as the following. We present our major contributions as thefollowing.
• Learning Algorithm: We devise a learning algorithm (Algorithm 1) to estimate the parameters(weight matrix and bias vector) of a single-layer ReLU neural network in the unsupervisedlearning framework where each output sample can potentially be an arbitrary outlier with afixed probability.
Our estimation algorithm uses GD along with either the median- or trimmedmean-based filters to mitigate the effect of outliers.
• Sample Complexity: Our algorithm requires ˜ O (cid:16) 1 p (cid:104) 1 p + 1 ϵ 2 (cid:105) log dδ (cid:17) samples to estimate thenetwork parameters within an error of ϵ with probability 1 − δ when the probability of a samplebeing uncorrupted is p ∈ (2 / 3 , 1] and the output dimension is d (Theorem 1).
We also characterizethe total variation distance between the estimated and true distributions.
(Corollary 1).• Lower bound: We derive a lower bound on the sample complexity which says that at leastΩ(1 /pϵ 2 ) samples are required to estimate the parameters up to an error of ϵ .
(Theorem 2).• Empirical validation (Section 5): We empirically evaluate the performance of our algorithm andshow that it is robust to the arbitrary outliers.
Also, we see that the performance of our algorithmimproves with the the probability of a sample being uncorrupted and the number of samples, andit increases slowly as the network output
dimension grows.
These observations from the empiricalresults are consistent with our theoretical results.
Overall, the results obtained in this paper show that parameter estimation of a single layer ReLUneural network is possible using robust gradient descent even in the presence of outliers.
ReLU Neural Network Learning Problem
Let the weight matrix of the ReLU neural network be denoted by W ∈ R d × m and the bias vectorby b ∈ R d .
The input to the neural network is denoted by the latent variable z ∈ R m .
We assumethat the variable z is drawn from the standard Gaussian distribution.
Thus, the output of the networkis the random vector x ∈ R d given by x = ReLU ( W z + b ) ∼ D ( W , b ) , where D ( W , b ) denotesthe distribution of x .
Our goal is to estimate the unknown parameters W and b of the distribution
D ( W , b ) using the knowledge of a corrupted set of data samples X = (cid:8) x ( n ) ∈ R d (cid:9) N n =1 .
Here,we assume that a data sample in X follows the Huber’s p -contamination model (Huber, 1964),i.e.
, a sample is drawn from D ( W , b ) with probability p , and it is an arbitrary outlier drawn froman unknown distribution D out with probability 1 − p .
Hence, a given sample x ∈ X follows thedistribution D p = p D
( W , b ) + (1 − p ) D out .
We make two observations about the learning problem.
Firstly, it is known that when all the samplesare from the true distribution, exponentially large number of samples are required to estimate thebias b , if it can take any value from the set R d (Wu et al., 2019).
Naturally, the requirement on thenumber of samples would be worse in the presence of arbitrary outliers and so, we assume b to benon-negative.
Thus, we assume the following.
Assumption 1.
The entries of b ∈ R d are all nonnegative.
Secondly, the weight matrix W may not be identifiable from the distribution D ( W , b ) .
In particular,for any unitary matrix Q ∈ R m × m , we have D ( W , b ) =
D ( W Q , b ) .
Since our goal is to learnthe distribution, learning either W or W Q is sufficient.
Thus, we focus on the learnability of theunderlying distribution and not the learnability of the neural network parameters.
Specifically, ourproposed algorithm estimates W W T ∈ R d × d and b ∈ R d .
Secondly,
We tackle these issues using a new formulation which we discuss next.
Robust Estimation Algorithm
Our algorithm is similar to that in (Wu et al., 2019) which considers the special case of p = 1 anduses SGD.
For the general case of p ≤ 1 , we combine the estimation framework with a robust filterto estimate the parameters.
To derive the robust estimation algorithm, we first consider a true sample x ∼ D ( W , b ) whose i -thelement is nonzero.
We have x i = ReLU ( W T
i z
+ b i ) ∼ N + ( b i , ∥ W i ∥ 2 ) ,
where N + ( b i , ∥ W i ∥ 2 )is the truncated normal distribution obtained by restricting the normal distribution
N ( b i , ∥ W i ∥ 2 )to the set of positive real numbers.
Hence, estimation of b i and ∥ W i ∥ is equivalent to estimatingthe parameters of a one-dimensional truncated normal distribution N + ( b i , ∥ W i ∥ 2 ) using positivesamples.
Therefore, we estimate the parameters in two steps.
In the first step, we estimate the i -thelement b i of the bias vector and the norm ∥ W i ∥ of the i -th row of the weight matrix, for i ∈ [ d ] .
The second step is the estimation of the angle θ ij between the i -th row W i ∈
R m and the j -th row
W j ∈ R m of the matrix W , for i, j ∈
[ d ] .
Then, the ( i, j ) -th entry of the symmetric matrix W W Tis given by ∥ W i ∥
∥ W j ∥ cos( θ ij ) .
The first step of the algorithm estimates the parameters of the univariate distribution given by
N + ( b i , ∥ W i ∥ 2 ) using the i -th element of the samples X
+ i
= { x i : x i > 0 , x ∈ X} via maximum likelihood estimation (Daskalakis et al., 2018).
Then, the parameter estimates are given by
[Equation]
ℓ ( µ, σ 2 ) where ℓ ( µ, σ 2 ) is the expected negative log likelihood that x i ∼ N + ( µ, σ 2 )and the expectation is with respect to the true distribution x i ∼ N + ( b i , ∥ W i ∥ 2 ) .
Further,from (Daskalakis et al., 2018), ℓ ( µ, σ 2 ) is a convex function of v = (cid:2) 1 /σµ/σ 2 (cid:3) ∈ R 2 .
Thus, wesolve the convex optimization problem of maximizing ℓ ( µ, σ 2 ) over v ,
[Equation]
The optimization problem in (1) can be solved using GD or SGD which is based on the gradient ofℓ ( v ) given by the relation (Daskalakis et al., 2018): ∇ ℓ ( v ) =
g − h ( v ) where
[Equation]
where µ = v 2 / v 1 , σ 2 = 1 / v 1 , and ϕ ( · ) and Φ( · ) denote the probability density function and thecumulative distribution function of the standard normal distribution, respectively.
The relation (3)follows directly from the closed form expressions of the first and second moments of the truncated
Gaussian distribution N + (cid:16) v 2 v 1 , 1 v 1 (cid:17) , which are functions of v only (Johnson et al., 1995).
We observe that g does not depend on v and only depends on the true distribution parameters( b i , ∥ W i ∥ 2 ) , whereas h ( v ) does not depend on the true distribution.
Therefore, the estimation of g ,which depends on the true distribution, can use the available data samples, and h ( v ) can be computedin closed form using the current iterate of v .
In other words, we compute the component of gradientg only once in the GD or SGD algorithm because it does not change across the iterations.
Thisobservation motivates us to use the GD algorithm to estimate the parameters instead of SGD.
SGDintroduces large variance due to the stochasticity and lower accuracy due to the outliers in everyiteration whereas GD introduces a small error that does not depend on the algorithm iterate.
This keyobservation and use of (3) to compute the gradient is the main difference between our algorithm andthe algorithm in (Wu et al., 2019), apart from the robust estimation aspect.
To estimate g , we partition X
+ i into batches of size N B and compute the batchwise estimate ˜ g ( b ) ,
[Equation]
where X + i,b ⊂ X + i is the b -th batch of samples.
We then combine the batchwise estimates using awell-known filter in the robust statistics literature such as the median or trimmed mean to handle theoutliers.
The median filter mitigates the effect of outliers by ensuring that if more than half of thebatchwise estimates lie in an interval around the true value g , then their median also lies in the sameinterval.
Similarly, the trimmed mean prunes the outliers by removing the vectors with relativelylarge and small values (controlled by its parameter) and computes the estimate of g as the mean ofthe remaining vectors.
Therefore, we obtain the gradient estimate ˜ g − h ( v ) as
[Equation]
where the function filter( · ) is either median or trimmed mean, and h ( v ) is given in (3).
Using thegradient estimate, the robust GD algorithm updates the k -th iterate v ( k ) as
[Equation] [Equation]
where γ ( k ) > 0 is the diminishing step size and P ( · ) projects the iterate into a bounded region D r as
[Equation]
The projection ensures that ℓ ( v ) is a strongly convex function of v , and the parameter r controls thestrong-convexity (Daskalakis et al., 2018)
(see Section 4 for more details).
The robust GD algorithmis summarized in Algorithm 2.
The role of r is further discussed in Section 4.
This completes thefirst step of our algorithm based on robust GD which is summarized in Algorithm 2.
Finally, using the estimates obtained via the robust GD algorithm, we estimate ˆ θ ij similar to (Wuet al., 2019) using (Williamson, Shmoys, 2011, Lemma 6.7).
Specifically, we have
[Equation]
where 1 ( · ) is the indicator function and ˆ b is the output of the robust GD algorithm.
The overalldistribution learning algorithm is given in Algorithm 1 where ˆ Σ denotes the estimate of W W T .
Input: Samples X = (cid:8) x ( n ) ∈ R d (cid:9) N n = 1 for i ∈
[ d ] do
X + i ← { x i : x ∈ X and x i > 0 }
Compute ˆ v using Algorithm 2 with input as X + i ˆ Σ i,i ← 1 / ˆ v ˆ b
i ← max { 0 , ˆ v 2 / ˆ v 1 } 6 for i < j ∈ [ d ] do
Compute ˆ θ ij using (9) ˆ Σ i,j ← (cid:113) ˆ Σ i,i ˆ Σ j,j cos(ˆ θ ij ) ˆ Σ j,i ← ˆ Σ i,j
Output: ˆ Σ ∈ R d × d and ˆ b ∈ R d
Algorithm
Input: Positive samples X + ⊂ R +
Parameters: Iterations K , step size γ ( k ) , projection parameter r , batch size N B 1 B ← |X + | /N
B 2 Compute { ˜ g ( b ) }
Bb =1 from X + using (4) 3 ˜ g ← filter (cid:16) ˜ g (1) , ˜ g (2) , . . .
, ˜ g
( B ) (cid:17) 4 v (0) ←5 for k = 1 , 2 , . . .
, K do µ ← v 2 ( k − 1) v 1 ( k − 1) ; σ 2 ← 1 v 1 ( k − 1)
Compute h
( v ( k − 1)) using (3)
Update v ( k ) using (6)
Output: v ( K ) ∈ R
Error Bounds for Parameter Estimation
This section provides our main result which characterizes the sample complexity for robust estimationof neural network parameters.
The analysis of distributed learning in the presence of arbitrary outliersis studied in Chen et al.
; Yin et al.
They assume that the available samples are splitinto a fixed number of batches and a constant fraction (<1/2) of batches are outliers.
However, oursetting assumes a probabilistic model where each data sample can be an outlier with probability 1 − pand there is no deterministic upper bound on the number of outliers.
Consequently, each batch canhave a mixture of true samples and outliers, and it is critical to choose the right batch size N B (see
Proposition 2 and its discussion for details).
We rely on two propositions to arrive at the main results.
The propositions establish the propertiesof the objective function of the optimization problem in (1) and the error bounds for the robust GDalgorithm presented in Algorithm 2.
Proposition 1.
There exist positive constants L and η that depend only on the projection parameterr ≥ 1 of Algorithm 2 such that the objective function ℓ ( v ) in (1) is an η − strongly convex and
L − smooth function of v when v ∈ D r defined in (7) .
Here, L is an increasing function of r whereasη is a decreasing function of r .
Proposition 1 proves that the projection parameter r controls the strong-convexity and smoothnessparameters of the objective function.
If r takes a large value, it leads to a small strong-convexityparameter η and a large smoothness parameter L .
We use this property to interpret the role ofparameter r in the algorithm performance using the next result presenting the error bounds of therobust GD algorithm.
To this end, we make the following assumptions to derive the error bounds:
Assumption 2.
The projection parameter r ≥ 1 is such that the minimizer of (1) , v ∗ ∈
D r where
D r is defined in (7) .
Assumption 3.
The step size of the robust GD algorithm in Algorithm 2 is fixed across the iterations,i.e., γ ( k ) =
γ , for k = 1 , 2 , . .
, K .
A large value of r ensures that the first assumption is satisfied.
However, if any prior knowledgeabout the true parameters is known, the parameter r can accordingly take smaller values.
The secondassumption is a guideline on how to choose the step size of the robust GD algorithm for the analysis.
Under the above assumptions, the error bound for the robust GD algorithm is as follows.
Proposition 2.
Consider the robust GD algorithm in Algorithm 2 based on the median filter, whichsolves (1) with input as X + .
Let p + ∈
(1 / 2 , 1] be the probability that a given sample in X + followsthe true distribution N + ( v ∗ 2 / v ∗ 1 , 1 / v ∗ 1 ) .
Assume that there exist ϵ ∈ (0 , 1) , δ ∈ (0 , 1 / 2) , andζ ∈ (0 , 1 − 2 δ ) such that the batch size N B satisfies
[Equation]
Then, under Assumptions 1 to 3, the output v ( K ) of our algorithm satisfies ∥ v ( K ) − v ∗ ∥ ≤ ϵ, withprobability at least 1 − δ if |X + | =
Ω (cid:16) N B ζ 2 log 1 δ (cid:17) and K = Ω(log 1 ϵ ) .
Here, all the order constants,the step size γ = 1 /L in Assumption 3, and the linear convergence rate Lη + L < 1 depend only on theprojection parameter r .
Also, η, L > 0 are defined in Proposition 1.
The above result indicates the role and suitable choices of the parameters: K , γ ( k ) , r , and N B asdiscussed next.
The result states the number of iterations K scales logarithmically with the inverseof the error
ϵ .
Also, Assumption 3 shows that the result holds when the step size is the same acrossall the iterations.
Finally, r should be large enough to satisfy Assumption 2.
However, a large rleads to slower convergence because the rate of convergence is an increasing function of r .
Finally,the algorithm gives an upper and lower bound on the batch size N B .
We note that for GD, theestimation error depends on the error in the first term of the gradient in (5), which is estimated usingthe batchwise gradient estimate in (4).
The error in the batchwise gradient estimate is contributed bythe outliers and the finite sample error (the difference between the sample moments computed usinga finite number of samples from a distribution and the true moment of the distribution).
With largebatch size, the number of batches without any outliers is also small.
Since the outliers are drawn froman arbitrary distribution D out , even if a batch contains one outlier, the error in the batchwise gradientestimate can be large.
This observation explains the upper bound on the batch size which depends onp + .
It is important to note that when there are no outliers (i.e., p + = 1 ), there is no upper bound onthe batch size.
Similarly, if the batch size is small, the batchwise gradient computed using batcheswithout outliers incurs a large finite sample error.
This observation intuitively explains the lowerbound on the batch size which is independent of p + .
In short, the upper and lower bounds on N Bbalances the tradeoff between the error due to the outliers and the finite sample error.
Further, we notethat the upper and lower bounds can be simultaneously achieved by choosing ϵ to be large enough.
We also note the restriction on p + which is not surprising.
This is because the median-based methodswork only if the number of outliers are smaller than that of the uncorrupted data samples, whichnaturally restricts the probability of outliers.
We next present our main theorem that discusses the overall complexity of our algorithm.
Theorem 1.
Consider the learning algorithm in Algorithm 1 that uses the meadian-based robust GD.
Let p ∈
(2 / 3 , 1] be the probability that a given sample follows the true distribution D (cid:16) W W T , b (cid:17) .
Assume that there exist ϵ ∈ (0 , 1) , δ ∈ (0 , 1 / 2) , and ζ ∈ (0 , 1 − 2 δ ) such that the batch size N B of
Algorithm 2 satisfies
[Equation]
Then, under Assumptions 1 to 3, the outputs ˆ Σ and ˆ b of Algorithm 1 satisfy(cid:13)(cid:13)(cid:13) ˆ Σ − W W T (cid:13)(cid:13)(cid:13) ≤ [ ϵ + (1 − p )]
∥ W ∥ 2 and (cid:13)(cid:13)(cid:13) ˆ b − b (cid:13)(cid:13)(cid:13) ≤ ϵ ∥ W ∥ , (12) with probability at least 1 − δ if the number of samples N =
˜ O (cid:16) 1 p (cid:104) 1 p + 1 ζ 2 ϵ 2 (cid:105) log dδ (cid:17) .
The algorithm runs in time ˜ O (cid:16) d 2 p (cid:104) 1 p + 1 ζ 2 ϵ 2 (cid:105) log dδ (cid:17) and space ˜ O (cid:16) dp (cid:104) 1 p + 1 ζ 2 ϵ 2 (cid:105) log dδ + d 2 (cid:17) .
All the orderconstants and the step size γ in Assumption 3 depend only on the algorithm parameter r .
With no outliers ( p = 1 ), our result is comparable with the existing error bounds from (Wu et al.
,2019, Theorem 1).
Specifically, for SGD in (Wu et al., 2019) with no outliers, the number of samples
[Table]
N = ˜
O (cid:0) 1 ϵ 2 log
dδ (cid:1) is sufficient to achieve
[Equation]
with probability at least 1 − δ for any ϵ, δ ∈ (0 , 1) ,
In our case, when p = 1 , there is no upper boundon N B and we choose N B = N = Ω (cid:0) 1 ϵ 2 log 3 dδ (cid:1) to achieve (13) with probability 1 − δ .
Thus, thetime and space complexities of our algorithm are identical to those of SGD in (Wu et al., 2019).
Wenext bound the total variation distance between the estimated distribution and the true distributionunder the restriction that W is a full-rank square matrix.
Corollary 1.
Consider the learning algorithm in Algorithm 1 that uses the meadian-based robust
GD.
Suppose that W ∈ R
d
×
d is full-rank with d > 1 and let κ be the condition number of W W T .
Let p > 1 − 12 κd be the probability that a given sample follows the true distribution D (cid:16) W W T , b (cid:17) .
Assume that there exist ϵ ∈ ( κd (1 − p ) , 1 / 2] , δ ∈ (0 , 1 / 2) , and ζ ∈ (0 , 1 − 2 δ ) such that the batchsize N B of Algorithm 2 satisfies N B =
˜ O (cid:16)
[Equation]
2 log 1 δ (cid:17) and N B ≤ 1 log(2 /p
[Equation]
log 2(1 − δ
[Equation]
1+ ζ .
Then,under Assumptions 1 to 3, the outputs ˆ Σ and ˆ b of Algorithm 1 satisfy
[Equation]
with probability at least 1 − δ if the number of samples N = ˜ O (cid:16) 1 p (cid:104)
[Equation]
Here, TV( · ) denotes the total variation distance between the argument distributions.
All the orderconstants and the step size γ in Assumption 3 depend only on the algorithm parameter r .
We note that when d > 1 , we have 1 − 12 κd ≥ 1 / 4 and the bound on p in Theorem 1 is automaticallysatisfied.
The last result of this section gives a lower bound on sample complexity of the problem of learning
ReLU NN described in Section 2.
We restrict the analysis to a specific class of ReLU distributionswhere W is a scaled identity matrix.
Theorem 2.
Consider the ReLU parameter estimation problem with p as the probability that a givensample follows the true distribution.
Suppose that the true distribution belongs to C =
{D ( W , b ) :
W = σ I , b ∈ R d , b i > 0 ∀ i } , where σ = O (1) .
Then, any algorithm that learns C to satisfy 253 (cid:13) (cid:13)(cid:13)
ˆ b − b (cid:13) (cid:13)(cid:13) ≤ ϵ ∥ W ∥ with success probability at least 2/3 requires Ω (cid:16) 1 pϵ 2 (cid:17) samples.
Comparing the sample complexity achieved by our algorithm (Theorem 1) and the above lowerbound, we can see that the second term of our sample complexity matches the derived bound upto log factors.
However, there is a gap between the sample complexity of our algorithm and thelower bound due to the first term that varies as 1 /p 2 (ignoring the log factors).
This is an interestingdirection for future work to see if there are better bounds.
Simulation Results
In this section, we provide numerical results to verify the performance of our algorithm.
In oursimulation setup, the columns of W are chosen as the left singular vectors of random matricesfrom the standard Gaussian distribution.
For b , we use a random vector from the standard normaldistribution whose negative values are replaced with zeros.
The mixture of samples are generated suchthat a sample comes from D ( W , b ) with probability p and from D out with probability 1 − p .
The
[Figure] [Figure]
outlier distribution D out = N (5 , 1) and the algorithm hyper-parameters are r = 3 and γ ( k ) =
1 k .
We use the batch-splitting approach to compute the gradient, inducing randomization.
Also, fromour experiments, we observe that the errors flatten certain number of iterations (see Figures 3 and4 in the supplementary material) around 1 / 100 -th of the number of positive output samples whichis chosen as the number of GD and SGD iterations K .
We compute two error metrics from theestimated parameters and the ground truth,
∥ ˆ Σ − W W T ∥ F / ∥ W ∥ 2 F and ∥ ˆ b − b ∥ 2 / ∥ W ∥ F .
Further,we compare our algorithm with two other schemes: the oracle schemes (estimation using the truesamples only) and schemes without a filter.
Our results are shown in Figures 1 and 2, and Table 1,and the observations from them are as follows.
Effect of the filters: From Figure 2, the GD schemes perform better than the corresponding SGDschemes.
Also from Figures 1 and 2, we infer that using filters along with GD or SGD reducesthe effect of the outliers, and the curves are closer to the oracle schemes.
We also infer that themedian-based approach performs slightly better than the trimmed mean-based approach.
Dependence on the probability of a sample being uncorrupted p : From Figures 1a and 2a, theperformance of all the schemes except the oracle schemes improves with p because the fraction ofoutliers in the observed samples decreases with increasing p .
However, the schemes without filtersshow a considerable difference in performance as they are not able to handle the outliers effectively.
The performance of oracle schemes does not change with p as they assume the knowledge of truesamples.
Further, all the schemes converge to the corresponding oracle schemes when p = 1 .
Dependence on the number of samples N and dimension d : Figures 1b and 2b show that the estimationperformance of the oracle schemes and the schemes with the filter improves with the number ofsamples N .
However, the schemes without a filter do not always improve with N because the numberof outliers also increases with N , which are not handled by the algorithm.
In Figures 1c and 2c,we varied the dimensions up to 500 and observed that there is a slight increase in the errors as thedimension increases for our proposed schemes as well as oracle schemes.
We also observe that theerrors increase as d increases for GD without filter due to the presence of arbitrary outliers.
Comparison of runtimes: From Table 1, the SGD schemes run faster as SGD utilizes only onesample for the gradient, whereas GD utilizes all the samples.
The filter-based schemes have highercomputation times than the ones without filters, but the runtimes of the trimmed mean-based schemesare significantly higher than those of the median-based schemes.
Overall, the median-based GD algorithm is the most effective approach to estimating the NNparameters in the presence of the outliers and is faster than the trimmed mean-based GD algorithm.
Also, the median-based scheme is parameter-free and enjoys solid theoretical guarantees.
Conclusion
In this paper we proposed an algorithm for the estimation of the parameters of a single-layer ReLUneural network from the truncated Gaussian samples where each sample was assumed to be anarbitrary outlier with a fixed probability.
Our only assumption was that the bias vector was nonnegative.
We analyzed the sample and time complexities of the GD-based estimation algorithmcombined with median-based filters to handle the outliers.
The efficacy of our approach was alsodemonstrated using numerical experiments.
Removing the non-negativity assumption on the biasvector and extending our results to multi-layer neural networks are two promising directions forfuture work.
Checklist
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] see Section
(b) Did you describe the limitations of your work?
[Yes] see Sections 3 and 4(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results..
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] see Sections 2and
(b) Did you include complete proofs of all theoretical results?
[Yes] see Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] see READMEfile in supplemental material
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] see Section 5 and Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes] see Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] we referred to thecode by Wu et al.
(b) Did you mention the license of the assets?
[N/A]
[Table]
[Table]
[Figure]