Learning Distributions Generated by Single-Layer ReLU Networks in the Presence of Arbitrary Outliers
Abstract
We consider a set of data samples such that a fraction of the samples are arbitrary outliers, and the rest are the output samples of a single-layer neural network with rectified linear unit (ReLU) activation.
Our goal is to estimate the parameters (weight matrix and bias vector) of the neural network, assuming the bias vector to be non-negative.
We estimate the network parameters using the gradient descent algorithm combined with either the median- or trimmed mean-based filters to mitigate the effect of the arbitrary outliers.
We then prove that ˜ O (cid:16) 1 p 2 + 1 ϵ 2 p (cid:17) samples and ˜ O (cid:16) d 2 p 2 + d 2 ϵ 2 p (cid:17) time are sufficient for our algorithm to estimate the neural network parameters within an error of ϵ when the outlier probability is 1 − p , where 2 / 3 < p ≤ 1 and the problem dimension is d (with log factors being ignored here).
Our theoretical and simulation results provide insights into the training complexity of ReLU neural networks in terms of the probability of outliers and problem dimension.
Introduction
Probability densities based on generative models are used in various applications such as image inpainting, deblurring, and generating new images.
In particular, generative models representing the distribution over natural images are used for image inpainting and to generate new images Van Oord et al.
A critical obstacle in generative modeling is building complex and expressive models that are tractable and scalable.
Hence, a fundamental challenge in machine learning, as well as in statistics, is to estimate a high dimensional distribution using a set of observed data samples generated by the distribution.
One solution technique is the deep generative method that models the unknown distribution as the output distribution of a neural network when the input of the neural network is drawn from a known distribution (for example, standard Gaussian in image generation by generative adversarial networks (GANs))
(Goodfellow et al., 2014; Arjovsky et al., 2017; Radford et al., 2015; Kingma, Welling, 2013; Van Oord et al., 2016).
Here, the unknown distribution is estimated by learning the neural network parameters from the data samples.
Several deep generative models like GANs (Goodfellow et al., 2014; Arjovsky et al., 2017; Radford et al., 2015), variational autoencoders (Kingma, Welling, 2013), and autoregressive models (Van Oord et al., 2016) have been proposed.
However, the guarantees associated with such models are generally unknown.
So, we address the problem of computing the sample complexity for the parameter estimation of a single-layer neural network with rectified linear unit (ReLU) activation using a corrupted dataset.
Here, the corrupted dataset refers to the model wherein the output samples consists of a fraction of arbitrary outliers introduced by an adversary (Byzantine).
Motivated by the deep generative models, we consider the unsupervised learning framework, i.e., we assume the knowledge of the output samples, but the corresponding inputs are unknown and drawn from the standard Gaussian distribution.
We start with a brief review of the related literature.
The algorithms for neural network parameter estimation have been widely studied under both supervised and unsupervised learning frameworks.
The estimation algorithms generally rely on the stochastic gradient descent (SGD)-based algorithm
(Goel et al., 2018; Allen-Zhu et al., 2019;
Chen et al., 2022; Oymak, 2019; Lei et al., 2020); or the gradient descent (GD)-based approach (Cao, Gu, 2019; Du et al., 2019).
Further, some works have considered the sample complexity of a single-layer NN with ReLU activation in the unsupervised learning framework (Mazumdar, Rawat, 2018; Wu et al., 2019).
It is known that when the bias vector is assumed to be a random vector, the column space of the weight matrix can be recovered within an error of O ( d ) when the output dimension is d (Mazumdar, Rawat, 2018).
Further, when the bias vector is assumed to be nonnegative, ˜ O
(1 /ϵ 2 ) samples and ˜ O
( d 2 /ϵ 2 ) iterations are sufficient to estimate the parameters of the ReLU network within an error of ϵ .
However, none of the above works have considered the problem of corrupted samples.
To the best of our knowledge, the estimation of a neural network in the presence of noise or outliers has been addressed only in the context of supervised learning (Bakshi et al., 2019;
Goel et al., 2019; Mukherjee, Muthukumar, 2020; Zhang et al., 2019; Frei et al., 2020; Vempala, Wilmes, 2019).
Our paper focuses on a special case of unsupervised learning with corrupted samples wherein we consider a single layer ReLU network.
Our neural network parameter estimation is also related to the area of robust statistics.
Mathematically, the problem of learning the distribution using a neural network is equivalent to estimating the parameters of a truncated Gaussian distribution (see Section 2 for details).
Robust statistics also deals with the estimation of high dimensional distributions like Gaussian, Gaussian product, and
Gaussian mixture distributions where a fraction of the samples are arbitrary outliers (Huber, 2004;
Hampel et al., 2011; Lai et al., 2016; Diakonikolas et al., 2019; Diakonikolas, Kane, 2019; Kane, 2021).
However, parameter estimation from truncated Gaussian samples in the noisy setting has not been studied in the robust statistics literature, and we address this literature gap.
We present our major contributions as the following.
• Learning Algorithm: We devise a learning algorithm (Algorithm 1) to estimate the parameters (weight matrix and bias vector) of a single-layer ReLU neural network in the unsupervised learning framework where each output sample can potentially be an arbitrary outlier with a fixed probability.
Our estimation algorithm uses GD along with either the median- or trimmed mean-based filters to mitigate the effect of outliers.
• Sample Complexity: Our algorithm requires ˜ O (cid:16) 1 p (cid:104) 1 p + 1 ϵ 2 (cid:105) log dδ (cid:17) samples to estimate the network parameters within an error of ϵ with probability 1 − δ when the probability of a sample being uncorrupted is p ∈ (2 / 3 , 1] and the output dimension is d (Theorem 1).
We also characterize the total variation distance between the estimated and true distributions.
(Corollary 1). • Lower bound: We derive a lower bound on the sample complexity which says that at least Ω(1 /pϵ 2 ) samples are required to estimate the parameters up to an error of ϵ .
(Theorem 2). • Empirical validation (Section 5): We empirically evaluate the performance of our algorithm and show that it is robust to the arbitrary outliers.
Also, we see that the performance of our algorithm improves with the the probability of a sample being uncorrupted and the number of samples, and it increases slowly as the network output dimension grows.
These observations from the empirical results are consistent with our theoretical results.
Overall, the results obtained in this paper show that parameter estimation of a single layer ReLU neural network is possible using robust gradient descent even in the presence of outliers.
ReLU Neural Network Learning Problem
Let the weight matrix of the ReLU neural network be denoted by W ∈ R d × m and the bias vector by b ∈ R d .
The input to the neural network is denoted by the latent variable z ∈ R m .
We assume that the variable z is drawn from the standard Gaussian distribution.
Thus, the output of the network is the random vector x ∈ R d given by x = ReLU ( W z + b ) ∼ D ( W , b ) , where D ( W , b ) denotes the distribution of x .
Our goal is to estimate the unknown parameters W and b of the distribution
D ( W , b ) using the knowledge of a corrupted set of data samples X = (cid:8) x ( n ) ∈ R d (cid:9) N n =1 .
Here, we assume that a data sample in X follows the Huber’s p -contamination model (Huber, 1964), i.e., a sample is drawn from D ( W , b ) with probability p , and it is an arbitrary outlier drawn from an unknown distribution D out with probability 1 − p .
Hence, a given sample x ∈ X follows the distribution D p = p D
( W , b ) + (1 − p ) D out .
We make two observations about the learning problem.
Firstly, it is known that when all the samples are from the true distribution, exponentially large number of samples are required to estimate the bias b , if it can take any value from the set R d (Wu et al., 2019).
Naturally, the requirement on the number of samples would be worse in the presence of arbitrary outliers and so, we assume b to be non-negative.
Thus, we assume the following.
Assumption 1.
The entries of b ∈ R d are all nonnegative.
Secondly, the weight matrix W may not be identifiable from the distribution D ( W , b ) .
In particular, for any unitary matrix Q ∈ R m × m , we have D ( W , b ) =
D ( W Q , b ) .
Since our goal is to learn the distribution, learning either W or W Q is sufficient.
Thus, we focus on the learnability of the underlying distribution and not the learnability of the neural network parameters.
Specifically, our proposed algorithm estimates W W T ∈ R d × d and b ∈ R d .
Secondly,
We tackle these issues using a new formulation which we discuss next.
Robust Estimation Algorithm
Our algorithm is similar to that in (Wu et al., 2019) which considers the special case of p = 1 and uses SGD.
For the general case of p ≤ 1 , we combine the estimation framework with a robust filter to estimate the parameters.
To derive the robust estimation algorithm, we first consider a true sample x ∼ D ( W , b ) whose i -th element is nonzero.
We have x i = ReLU ( W T
i z
+ b i ) ∼ N + ( b i , ∥ W i ∥ 2 ) ,
where N + ( b i , ∥ W i ∥ 2 ) is the truncated normal distribution obtained by restricting the normal distribution
N ( b i , ∥ W i ∥ 2 ) to the set of positive real numbers.
Hence, estimation of b i and ∥ W i ∥ is equivalent to estimating the parameters of a one-dimensional truncated normal distribution N + ( b i , ∥ W i ∥ 2 ) using positive samples.
Therefore, we estimate the parameters in two steps.
In the first step, we estimate the i -th element b i of the bias vector and the norm ∥ W i ∥ of the i -th row of the weight matrix, for i ∈ [ d ] .
The second step is the estimation of the angle θ ij between the i -th row W i ∈
R m and the j -th row
W j ∈ R m of the matrix W , for i, j ∈
[ d ] .
Then, the ( i, j ) -th entry of the symmetric matrix W W T is given by ∥ W i ∥
∥ W j ∥ cos( θ ij ) .
The first step of the algorithm estimates the parameters of the univariate distribution given by
N + ( b i , ∥ W i ∥ 2 ) using the i -th element of the samples X
+ i
= { x i : x i > 0 , x ∈ X} via maximum likelihood estimation (Daskalakis et al., 2018).
Then, the parameter estimates are given by arg min µ,σ 2 ℓ ( µ, σ 2 ) where ℓ ( µ, σ 2 ) is the expected negative log likelihood that x i ∼ N + ( µ, σ 2 ) and the expectation is with respect to the true distribution x i ∼ N + ( b i , ∥ W i ∥ 2 ) .
Further, from (Daskalakis et al., 2018), ℓ ( µ, σ 2 ) is a convex function of v = (cid:2) 1 /σµ/σ 2 (cid:3) ∈ R 2 .
Thus, we solve the convex optimization problem of maximizing ℓ ( µ, σ 2 ) over v ,
[Equation]
The optimization problem in (1) can be solved using GD or SGD which is based on the gradient of ℓ ( v ) given by the relation (Daskalakis et al., 2018): ∇ ℓ ( v ) =
g − h ( v ) where
[Equation]
where µ = v 2 / v 1 , σ 2 = 1 / v 1 , and ϕ ( · ) and Φ( · ) denote the probability density function and the cumulative distribution function of the standard normal distribution, respectively.
The relation (3) follows directly from the closed form expressions of the first and second moments of the truncated
Gaussian distribution N + (cid:16) v 2 v 1 , 1 v 1 (cid:17) , which are functions of v only (Johnson et al., 1995).
We observe that g does not depend on v and only depends on the true distribution parameters ( b i , ∥ W i ∥ 2 ) , whereas h ( v ) does not depend on the true distribution.
Therefore, the estimation of g , which depends on the true distribution, can use the available data samples, and h ( v ) can be computed in closed form using the current iterate of v .
In other words, we compute the component of gradient g only once in the GD or SGD algorithm because it does not change across the iterations.
This observation motivates us to use the GD algorithm to estimate the parameters instead of SGD.
SGD introduces large variance due to the stochasticity and lower accuracy due to the outliers in every iteration whereas GD introduces a small error that does not depend on the algorithm iterate.
This key observation and use of (3) to compute the gradient is the main difference between our algorithm and the algorithm in (Wu et al., 2019), apart from the robust estimation aspect.
To estimate g , we partition X
+ i into batches of size N B and compute the batchwise estimate ˜ g ( b ) ,
[Equation]
where X + i,b ⊂ X + i is the b -th batch of samples.
We then combine the batchwise estimates using a well-known filter in the robust statistics literature such as the median or trimmed mean to handle the outliers.
The median filter mitigates the effect of outliers by ensuring that if more than half of the batchwise estimates lie in an interval around the true value g , then their median also lies in the same interval.
Similarly, the trimmed mean prunes the outliers by removing the vectors with relatively large and small values (controlled by its parameter) and computes the estimate of g as the mean of the remaining vectors.
Therefore, we obtain the gradient estimate ˜ g − h ( v ) as
[Equation]
where the function filter( · ) is either median or trimmed mean, and h ( v ) is given in (3).
Using the gradient estimate, the robust GD algorithm updates the k -th iterate v ( k ) as
[Equation]
where γ ( k ) > 0 is the diminishing step size and P ( · ) projects the iterate into a bounded region D r as
[Equation]
{ max [Equation] [Equation] The projection ensures that ℓ ( v ) is a strongly convex function of v , and the parameter r controls the strong-convexity (Daskalakis et al., 2018)
(see Section 4 for more details).
The robust GD algorithm is summarized in Algorithm 2.
The role of r is further discussed in Section 4.
This completes the first step of our algorithm based on robust GD which is summarized in Algorithm 2.
Finally, using the estimates obtained via the robust GD algorithm, we estimate ˆ θ ij similar to (Wu et al., 2019) using (Williamson, Shmoys, 2011, Lemma 6.7).
Specifically, we have
[Equation]
where 1 ( · ) is the indicator function and ˆ b is the output of the robust GD algorithm.
The overall distribution learning algorithm is given in Algorithm 1 where ˆ Σ denotes the estimate of W W T .
Input: Samples X = (cid:8) x ( n ) ∈ R d (cid:9) N n = 1 for i ∈
[ d ] do
X + i ← { x i : x ∈ X and x i > 0 }
Compute ˆ v using Algorithm 2 with input as X + i ˆ Σ i,i ← 1 / ˆ v ˆ b
i ← max { 0 , ˆ v 2 / ˆ v 1 } 6 for i < j ∈ [ d ] do
Compute ˆ θ ij using (9) ˆ Σ i,j ← (cid:113) ˆ Σ i,i ˆ Σ j,j cos(ˆ θ ij ) ˆ Σ j,i ← ˆ Σ i,j
Output: ˆ Σ ∈ R d × d and ˆ b ∈ R d
Algorithm 2: Robust GD
Input: Positive samples X + ⊂ R +
Parameters: Iterations K , step size γ ( k ) , projection parameter r , batch size N B 1 B ← |X + | /N
B 2 Compute { ˜ g ( b ) }
Bb =1 from X + using (4) 3 ˜ g ← filter (cid:16) ˜ g (1) , ˜ g (2) , . . .
, ˜ g
( B ) (cid:17) 4 v (0) ←5 for k = 1 , 2 , . . .
, K do µ ← v 2 ( k − 1) v 1 ( k − 1) ; σ 2 ← 1 v 1 ( k − 1)
Compute h
( v ( k − 1)) using (3)
Update v ( k ) using (6)
Output: v ( K ) ∈ R
Error Bounds for Parameter Estimation
This section provides our main result which characterizes the sample complexity for robust estimation of neural network parameters.
The analysis of distributed learning in the presence of arbitrary outliers is studied in Chen et al.
; Yin et al.
They assume that the available samples are split into a fixed number of batches and a constant fraction (<1/2) of batches are outliers.
However, our setting assumes a probabilistic model where each data sample can be an outlier with probability 1 − p and there is no deterministic upper bound on the number of outliers.
Consequently, each batch can have a mixture of true samples and outliers, and it is critical to choose the right batch size N B (see
Proposition 2 and its discussion for details).
We rely on two propositions to arrive at the main results.
The propositions establish the properties of the objective function of the optimization problem in (1) and the error bounds for the robust GD algorithm presented in Algorithm 2.
Proposition 1.
There exist positive constants L and η that depend only on the projection parameter r ≥ 1 of Algorithm 2 such that the objective function ℓ ( v ) in (1) is an η − strongly convex and
L − smooth function of v when v ∈ D r defined in (7) .
Here, L is an increasing function of r whereas η is a decreasing function of r .
Proposition 1 proves that the projection parameter r controls the strong-convexity and smoothness parameters of the objective function.
If r takes a large value, it leads to a small strong-convexity parameter η and a large smoothness parameter L .
We use this property to interpret the role of parameter r in the algorithm performance using the next result presenting the error bounds of the robust GD algorithm.
To this end, we make the following assumptions to derive the error bounds:
Assumption 2.
The projection parameter r ≥ 1 is such that the minimizer of (1) , v ∗ ∈
D r where D r is defined in (7) .
Assumption 3.
The step size of the robust GD algorithm in Algorithm 2 is fixed across the iterations, i.e., γ ( k ) =
γ , for k = 1 , 2 , . .
, K .
A large value of r ensures that the first assumption is satisfied.
However, if any prior knowledge about the true parameters is known, the parameter r can accordingly take smaller values.
The second assumption is a guideline on how to choose the step size of the robust GD algorithm for the analysis.
Under the above assumptions, the error bound for the robust GD algorithm is as follows.
Proposition 2.
Consider the robust GD algorithm in Algorithm 2 based on the median filter, which solves (1) with input as X + .
Let p + ∈
(1 / 2 , 1] be the probability that a given sample in X + follows the true distribution N + ( v ∗ 2 / v ∗ 1 , 1 / v ∗ 1 ) .
Assume that there exist ϵ ∈ (0 , 1) , δ ∈ (0 , 1 / 2) , and ζ ∈ (0 , 1 − 2 δ ) such that the batch size N B satisfies
[Equation]
Then, under Assumptions 1 to 3, the output v ( K ) of our algorithm satisfies ∥ v ( K ) − v ∗ ∥ ≤ ϵ, with probability at least 1 − δ if |X + | =
Ω (cid:16) N B ζ 2 log 1 δ (cid:17) and K = Ω(log 1 ϵ ) .
Here, all the order constants, the step size γ = 1 /L in Assumption 3, and the linear convergence rate Lη + L < 1 depend only on the projection parameter r .
Also, η, L > 0 are defined in Proposition 1.
The above result indicates the role and suitable choices of the parameters: K , γ ( k ) , r , and N B as discussed next.
The result states the number of iterations K scales logarithmically with the inverse of the error ϵ .
Also, Assumption 3 shows that the result holds when the step size is the same across all the iterations.
Finally, r should be large enough to satisfy Assumption 2.
However, a large r leads to slower convergence because the rate of convergence is an increasing function of r .
Finally, the algorithm gives an upper and lower bound on the batch size N B .
We note that for GD, the estimation error depends on the error in the first term of the gradient in (5), which is estimated using the batchwise gradient estimate in (4).
The error in the batchwise gradient estimate is contributed by the outliers and the finite sample error (the difference between the sample moments computed using a finite number of samples from a distribution and the true moment of the distribution).
With large batch size, the number of batches without any outliers is also small.
Since the outliers are drawn from an arbitrary distribution D out , even if a batch contains one outlier, the error in the batchwise gradient estimate can be large.
This observation explains the upper bound on the batch size which depends on p + .
It is important to note that when there are no outliers (i.e., p + = 1 ), there is no upper bound on the batch size.
Similarly, if the batch size is small, the batchwise gradient computed using batches without outliers incurs a large finite sample error.
This observation intuitively explains the lower bound on the batch size which is independent of p + .
In short, the upper and lower bounds on N B balances the tradeoff between the error due to the outliers and the finite sample error.
Further, we note that the upper and lower bounds can be simultaneously achieved by choosing ϵ to be large enough.
We also note the restriction on p + which is not surprising.
This is because the median-based methods work only if the number of outliers are smaller than that of the uncorrupted data samples, which naturally restricts the probability of outliers.
We next present our main theorem that discusses the overall complexity of our algorithm.
Theorem 1.
Consider the learning algorithm in Algorithm 1 that uses the meadian-based robust GD.
Let p ∈
(2 / 3 , 1] be the probability that a given sample follows the true distribution D (cid:16) W W T , b (cid:17) .
Assume that there exist ϵ ∈ (0 , 1) , δ ∈ (0 , 1 / 2) , and ζ ∈ (0 , 1 − 2 δ ) such that the batch size N B of
Algorithm 2 satisfies
[Equation]
Then, under Assumptions 1 to 3, the outputs ˆ Σ and ˆ b of Algorithm 1 satisfy [Equation] [ ϵ + (1 − p )] ∥ W ∥
[Equation]
(cid:13)(cid:13)(cid:13) ˆ [Equation] [Equation] 12) with probability at least 1 − δ if the number of samples N =
˜ O (cid:16) 1 p (cid:104) 1 p + 1 ζ 2 ϵ 2 (cid:105) log dδ (cid:17) .
The algorithm runs in time ˜ O (cid:16) d 2 p (cid:104) 1 p + 1 ζ 2 ϵ 2 (cid:105) log dδ (cid:17) and space ˜ O (cid:16) dp (cid:104) 1 p + 1 ζ 2 ϵ 2 (cid:105) log dδ + d 2 (cid:17) .
All the order constants and the step size γ in Assumption 3 depend only on the algorithm parameter r .
With no outliers ( p = 1 ), our result is comparable with the existing error bounds from (Wu et al., 2019, Theorem 1).
Specifically, for SGD in (Wu et al., 2019) with no outliers, the number of samples
N = ˜
O (cid:0) 1 ϵ 2 log
dδ (cid:1) is sufficient to achieve
[Equation]
and
[Equation]
(13) with probability at least 1 − δ for any ϵ, δ ∈ (0 , 1) , In our case, when p = 1 , there is no upper bound on N B and we choose N B = N = Ω (cid:0) 1 ϵ 2 log 3 dδ (cid:1) to achieve (13) with probability 1 − δ .
Thus, the time and space complexities of our algorithm are identical to those of SGD in (Wu et al., 2019).
We next bound the total variation distance between the estimated distribution and the true distribution under the restriction that W is a full-rank square matrix. [Figure]
Corollary 1.
Consider the learning algorithm in Algorithm 1 that uses the meadian-based robust
GD.
Suppose that W ∈ R
d
×
d is full-rank with d > 1 and let κ be the condition number of W W T .
Let p > 1 − 12 κd be the probability that a given sample follows the true distribution D (cid:16) W W T , b (cid:17) .
Assume that there exist ϵ ∈ ( κd (1 − p ) , 1 / 2] , δ ∈ (0 , 1 / 2) , and ζ ∈ (0 , 1 − 2 δ ) such that the batch size N B of Algorithm 2 satisfies N B =
˜ O (cid:16)
[Equation]
and N B ≤
[Equation]
Then, under Assumptions 1 to 3, the outputs ˆ Σ and ˆ b of Algorithm 1 satisfy
[Equation]
with probability at least 1 − δ if the number of samples N = ˜ O
[Equation]
Here, TV( · ) denotes the total variation distance between the argument distributions.
All the order constants and the step size γ in Assumption 3 depend only on the algorithm parameter r .
We note that when d > 1 , we have 1 − 12 κd ≥ 1 / 4 and the bound on p in Theorem 1 is automatically satisfied.
The last result of this section gives a lower bound on sample complexity of the problem of learning
ReLU NN described in Section 2.
We restrict the analysis to a specific class of ReLU distributions where W is a scaled identity matrix.
Theorem 2.
Consider the ReLU parameter estimation problem with p as the probability that a given sample follows the true distribution.
Suppose that the true distribution belongs to C =
{D ( W , b ) :
W = σ I , b ∈ R d , b i > 0 ∀ i } , where σ = O (1) .
Then, any algorithm that learns C to satisfy (cid:13)(cid:13)(cid:13)
[Table] ≤ [Table] ∥ W ∥ with success probability at least 2/3 requires Ω (cid:16) 1 pϵ 2 (cid:17) samples
Comparing the sample complexity achieved by our algorithm (Theorem 1) and the above lower bound, we can see that the second term of our sample complexity matches the derived bound up to log factors.
However, there is a gap between the sample complexity of our algorithm and the lower bound due to the first term that varies as 1 /p 2 (ignoring the log factors).
This is an interesting direction for future work to see if there are better bounds.
[Figure]
rr
[Figure]
r i n
[Figure]
t s [Figure] [Table]  Simulation Results
In this section, we provide numerical results to verify the performance of our algorithm.
In our simulation setup, the columns of W are chosen as the left singular vectors of random matrices from the standard Gaussian distribution.
For b , we use a random vector from the standard normal distribution whose negative values are replaced with zeros.
The mixture of samples are generated such that a sample comes from D ( W , b ) with probability p and from D out with probability 1 − p .
The outlier distribution D out = N (5 , 1) and the algorithm hyper-parameters are r = 3 and γ ( k ) =
1 k .
We use the batch-splitting approach to compute the gradient, inducing randomization.
Also, from our experiments, we observe that the errors flatten certain number of iterations (see Figures 3 and 4 in the supplementary material) around 1 / 100 -th of the number of positive output samples which is chosen as the number of GD and SGD iterations K .
We compute two error metrics from the estimated parameters and the ground truth,
∥ ˆ Σ − W W T ∥ F / ∥ W ∥ 2 F and ∥ ˆ b − b ∥ 2 / ∥ W ∥ F .
Further, we compare our algorithm with two other schemes: the oracle schemes (estimation using the true samples only) and schemes without a filter.
Our results are shown in Figures 1 and 2, and Table 1, and the observations from them are as follows.
Effect of the filters: From Figure 2, the GD schemes perform better than the corresponding SGD schemes.
Also from Figures 1 and 2, we infer that using filters along with GD or SGD reduces the effect of the outliers, and the curves are closer to the oracle schemes.
We also infer that the median-based approach performs slightly better than the trimmed mean-based approach.
Dependence on the probability of a sample being uncorrupted p : From Figures 1a and 2a, the performance of all the schemes except the oracle schemes improves with p because the fraction of outliers in the observed samples decreases with increasing p .
However, the schemes without filters show a considerable difference in performance as they are not able to handle the outliers effectively.
The performance of oracle schemes does not change with p as they assume the knowledge of true samples.
Further, all the schemes converge to the corresponding oracle schemes when p = 1 .
Dependence on the number of samples N and dimension d : Figures 1b and 2b show that the estimation performance of the oracle schemes and the schemes with the filter improves with the number of samples N .
However, the schemes without a filter do not always improve with N because the number of outliers also increases with N , which are not handled by the algorithm.
In Figures 1c and 2c, we varied the dimensions up to 500 and observed that there is a slight increase in the errors as the dimension increases for our proposed schemes as well as oracle schemes.
We also observe that the errors increase as d increases for GD without filter due to the presence of arbitrary outliers.
Comparison of runtimes: From Table 1, the SGD schemes run faster as SGD utilizes only one sample for the gradient, whereas GD utilizes all the samples.
The filter-based schemes have higher computation times than the ones without filters, but the runtimes of the trimmed mean-based schemes are significantly higher than those of the median-based schemes.
Overall, the median-based GD algorithm is the most effective approach to estimating the NN parameters in the presence of the outliers and is faster than the trimmed mean-based GD algorithm.
Also, the median-based scheme is parameter-free and enjoys solid theoretical guarantees.
Conclusion
In this paper, we proposed an algorithm for the estimation of the parameters of a single-layer ReLU neural network from the truncated Gaussian samples where each sample was assumed to be an arbitrary outlier with a fixed probability.
Our only assumption was that the bias vector was nonnegative.
We analyzed the sample and time complexities of the GD-based estimation algorithm combined with median-based filters to handle the outliers.
The efficacy of our approach was also demonstrated using numerical experiments.
Removing the non-negativity assumption on the bias vector and extending our results to multi-layer neural networks are two promising directions for future work.