F OURIER N ETWORKS FOR U NCERTAINTY ESTIMATES AND O UT - OF - DISTRIBUTION DETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
A simple method for obtaining uncertainty estimates for Neural Network classiﬁers (e.g. for out-of-distribution detection) is to use an ensemble of independently trained networks and average the softmax outputs.
While this method works, its results are still very far from human performance on standard data sets.
We investigate how this method works and observe three fundamental limitations: “Unreasonable” extrapolation, “unreasonable” agreement between the networks in an ensemble, and the ﬁltering out of features that distinguish the training distribution from some out–of–distribution inputs, but do not contribute to the classiﬁcation.
To mitigate these problems we suggest “large” initializations in the ﬁrst layers and changing the activation function to sin( x ) in the last hidden layer.
We show that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of- distribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).
I NTRODUCTION
When a neural network classiﬁes inputs, we often need to have some measure of the uncertainty involved in the prediction.
In particular, in safety critical applications we would like to know when the classiﬁer received an input that is different from the inputs it has been trained on - in such cases it could be dangerous to just use the “best guess”, instead we may want to choose some safe action or involve a human.
One solution for this “out-of-distribution detection” problem could be to add a label “unknown” and augment the training set with inputs that do not belong to any of the labels of the classiﬁer and which get the new label “unknown”.
However, the danger is that the classiﬁer learns the particular type of “known unknowns” it was trained on and does not generalize to the “unknown unknowns”.
To avoid this, we treat out-of-distribution detection as a “one class classiﬁcation” problem, i.e. we want to learn the input distribution without help of out of distribution inputs.
The simplest approach is to use the softmax output of a neural net as a conﬁdence measure.
It is well known that the softmax output tends to be “overconﬁdent”, so we cannot interpret it directly as a probability for the chosen class.
However, it still tends to be more conﬁdent for samples from the correct distribution than for outliers, so setting a threshold for the softmax output can to some extent distinguish between “in distribution” and “out-of-distribution” samples.
This is the “baseline method” for outlier detection suggested in (Hendrycks & Gimpel, 2017).
In (Lakshminarayanan et al., 2017) this baseline is improved by using an ensemble of classiﬁers (and adversarial training):
Often each random initialization of a neural network gives an overconﬁdent classiﬁer, but different classiﬁers disagree - averaging the softmax output over an ensemble of classiﬁers then gives an improved signal.
Other, more computationally demanding approaches are Bayesian Neural Networks ((Neal, 1996), (Barber & Bishop, 1998), (Blundell et al., 2015)), nearest neighbor methods (e.g. (Mandelbaum &
Weinshall, 2017), (Jiang et al., 2018), (Papernot & McDaniel, 2018), (Frosst et al., 2019)).
For some further (less closely) related works, see appendix A.
We investigate limitations of the “ensemble averaged softmax output” method, as this seems to hit the sweet spot of being easy and fast, but still giving good results.
To overcome the observed limitations and further improve this method, we propose two changes:
• Changing the activation function of the last hidden layer to the sin( x ) function.
(We will see in the next section how this can guard against overgeneralization.)
• Use larger than usual initialization, to increase the chances of obtaining more diverse networks for an ensemble.
In the next three sections we go through three observed problems with ensembles of ReLU networks and explain how these changes mitigate them, and in the following sections we evaluate the suggested changes on standard data sets (MNIST, CIFAR10, etc.).
We evaluate this uncertainty mainly with view to out-of-distribution detection: If we have meaningful uncertainties, the classiﬁer should in particular “know when it does not know the answer”.
This means it should display uncertainty for inputs that do not “belong to the same type of input” as the training examples.
By selecting different thresholds for the uncertainty, we can plot the ROC curve by recording for each threshold the fraction of samples below the threshold for “in distribution” ( x ) and for “out-of-distribution” ( y ) inputs.
In particular, ﬁgure 1 gives this evaluation for our method against the standard ensemble methods on two standard data sets.
P ROBLEM : “U NREASONABLE ” EXTRAPOLATION
We ﬁrst look at the simplest possible example of classiﬁcation: The training set consists only of one point x 0 ∈ R d which has label “1” (we assume there is another label, for which we do not have a sample).
Training standard ReLU networks with one hidden layer on just one point x 0 = 2 ∈ R we get something like ﬁgure 1: Gradient descent ﬁnds a simple functions describing the data, which will (with very high probability) be an increasing function of x .
So for larger values of x the network will become arbitrarily certain that the label of x should also be “1”.
(See appendix C for more details.)
It has been observed in (Hein et al., 2018) that this is true in general:
Given any ReLU network trained on some data for a classiﬁcation problem in R d , it will describe a piecewise linear function.
If we look at the values of this function along a ray to inﬁnity, this will be a linear function when we are far enough from the origin, and in general this means the network will become arbitrarily certain of its classiﬁcation for all points on the ray far enough from the origin.
This is clearly not desirable, we would rather get a low conﬁdence when we are far from the training data.
(Here “far from the training data” is a convenient aspect that can easily be checked mathematically.
Since the data are often bounded, it is actually more relevant in practice that this behavior starts as we leave the range of the training data.
As an illustration, see ﬁgures 3 and 4.)
Going back to our simplest possible special case, what should our conﬁdence be that other points belong to the same label?
Of course, there is no unique “correct” answer, but a reasonable model could be that our conﬁdence decays like a Gauss function around this point (cid:126)x 0 , i.e. is proportional to
[Equation]
for some σ that deﬁnes the scale on which our conﬁdence decays.
One way to force such a decay would be to use directly Gauss functions (equation 1) as activation functions, which leads to RBF networks which we could consider as a form of nearest neighbor methods.
While they recognize reliably points that are not close to one of the (cid:126)x 0 in (equation 1), they do not generalize as well as ReLU networks (see appendix G), so we will use a more indirect approach.
Using that the Fourier transform of such a Gauss function is again of this shape, we can write equation 1 also as an expectation:
[Equation]
where Σ is the diagonal covariance matrix σ − 2 · I .
When we approximate the expectation by a ﬁnite sum over N samples (cid:126)w i from N (0 , Σ) we get an expression of the form
[Equation]
where u i = 1 /N and the b i are chosen such that the cos( (cid:126)w i · (cid:126)x 0 + b i ) = 1 .
This function can be interpreted as the output of a neural net with one hidden layer of N neurons with activation function cos( x ) (we will later shift the phase b i and use sin( x ) ).
(The output is one dimensional since we only care about the logits for the one label that appears in our training set.)
This is the main idea in the proof of the following result (see appendix C for details and proof):
Proposition 1 : Assume the weight vectors (cid:126)w i are sampled from a normal distribution N (0 , Σ) and then ﬁxed, the numbers b i and u i are sampled independently from some distribution (not identically zero) with ﬁnite second moments (e.g. a normal distribution) and then trained on one data point (cid:126)xwith label 1, then Gradient Descent (for the usual cross-entropy loss) will make the network’s output converge to a function that is approximately proportional to e − (cid:126)x T Σ (cid:126)x/ 2 .
This approximation becomes exact when the number of neurons approaches inﬁnity, or when we take the expected value.
(cid:4)
We also have for general training sets (see appendix C for a proof):
Proposition 2 : Under the same idealized conditions (inﬁnite number of neurons, weights (cid:126)w i frozen), but arbitrary ﬁnite training set, and assuming the training converged, the function goes to 0 for | (cid:126)x | → ∞ .
(cid:4)
To get an exact formula, we froze the weights (cid:126)w i .
In practice we would train them as well, but at least experimentally we do not see a difference when the (cid:126)w i are large: They do not get changed by a large amount, and the result still looks the same.
An example is given in the two plots of the right side of ﬁgure 2.
When we train on a training set of a few points, we see experimentally that (under the same idealized conditions: Large number of hidden neurons, frozen (cid:126)w i ) we get an approximation to a weighted sum of Gaussian functions at these points (see appendix E).
So it may seem we constructed a complicated approximation to a nearest neighbor classiﬁer.
We now switch the activation function to sin( x ) , this makes no difference for the function if we compensate by shifting the b i by π/ 2 , but the region of “small initialization” | (cid:126)w i | , b i ≈ 0 is now similar to a linear network: The input to the hidden neurons i will be (cid:126)w i · (cid:126)x ≈ 0 , and sin( (cid:126)w i · (cid:126)x )
≈ (cid:126)w i · (cid:126)x .
As training progresses, | x | usually increases.
For ReLU activation, this increase is not bounded (if we are on the side (cid:126)w i · (cid:126)x > 0 ), but for sin the increase will usually stop at the ﬁrst maximum or minimum of sin , i.e. around ≈ ± π/ 2 (or also at a different minimum/maximum if we start from higher initialization, see appendix P for experimental examples).
This means that on the training set, features (i.e. the output of the sin( x ) neurons in the last hidden layer) get a large value by tuning the input to around x ≈ ± π/ 2 , which is difﬁcult to achieve “accidentally” for input that is not in the training set.
By contrast, for ReLU (or tanh) activation, the inputs only have to achieve any high value.
We will call networks in which the last hidden layer uses the sin( x ) activation function “Fourier networks”.
So the Fourier network is somewhere between a ReLU network (small initialization) and a nearest neighbor classiﬁer (large initialization).
In particular for intermediate initializations it generalizes better than a nearest neighbor classiﬁer, but still detects more outliers than a ReLU network, which leads to an out-of-distribution detection which is better than either of them, as we will see later (see in particular appendix G for a comparison to nearest neighbor classiﬁers).
P ROBLEM : “U NREASONABLE ” SIMILARITY OF NETWORKS IN THE ENSEMBLE
In section 2 we considered only one network, and saw that ReLU networks become arbitrarily certain far from the training set.
However, averaging the softmax output over an ensemble of networks could ﬁx that: Even if each network is very certain about its prediction, the ensemble could give low conﬁdence if the different networks disagree.
This indeed happens, but less often than one might expect.
To illustrate this, we generate the input in ﬁgure 3, which has 20 clusters of points with labels “red”, “green”, or “blue”.
Following images: Average output of an ensemble of 50 networks.
We take an ensemble of 50 networks that were independently trained on the same data set, and use the average “probabilities” for the 3 possible labels as RGB values for the corresponding pixels.
In the second image (ReLU networks) the ensemble is still overconﬁdent in most of the area, whereas on the right the ensemble of Fourier networks is only conﬁdent close to the input.
The reason for the overconﬁdence of the ReLU ensemble is that the individual networks tend to agree even far from the sample points.
On the other hand, the Fourier networks ﬁnd different sparse
Fourier interpolations to describe the input, so their output only agrees close to the inputs, see ﬁgure 4.
For another 2-dimensional example, see also appendix N.
As a higher dimensional illustration, we train 50 ReLU networks with standard initializations on
MNIST images with labels from 0 to 4.
Evaluated on MNIST images for labels 5 to 9, for a quarter of all images all 50 classiﬁers agree on the same label (which is of course wrong, since it is one of known labels 0 – 4).
If we use smaller initialization, all 50 classiﬁers agree even for the majority of inputs.
We can understand why this similarity of independently initialized networks is particularly prevalent for “small” initializations: We expect that Gradient Descent ﬁnds local optima “close to” the initial point.
If we start with high random initializations, we may have a better chance that they are far apart and converge to different solutions, but for small initialization the networks are likely to converge to the same closest solution.
This can be made precise for ReLU networks with one hidden layer and “inﬁnitesimal” initialization, see appendix H.
As a method between RBF networks and Fourier networks, we can also try using a Gauss function as an activation function.
While this does show some of the beneﬁts of Fourier networks, extrapolating with Gauss functions predictably reverts to zero, whereas the sum of periodic functions become essentially random far from the training distribution, resulting in far more diverse ensembles, see appendix F, ﬁgure 10.
P ROBLEM : C ONSTANT FUNCTIONS ON THE TRAINING MANIFOLD
The third problem we see may be a general problem for using networks trained as classiﬁers also for out–of–distribution detection: If we have a feature (function) that is constant on the training distribution, but takes on other values out–of–distribution, we can use it to detect out–of–distribution inputs.
However, when we train for distinguishing between different parts of the input distribution, such a feature is useless and may be dropped during training.
How do such features arise?
One simple possibility is that some part of the input is constant.
For example, for MNIST images, the corner pixels are always white.
But in notMNIST or fashionMNIST this is not always true (see e.g. the ”E” in ﬁgure 17), and thus this gives an easy way to detect images not in MNIST.
Another possibility is that there may be different ways to come to the conclusion that an input gets a certain label L .
For example, for MNIST we usually can already guess the input has label ”0” when we only see the top or the bottom half of the image.
Let us say features f
and g get valueif their clues to detect label L is present in the image, and are 0 otherwise.
Then f − g is alwayson the training set (as they are
if and only if the image is of label L ),
but it may take on different values out–of–distribution.
(For some preliminary experiments with forcing such different features into separate networks, see appendix O).
Why would such features be “dropped”?
If we start from small initialization, only weights that are useful for distinguishing between labels will be increased.
So e.g. if the corners are always white, there is no incentive to change the weight that connects a corner pixel to the next layers, and if we start from small initialization, these pixels will never contribute to the end result.
So an obvious mitigation is to use larger initialization: Although the weights will still not be changed, the pixel will likely have an inﬂuence on the end result, and in different networks of the ensemble this inﬂuence will be different.
So if the pixel is not white, this may lead to the different classiﬁers disagreeing about the label.
E XPERIMENTS : T WO HIDDEN LAYERS , FULLY CONNECTED
We follow the example given in (Lakshminarayanan et al., 2017) and look at fully connected networks with two hidden layers of 200 neurons, which we train on MNIST.
For the baseline network, we use the model presented in (Lakshminarayanan et al., 2017), namely ensembles of networks with
ReLU activations and the standard initialization of the weights ( σ = (cid:112) 2 /n with n = 784 for the ﬁrst layer, n = 200 for the second layer).
For the Fourier network we change the activation function of the deeper layer to sin and use σ 1 = 0 .
75 and σ 2 = 0 . 0002 .
We train the networks on the training set of MNIST (60000 images), and evaluate them both on the test set of MNIST and on other data sets, here we use notMNIST (see (Bulatov, 2011)) and fashionMNIST (see (Xiao et al., 2017)) as the “outlier” data sets.
For other data sets see appendix
K).
The ROC curves that are obtained this way are presented in ﬁgure 1.
The variation of these curves is captured by ﬁgure 15 in appendix I.
While using an ensemble of ReLU nets gives a very substantial improvement over using one ReLU net, the result is still not close to what humans would achieve: The classiﬁers used here have some 98% accuracy, if we allow not recognizing an image as MNIST digit in 2% of the cases, we might expect to recognize that the printed letters of the alphabet are no handwritten digits in most cases, but in the above graph 2% error rate for MNIST corresponds to “only” 80% success rate on notMNIST.
However, the ensembles of Fourier networks recognizes almost all of notMNIST images if we allow 2% error rate on MNIST.
In appendix M, ﬁgure 22 shows the corresponding results for training on the more challenging data set fashionMNIST, the results are qualitatively the same.
In the literature, the performance of such “out-of-distribution” detection is often measured by the area under the ROC curve, in Table 1 are numbers for MNIST vs. notMNIST.
[Table]
(Hendrycks & Gimpel, 2017)
[Table]
(Liang et al., 2018)
[Table]
(Malinin & Gales, 2018) [Table] Some of the approaches in Table 1 use outliers for training/tuning the model, so they are dependant on how similar the outliers used for training are to those used at test time. For more details about the baselines we refer the reader to Appendix R.
E XPERIMENTS : E FFECT OF INITIALIZATION VS . ACTIVATION
If we use only one hidden layer, we can only vary the σ for weights between the input and the hidden layer (we keep the initialization of the output layer at the standard σ = (cid:112) 2 /n ).
This makes it easy to plot how much of the effect is due to “high initialization” vs. the choice of activation function.
We plot the area under the ROC curve in dependence on initialization: On the x -axis we plot log( σ ) , on the y -axis the negative logarithm of the “error” := area above the ROC curve.
The error bars correspond to two standard deviations (we has used randomly sampled ensembles of 10 networks out of 50 pre-computed networks).
We see in the ﬁrst three images of ﬁgure 5 that notMNIST images are the easiest to detect, and ﬂipped digits from MNIST are the most difﬁcult, but in each case the Fourier networks clearly outperform the ReLU networks.
One reason why notMNIST is particularly easy probably is that its images often can be recognized by just looking at pixels close to the corners:
In MNIST these pixels are always white, but in notMNIST often not.
To eliminate this shortcut, we also consider “masked” versions of notMNIST and fashionMNIST in which these pixels are set to white (see appendix K for details).
The rightmost image in ﬁgure 5 shows the corresponding graphs for the
Even with shortcut”
The vertical line corresponds to σ = 0 . 2 , which seems to be close to optimal for Fourier networks for all test sets.
This corresponds to a standard deviation of around 1.4 for the input to the sin( x ) activation function (for our encoding of the MNIST pixels, which we scale into the interval [ − 1 , 1] ).
We also see that “larger than usual” initialization alone already helps for ReLU networks, but using the sin( x ) activation function gives a signiﬁcant additional beneﬁt.
(See also appendix N for more general initialization schemes.)
E XPERIMENTS : C ONVOLUTIONAL N ETWORKS ON SVHN VS CIFAR
The same phenomena described in the previous section for fully-connected networks apply for convolutional networks as well, thus allowing one to reap the beneﬁts of using Fourier networks for out-of-distribution detection on more complex image datasets.
To illustrate this, we train ensembles of classiﬁers on the Street View House Numbers dataset (SVHN)
(Netzer et al., 2011) and use CIFAR10 (Krizhevsky et al., 2009) as out-of-distribution samples.
Once again, we compare against a baseline of ReLU networks, ﬁrst described as performing well on this task in (Lakshminarayanan et al., 2017).
Three different convolutional architectures have been considered, with increasing complexity: i) a 2-layer convolutional block followed by a fully-connected layer with ReLU activations and one with Sine activations dubbed SimpleCNN , ii) a network containing the ﬁrst 4 layers from the classic
VGG16 architecture (Simonyan & Zisserman, 2015) followed by a sine fully-connected layer referred to as ShallowVGG , and iii) the full-ﬂedged VGG16 network.
Appendix J contains the exact hyperparameter values used for each network.
Table 2 presents the best results obtained using this setup for both the ReLU and Fourier networks.
Out-of-distribution samples are detected using two approaches, by selecting a threshold for either the maximum probability as obtained after aggregating the outputs of the softmax layers in an ensemble (presented in appendix Q) or for the entropy of the ensemble’s prediction (presented in Table 2).
For evaluation, we used the area under the ROC curve ( AUROC ↑ ) as well as the false positive rate at 80% true positive rate ( FPR80 ↓ ).
The arrows indicate how the metrics change as the model gets better (i.e. ↑ if a larger value is better and ↓ if a smaller value is better).
Additional results are presented in appendix Q.
One other way in which one can assess qualitatively the performance of an out-of-distribution detection approach is by inspecting the distribution of the entropy of the softmax outputs over in-distribution and out-of-distribution samples.
Ideally, the entropy for in-distribution samples stays close to 0 (meaning that the model is certain about its predictions) while the one for outof-distribution samples is pushed as far from 0 as possible (the maximum entropy that a 10-class classiﬁer can achieve is log 10 ≈ 2 . 3 ).
Figure 6 shows how Fourier networks compare to regular ReLU networks for the Shallow VGG architectures (with σ = 0 .
01 used as the standard deviation of the initialization distribution of the last layer).
The Fourier networks make lower certainty predictions about out-of-distribution samples.
[Table]
C ONCLUSION
We suggested using the sin( x ) activation function in the last layer and larger than usual weight initializations to mitigate three problems we saw in using ensembles of ReLU networks for out–of– distribution detection:
• “Unreasonable” extrapolation: While ReLU networks tend to get more conﬁdent away from the training set, Fourier networks get decreasing conﬁdence in the average.
• “Unreasonable” agreement between the networks in an ensemble: Larger initialization can make the networks more diverse, and the mixture of sin ( x ) functions with different frequencies makes the networks more random away from the training set.
• Filtering out of features that distinguish the training distribution from some out–of– distribution inputs, but do not contribute to the classiﬁcation: Larger initialization can preserve such features, they do not inﬂuence the in–distribution classiﬁcation, but create diversity in the ensemble when applied to out–of–distribution input.
We showed that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-distribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).
A A PPENDIX A: M ORE RELATED WORKS
Related to (Lakshminarayanan et al., 2017) is the suggestion of “MC dropout” (Gal & Ghahramani, 2016), here the ensemble of networks is replaced by one network, from which different predictions are produced by Dropout at prediction time.
This is faster in training and evaluation, but in general not as effective as training the networks independently.
Another method that is also suggested in (Lakshminarayanan et al., 2017) is adversarial training, but its effect seems to be small compared to the effect of using ensembles instead of single networks.
Using a RBF layer on top of a normal network was ﬁrst suggested in (Y. LeCun & Haffner, 1998), and is further investigated in (Zadeh & Hosseini, 2018).
While we are avoiding the use of “outliers” as training input, it can be helpful.
A recent work in this direction is (Hendrycks et al., 2019).
“Interesting outliers” can also be generated by a GAN, see (Lee et al., 2018) and (Kliger & Fleishman, 2018).
We focused on the use of uncertainties for out–of–distribution detection, and for this purpose it does not matter whether the “probabilities” are really correct (as long as they are higher for in–distribution than for out–of–distribution).
However, if one wants “correctly scaled” probabilities, one can try to learn the right scale from the training data, see (Pawlowski et al., 2018).
We used the classifying networks and their “probability” output also for one–class–classiﬁcation, as this is the computationally cheapest solution.
But another intuitive approach is to use a generative model to produce a probability that an input belongs to the training distribution (Bishop, 1994).
However, (Choi et al., 2018) and (Nalisnick et al., 2018) showed how this can fail and actually produce higher probabilities for out of distribution inputs.
An interesting method that ﬁxes this problem was recently described in (Ren et al., 2019).
For more works related to sin( x ) as activation function, see the next section.
B A CCURACY AND TRAINING WITH SIN AS ACTIVATION FUNCTION
We did not investigate in detail the general behavior of the sin( x ) –function as an activation function with respect to the usual accuracy and training behavior, but in our experiments, we did not see any signiﬁcant difference to ReLU ( x ) networks inside the training distribution.
For example, the table below shows the classiﬁcation accuracy of ensembles of 5 ReLU networks and Fourier networks on the SVHN test set.
For each case, the accuracy is reported for the ensemble with the best out-of-distribution detection performance (the same that are presented in Table 2).
[Table]
These observations are consistent with the remarks in chapter 6.3.3.
“Other Hidden Units” in (Goodfellow et al., 2016).
It is usually said that ReLU works much better than other activation functions for deep neural nets, but that is less relevant for us since we only use sin( x ) in the last layer.
Nevertheless, there may be special cases in which sin( x ) behaves better or worse than ReLU, as has been observed in e.g. (Sopena et al., 1999) and (Parascandolo et al., 2017).
Another way to formulate what a sin( x ) –layer does, is that it computes a sparse Fourier approximation.
Sparse Fourier approximation has been extensively studied (e.g. (Gilbert et al., 2002), (Candes et al., 2006)), but in a quite different context: Usually the aim is to fully reconstruct a function from a small amount of input / output data.
This is possible when the function is known to have a sparse
Fourier representation.
In our context it is unlikely that our functions have a sparse Fourier representation, and instead of trying to reconstruct the function the aim is rather to use different sparse
Fourier interpolations to show the uncertainty we have in areas where we do not have data.
C P ROOFS OF P ROPOSITION 1 AND
We study the Gradient Descent dynamic of a neural net with
• input (cid:126)x ∈ R d • one hidden layer with N neurons with activation function cos , connected to the input layer by ﬁxed weights (cid:126)w i ∈ R d , i = 1 , 2 , ..., N • one output f ( (cid:126)x ) ∈ R , connected to the hidden layer with weights u i ∈ R .
The output of this neural net at the point (cid:126)x is given by
[Equation]
(We are using only one output dimension since for Proposition 1 we will be training it only on one data point, so the outputs for other labels do not matter for now.)
We use the cross entropy loss, which is for a data set of one point (cid:126)x 0 with the correct label given by
[Equation]
The derivative of L for some parameter θ (i.e. one of the b i or u i ) is
[Equation]
With d i :=
(cid:126)w i · (cid:126)x 0 the last derivative is
[Equation]
We will assume “inﬁnitesimal” learning rate, compensated by “inﬁnitely many” steps, so we get a differential equation for the parameters, and a learning “velocity” α ( t ) .
Then we get as differential equation
[Equation]
We assume we can train as long as we want, and we are not interested in how fast this procedure is proceeding.
This means we are only interested in the direction of the vectors in (equation 2) and can as well look at the re-scaled vectors giving the differential equation
[Equation]
These equations are independent for each neuron i , we we can ﬁx a neuron i and drop the subscript.
Introducing the function
[Equation]
and using the dot notation for the time derivative we get the simple differential equation
[Equation]
(See ﬁgure 7 for a plot of the corresponding vector ﬁelds.)
Lemma 1: For (Lebesgue–)almost all ( c 0 , u 0 ) ∈ [ − 1 , 1] ×
R the solution curve of (equation 4) starting at t = 0 at ( c 0 , u 0 ) will go either to (1 , + ∞ ) or to ( − 1 , −∞ ) for t → ∞ .
Proof of Lemma 1:
Since we have a symmetry ( c, u ) (cid:55)→ ( − c, − u ) , it is enough to consider u 0 ≥ 0 .
For c 0 ∈ {± 1 } we obviously have c constant and u ( t ) =
u 0 + t · c 0 → ±∞ .
So we will assume | c | < 1 .
For 0 < c 0 < 1 , 0 < u 0 the c ( t ) is increasing, hence we must have u ( t ) ≥ u 0 + t · c 0 → + ∞ .
So we can assume also u t ≥ 1 .
Then c ( t ) must be above the solution to the differential equation
[Equation]
which has the solutions
[Equation]
so we also must have 1 ≥ c ( t ) ≥ f
( t ) → 1 , hence c ( t ) → 1 .
For c 0 = 0 , u 0 > 0 or u 0 = 0 , c 0 > 0 we move for small t > 0 into the domain c > 0 , u > 0 , so we know that for them as well we have c ( t ) → 1 , u ( t ) → + ∞ .
By symmetry, for u 0 = 0 , c 0 < 0 we will have c ( t ) → − 1 , u ( t ) → −∞ .
For c 0 = 0 , u 0 =
the functions c, u are constant 0, so this is a point we have to exclude.
So the remaining part is − 1 < c 0 < 0 , 0 < u 0 .
From (equation 4) we see that in that case ˙ c > 0 and ˙ u < 0 .
So a solution curve starting in the quadrant − 1 < c 0 < 0 , 0 < u 0 has to leave this quadrant by either
Going through a point c = 0 , u > 0 , (light blue in ﬁgure 7), or 2.
Going through a point c < 0 , u = 0 , (light red in ﬁgure 7), or 3.
approaching c = 0 , u = 0 (red line in ﬁgure 7).
We have already seen that the ﬁrst two possibilities lead to c ( t ) , u ( t ) → 1 , + ∞ and c ( t ) , u ( t ) → − 1 , −∞ respectively, so it remains to show that the points that will approach c = 0 , u = 0 are a null set.
In fact, we show that for each − 1 < c 0 < 0 there can be at most one u 0 with that property: From (equation 4) we see that the tangent to a solution curve through a point ( c 0 , u 0 ) has slope u · c/ (1 − c ) .
If we look at the vertical distance between two solution curves in our quadrant, this means that with increasing c their distance also increases.
Therefore, two curves with positive distance at c 0 cannot converge both to (0 , 0) , and for each given c 0 there can be at most one u 0 in our quadrant such that ( c 0 , u 0 ) belongs to the third set above.
(cid:4)
With Lemma 1 we can now prove:
Proposition 1 : Assume the weight vectors (cid:126)w i are sampled from a normal distribution N (0 , Σ) and then ﬁxed, the numbers b i and u i are sampled independently from some distribution (not identically zero) with ﬁnite second moments (e.g. a normal distribution) and then trained on one data point (cid:126)xwith label 1, then Gradient Descent (for the usual cross-entropy loss) will make the network’s output converge to a function that is approximately proportional to e − (cid:126)x T Σ (cid:126)x/ 2 .
This approximation becomes exact when the number of neurons approaches inﬁnity, or when we take the expected value.
Proof:
We assume that we have trained equation 2 for some time T long enough such that we are in the situation of Lemma 1, by symmetry (changing u
i (cid:55)→ − u i , b i (cid:55)→ b i + π if necessary) we can assume that we have always
u i (cid:29) 0 and cos( d i + b i ) ≈ 1 .
Setting Z := (cid:80) i u i , we have
[Equation]
(For the second approximation we use that we have sampled the (cid:126)w i independent from the u i .)
This is then the Fourier transform of the Gauss function and we get
[Equation]
The approximation (equation 5) becomes exact when we train inﬁnitely long, and (equation 6) when we have inﬁnitely many neurons (or take the expected value).
[Equation]
Proposition 2 : Under the same idealized conditions (inﬁnite number of neurons, weights (cid:126)w i frozen), but arbitrary ﬁnite training set, and assuming the training converged, the function goes to 0 for | (cid:126)x | → ∞ .
Proof: We assume we trained on M points x 1 , ..., x M (with the correct label) and use a shallow
Fourier net with N ≈ ∞ neurons.
We also assume that the training “converged” in the sense that the b j converged (the u j keep growing).
Analog to the above case of one training point, we have
[Equation]
With
[Equation]
we can write this using the imaginary part (cid:61) :
[Equation]
If E ( (cid:126)w j ) = 0 , the value of L does not depend on b j ; we ﬁrst show that this only happens for a (Lebesque–)Null set of (cid:126)w .
The real part (cid:60) ( E ) is a real analytic function, and its value at 0 does not vanish since
[Equation]
Therefore the zero set of (cid:60) ( E ) has Lebesgue measure zero (an exercise in Calculus II, according to, and solved in (Mityagin, 2015)).
This implies that the even smaller zero set Z :=
{ (cid:126)w ∈ R d | E ( (cid:126)w ) = 0 } also has Lebesgue measure zero.
Using the arg function
[Equation]
we can deﬁne the continuous function
[Equation]
which makes
[Equation]
a well deﬁned continuous function.
Now for (cid:126)w ∈ R d \ Z and u j (cid:54) =
[Equation]
depends on b j ∈ R / 2 π , and L is minimal if this derivative crosses from negative to positive values around b j , which happens exactly when
[Equation]
So for almost all (cid:126)w j the weight (cid:126)w j determines the phase b j in a way that depends continuously on (cid:126)w j .
With this our function becomes
[Equation]
and after rescaling by dividing by Z := (cid:80) i | u i | this converges in the limit of inﬁnitely many neurons to
[Equation]
With the Gauss function
[Equation]
giving the probability density function for N ( [Equation] Σ) this is
[Equation]
which is up to a multiplicative constant the real part of the Fourier transform of the function
[Equation]
Since | e − iB ( (cid:126)w ) | = 1 and φ Σ ∈ L 1 ( R d ) , this function is also in L 1 ( R d ) , so the Riemann–Lebesgue
Lemma gives that f ( (cid:126)x ) → 0 for | (cid:126)x | → ∞ as claimed.
(cid:4)
D R E LU NETWORKS IN THE SIMPLEST 1-D PROBLEM
After randomly initializing the weight w i ∼ N (0 , σ ) and bias b i ∼ N (0 , σ ) for neuron i , the location of the resulting kink of ReLU ( w i · x + b i ) is at − b
i /w i , so it is Cauchy-distributed with center 0 and part of them will fall on the right hand side of x 0 , and another part on the left hand side.
Training the parameters tries to increase the value at x 0 , but it will not move the kink across x 0 .
So we will with high probability (probability 1 for inﬁnitely wide networks) see the sum of functions of which some increase for large positive x , others increase for very negative X , so in general the expected picture is indeed curving upward on both sides, like the experimental result in the ﬁrst image of ﬁgure 2.
E F EW POINTS
We construct as small set of points around -5, -2, 3, 9:
[Equation]
The Fourier approximation gives ﬁgure 8.
[Figure]
We can deﬁne “Gauss networks” for which we take as activation functions Gauss functions
[Equation]
Using this function instead of ReLU ( x ) or sin( x ) gives in the 2-dim toy example the ﬁgure(compare with ﬁgure 4).
G C OMPARISON TO N EAREST N EIGHBOR METHODS
The “nearest neighbor” classiﬁer assigns a label L to a test input if the closest training input had label L .
We can also use the “nearest neighbor” method for out of distribution detection:
For a threshold r we call an input “in distribution” if and only if its nearest neighbor in the training set has distance ≤ r .
Varying the r then gives a ROC curve, analogous to varying the threshold θ for the softmax output of a network ensemble.
There are many variations of this simple method ( k –nearest neighbor, RBF networks, adjusting the metric to the data), but for the general principles we want to discuss here the differences do not matter and we will use the basic implementation.
In (Bishop, 1995), chapter 5.9, pp.
183–186 the author writes about the “major potential difﬁculties with radial basis function networks” stemming from the “localized nature” of the hidden units (in our implementation of the nearest neighbor method the hidden units would correspond to the training points).
The problem is that nearest neighbor methods “just memorize” training points, and only generalize to nearby points – this requires that we have a training point “near” any possible input point of our training distribution.
On the other hand, “normal” networks learn “rules”, which can generalize the “important parts” and ignore the “noise” that does not contribute to the classiﬁcation.
This can be a problem for nearest neighbor methods mainly in high dimensions, but as a ﬁrst illustration we reproduce the two–dimensional toy problem from the book: Assume we have three classes (red, green, blue) that can be distinguished by their x 1 –coordinate, but we also have a random x 2 –coordinate with high variance (see ﬁgure 11).
If the red / green / blue stripes have length δ along the x 1 –axis, a nearest neighbor classiﬁer or RBF network may pick circles of radius around δ/ 2 to characterize the points that belong to the same stripe.
However, for that we need to have enough points also in x 2 direction, if there is a gap of much more than δ , we will have points in a stripe that are not recognized as belonging to the training distribution (or to the corresponding label).
On the other hand, a ReLU network will learn some regions and (over)conﬁdently classify all points in these regions as red / green / blue.
We can demonstrate in this toy example how the Fourier network ”interpolates” between a “ReLU behavior” and a “Nearest Neighbor behavior” when we adjust the standard deviation used in the weights’ initialization of a Fourier network: For small initializations, we are close to the “generalizing” ReLU behavior, large initializations give something close to a “memorizing” Nearest Neighbor behavior.
We show in ﬁgure 12 how an ensemble of ReLU networks and ensembles of Fourier Networks generalize to the 2D plane.
The upper row gives the classiﬁcation around the input, the lower row zooms out, in particular showing that for Fourier networks the ensembles are not extending their conﬁdence region to inﬁnity (as the ReLU networks do).
To illustrate the resulting effect in higher dimensions, we add some “random background” to the
MNIST and fashionMNIST images (see ﬁgure 13).
The “random background” plays the role of the x 2 coordinate above: For a new input, the nearest neighbor classiﬁer now has to ﬁnd a training image that is close to the new one both in the digit and in the background.
This is of course much harder (even with 60000 training points), and correspondingly the classiﬁcation accuracy of the nearest neighbor classiﬁer drops from 96.9% to 76.2% (see table 4).
For ReLU (and Fourier) nets, we also see a drop, but only to 91.3%, the classiﬁer just learns to “ignore” the background.
(The classiﬁcation accuracies reported here are the accuracy of the averaged softmax output of the networks in an ensemble.)
For out of distribution detection, we see the same effect – for the nearest neighbors method the area under the ROC curve drops from 98.9% to 77%, but for ReLU nets only from 91.8% to 91.1%.
The
Fourier networks are supposed to combine the best of both approaches, and indeed they give better out–of–distribution detection than either method in both cases.
[Table]
As a second example
[Table]
we produce 60000 wider images by randomly selecting 4 MNIST digits and combining them to one new image.
As the label of these images we take the label of the ﬁrst digit.
For out–of–sample images we replace the ﬁrst digit by a fashionMNIST image
[Table]
see ﬁgure 14.
Again, this makes it much harder for nearest neighbor method — now it has to ﬁnd images in which all 4 digits are similar (which is still possible with 60000 training images
[Table]
but harder),
whereas the
ReLU classiﬁer can focus on the pixels in the ﬁrst quarter of the image and learn to ignore the rest, see table 5 for the results.
An interesting philosophical question arises when we produce images by exchanging the last digit by a fashionMNIST image:
Are the resulting images “outliers” or not?
In any case, the results in table 5 show that the ReLU classiﬁers now do not ﬂag many “outliers”, which is in agreement with our expectation that they focus on the left quarter of the image and mostly ignore the rest
[Table]
Neighbor [Table] H R E LU NETWORKS WITH INFINITESIMAL INITIALIZATION For given training data and inﬁnitesimal initialization, there are only ﬁnitely many possibilities to which networks can converge (independent of network size), see (Maennel et al., 2018).
The arguments given in that paper show also that with increasing number of weights we should with high probability end up at the same possibility.
I M ORE RESULTS ON MNIST I. V ARIATION OF ROC CURVES FOR MNIST VS NOT MNIST AND FASHION MNIST
The ROC curves obtained when considering MNIST as in-distribution and notMNIST samples as out-of-distribution and using two-layer classiﬁers for training were shown in ﬁgure 1.
To visualize the variation of these curves, we compute 50 networks, and apply this method 10 times on ensembles of 10 randomly selected networks, this gives the left hand graph in ﬁgure 15.
(The right hand graph in ﬁgure 15 shows the corresponding results for testing for “outliers” from fashionMNIST.
We also get similar ﬁgures when training on fashionMNIST, see appendix M.)
I. E NTROPY HISTOGRAMS
To compare our method also to the evaluation given in (Lakshminarayanan et al., 2017), we compute for each image the entropy of the softmax output and plot a histogram over the MNIST and the notMNIST data set for the ensemble sizes 1,5,10.
The ﬁrst three columns of ﬁgure 16 (copied from ﬁgure 3 in (Lakshminarayanan et al., 2017)) show the resulting entropy of the softmax output for
ReLU networks, ﬁrst using the ensemble as is, then with additional adversarial training, and in the third column the corresponding results for “dropout at test time”.
For input from the MNIST test set (ﬁrst row, blue curves) the entropy is almost always close to 0, i.e. the classiﬁer puts almost all probability on one label.
For input from the notMNIST data set the classiﬁer ensembles distribute the probability over several labels, resulting in much higher entropy.
Use of adversarial training (second column) increases the entropy even a bit further for the notMNIST examples.
The fourth column shows the results for the corresponding Fourier networks.
On the MNIST test set the behavior is basically the same as the usual network, but on notMNIST even one network often is unsure about the label, and an ensemble of 10 networks gives most of the time an entropy around 2 (the maximum would be log(10)
≈ 2 .
3 for the uniform distribution across all 10 labels).
I. C ONVOLUTIONAL NETWORKS ON MNIST
Since MNIST vs. notMNIST is too easy to proﬁt from better classiﬁers, we use a more difﬁcult problem: Train on classes 0-4, see whether we detect classes 5-9 as “out of distribution”.
For MNIST we use two convolutional layers with 7 × 7 windows and max pooling in 3 × 3 windows with stride 2, and add two fully connected layers with 200 / 100 neurons.
We also use data augmentation by randomly cropping to 24 × 24 pixels.
For the Fourier network we change the activation function of the last layer to sin( x ) .
First experiments seem to indicate that a good way to choose the initialization for the different layers of the Fourier network is to start with small initializations for the ﬁrst layers and the last layer, and then choose the second last initialization as large as possible (see appendix J.5 for details). With this setup we get for the area under the ROC curve:
[Table]
J D ETAILED SPECIFICATIONS OF THE N EURAL N ETWORKS AND THEIR PARAMETERS J. O NE DIMENSIONAL TOY EXAMPLE
• Input data: Only one point at x 0 = 2 with label ”1”.
• Network: One hidden layer with 500 neurons, weights initialized with σ = 0 .
≈ (cid:112) 2 / 500 .
• Learning: – ReLU: 500 steps with learning rate 0.
– sin: 5000 steps with learning rate 0.
J. T WO DIMENSIONAL TOY EXAMPLE
• Input data: 20 centers randomly chosen in − 10 ≤ x ≤ 10 , − 10 ≤ y ≤ 10 .
To each center one of the labels ”Red”, ”Green”, ”Blue” was randomly chosen.
For each center ( x, y ) there were 50000 points randomly drawn from a normal distribution N ( µ, Σ) , where Σ = R · D 2 for a randomly drawn rotation matrix R and D the diagonal matrix with entries σ 1 = 0 .
2 and σ 2 = 0 .
• Network: One hidden layer with 500 neurons, activation function sin or ReLU.
Weights (including bias terms) initialized with σ = 12 .
• Learning: 26 epochs, batch size 256, learning rate 0.02.
J. 784 DIM INPUT , ONE HIDDEN LAYER ( FULLY CONNECTED ), TRAINED ON MNIST
Input: 764 pixels, normalized to be in [ − 1 , 1] Hidden layers: One
hidden
layer of
100 neurons
Initialization: σ
∈ { 0 . 002 ,
[Table]
Input: 764 pixels, normalized to be in [ − 1 , 1]
Hidden layers: Two hidden layer of 200 neurons, ﬁrst layer: ReLU, second layer: ReLU / sin.
Initialization:
• For ReLU network: σ 1 = 0 .
07 , σ 2 = 0 .
• For Sine network: σ 1 = 0 .
75 , σ 2 = 0 .
Parameters searched: For ReLU this is the standard initialization σ = 2 / (cid:112) ( n ) .
For Sine networks, initial tests gave the result that for a given σ 2 the σ 1 should be as large as possible such that the training still converges.
Then I tried σ 2 ∈ { 0 . 0002 , 0 .
001 , 0 . 01 } and σ 1 ∈ { 0 . 25 , 0 . 5 , 0 .
Learning rate was 0.01.
Best results: ( σ 1 , σ 2 , area under ROC curve for MNIST vs. notMNIST): 0.5, 0.0002, 0.
J
[Table] 784 DIM INPUT , CONVOLUTIONAL NETWORKS , TRAINED ON MNIST
Input: 764 pixels, normalized to be in [ − 1 , 1] ,
Preprocessing: 4 times random cropping from 28 × 28 to 24 × 24 .
Hidden layers:
• Convolution, 7 × 7 windows, 1 → 16 channels (to size 24 × 24 × 16 )
• Max pooling in 3 × 3 windows with stride 2 (to size 12 × 12 × 16 ) • Convolution, 5 × 5 windows, 16 → 32 channels (to size 12 × 12 × 32 ) • Max pooling in 3 × 3 windows with stride 2 (to size 6 × 6 × 32 ) • Fully connected, 200 neurons, ReLU activation function • Fully connected, 100 neurons, ReLU or sin( x ) activation function
Batch size:
Learning rate: 25 epochs 0.002, 20 epochs 0.001, 20 epochs 0.
Initialization: Given by σ = ( σ 1 , σ 2 , σ 3 , σ 4 ) .
For Fourier networks:
After initial attempts systematic evaluation of area under ROC curve for MNIST 0-4 vs. MNIST 5-9 for
• σ = (0 .
001 , σ 3 , 0 . 0001) , best result: 99.1% for σ 3 =• σ = (0 . 002 , 0 .
001 , σ 3 , 0 . 5) , best result: 99.1% for σ 3 = 1 .•
σ
=
1 , σ 3 , 0 . 001) , best result: 98.8% for σ 3 = 1 . 75 .
For ReLU networks:
After initial attempts systematic evaluation of area under ROC curve for MNIST 0-4 vs. MNIST 5-9 for
• σ = (0 . 01 , 0 . 01 , σ 3 , 1 . 0) , best result: 97.8% for σ 3 = 0 .•
σ = (0 . 01 , 0 .
01 , σ 3 , 0 . 6) , best result: 97.4% for σ 3 = 0 .•
σ = ( σ 1 , σ 1 , 0 . 1 , 0 . 1) , best result: 97.7% for σ 1 = 0 .•
σ = ( σ 1 , σ 1 , σ 1 , σ 1 ) , best result: 98.1% for σ 1 = 0 .
J. S IMPLE CNN, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0 , 1] ,
layers
• Convolution, 7 × 7 windows, 1 → 16 channels (to size 32 × 32 × 16 )
• Max pooling in 3 × 3 windows with stride 2 (to size 16 × 16 × 16 ) • Convolution, 5 × 5 windows, 16 → 32 channels (to size 16 × 16 × 32 )
• Max pooling in 3 × 3 windows with stride 2 (to size 8 × 8 × 32 ) • Fully connected, 200 neurons, ReLU activation function • Fully connected, 200 or 100000 neurons, ReLU or sin( x ) activation function
Batch size:
Learning rate: 0.01 (for 200 neurons on the last layer) and 0.001 (for 100000 neurons).
Halved every 40 epochs.
Best σ for initializing the last layer: 0.75.
J. S HALLOW VGG, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0 , 1] , Hidden
layers:
• the ﬁrst 4 convolutional blocks of the VGG16 architecture (including the corresponding max pooling layers) (to size 8 ×
8 × 128 ) •
Fully connected, 200 or 100000 neurons, ReLU or sin( x ) activation function Batch size  Learning rate: 0.001.
Halved every 40 epochs.
Best
σ for initializing the last layer: 0.01.
J. VGG16, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0 , 1] ,
Hidden layers:
• all the convolutional blocks of the original VGG16 architecture (to size 1 × 1 × 512 ) • Fully connected, 200 neurons, ReLU activation function • Fully connected, 100 or 100000 neurons, ReLU or sin( x ) activation function Batch size:Learning rate: 0.001.
Halved every 40 epochs.
Best σ for initializing the last layer: 0.25 and 0.1 for sin ( x ) and ReLU respectively, for 100 neurons on the last layer and 0.002 for 100000 neurons.
K D EFINITION OF THE DATA SETS
As input distributions that are “clearly different” from MNIST we can use the fashionMNIST ((Xiao et al., 2017)) and notMnist ((Bulatov, 2011)) data sets, see ﬁgure 17.
We also use some constructed data sets:
• “Masked fashionMNIST”:
In MNIST some pixels are always or almost always white, see ﬁgure 19.
Out of fashionMNIST we derive this data set by setting all pixels to 0 (white) that have an average gray value of below 0.01.
• “Masked notMNIST”:
This is derived from notMNIST in the same way.
• Circles and lines: Two lines and two circles in random positions.
• Flipped MNIST:
Derived from MNIST by mirroring the digits 3,4,6,7,9 horizontally and the digits 4,6,7, vertically.
• IID noise: Uniform iid pixel values from [0.256].
• Independent pixels: Each pixel is randomly sampled from pixels at the same position at MNIST training images, independent of the other pixel values.
• PseudoMNIST: Here we try to mimic not only the value distribution for each pixel, but also local correlations.
We start with an image in which each pixel is independently sampled like in the previous distribution.
Then we perform 1000 (PseudoMNIST1) or 1500 (PseudoMNIST2) updates of the following form: We randomly select a 3 × 3 window, and ﬁnd the closest match in 1000 randomly sampled images from MNIST, where “closest” is with respect to the L2 distance of the 8 outer pixels of the 3 × 3 window.
Then we replace the middle pixel in our image with the value of the middle pixel in this “closest match” from MNIST.
We will also use a “masked version” with a mask from MNIST: see ﬁgure 19 for the origin of the mask, and ﬁgure 18 for examples.
L T RAINING ON MNIST, USING OTHER OUTLIER SETS
With these data sets we can again compute the ROC curves that show how well we can distinguish the training set from the outlier set by setting a threshold on the “conﬁdence”.
As conﬁdence we take again the averaged softmax output of the predicted label over an ensemble of 10 classiﬁers, in which the weights are randomly sampled from a normal distribution with mean 0 and standard
In some cases it is easy to distinguish MNIST samples from outliers by just looking at pixels close to the corners:
In MNIST these pixels are always white but in notMNIST often not.
To eliminate this shortcut we also consider “masked” versions of notMNIST and fashionMNIST in which these pixels are set to white (see appendix K for details).
In ﬁgure 21 we show the corresponding graphs for the masked versions and for images in which each pixel is independently sampled from MNIST images so each pixel has the same value distribution as in MNIST.
Even with the “looking at the corners shortcut” disabled, the Fourier networks still perform well.
M T RAINING ON FASHION MNIST
We use the same two hidden layer network architecture, initializations, and learning rates as in ﬁgure 1, but train on fashion MNIST, the result is in ﬁgure 22 below.
Since this is a more challenging data set, the same architecture gives not as good results as for MNIST (note the x -axis extends now to 0.2), but the difference between ReLU and sin( x ) networks is comparable, resulting in graphs (ﬁgure 22) that look very similar to ﬁgure 1.
We can also plot the area under the ROC curve for different initializations of a network with one hidden layer - this gives ﬁgure 23.
Again this is qualitatively the same result as in ﬁgure 5 or in ﬁgure20 above.
N I NFLUENCE OF THE DISTRIBUTION FROM WHICH WEIGHTS ARE SAMPLED
Our method for getting a variety of networks relies on different randomly sampled initializations giving different networks.
According to (Sirignano & Spiliopoulos, 2018) this actually no longer happens if the number of neurons is very large: Then the networks converge to an idealized network in which there are inﬁnitely many neurons that have their weights distributed according to the probability distribution from which we sample.
To illustrate this potential problem visually, we look at a very simple 2-dimensional toy example, in which we can again plot how the classiﬁers generalize from the training distribution to other points, like we did in section 3.
As input we use only a line, with labeled training samples given as blue dots in the middle and red dots at the end (ﬁgure 24).
We will use a ReLU network with one hidden layer of 100 neurons to classify all points in the surrounding area of the plane.
Colors encode the “probabilities” that the softmax computes from the output layer, with blue / red signifying “probability 0/1 of being blue”.
The result is in ﬁgure 25.
As in section 3, we see that most points belong with high conﬁdence to on of the labels (here they are either blue or red), and there is only limited variation for classiﬁers initialized from different
If we change the activation function of the hidden layer from to ReLU to sin( x ) , we obtain different results outside of the training distribution (ﬁgure 26).
While individual points are still mostly red or blue, i.e. the individual classiﬁers are still “overconﬁdent”, the different classiﬁers now tend to disagree and averaging the softmax outputs over several of them we get much more realistic conﬁdences for the Fourier networks than for the ReLU networks: Now the ensemble is only conﬁdent about the label for points in the vicinity of the training inputs.
To demonstrate the potential problem with very wide networks, we now use 20000 neurons (instead of the 100 we used in the above pictures).
Now the results of different initializations become more similar to each other, see ﬁgure 27.
So in this case, we can no longer rely on different sampling from the same distribution, but need to change the distribution itself.
For higher dimensional input, this effect may be more theoretical because the number of neurons required to get into that regime is no longer realistic.
However, even for a moderate number of neurons it may make sense to vary the distribution of initial weights from which we sample to obtain a greater diversity.
Here is an example in our case of 2-dimensional input, 100 neurons, ReLU activation:
The weights in the 2-dimensional case have 3 coordinates: Two for the direction and one for the bias.
We can now sample the “direction” weights according to a Normal Distribution with one standard deviation σ 1 , and the “bias” weight according to a Normal Distribution with a different standard deviation σ 2 .
Different pairs ( σ 1 , σ 2 ) describe different distributions from which we initialize the weights.
If we vary now our weight initialization by varying ( σ 1 , σ 2 ) , the resulting networks seem to converge to different classiﬁers, see ﬁgure 28.
In particular, the networks initialized by σ 1 , σ 2 = (5 , 5) could in theory contain the same initializations as the networks initialized by σ 1 , σ 2 = (0 . 01 , 1) , but the emphasis shifts to different networks:
The latter give blue cones extending downwards most of the time, whereas such a result is not among the 50 samples I obtained for σ 1 , σ 2 = (5 , 5) .
So even for not extremely large networks it would make sense to restrict the distribution of weights from which we sample in different ways to obtain a more diverse set of networks.
O S EPARATING DIFFERENT FEATURES FOR THE SAME LABEL
We try restricting the weights in a neural network classiﬁer to certain subspaces – not only during initialization, but throughout the training.
The hope would again be that classiﬁers with different restrictions develop different features which generalize in different ways to unseen examples, giving more variety for out-of-distribution samples.
In the MNIST case, we evaluate two strategies of restricting the weights of the ﬁrst layer:
• Only allow access to half of the pixels
If each classiﬁer sees only half of the image, they usually still can ﬁgure out which digit it is:
• Only allow access to “line detectors” with certain directions.
We implement this by convolution with a 7 × 7 –matrix which corresponds to a stripe of a certain direction.
We allow 4 (of 10) directions per location, and to reduce the dimension of the resulting weight space we subsample the locations to a grid, see ﬁgure 30.
This again gives a vector space of dimension which is roughly half of the original dimension.
This seems to give a signiﬁcant increase of the in particular for
“Outliers” sin sin with half input sin with stripes combined
MNIST 596.5% 97.3% 97.5% 97.6% fashionMNIST 99.3% 99.5% 99.6% 99.7% notMNIST 99.9% 99.9% 99.9% 99.9%
Same test with restriction to half images on fashionMNIST:
The classiﬁer is trained on fashionMNIST 0-4.
[Table]
We have also applied the same approach when training convolutional networks on SVHN.
The table below shows the area under ROC curve obtained with a Simple CNN and a Shallow VGG.
By applying a square mask on one of the corners of the image (top left, top right, bottom left, bottom right)
we obtain an ensemble of constrained classiﬁers that is slightly more robust to out-ofdistribution samples.
[Table]
P I NPUT TO ACTIVATION FUNCTION
In this network the ﬁrst hidden layer has ReLU neurons, and the second hidden layer has sinNeurons. We plot the activation observed as input to the second layer, for the 5 neurons which have the largest weight going to the output of “0” (left), and the for the 5 neurons with the smallest weight going to the output of “0” (right) for all images that are indeed a “0”. We observe that the peaks are separated by 2 π , the left at the maxima of sin, the right at the minima of sin. A similar pattern arises for other images from the training set.
However, when we plot the input to the sin activation for images from notMnist, we no longer see the peaks at maxima / minima of sin, suggesting that the output for these images will be “random”:
Q M ORE EXPERIMENTAL RESULTS FOR SVHN VS CIFAR
Figure 33 shows which are the regions of optimal initialization for the ﬁnal fully-connected layer for both ReLU and Fourier networks.
Note that if the initialization is too large, gradient descent on the Simple VGG Fourier network will sometimes fail to converge.
The distribution of the entropy of the softmax outputs over in-distribution and out-of-distribution samples for a Simple CNN network is shown in Figure 34.
Results are presented for both regular
ReLU and sin ( x ) activations for the Simple CNN architectures (with σ = 0 .
75 used as the standard deviation of the initialization distribution of the last layer).
The curves correspond to ensembles of 1, 5, and 10 models.
Table 7 presents the best results obtained using convolutional networks trained on SVHN for both the ReLU and Fourier setup.
Out-of-distribution samples are detected using two approaches, by selecting a threshold for either the maximum probability as obtained after aggregating the outputs of the softmax layers in an ensemble or for the entropy of the ensemble’s prediction.
For evaluation, we used the following metrics:
• the area under the ROC curve ( AUROC ↑ ) • the false positive rate at 80% true positive rate ( FPR80 ↓ ) • the negative log-likelihood score ( NLL ↓ ) • the Brier score ( Brier ↓ )
In order to be closer to the ideal case, which is the premise for theoretical results proved above, we also report numbers for the same network architectures with signiﬁcantly more neurons on the ﬁnal fully-connected layer (i.e. 100,000).
For each conﬁguration, the table contains the numbers for the model with the best value for the initialization σ .
The dependency between the area under the ROC curve and the initialization strength is presented in ﬁgure [Table]
R C OMPARISON WITH OTHER OOD DETECTION METHODS
[Table]
(Hendrycks & Gimpel, 2017)
[Table]
(Liang et al., 2018)
[Table]
ours
[Table]
(Malinin & Gales, 2018)
[Table]
ours
[Table]
(Hendrycks & Gimpel, 2017) use a slightly larger network and GELU activation functions, (Liang et al., 2018) train also on images that do not belong to MNIST, but also not on the “notMNIST” images that are used for evaluation.
Similarly, (Malinin & Gales, 2018) use samples from a set of outliers to tune the model; the more similar these are to the outliers used for testing, the better the performance of the model is.
Table 9 shows the results obtained using the Dirichlet Prior Network model (DPN) for various settings.
In all instances we used the default hyperparameter values (suggested either in the paper or in the code released by the authors 1 ).
It is important to note that for the setting on which the Prior Network model performs best half of the out-of-distribution test samples are drawn from the outlier distribution that was used for ﬁne-tuning the model (namely the distribution of FashionMNIST samples from classes 0-4, since we use the whole test set of FashionMNIST as OOD samples at test time).
Moreover, for the best performing DPN results we tweaked the original setup and allowed the model to use the FashionMNIST training samples belonging to classes 0-4, instead of the (fewer) test samples, as prescribed by the authors of the paper.
However, even on this signiﬁcantly simpliﬁed setup, the DPN baseline falls behind the Fourier network ensemble even though the latter never sees any outliers during training.
[Table]