S UPERSEDING M ODEL S CALING BY P ENALIZING D EAD U NITS AND P OINTS WITH S EPARATION C ONSTRAINTS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this article, we study a proposal that enables to train extremely thin (4 or 8 neurons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modiﬁcation such as Residual or Dense connections, data normalization or model scaling.
We accomplish that by alleviating two problems.
One of them are neurons whose output is zero for all the dataset, which renders them useless.
This problem is also known as dead neurons .
The other is a less studied problem, dead points .
Dead points refers to data points that are mapped to zero during the forward pass of the network.
As such, the gradient generated by those points is not propagated back past the layer where they die, thus having no effect in the training process.
In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both.
As an additional beneﬁt, the proposed method allows to initialize the network weights with constant values or even zero and still allowing the network to converge to reasonable results.
We show very promising results on a toy, MNIST, and CIFAR-10 datasets.
I NTRODUCTION
The success of Deep Neural Networks (DNN for short) is linked to its ability to learn abstract representations from input data in a hierarchical fashion (LeCun et al.
; Ramachandran et al.
However, the concepts of depth and width in networks are often used as instrumental elements to address different DNN pathologies during the learning process.
Examples of these are: vanishing gradient (Hochreiter (1991; 2001)), exploding gradient (Pascanu et al.
dead units (Maas (2013); Douglas & Yu (2018); Guerraoui et al. (2017)), or the degradation problem (He et al.
(2015b)).
In order to address the former issues, we ﬁnd many methods and techniques that we can roughly classify in two families: data manipulation and architectural modiﬁcations .
The most commonly used data manipulation technique is data normalization on the output of the layers, for example using batch normalization (Ioffe & Szegedy (2015)).
Examples of architectural modiﬁcations include the use of additional connections, as done in ResNets (He et al.
(2015b))
or DenseNets (Huang et al.
; unit augmentation, as in leaky - ReLU (Maas (2013)), PReLU (He et al.
(2015a)),
C-ReLU (Shang et al.
(2016)), or linked neurons (Riera & Pujol (2017));
or the increment of layer’s width with depth (Zagoruyko & Komodakis (2016); Szegedy et al.
This last widely used approach effectively increases the the size of the network requiring larger computational power.
This has led to several works (Hasanpour et al.
; Tan & Le (2019))
that address it by offering heuristics on how to scale the network.
Recent research (Liu et al.
(2018)) suggests that this inverted pyramidal architecture is not optimal.
Besides of the width scaling strategy, the concept of dead units 1 is of particular interest to this work.
A dead neuron is deﬁned as the neuron with a constant or zero output for all training data points.
This effectively renders this unit ineffective during the learning process.
In (Lu et al., 2019) it is shown that as the depth increases the probability of ﬁnding dead neurons also increases, to the point that the entire network can be dead even at initialization.
Additionally, as expected, as the width of a layer increases, the probability of having dead neurons decreases.
In this article we characterize another pathology, the dead point.
The dead point is a dual concept to the dead neuron.
A dead point corresponds to a data point that does not reach the output of the network.
This is, the activation is zero for all the units in a given layer.
As a result, this data point will have no inﬂuence in the training process.
As in the case of dead units, both condition are not recoverable by back-propagation.
This virtually reduces the size of the training set.
In order to solve the former issues, we propose and present a geometrical optimization constraint that is added to the loss function.
As a result of adding this constraint we ensure that all neurons are active and no points are dead.
The rationale behind the geometrical constraint is to control that all units/neurons and data points are alive by constraining pre-activation values in such a way that the hyperplanes associated to the non-linearity of each ReLU neuron separates, at least, one data point from the rest of the dataset.
This has the additional effect of enabling the learning process to propagate all information through all the network avoiding the instrumental need of using additional connections or inverted pyramidal architectures.
We test our proposal in a series of controlled experiments to showcase the effect of applying the proposed Separation Constraints .
We ﬁnd that we can arbitrarily increase the network depth using the same constant width when compared to standard feedforward network and Batch Normalization
Ioffe & Szegedy (2015) networks even when the width of the layer is extremely small ( 4 orneurons).
We additionally provide evidences that using this same approach we can initialize network parameters to zero and still achieve reasonable performance.
These promising results gives insight on learning dynamics and suggest potential lines for future checks and research.
The article is organized as follows: Section 2 introduce the characterization of dead neurons and dead points, Section 3 introduces the two geometrical constraints that ensure that neurons and data points are alive, Section 4 describes the experiments and result, and, ﬁnally, Section 5 concludes the paper and suggests future lines of research.
C HARACTERIZING DEAD NEURONS AND DEAD POINTS
A standard feed-forward ReLU DNN (LeCun et al., 2015) F can be formally written as a multivalued real function, F ( x ) , that is created by composing a collection of D vector layer functions (cid:96) :
R n k − 1 → R n k .
Layer k is deﬁned as the sum of a collection of scalar functions ( units ): (cid:96) k
[Equation]
j
[Equation] [Equation]
that afﬁnely depend on a weight vector w kj ∈ R n k and a bias parameter b kj ∈ R .
When using rectiﬁed linear units this value is truncated on negative values:
[Equation]
Considering the hyperplane deﬁned by w kj · x + b kj = 0 , each unit deﬁnes a partition of the space
R n k in two sets: the upper part of unit u kj and the lower part of u kj :
[Equation]
We deﬁne the afﬁne component of a layer function (cid:96) k as the intersection of the upper parts of its units, and its zero set as the intersection of the lower parts, as follows:
[Equation]
Remark 2.1 (Dead unit) .
In a given ReLU -DNN F : R n → R k , we say that the j -th unit of layer (cid:96) k , u kj is dead with respect to a data set X ⊂ R n k
if and only if
X ⊂ lower ( u k j ) .
[Equation]
Observe that if a unit is dead, the output of the unit will be zero for the entire dataset, rendering the unit useless.
Moreover, since the gradient is zero as well, it remains in this state for the rest of the training (See Lu et al.
; Shin & Karniadakis (2019)).
This effectively reduce the network learning capacity.
Remark 2.2 (Dead point with respect to a layer) .
Given ReLU -DNN F : R n → R k , we say that a point x ∈ X is dead with regards to layer (cid:96) k
if
[Equation]
Remark 2.3 (Dead point) .
In particular, if X ⊂ R n , we say that point x ∈ X is dead with respect to a network F with depth D if it gets mapped to the zero set of a layer in its transit through the network.
This is
[Equation]
Any dead point fulﬁlling Equation 7 will show zero gradient in the layers previous to (cid:96) k .
This hinders the learning process by effectively reducing the data set size.
Again, this condition is not reversible using standard back-propagation.
In a similar fashion to the case with dead units, the probability of ﬁnding a dead point increases with network’s depth and decreases with layer’s width.
I NTRODUCING SEPARABILITY CONSTRAINTS
In this section we introduce the desiderata for units and points to remain alive.
Then, we proceed to formulate the separability contraints that fulﬁll the desired conditions.
Let us introduce the concept of a separating unit with respect to an arbitrary set X .
Deﬁnition 3.1 (Separating Unit) .
Given an arbitrary set X ⊂ R n k , we say that the j -th unit on layer k , u kj , is able to separate through X if the following predicate is satisﬁed:
[Equation]
Thus, by construction, a separating unit can not be dead, and if R X ( u kj ) is valid, u kj can not degrade set X to zero.
In other words, this condition ensures that each unit always separates at least one data point.
In terms of points, we can deﬁne a separating point as follows:
Deﬁnition 3.2 (Separating point) .
Given an arbitrary set X ⊂ R n k , we say that point x ∈ X is separating a layer function (cid:96) k if there exist indices j, l ∈ { 1 , . . .
, m k } for which the following predicate is satisﬁed.
[Equation]
Again, by construction, a separating point ensures that each point in the data set has at least one unit in each layer with an activation different to zero and another with an activation equal to zero.
3. M ODELLING U NIT - BASED S EPARATION C ONSTRAINT ( S EP -U )
Unit based separation contraint ( Sep-U ) is designed to model predicate 8 with the goal of avoiding the presence of dead units.
Given a unit u kj in layer (cid:96) k : R n k
→ R n k +1 from a ReLU DNN F of depth D , predicate 8 can be simply modelled imposing the following constraints,
[Equation]
These strict inequalities can not be directly optimized.
It is easily to see that these can be rewritten as
[Equation]
The use of the former constraints makes most problems unfeasible.
Thus, in a similar fashion to soft-margin SVM (Cortes & Vapnik, 1995), we introduce a set of positive slack variables { ξ + jk , ξ − jk } that account for constraint violations as follows,
[Equation]
for k = 1 , . . .
, D and n k the number of units of layer k .
The intuition behind the introduction of these constraints is as follows: by minimizing ξ + at least one pre-activation value is forced to be greater (or equal) than 1 .
Simmetrically, minimizing ξ − promotes at least one pre-activation to be below − 1 .
This effectively fulﬁlls Predicate 8 and penalize the apparition of dead units.
At a global network scale, we can aggregate all the slack variable in a single optimization objective as follows,
[Equation]
3. M ODELLING P OINT B ASED S EPARATION C ONSTRAINT ( S EP -P )
The derivation of the Point Based Separation Constraints ( Sep-P ) follows a parallel process to
Sep-U .
In order to avoid the presence of dead points it sufﬁces to fulﬁll Predicate 9.
Similarly to the former derivation, we introduce a set of slack variables for each each point on the batch.
That is, given x i ∈ X , and u k 1 , . . .
, u kn unit functions in a layer (cid:96) k , we deﬁne slack variables ξ − ik , ξ + ik in the context of the following constraints,
[Equation]
Observe that the minimization of the slacks makes that for any data point at least one activation is above 1 and another is below -1.
We can summarize all the point-based slack variables in a single optimization objective as follows,
[Equation]
3. T RAINING WITH S EPARATING C ONSTRAINTS
The new optimization objectives can now be added to the original loss objective using a simple scalarization (Boyd & Vandenberghe (2004)) as follows,
[Equation]
where λ is a hyper-parameter that introduces a trade-off between the constraint fulﬁllment and the main loss function.
In terms of memory complexity , the former constraints introduce a very small memory overhead.
In particular, Sep-U places a pair of constraints on each of the units of the network, so the complexity with respect to Sep-U scales with the size of the network as 2 (cid:80)
Dk =1 n k .
Alternatively,
Sep-P places a pair of constraints in each of the points of the data set or selected subset X .
In practice, one can use X as the training batch.
Thus, the memory complexity scales with the size of the batch and the number of layers, i.e. 2 D | X | .
Furthermore, since the resulting gradient of both types of constraints depends only on the input of the layer that is already computed in the forward pass, we only add the cost of storing the slacks.
Therefore, the total cost in terms of number of constraint is the addition of the former terms,
i.e. 2 (cid:80) Dk =1 n k + 2 D | X | .
E XPERIMENTS AND R ESULTS
In this section we explore the application of the proposed constraints in different datasets.
For that task we train all methods with different choices of depth and width parameters.
The network architecture used is rectangular , i.e. networks with a ﬁxed layer width for all the layers.
Datasets: Due to the large amount of computational resources required for the depth and width grid training, we are forced to chose three controlled datasets in our experimentation: the MOONS dataset (sampling 100 points, 85 for training and 15 for validation), the MNIST dataset as described in
LeCun and Cortes.
LeCun & Cortes (2010) and the CIFAR-10 dataset described in Krizhevsky (2009).
Experimental setting: We compare the combination of Sep-U and Sep-P ( Sep-UP from now on) to feed-fordward ReLU networks (Glorot et al., 2011) and Batch Normalization as described in
Ioffe & Szegedy (2015) using the same architecture.
For the MOONS dataset, we use depths from 1 to 120 in steps of 10 , and width from 1 to 25 in steps of 1 between 1 and 5 , and steps of 5 afterwards.
In the case of the MNIST dataset, we use depth fromto 64 and width from 2 to 8 in steps of 4 .
Finally, for CIFAR-10 we use depths in { 2 , 10 , 25 , 30 , 40 } with a ﬁxed width of 10 due memory constraints.
Training Parameters:
All the networks used were optimized using Adam (Kingma & Ba, 2014).
More speciﬁcally, for the MOONS dataset we used a learning rate of 0 .
01 for 5000 epochs and a batch size of 85 .
Meanwhile, for both MNIST and CIFAR-10 we used a learning rate of 0 .
0001 forepochs and a batch size of 1024 .
We used convolutional layers with a (3 , 3) kernel size for both
MNIST and CIFAR-10 .
We used λ MOONS =
10 − 4 λ MNIST =
λ
CIFAR = 10 − 8 for Sep-UP .
Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015), ﬁxing the random seed to an arbitrary value of 10 .
As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010) for all the methods and datasets.
4. R ESULTS
Figure 1 shows the results obtained in the MOONS dataset.
Our proposal Sep-UP is able to train networks successfully without increasing the width up to 60 layers deep (see Figures 1c and 1f), while ReLU breaks down at only 30 layers (see Figures 1a and 1d) and ReLU + BN suffers from severe accuracy degradation (see Figures 1b and 1b).
Observe that no conﬁguration with lower width than 2 - 3 is successful in achieving maximum accuracy.
We understand that there exists a minimum width required and this is related to the complexity of the problem.
When using wider layers, the rest of the width is instrumentally used to enable the training of deeper networks.
As previously commented, the larger the layer’s width, the higher the chances of ﬁnding active units that do not cause dead points and dead units during initialization.
In opposition, Sep-UP succesfully overcomes that constraint.
Notice that though Sep-UP is superior to all its competitors, it starts showing performance degradation after reaching depth 60 with the minimum width of 3 .
Considering that the number of parameters increases with the depth and width of the network and that we have a ﬁnite number of trained epochs, we conjecture that the displayed degradation is strictly due to the lack of convergence of the constrained network.
The separation constraint also proves successful on convolutional networks, as tested in MNIST and CIFAR-10 datasets.
Figure 2 shows a similar behaviour to the MOONS dataset (Figure1). [Figure] [Figure]
ReLU breaks down after a few layers, ReLU + BN delays the degradation of accuracy, while
Sep-UP remains functional regardless of the depth.
In the case of our experiments with the
CIFAR-10 dataset (as presented on Figure 3) all the methods degrade with depth, but Sep-UP is the most robust.
In regards to accuracy ReLU performs best closely followed by Sep-UP , while
ReLU + BN clearly overﬁts.
The poorer accuracy values shown are due to a limited choice of width, clearly inferior to the minimum required by the dataset [Figure] and a potential lack of convergence of the separating constraints.
Additional results in the Appendix further elaborate the contributions of each term Sep-U and
Sep-P independently.
40 Depth
ReLU
ReLU-BN
Sep-UP 0.
(a [Figure]
Training accuracy    40 Depth
ReLU
ReLU-BN
Sep-UP 0.
[Figure]
4. R ESULTS USING Z ERO I NITIALIZATION
In order to test the invariace to initialization scheme of Sep-UP , we use Zero Initialization.
As its name states, in this initialization scheme all weights and biases are set to zero.
However, a small variation of the scheme must be introduced in order to break symmetry for the constraints to apply.
Since all the units are initialized to the same value (zero), we use Annealed Dropout (Rennie et al., 2014).
Additionally, instead of adding ξ + and ξ − pairs as in Equations 13 and 15, we use a convex combination with a small perturbation ρ .
In our experiments, we use a value of ρ = 0 . 01 .
[Equation]
Figure 4 summarizes the results found using zero initalization with our constraint formulation.
In comparison to Glorot, we observe that zero initalization requires wider networks.
Indeed, at a depth of 60 , Zero initialization requires a width of 25 units while the Glorot scheme work with onlyunits.
However, if we contrast ReLU + BN to our constraint formulation we ﬁnd that it is unable to train networks of past depth 50 , while Zero initialization achieves 70 layers.
Figure 4 shows the results using zero initialization. Although reported values and behavior are slightly worse compared to Glorot initialization, our results are promising.
Notice how at depth of 60 layers, the zero initialization requires a width of 25 units in contrast to 2 required for Glorot. In addition, zero initialization rquires only 70 layers to reach perfect accuracy in contrast to the 100 required in Glorot (see Figures 1c and 1f).
Moreover, in comparison to ReLU + BN , zero initialized networks show superior performance and behavior (check Figures 1b and 1e).
C ONCLUSIONS
Through the Separation Constraints, we have shown that deeper networks can be trained without increasing the width of the layers.
Moreover, this increment can be done using very small width values.
In this sense we consider that effective training of deeper networks can be achieved by better accommodating the network to the input data.
This departs from many proposal that achieve similar effects by modifying the architecture of the network or manipulating the data.
We believe that this work shows an alternative research path in the pursuit of effective and efﬁcient learning techniques for deep neural networks.
This also opens the possibility of avoiding the use of pyramidal architectures commonly employed in DNNs, thus removing the computational burden caused by the additional units and reducing the dimensionality of the internal representations. [Figure]
We additionally show that our proposal enables the use of Zero Initialization.
The results are promising, and though further experimentation is still needed, they still manage to surpass Batch Normalization in terms of depth, width, and accuracy.
Nevertheless, the modiﬁcations used to break the symmetry of constraints and units need further consideration in order to achieve the same performance than random initialization.
Despite this article provides no information about the dynamics of the training process using Separation Constraints, preliminary revision shows interesting properties.
In particular, training displays slight instabilities which appear as the units/points are reactivated.
Although this does not hinder the training process, further study is needed in order to guarantee a smoother convergence.
The extension of this work to other activation functions is still to be explored.
However, we conjecture that in cases where the activation function display a ﬂat region, e.g. ELU, tanh, or even sigmoid, the current proposal can be applied with minor changes.
Finally, while the separation constraints prevent the vanishing gradient effect, the exploding gradient problem still remains.
Extending the Separation Constraint with an upper bound on the magnitudes of the pre-activation, similar to (cid:15) -insensitive loss, might address it.
It could be also helpful to explore other activation functions whose gradient vanishes with high pre-activations, such as the logistic family
A A PPENDIX A. U NIT BASED AND P OINT BASED S EPARATION C ONSTRAINT E XPERIMENT
In order to justify the decision of combining Sep-U and Sep-P into Sep-UP , we provide Figure 5 showing the same experiment than Figure 1 for Sep-U and Sep-U . We ﬁnd how although Sep-U is able to train deeper and thinner networks it suffers from inferior accuracy (see Figures 5a and 5c), requiring the use of the layer width increase technique.
Contrarily, Sep-P is able to train without increasing the width from 1 to 40 layers deep, but it breaks down afterwards.
Provided how complementary are their strengths and weaknesses, it is sensible to combine them, resulting in the superior performance (larger constant width area together with greater accuracy in higher depths) of
Sep-UP , see Figures 1c and 1f.
Depth
W i d t h 0. [Figure] Depth W i d t h 0.
[Figure]
train
Depth 1234510152025 W i d t h 0.
[Figure]
Depth 1234510152025 W i d t h 0. [Figure] -U enables train thinner and deeper networks (120 layers of single unit), but with reduced performance compared to Sep-UP . It also requires increasing the width of the layers in order to achieve perfect accuracy as ReLU + BN .
Alternatively, Sep-P shows a constant width area between 1 and 40 layers, but its performance is much worse above. This justiﬁes the decision of combining them both into Sep-UP to combine their strengths. A. P ROBABILITY OF A POINT TO BE DEAD IN A NEURAL NETWORK
We conjecture that difﬁculty arising when training neural network as the depth increases or the width decreases, is due the incorrect positioning of the hyperplanes of the units after initialization which causes the apparition of dead points (recall Remark 2.3), that ultimately break back-propagation. In this section we offer a formal treatment of this idea. Theorem A.1 (The probability of a point to be dead increases with depth and decreases with width) . Let F : R n 1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions (cid:96) :
R n k − 1 → R n k with k ∈ 1 , . .
, D , we denote D as the depth of the network.
The layers are composed by a collection of n k ∈ N ReLU unit functions u : R n k − 1 → R + , we denote n k as the width of the layer.
Let the network be initialized following a random distribution.
Then, given a point x ∈ R n 1 we say that the probability of the point dying (Recall Deﬁnition 7) increases with D and decreases with n k .
Lemma A.2 (Probability of a point to be dead for a ReLU unit follows Bernoulli) .
Let be a point x ∈ R n 1 and a ReLU unit u parametrized by a weight vector w ∈ R n and a bias b ∈ R initialized following a random distribution, then the probability of point x being dead for u follows a Bernoulli distribution of unknown parameter p .
[Equation]
Proof.
Since the unit is initialized randomly according to a given distribution (i.e. normal, uniform,
Glorot, ...), the probability of its hyper-plane parametrized by w , b facing x is an unknown p according to the distribution used.
Therefore, the probability of x belonging to lower ( u ) (Recall Equation 3) must be p .
As the lower set (Recall Deﬁnition 3) pre-activation is truncated, its gradient with respect the parameters will be zero (See Rey-Torres et al.
(2019, Section 2)).
Therefore, any point belonging to lower ( u ) will be dead in regards to u .
Lemma A.3 (Probability of a point to be dead for a ReLU layer) .
Let (cid:96) k be a layer composed of n k ∈ N ReLU units u initialized following the same random distribution.
Let be a point x ∈ R n k − 1 .
Then the probability of x being dead for (cid:96) k is:
[Equation]
Proof.
Similarly to the unit case, points belonging to the zero set of a layer (Recall Deﬁnition 4) have zero gradient with respect the parameters of the layer due the truncation (See Rey-Torres et al.
(2019, Section 2)).
The probability of a point belonging to the zero set of a layer P ( x ∈ Z ( (cid:96) k )) is equal to the probability of belonging to all the lowers of the units of the layer P ( ∀ u ∈ (cid:96) | x ∈ lower ( u )) .
Since unit parameters are sampled independently during initialization and the probability of a point being dead for a unit follows Bernoulli, then the probability of a point belonging to Z ( (cid:96) k ) is the product of the probabilities of belonging to the lower of each of the units.
Lemma A.4 (The probability of a point to be dead for a layer diminishes with its width) .
Let (cid:96) k be a layer composed of n k ∈ N ReLU units u initialized following a random distribution.
Let be a point x ∈ R n k − 1 .
Then the probability of x being dead diminishes with n k :
[Equation]
Proof.
The probability of a point being dead for a layer is the multiplication of the probabilities of being dead of each of its units, recall Lemma A.3.
Therefore, as the probabilities are lower thansince the upper set of a unit is never empty, in the limit where n k tends to inﬁnity the probability tends to zero.
Note how Lemma A.4 can explain why increasing width allows us to train deeper networks (See
Hasanpour et al.
; Huang et al.
, it simply improves the chances of ﬁnding a hyperplane that allows the points to live.
Deﬁnition A.1 (Dead point set in a neural network) .
Let F : R n 1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions (cid:96) :
R n k − 1 → R n k with k ∈ 1 , . .
, D , we denote D as the depth of the network.
The layers are composed by a collection of n k ∈ N
ReLU unit functions u : R n k − 1 → R + , we denote n k as the width of the layer.
Then, we deﬁne the set of dead points of a neural network following Equation 7 as.
[Equation]
Lemma A.5 (Probability of a point to be dead for a ReLU feed-forward network) .
Let F : R n 1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions (cid:96) :
R n k − 1 → R n k with k ∈ 1 , . .
, D , we denote D as the depth of the network.
The layers are composed by a collection of n k ∈ N ReLU unit functions u : R n k − 1 → R + , we denote n k as the width of the layer.
Let the network be initialized following a random distribution.
Then, given a point x ∈ R n 1 we say that the probability of the point dying (Recall Deﬁnition 7) is
[Equation]
Proof.
The probability of not dying in a given layer is equal to 1 minus the probability of dying.
[Equation]
As the sampling of the parameters during intialization is independent among layers, the probabiliity of not dying in any of the layers of the network is the multiplication of the probabilities of not dying in each of the layers.
[Equation]
The probability of a point to be dead for a network is equal to the probability of being dead at least in one layer, see 7.
This is equivalent to 1 minus the probability of not dying in any of the layers.
[Equation]
Plugging Equation 24 into Equation 25 results in Equation 22.
Lemma A.6 (The probability of a point to be dead for a layer increases with its depth) .
Let F : R n 1 → R n D be a feed-forward neural network composed by a collection of D ∈ N layer functions (cid:96) :
R n k − 1 → R n k with k ∈ 1 , . .
, D , we denote D as the depth of the network.
Let the network be initialized following a random distribution.
Then, given a point x ∈ R n 1 we say that the probability of the point dying as it traverses the network (Recall Deﬁnition 7) increases with D .
[Equation]
Proof.
Since the hyper-planes of the units split the space in two half-spaces, the probability of point not being dead is always lower than 1 .
[Equation]
Therefore, the limit of the probability of x not dying in any of the layers when D tends to inﬁnity is zero, recall Equation 24.
[Equation]
Which when plugged into Equation 22 leads to Equation 26.
A. E FFECT OF THE SEPARATION ON THE INTERNAL REPRESENTATIONS
In this section we explore the effect of the proposed constraints in the activation of the network.
For that task we train all methods ﬁxing the architecture at 50 layers of 4 units each.
We use the
MOONS dataset for easy visualization, in the same conﬁguration than Section 4(sampling 100 points, 85 for training and 15 for validation) .
Experimental setting: We compare the combination of Sep-U , Sep-P and Sep-UP to feedfordward ReLU networks (Glorot et al., 2011) and Batch Normalization as described in Ioffe &
Szegedy (2015) using the same architecture.
Training Parameters:
All the networks used were optimized using Adam (Kingma & Ba, 2014).
We used a learning rate of 0 . 01 for 1000 epochs and a batch size of 85 .
We used λ = 10 − 4 when required.
As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010).
We also test the use of the Separation Constraints alone, without categorical crossentropy, to assess its effect by itself.
Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015), ﬁxing the random seed to an arbitrary value of 10 .
Table 1 summarizes the results.
Figures 6, 7, 11, 10 and 8 show the activation plots for each of the methods tested.
A.3. A CTIVATION COLLAPSE
[Table]
loss (for train and validation sets) for ReLU , ReLU + BN , and Sep-Cons in all its variants.
In terms of accuracy ReLU reaches a trivial accuracy of 0 .
51 while ReLU + BN reaches 0 .
60 , both architectures fail to solve the problem, see Table 1.
When comparing the results presented in
Figures 6 and 7, we ﬁnd that at the fourth layer ReLU has collapsed the points of the dataset over a line (parts 6b and 6c), while ReLU + BN still manages to warp the dataset.
However, both methods at layer 25th have collapsed the dataset into a small number of points (see Figures 7d, 7e for ReLU + BN and Figures 6d and 6e for ReLU ) that is then pushed to the point zero for ReLU (as shown in the feature layer: 6f and 6g), while ReLU + BN collapses to few points (Figures 7f and 7g).
This collapse is congruent with our expectations, displaying the born-dead network condition from Lu et al.
Note how layer after layer its effect is worse, as we predicted in A.2.
In addition, since the gradient is only back-propagated through the points lying on the upper sets of units, geometric collapse also stops learning.
Notice that the standarization and the afﬁne transformation (dependent on γ and β ) from ReLU + BN are not able to prevent from killing the dataset.
Thus, training thin and deep neural networks is difﬁcult with ReLU or ReLU + BN .
On the extent of our experimentation ReLU networks featuring separation constraints are able to solve the problem with higher degrees of accuracy than ReLU and ReLU + BN (see the ﬁgures in Table 1).
Moreover, Figures 8h, 11h and 10h), show that the separation function behaves intuitively in the sense of Hauser & Ray (2017).
The internal representations not only are non-trivial (as in ReLU + BN ) but also preserve geometrical structures like shape and connectivity, as shown in Figures 11d, 10d or 8e.
Indeed,
Figures 11c, 8c and 10c showcase how at the 4th layers a much a solution is already found.
This proves that the gradient of the main loss is back-propagated to the input, unlike ReLU and ReLU +
BN (Figures 6b, 6b, 7b, 7c). [Table] [Figure] [Figure]
[Figure]
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+
[Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
L and Sep-UP , versus ReLU and ReLU + BN .
Notice how effectively ReLU and ReLU +
BN collapse the dataset into few points whereas Sep-UP force the network to learn representations that preserve geometrical structure useful for back-propagation.
The constraints enforce richer representations when used without main loss (cross-entropy), see Figure 9.
whereas the representation reached by Sep-UP is more complex, yet preserves connectivity.
Furthermore, both avoid mapping the entire dataset to few values as observed with
ReLU (Figures 9e and 9f) and ReLU + BN (Figures 9b and 9c).
The only exception is Sep-U that apparently solves the problem at layer 25th on Figures 11d and 11e, yet collapses the dataset in two points at the feature layer (Figures 11f and 11f).
Our intuition is is that, although Sep-U prevents dead units, it cannot prevent points from falling into the intersection lower set of the units of a layer, resulting in a dead point ).
The Sep-P separation constraint was designed precisely to prevent the presence of dead points.
However, Sep-P allows afﬁne and dead units (as shown in Figures 10d and 10e), so the the decision function reached becomes too linear (see Figure 10h).
Therefore, if we combine both Sep-U and Sep-P we should have best of both worlds, as we say in Section A.1.
Figure 8, shows how Sep-UP is able to separate both classes perfectly (recall the 0 .
93 in accuracy from Table 1).
Furthermore Sep-UP , produces a feature layer that does not concentrates the dataset in two points like Sep-U , nor distributes the dataset linearly like
Sep-P .
[Figure]
in the MOONS dataset for 1000 epochs.
The network displays a richer internal representation without collapsing the dataset like Sep-U or ReLU + BN .
However, plenty of dead units appear since they are not penalized, causing underﬁtting.
[Figure]
Notice how dead units have been reduced.
The interal representations are much richer than ReLU or ReLU + BN .
Despite collapsing the dataset in two points at the feature layer, the classiﬁcation performed in the output layer is approximately correct.
We conjecture that this is due the dead point addressed with Sep-P .
A. R ELATION BETWEEN SEPARATION CONSTRAINTS AND MAIN LOSS
In this section we explore the effect of the proposed constraints in the convergence of the network during optimization.
We train a network of 50 layers of 4 units each using Sep-UP .
We use the
MOONS dataset, in the same conﬁguration than Section 4 (sampling 100 points, 85 for training and 15 for validation).
Experimental setting: We analyze the interaction between of the main loss (cross-entropy) and the loss arising from the Separation Constraints.
We plot them, together with training accuracy in
Figure 12.
Below we attach the decision boundaries at the most relevant epochs.
Training Parameters:
The networks wes optimized using Adam (Kingma & Ba, 2014).
We used a learning rate of 0 .
01 for 17000 epochs and a batch size of 85 .
We used λ = 10 − 4 .
As initialization scheme, we used Glorot uniform from Glorot & Bengio (2010).
Our experiments were conducted using Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015), ﬁxing the random seed to an arbitrary value of 10 .
We see at Figure 12 how the use of the separation constraints enables back-propagation, and thus cross-entropy minimization.
According to it, the training begins minimizing the constraint loss until a certain level is reached (approximately 10 − 2 circa epoch 250).
During this phase, the cross-entropy loss stays constant.
Only when this critical point in the constraint loss is reached, back-propagation is restored and the accuracy starts climbing up to 100%.
However, using an this additional loss (from the separation constraints) induces aggressive transient states (e.g. instabilities) during training (approximately around epochs 6200 , 11883 and 16432 ).
We see also how after each of the transient states, the network ﬁnds a new solution different from earlier ones.
Moreover, those solutions show an increasing level of complexity, in the form of smoother and more rouded boundaries.
For instance, the ﬁrst solution (Figure 12f) is linear and the second is composed of few lines (Figure 12h, whereas the last and second to last (Figure 12ai and 12ao) are much more curved.
We conjecture that this behaviour must be due dead units and points being revived during training.
As a dead unit or point comes back from being dead, the weight conﬁguration set by the Separation Constraints is likely to be unrelated to cross-entropy.
That causes a break down in the convergence that we can see around epochs 6200 , 11883 and 16432 .
As back-propagation is now working for those points or units, their error is eventually corrected by normal cross-entropy minimization and the network converges to a solution again.
However, as now there is an additional working unit or point involved, the network has greater representational expressiveness, which results in a more complex decision boundary.
[Figure]
[Figure]
[Figure]
[Table]
[Table]
[Table]
(e
[Table]
[Table]
[Table]
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (
[Table]
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E-
[Table]
1.05E
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
3.75E-01 4.00E-01 4.25E4.50E-01 4.75E-01 5.00E5.25E-01 5.50E-01 5.75E6.00E- (k)0.2 0.0 0.2 0.4 0.6 0.
2.00E-01 3.00E-01 4.00E5.00E
[Table]
4.60E-01 4.80E5.00E-01 5.20E5.40E-01 5.60E5.80E-01 6.00E-
(x)0.2 0.0 0.2 0.4 0.6 0.
4.25E-01 4.50E-01 4.75E5.00E-01 5.25E-01 5.50E5.75E-01 6.00E-01 6.25E- (y)0.2 0.0 0.2 0.4 0.6 0.
4.06E-01 4.06E4.06E-01 4.06E4.06E-01 4.06E- (z)0.2 0.0 0.2 0.4 0.6 0.
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (aa)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ab)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ac)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ad)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ae)0.2 0.0 0.2 0.4 0.6 0.
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (af)0.2 0.0 0.2 0.4 0.6 0.
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (ag)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ah)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ai)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (aj)0.2 0.0 0.2 0.4 0.6 0.
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (ak)0.2 0.0 0.2 0.4 0.6 0.
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (al)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (am) 16436 0.2 0.0 0.2 0.4 0.6 0.
0.00E+00 1.50E3.00E-01 4.50E6.00E-01 7.50E9.00E-01 1.05E+ (an)0.2 0.0 0.2 0.4 0.6 0.
{+0.00E+00 1.50E3.00E-01 4.50E6.00E-
+}
7.50E9.00E-01 1.05E+ (ao)0.2 0.0 0.2 0.4 0.6 0.
1.00E+00 1.00E+00 1.00E+
1.00E+00 1.00E+00 1.00E+
1.00E+00 1.00E+00 1.00E+ (ap)