D EEP L EARNING AS A M IXED C ONVEXC OMBINATORIAL O PTIMIZATION P ROBLEM
Anonymous authors
Paper under double-blind review
A BSTRACT
As neural networks grow deeper and wider, learning networks with hard-threshold activations is becoming increasingly important, both for network quantization, which can drastically reduce time and energy requirements, and for creating large integrated systems of deep networks, which may have non-differentiable components and must avoid vanishing and exploding gradients for effective learning.
However, since gradient descent is not applicable to hard-threshold functions, it is not clear how to learn them in a principled way.
We address this problem by observing that setting targets for hard-threshold hidden units in order to minimize loss is a discrete optimization problem, and can be solved as such.
The discrete optimization goal is to ﬁnd a set of targets such that each unit, including the output, has a linearly separable problem to solve.
Given these targets, the network decomposes into individual perceptrons, which can then be learned with standard convex approaches.
Based on this, we develop a recursive mini-batch algorithm for learning deep hardthreshold networks that includes the popular but poorly justiﬁed straight-through estimator as a special case.
Empirically, we show that our algorithm improves classiﬁcation accuracy in a number of settings, including for AlexNet and ResNeton ImageNet, when compared to the straight-through estimator.
I NTRODUCTION
The original approach to neural classiﬁcation was to learn single-layer models, like the perceptron (Rosenblatt, 1958).
However, it proved difﬁcult to extend these methods to multiple layers, because hard-threshold units, having zero derivative almost everywhere and being discontinuous at the origin, cannot be trained by gradient descent.
Instead, the community turned to multilayer networks with soft activation functions, such as the sigmoid and, more recently, the ReLU, for which gradients can be computed efﬁciently by backpropagation (Rumelhart et al., 1986).
This approach has enjoyed remarkable success, enabling researchers to train networks with hundreds of layers and learn models that have signiﬁcantly higher accuracy on a variety of tasks than any previous approach.
However, as networks become deeper and wider, there has been a growing trend towards using hard-threshold activations for quantization purposes, where they enable binary or low-precision inference (e.g., Hubara et al.
; Rastegari et al.
; Zhou et al.
Lin & Talathi (2016); Zhu et al.
and training (e.g., Lin et al.
; Li et al.
; Tang et al.
; Micikevicius et al.
(2017)), which can greatly reduce the energy and computation time required by modern deep networks.
Beyond quantization, the scale of the output of hard-threshold units is independent of (or insensitive to) the scale of their input, which can alleviate vanishing and exploding gradient issues and should help avoid some of the pathologies that occur during low-precision training with backpropagation (Li et al., 2017).
Avoiding these issues is crucial for developing large systems of deep networks that can be used to perform even more complex tasks.
For these reasons, we are interested in developing well-motivated and efﬁcient techniques for learning deep neural networks with hard-threshold units.
In this work, we propose a framework for learning deep hard-threshold networks that stems from the observation that hard-threshold units output discrete values, indicating that combinatorial optimization may provide a principled method for training these networks.
By specifying a set of discrete targets for each hidden-layer activation, the network decomposes into many individual perceptrons, each of which can be trained easily given its inputs and targets.
The difﬁculty in learning a deep hard-threshold network is thus in setting the targets so that each trained perceptron – including the output units – has a linearly separable problem to solve and thus can achieve its targets.
We show that networks in which this is possible can be learned using our mixed convex-combinatorial optimization framework.
Building on this framework, we then develop a recursive algorithm, feasible target propagation ( FTPROP ), for learning deep hard-threshold networks.
Since this is a discrete optimization problem, we develop heuristics for setting the targets based on per-layer loss functions.
The mini-batch version of FTPROP can be used to explain and justify the oft-used straight-through estimator (Hinton, 2012;
Bengio et al., 2013), which can now be seen as an instance of FTPROP with a speciﬁc choice of per-layer loss function and target heuristic.
Finally, we develop a novel per-layer loss function that improves learning of deep hard-threshold networks.
Empirically, we show improvements for our algorithm over the straight-through estimator on CIFAR10 for two convolutional networks and on
ImageNet for AlexNet and ResNet-18, with multiple types of hard-threshold activation.
R ELATED WORK
The most common method for learning deep hard-threshold networks is to use backpropagation with the straight-through estimator (STE) (Hinton, 2012; Bengio et al., 2013), which simply replaces the derivative of each hard-threshold unit with the identity function.
The STE is used in the quantized network literature (see citations above) to propagate gradients through quantized activations, and is used in Shalev-Shwartz et al.
(2017) for training with ﬂat activations.
Later work generalized the
STE to replace the hard-threshold derivative with other functions, including saturated versions of the identity function (Hubara et al., 2016).
However, while the STE tends to work quite well in practice, we know of no rigorous justiﬁcation or analysis of why it works or how to choose replacement derivatives.
Beyond being unsatisfying in this regard, the STE is not well understood and can lead to gradient mismatch errors, which compound as the number of layers increases (Lin & Talathi, 2016).
We show here that the (saturated) STE is a special case of our framework, thus providing a principled justiﬁcation for it and a basis for exploring and understanding alternatives.
Another common approach to training with hard-threshold units is to use randomness, either via stochastic neurons (e.g., Bengio et al.
; Hubara et al.
or probabilistic training methods, such as those of Soudry et al.
(2014) or Williams (1992), both of which are methods for softening hard-threshold units.
In contrast, our goal is to learn networks with deterministic hard-threshold units.
Finally, target propagation (LeCun, 1986; 1987; Carreira-Perpi ˜ n ´ an & Wang, 2014; Bengio, 2014;
Lee et al., 2015; Taylor et al., 2016) is a method that explicitly associates a target with the output of each activation in the network, and then updates each layer’s weights to make its activations more similar to the targets.
Our framework can be viewed as an instance of target propagation that uses combinatorial optimization to set discrete targets, whereas previous approaches employed continuous optimization.
The MADALINE Rule II (MRII) algorithm (Winter & Widrow, 1988) can also be seen as a special case of our framework and of target propagation, where only one target is set at a time.
L EARNING DEEP NETWORKS WITH HARD - THRESHOLD UNITS
Given a dataset D = { ( x ( i ) , t ( i ) ) } mi =1 with vector-valued inputs x ( i ) ∈ R n and binary targets t ∈ {− 1 , +1 } , we are interested in learning an (cid:96) -layered deep neural network with hard-threshold units y =
[Equation]
. . . )) , (1) with weight matrices W = { W d : W d ∈ R n d × n d − 1 } (cid:96)d =1 and element-wise activation function g ( x ) =
sign( x ) , where sign is the sign function such that sign( x )
=
if x > 0 and −otherwise.
Each layer d has n d units, where we deﬁne n 0 = n for the input layer, and we let h d
=
g
( W d . . .
g ( W 1 x ) . . . )
denote the output of each hidden layer, where h d = ( h d 1 , . .
, h dn d ) and h dj ∈ {− 1 , +1 } for each layer d and each unit j .
Similarly, we let z d = W d g ( . . .
g ( W 1 x ) . . . )
denote the pre-activation output of layer d .
For compactness, we have incorporated the bias term into the weight matrices.
We denote a row or column of a matrix W d as W d, : j and W d,j : , respectively, and the entry in the j th row and k th column as W d,jk .
Using matrix notation, we can write this model as Y = f
( X ; W ) =
g ( W (cid:96) . . .
g ( W 1 X ) . . . ) , where X is the n × m matrix of dataset instances and
Y is the n (cid:96) × m matrix of outputs.
We let T (cid:96) denote the matrix of ﬁnal-layer targets, H d denote the n d × m matrix of hidden activations at layer d , and Z d denote the n d × m matrix of pre-activations at layer d .
Our goal will be to learn f by ﬁnding the weights W that minimize an aggregate loss
L ( Y, T (cid:96) ) =
(cid:80) mi =1 L ( y ( i ) , t ( i ) ) for some convex per-instance loss L ( y, t ) .
In the simplest case, a hard-threshold network with no hidden layers is a perceptron Y =
g ( W 1 X ) , xx
W
W 2 hhhyytt 22 set Txxth
W 1 , 1: x xth
W 1 , 2: x xth
W 1 , 3: yt 21 ttt
W 2 , 1: yt 22 ttt
W 2 , 2: as introduced by Rosenblatt (1958).
The goal of learning a perceptron, or any hard-threshold network, is to classify unseen data.
A useful ﬁrst step is to be able to correctly classify the training data, which we focus on here for simplicity when developing our framework; however, standard generalization techniques such as regularization are easily incorporated into this framework.
Since a perceptron is a linear classiﬁer, it is only able to separate a linearly-separable dataset.
Deﬁnition 1.
A dataset { ( x ( i ) , t ( i ) ) } mi =1 is linearly separable iff there exists a vector w ∈ R n and a real number γ >
such that ( w · x ( i ) )
t ( i ) > γ for all i = 1 . . .
m .
When a dataset is linearly separable, the perceptron algorithm is guaranteed to ﬁnd its separating hyperplane in a ﬁnite number of steps (Novikoff, 1962), where the number of steps required is dependent on the size of the margin γ .
However, linear separability is a very strong condition, and even simple functions, such as XOR, are not linearly separable and thus cannot be learned by a perceptron (Minsky & Papert, 1969).
We would thus like to be able to learn multilayer hard-threshold networks.
Consider a simple single-hidden-layer hard-threshold network Y =
f ( X ; W ) =
g ( W 2 g ( W 1 X ))
=
g ( W 1 H 1 ) for a dataset D =
( X, T 2 ) , where H 1 = g ( W 1 X ) are the hidden-layer activations.
Clearly,
Y and H 1 are both collections of (single-layer) perceptrons.
Backpropagation cannot be used to train the input layer
’s weights W 1 , but since each hidden activation h 1 j is the output of a perceptron, then if we knew the value t 1 j ∈ {− 1 , +1 } that each hidden unit should take for each input x ( i ) , we could use the perceptron algorithm to set W 1 to produce these values.
We refer to t 1 j as the target of h 1 j .
Given a matrix of hidden-layer targets T 1 ∈ {− 1 , +1 } n 1 × m , each layer (and in fact each perceptron in each layer) can be learned separately, as they no longer depend on each other, where the goal of perceptron learning is to update the weights of each layer d so that its activations H d equal its targets
T d given inputs T d − 1 .
Figure 1 shows an example of this decomposition.
We denote the targets of an (cid:96) -layer network as T =
{ T 1 , . . .
, T (cid:96) } , where T k for k = 1 . . .
(cid:96) − 1 are the hidden-layer targets, T (cid:96) are the dataset targets, and we often let T
= X for notational convenience.
Auxiliary-variable-based approaches, such as ADMM (Taylor et al., 2016; Carreira-Perpi ˜ n ´ an & Wang, 2014) and other target propagation methods (LeCun, 1986; Lee et al., 2015) use a similar process for decomposing the layers of a network; however, these focus on continuous variables and impose (soft) constraints to ensure that each activation equals its auxiliary variable.
We take a different approach here, inspired by the combinatorial nature of the problem and the perceptron algorithm.
Since the ﬁnal layer is a perceptron, the training instances can only be separated if the hidden-layer activations H 1 are linearly separable with respect to the dataset targets T 2 .
Thus, the hidden-layer targets T 1 must be set such that they are linearly separable with respect to the dataset targets T 2 , since the hidden-layer targets T 1 are the intended values of their activations H 1 .
However, in order to ensure that the hidden-layer activations H 1 will equal their targets T 1 after training, the hidden-layer targets T 1 must be able to be produced (exactly) by the ﬁrst layer, which is only possible if the hidden-layer targets T 1 are linearly separable with respect to the inputs X .
Thus, a sufﬁcient condition for f ( X ; W ) to separate the data is that the hidden-layer targets induce linear separability in all units in both layers of the network.
We refer to this property as feasibility.
Deﬁnition 2.
A setting of the targets T = { T 1 , . . .
, T (cid:96) } of an (cid:96) -layer
deep hard-threshold network f
( X ; W ) is feasible for a dataset D =
( X, T (cid:96) )
iff for each unit j = 1 . . .
n d in each layer d = 1 . . .
(cid:96) the dataset formed by its inputs T d − 1 and targets T d,j : is linearly separable, where T 0 (cid:44) X .
Feasibility is a much weaker condition than linear separability, since the output decision boundary of a multilayer hard-threshold network with feasible targets is in general highly nonlinear.
It follows from the deﬁnition of feasibility and convergence of the perceptron algorithm that if a feasible setting of a network’s targets on a dataset exists, the network can separate the training data.
Proposition 1.
Let D =
( x ( i ) , t ( i ) ) } be a dataset and let f ( X ; W ) be an (cid:96) -layer hard-threshold network with feasible targets T
= { T 1 , . .
, T (cid:96) } in which each layer d of f was trained separately with inputs T d − 1 and targets T d , where T 0 (cid:44) X , then f will correctly classify each instance x
( i ) , such that f ( x ( i ) ; W ) t ( i ) > 0 for all i
= 1 . . .
m .
Learning a deep hard-threshold network thus reduces to ﬁnding a feasible setting of its targets and then optimizing its weights given these targets, i.e., mixed convex-combinatorial optimization.
The simplest method for this is to perform exhaustive search on the targets.
Exhaustive search iterates through all possible settings of the hidden-layer targets, updating the weights of each perceptron whose inputs or targets changed, and returns the weights and feasible targets that result in the lowest loss.
While impractical, exhaustive search is worth brieﬂy examining to better understand the solution space.
In particular, because of the decomposition afforded by setting the targets, exhaustive search over just the targets is sufﬁcient to learn the globally optimal deep hard-threshold network, even though the weights are learned by gradient descent.
Proposition 2.
If a feasible setting of a deep hard-threshold network’s targets on a dataset D exists, then exhaustive search returns the global minimum of the loss in time exponential in the number of hidden units.
Learning can be improved and feasibility relaxed if, instead of the perceptron algorithm, a more robust method is used for perceptron learning.
For example, a perceptron can be learned for a non-linearlyseparable dataset by minimizing the hinge loss L ( z, t ) = max(0 , 1 − tz ) , a convex loss on the perceptron’s pre-activation output z and target t that maximizes the margin when combined with L2 regularization.
In general, however, any method for learning linear classiﬁers can be used.
We denote the loss used to train the weights of a layer d as L d , where the loss of the ﬁnal layer L (cid:96) is the output loss.
At the other end of the search spectrum is hill climbing.
In each iteration, hill climbing evaluates all neighboring states of the current state (i.e., target settings that differ from the current one by only one target) and chooses the one with the lowest loss.
The search halts when none of the new states improve the loss.
Each state is evaluated by optimizing the weights of each perceptron given the state’s targets, and then computing the output loss.
Hill climbing is more practical than exhaustive search, since it need not explore an exponential number of states, and it also provides the same local optima guarantee as gradient descent on soft-threshold networks.
Proposition 3.
Hill climbing on the targets of a deep hard-threshold network returns a local minimum of the loss, where each iteration takes time linear in the size of the set of proposed targets.
Exhaustive search and hill climbing comprise two ends of the discrete optimization spectrum.
Beam search, which maintains a beam of the most promising solutions and explores each, is another powerful approach that contains both hill climbing and exhaustive search as special cases.
In general, however, any discrete optimization algorithm can be used for setting targets.
For example, methods from satisﬁability solving, integer linear programming, or constraint satisfaction might work well, as the linear separability requirements of feasibility can be viewed as constraints on the search space.
We believe that our mixed convex-combinatorial optimization framework opens many new avenues for developing learning algorithms for deep networks, including those with non-differentiable modules.
In the following section, we use these ideas to develop a learning algorithm that hews much closer to standard methods, and in fact contains the straight-through estimator as a special case.
F EASIBLE TARGET PROPAGATION
The open question from the preceding section is how to set the hidden-layer targets.
Generating good, feasible targets for the entire network at once is a difﬁcult problem; instead, an easier approach is to propose targets for only one layer at a time.
As in backpropagation, it makes sense to start from the output layer, since the ﬁnal-layer targets are given, and successively set targets for each upstream layer.
Further, since it is hard to know a priori if a setting of a layer’s targets is feasible for a given network architecture, a simple alternative is to set the targets for a layer d and then optimize the upstream weights (i.e., weights in layers j ≤ d ) to check if the targets are feasible.
Since the goals when optimizing a layer’s weights and when setting its upstream targets (i.e., its inputs) are the same – namely, to induce feasibility – a natural method for setting target values is to choose targets that reduce the layer’s loss L d .
However, because the targets are discrete, moves in target space are large and non-smooth and cannot be guaranteed to lower the loss without actually performing the move.
Thus, heuristics are necessary.
We discuss these in more detail below.
Determining feasibility of the targets at layer d can be done by recursively updating the weights of layer d and proposing targets for layer d − 1 given the targets for layer d .
This recursion continues until the input layer is reached, where feasibility (i.e., linear separability) can be easily determined by optimizing that layer’s weights given its targets and the dataset inputs.
The targets at layer d can then be updated based on the information gained from the recursion and, if the upstream weights were altered, based on the new outputs of layer d − 1 .
We call this recursive algorithm feasible target propagation , or FTPROP .
Pseudocode is shown in Algorithm 1.
As the name implies, FTPROP is a form of target propagation (LeCun, 1986; 1987; Lee et al., 2015) that uses discrete instead of continuous optimization to set targets.
FTPROP is also highly related to RDIS (Friesen & Domingos, 2015), a powerful nonconvex optimization algorithm based on satisﬁability (SAT) solvers that recursively chooses and sets subsets of variables in order to decompose the underlying problem into simpler subproblems.
While RDIS is applied only to continuous problems, the ideas behind RDIS can be generalized to discrete variables via the sum-product theorem (Friesen & Domingos, 2016).
This suggests an interesting connection between
FTPROP and SAT that we leave for future work.
Algorithm 1 Train an (cid:96) -layer
hard-threshold network Y =
f ( X ; W ) on dataset D =
( X, T (cid:96) ) with feasible target propagation ( FTPROP ) using loss functions L = { L d } (cid:96)d =1 .
1: initialize weights W =
{ W 1 , . . .
, W (cid:96) } randomly 2: initialize targets T 1 , . .
, T (cid:96) − 1 as the outputs of their hidden units in f ( X ; W ) 3: set T
X and set T ← { T 0 , T 1 , . .
, T (cid:96) } 4: FTPROP
( W, T, L, (cid:96) ) // train the network by searching for a feasible target setting 5: function FTPROP (weights W , targets T , losses L , and layer index d ) 6: optimize W d with respect to layer loss L d ( Z d , T d ) // check feasibility; Z d = W d T d −7: if activations H d = g ( W d T d − 1 ) equal the targets T d then return True // feasible 8: else if this is the ﬁrst layer (i.e., d = 1 ) then return False // infeasible 9: while computational budget of this layer not exceeded do // e.g., determined by beam search 10:
T d − 1 ← set targets for upstream layer to reduce layer loss L d ( Z d , T d ) 11: if FTPROP ( W, T, L, d − 1) then // check if targets T d − 1 are feasible 12: optimize W d with respect to layer loss L d ( Z d , T d ) 13: if activations H d = g ( W d T d − 1 ) equal the targets T d then return True // feasible 14: return False
Of course, modern deep networks will not always have a feasible setting of their targets for a given dataset.
For example, a convolutional layer imposes a large amount of structure on its weight matrix, making it less likely that the layer’s input will be linearly separable with respect to its targets.
Further, ensuring feasibility will in general cause learning to overﬁt the training data, which will worsen generalization performance.
Thus, we would like to relax the feasibility requirements.
In addition, there are many beneﬁts of using mini-batch instead of full-batch training, including improved generalization gap (e.g., see LeCun et al.
(2012) or Keskar et al.
, reduced memory usage, the ability to exploit data augmentation, and the prevalence of tools (e.g., GPUs) designed for it.
Fortunately, it is straightforward to convert FTPROP to a mini-batch algorithm and to relax the feasibility requirements.
In particular, since it is important not to overcommit to any one mini-batch, the mini-batch version of FTPROP (i) only updates the weights and targets of each layer once per mini-batch; (ii) only takes a small gradient step on each layer’s weights, instead of optimizing them fully; (iii) sets the targets of the downstream layer in parallel with updating the current layer’s weights, since the weights will not change much; and (iv) removes all checks for feasibility.
We call this algorithm FTPROP - MB and present pseudocode in Algorithm 2 in Appendix A. FTPROP - MB closely resembles backpropagation-based methods, allowing us to easily implement it with standard libraries.
3. T ARGET HEURISTICS
When the activations of each layer are differentiable, backpropagation provides a method for telling each layer how to adjust its outputs to improve the loss.
Conversely, in hard-threshold networks, target propagation provides a method for telling each layer how to adjust its outputs to improve the next layer’s loss, as long as the targets are set effectively.
While gradients cannot propagate through hard-threshold units, the derivatives within a layer can still be computed.
An effective and efﬁcient
[Figure]
QuantizedReLU
[Figure]
heuristic for setting the target t dj for an activation h dj of layer d is to use the (negative) sign of the partial derivative of the next layer’s loss.
Speciﬁcally, we set t dj =
r
( h dj ) , where
[Equation]
and Z d +1 is either the pre-activation or post-activation output, depending on the choice of loss.
When used to update only a single target at a time, this heuristic will often set the target value that correctly results in the lowest loss.
In particular, when L d +1 is convex, its negative partial derivative with respect to h dj by deﬁnition points in the direction of the global minimum of L d +1 .
Without loss of generality, let h dj
= − 1 .
Now, if r ( h dj )
= − 1 , then it follows from the convexity of the loss that ﬂipping h dj and keeping all other variables the same would increase L d +1 .
On the other hand, if r ( h dj ) =
+1 , then ﬂipping h dj may or may not reduce the loss, since convexity cannot tell us which of h dj
=
+1 or h dj = − 1 results in a smaller L d +1 .
However, the discrepancy between h dj and r ( h dj ) indicates a lack of conﬁdence in the current value of h dj .
A natural choice is thus to set t dj to push the pre-activation value of h dj towards 0 , making h dj more likely to ﬂip.
Setting t dj
= r ( h dj ) =
+1 accomplishes this.
We note that, while this heuristic performs well, there is still room for improvement, for example by extending r ( · ) to better handle the h dh (cid:54) =
r ( h dj ) case or by combining information across the batch.
We leave such investigations for future work.
3. L AYER LOSS FUNCTIONS
The hinge loss, shown in Figure 2a, is a robust version of the perceptron criterion and is thus a natural per-layer loss function to use for ﬁnding a feasible (or nearly feasible) setting of the targets and weights.
However, in preliminary experiments, we found that learning tended to stall and become erratic over time when using the hinge loss for each layer.
We attribute this to two separate issues.
First, the hinge loss is sensitive to noisy data and outliers (Wu & Liu, 2007), which can cause learning to focus on instances that are unlikely to ever be classiﬁed correctly, instead of on instances near the separator.
Second, since with convolutional layers and large, noisy datasets it is unlikely that a layer’s inputs are entirely linearly separable, it is thus important to prioritize some targets over others.
Ideally, the highest priority targets would be those with the largest effect on the output loss.
The ﬁrst issue can be solved by saturating (truncating) the hinge loss, thus making it less sensitive to outliers (Wu & Liu, 2007).
The saturated hinge loss, shown in Figure 2b, is sat hinge( z, t ; b ) = max(0 , 1 − max( tz, b )) for some threshold b , where we set b = − 1 to make its derivative symmetric.
The second problem can be solved in a variety of ways, including randomly subsampling targets or weighting the loss associated with each target according to some heuristic.
The simplest and most accurate method that we have found is to weight the loss for each target t dj by the magnitude of the partial derivative of the next layer’s loss L d +1 with respect to the target’s hidden unit h dj , such that
[Equation]
While the saturated hinge loss works well, if the input z dj ever moves out of the range [ − 1 , +1] then its derivative will become zero and the unit will no longer be trainable.
To avoid this, we propose the soft hinge loss , shown in Figure 2c, where soft hinge( z, t ) =
tanh( − zt ) + 1 .
Like the saturated hinge, the soft hinge has slope 1 at the threshold and has a symmetric derivative; however, it also beneﬁts from having a larger input region with non-zero derivative.
Note that Bengio et al.
(2013) report that using the derivative of a sigmoid as the STE performed worse than the identity function.
Based on our experiments with other loss functions, including variations of the squared hinge loss and the log loss, this is most likely because the slope of the sigmoid is less than unity at the threshold, which causes vanishing gradients.
Loss functions with asymmetric derivatives around the threshold also seemed to perform worse than those with symmetric derivatives (e.g., the saturating and soft hinge losses).
In our experiments, we show that the soft hinge loss outperforms the saturated hinge loss for both sign and quantized ReLU activations, which we discuss below.
3. R ELATIONSHIP TO THE STRAIGHT - THROUGH ESTIMATOR
When each loss term in each hidden layer is scaled by the magnitude of the partial derivative of its downstream layer’s loss and each target is set based on the sign of the same partial derivative, then target propagation transmits information about the output loss to every layer in the network, despite the hard-threshold units.
Interestingly, this combination of loss function and target heuristic can exactly reproduce the weight updates of the straight-through estimator (STE).
Speciﬁcally, the weight updates that result from using the scaled saturated hinge loss from (3) and the target heuristic in (2) are exactly those of the saturated straight-through estimator (SSTE) deﬁned in Hubara et al.
(2016), which replaces the derivative of sign( z ) with 1 | z |≤ 1 , where 1 ( · ) is the indicator function.
Other STEs correspond to different choices of per-layer loss function.
This connection provides a justiﬁcation for existing STE approaches, which can now be seen as instances of FTPROP with a speciﬁc choice of per-layer loss function and target heuristic.
We believe that this will enable more-principled investigations and extensions of these methods in future work.
3. Q UANTIZED ACTIVATIONS
Straight-through estimation is also commonly used to backpropagate through quantized variants of standard activations, such as the ReLU.
Figure 2d shows a quantized ReLU (qReLU) withevenly spaced thresholds.
The simplest and most popular straight-through estimator (STE) for qReLU is to use the derivative of the saturated (or clipped) ReLU ∂ sat ReLU( x ) ∂x = 1 0 <x< 1 , where sat ReLU( x ) = min(1 , max( x, 0)) .
However, if we instead consider the qReLU activation from the viewpoint of FTPROP , then the qReLU becomes a (normalized) sum of step functions qReLU( z ) =
1 k (cid:80) k − 1 i =0 step( z − i k − 1 ) , where step( z )
=
if z > 0 and 0 otherwise, and is a linear transformation of sign( z ) .
The resulting derivative of the sum of saturated hinge losses (one for each step function) is shown in red in Figure 2d, and is clearly quite different than the STE described above.
In initial experiments, this performed as well as or better than the STE; however, we achieved additional performance improvements by using the softened approximation shown in yellow in Figure 2d, which is simply the derivative of a soft hinge that has been scaled and shifted to match the qReLU domain.
This is a natural choice because the derivative of a sum of a small number of soft hinge losses has a shape similar to that of the derivative of a single soft hinge loss.
E XPERIMENTS
We evaluated FTPROP - MB with soft hinge per-layer losses (FTP-SH) for training deep networks with sign and 2- and 3-bit qReLU activations by comparing models trained with FTP-SH to those trained with the saturated straight-through estimators (SSTEs) described earlier (although, as discussed, these
SSTEs can also be seen as instances of FTPROP - MB ).
We also trained each model with full-precision
ReLU and saturated ReLU activations as baselines.
We did not use weight quantization because our main interest is training with hard-threshold activations, and because recent work has shown that weights can be quantized with little effect on performance (Hubara et al., 2016; Rastegari et al., 2016; Zhou et al., 2016).
We tested these training methods on the CIFAR-10 (Krizhevsky, 2009) and ImageNet (ILSVRC 2012)
(Russakovsky et al., 2015) datasets.
On CIFAR-10, we trained a simple 4 -layer convolutional network and the 8 -layer convolutional network of Zhou et al.
On
ImageNet, we trained AlexNet (Krizhevsky et al., 2012), the most common model in the quantization literature, and ResNet-18 (He et al., 2015a).
Experiment details are provided in Appendix B, along with training curves for all experiments.
[Table] [Table]
layer convnet
[Table]
[Table]
[Table]
layer
[Table]
Test accuracies for the 4-layer and 8-layer convolutional network on CIFAR-10 are shown in Table 1. For the simpler 4-layer model, FTP-SH shows a consistent 0 . 5 - 1 % accuracy gain over SSTE for the entire training trajectory, resulting in the 0 .
7 % improvement shown in Table 1.
However, for the 2-bit qRELU activation, SSTE and FTP-SH perform nearly identically in the 4-layer model.
Conversely, for the more complex 8-layer model, the FTP-SH accuracy is only 0 .
3 % above SSTE, but for the qReLU activation FTP-SH achieves a consistent 1 .
4 % improvement over SSTE.
We posit that the decrease in performance gap for the sign activation when moving from the 4- to 8-layer model is because both methods are able to effectively train the higher-capacity model to achieve close to its best possible performance on this dataset, whereas the opposite is true for the qReLU activation; i.e., the restricted capacity of the 4-layer model limits the ability of both methods to train the more expressive qReLU effectively.
If this is true, then we expect that FTP-SH will outperform SSTE for both the sign and qReLU activations on a harder dataset.
Unsurprisingly, none of the low-precision methods perform as well as the high-precision methods; however, the narrowness of the performance gap between 2-bit qReLU with FTP-SH and full-precision ReLU is encouraging.
4. I MAGE N ET
The results from the ImageNet experiments are also shown in Table 1.
As predicted from the CIFAR10 experiments, we see that FTP-SH improves test accuracy on AlexNet for both sign and 2-bit qReLU activations on the more challenging ImageNet dataset.
This is also shown in Figure 3, which plots the top-1 train and test accuracy curves for the six different activation functions for AlexNet on
ImageNet.
The left-hand plot shows that training sign activations with FTP-SH provides consistently better test accuracy than SSTE throughout the training trajectory, despite the hyperparameters being optimized for SSTE.
This improvement is even larger for the 2-bit qReLU activation in the right- [Figure] hand plot, where the FTP-SH qReLU even outperforms the full-precision ReLU for part of its trajectory, and outperforms the SSTE-trained qReLU by almost 2 %.
Interestingly, we ﬁnd that the saturated ReLU outperforms the standard ReLU by almost a full point of accuracy.
We believe that this is due to the regularization effect caused by saturating the activation.
This may also account for the surprisingly good performance of the FTP-SH qReLU relative to full-precision ReLU, as hard-threshold activations also provide a strong regularization effect.
Finally, we ran a single experiment with ResNet-18 on ImageNet, using hyperparameters set from previous works that used SSTE, to check (i) whether the soft hinge loss exhibits vanishing gradient behavior due to its diminishing slope away from the origin, and (ii) to evaluate the performance of FTPSH for a less-quantized ReLU (we used k = 5 steps, which is less than the full range of a 3-bit ReLU).
While FTP-SH does slightly worse than SSTE for the sign function, we believe that this is because the hyperparameters were tuned for SSTE and not due to vanishing gradients, as we would expect much worse accuracy in that case.
Results from the qReLU activation provide further evidence against vanishing gradients as FTP-SH for qReLU outperforms SSTE by almost 4 % in top-1 accuracy (Table 1).
C ONCLUSION
In this work, we presented a novel mixed convex-combinatorial optimization framework for learning deep neural networks with hard-threshold units.
Combinatorial optimization is used to set discrete targets for the hard-threshold hidden units, such that each unit only has a linearly-separable problem to solve.
The network then decomposes into individual perceptrons, which can be learned with standard convex approaches, given these targets.
Based on this, we developed a recursive algorithm for learning deep hard-threshold networks, which we call feasible target propagation ( FTPROP ), and an efﬁcient mini-batch version ( FTPROP - MB ).
We showed that the commonly-used but poorly-justiﬁed straight-through estimator (STE) is the special case of FTPROP - MB that results from using a saturated hinge loss at each layer and our target heuristic.
Finally, we deﬁned the soft hinge loss and showed that FTPROP - MB with a soft hinge loss at each layer improves classiﬁcation accuracy for multiple models on CIFAR-10 and ImageNet when compared to the STE.
In future work we plan to develop novel target heuristics and layer loss functions by investigating connections between our framework, constraint satisfaction, and satisﬁability.
We also intend to further explore the beneﬁts of deep networks with hard-threshold units.
In particular, while recent research clearly shows their ability to reduce computation and energy requirements, they should also be less susceptible to vanishing and exploding gradients and may be less susceptible to covariate shift and adversarial examples
A M INI - BATCH FEASIBLE TARGET PROPAGATION
L d
(cid:96)d =1 .
1: initialize weights W =
{ W 1 , . . .
, W (cid:96) } randomly 2: for each minibatch ( X b , T b ) from D do 3: initialize targets T 1 , . . .
, T (cid:96) − 1 as the outputs of their hidden units in f ( X b ; W ) //
forward pass 4: set T 0 ← X b , set T (cid:96) ← T b , and set T ← { T 0 , . .
, T (cid:96) } 5:
FTPROP - MB ( W, T, L, (cid:96) ) 6: function FTPROP - MB (weights W , targets T , losses L , and layer index d ) 7:
ˆ T d − 1 ← set targets for upstream layer based on current weights W d and loss L d ( Z d , T d ) 8: update W d with respect to layer loss L d ( Z d , T d ) // where Z d = W d T d − 1 = W d H d −9: if d > 1 then FTPROP - MB ( W, { T 0 , . .
, ˆ T d − 1 , . . .
, T (cid:96) } , L, d − 1)
B E XPERIMENT DETAILS
All experiments were performed using PyTorch ( http://pytorch.org/ ).
CIFAR-
On CIFAR-10, which has 50 K training images and 10 K test images divided into 10 classes, we trained both a simple 4 -layer convolutional network and a deeper 8 -layer convolutional network used in (Zhou et al., 2016) with the above methods and then compared their top-1 accuracies on the test set.
We pre-processed the images with mean / std normalization, and augmented the dataset with random horizontal ﬂips and random crops from images padded with 4 pixels.
Hyperparameters were chosen based on a small amount of exploration on a validation set.
The ﬁrst network we tested on CIFAR-10 was a simple 4-layer convolutional network (convnet) structured as: conv (32) → conv (64) → fc (1024) → fc (10) , where conv ( c ) and fc ( c ) indicate a convolutional layer and fully-connected layer, respectively, with c channels.
Both convolutional layers used 5 × 5 kernels.
Max-pooling with stride 2 was used after each convolutional layer, and a non-linearity was placed before each of the above layers except the ﬁrst.
Adam (Kingma & Ba, 2015) with learning rate 2 . 5 e-
4 and weight decay
e- 4 was used to minimize the cross-entropy loss for 300 epochs.
The learning rate was decayed by a factor of 0 .
1 after 200 and 250 epochs.
In order to evaluate the performance of FTPROP - MB with the soft hinge loss on a deeper network, we adapted the 8-layer convnet from Zhou et al.
(2016) to CIFAR-10.
This network has 7 convolutional layers and one fullyconnected layer for the output and uses batch normalization (Ioffe & Szegedy, 2015) before each non-linearity.
We optimized the cross-entropy loss with Adam using a learning rate of 1 e- 3 and a weight decay of 1 e- 7 for the sign activation and 5 e- 4 for the qReLU and baseline activations.
We trained for 300 epochs, decaying the learning rate by 0 .
1 after 200 and 250 epochs.
I MAGE N ET (ILSVRC 2012)
On ImageNet, a much more challenging dataset with roughly 1 . 2 M training images and 50 K validation images divided into 1000 classes, we trained AlexNet, the most commonly used model in the quantization literature, with different activations and compared top-1 and top-5 accuracies of the trained models on the validation set.
As is standard practice, we treat the validation set as the test data.
Images were resized to 256 × 256 , mean / std normalized, and then randomly cropped to 224 × 224 and randomly horizontally ﬂipped.
Models are tested on centered 224 × 224 crops of the test images.
Hyperparameters were set based on Zhou et al.
(2016) and Zhu et al.
(2017), which both used SSTE to train AlexNet on ImageNet.
We trained the Zhou et al.
(2016) variant of AlexNet (Krizhevsky et al., 2012) on ImageNet with sign, 2-bit qReLU, ReLU, and saturated ReLU activations.
This version of AlexNet removes the dropout and replaces the local contrast normalization layers with batch normalization.
Our implementation does not split the convolutions into two separate blocks.
We used the Adam optimizer with learning rate 1 e- 4 on the cross-entropy loss forepochs, decaying the learning rate by 0 .
1 after 56 and 64 epochs.
For the sign activation, we used a weight decay of 5 e- 6 as in Zhou et al.
For the ReLU and saturated ReLU activations, which are much more likely to overﬁt, we used a weight decay of 5 e- 4 , as used in Krizhevsky et al.
For the qReLU activation, we used a weight decay of 5 e- 5 , since it is more expressive than sign but less so than ReLU.
As with AlexNet, we trained ResNet-18 (He et al., 2015b) on ImageNet with sign, qReLU, ReLU, and saturated
ReLU activations; however, for ResNet-18 we used a qReLU with k = 5 steps (i.e., 6 quantization levels, requiring 3 bits).
We used the ResNet code provided by PyTorch.
We optimized the cross-entropy loss with
SGD with learning rate 0 . 1 and momentum 0 .
9 for 90 epochs, decaying the learning rate by a factor of 0 .
1 after 30 and 60 epochs.
For the sign activation, we used a weight decay of 5 e − 7 .
For the ReLU and saturated ReLU activations, we used a weight decay of 1 e- 4 . For the qReLU activation, we used a weight decay of 1 e- 5 . T RAINING CURVES FOR CIFAR-  [Figure]
[Figure] T RAINING CURVES FOR I MAGE N ET [Figure] [Figure]