Paper:
ENERGY-BASED SPHERICAL SPARSE CODING
In this paper, we explore an efficient variant of convolutional sparse coding with unit norm code vectors where reconstruction quality is evaluated using an inner product (cosine distance). To use these codes for discriminative classification, we describe a model we term Energy-Based Spherical Sparse Coding (EB-SSC) in which the hypothesized class label introduces a learned linear bias into the coding step. We evaluate and visualize performance of stacking this encoder to make a deep layered model for image classification.
1 INTRODUCTION
Sparse coding has been widely studied as a representation for images, audio and other vectorial data. This has been a highly successful method that has found its way into many applications, from signal compression and denoising (Donoho, 2006; Elad & Aharon, 2006) to image classification (Wright et al., 2009), to modeling neuronal receptive fields in visual cortex (Olshausen & Field, 1997). Since its introduction, subsequent works have brought sparse coding into the supervised learning setting by introducing classification loss terms to the original formulation to encourage features that are not only able to reconstruct the original signal but are also discriminative (Jiang et al., 2011; Yang et al., 2010; Zeiler et al., 2010; Ji et al., 2011; Zhou et al., 2012; Zhang et al., 2013).
While supervised sparse coding methods have been shown to find more discriminative features leading to improved classification performance over their unsupervised counterparts, they have received much less attention in recent years and have been eclipsed by simpler feed-forward architectures.
This is in part because sparse coding is computationally expensive. Convex formulations of sparse coding typically consist of a minimization problem over an objective that includes a least-squares (LSQ) reconstruction error term plus a sparsity inducing regularizer.
Because there is no closed-form solution to this formulation, various iterative optimization techniques are generally used to find a solution (Zeiler et al., 2010; Bristow et al., 2013; Yang et al., 2013; Heide et al., 2015). In applications where an approximate solution suffices, there is work that learns non-linear predictors to estimate sparse codes rather than solve the objective more directly (Gregor & LeCun, 2010). The computational overhead for iterative schemes becomes quite significant when training discriminative models due to the demand of processing many training examples necessary for good performance, and so sparse coding has fallen out of favor by not being able to keep up with simpler non-iterative coding methods.
In this paper we introduce an alternate formulation of sparse coding using unit length codes and a reconstruction loss based on the cosine similarity. Optimal sparse codes in this model can be computed in a non-iterative fashion and the coding objective lends itself naturally to embedding in a discriminative, energy-based classifier which we term energy-based spherical sparse coding (EBSSC). This bi-directional coding method incorporates both top-down and bottom-up information where the features representation depends on both a hypothesized class label and the input signal. Like Cao et al. (2015), our motivation for bi-directional coding comes from the “Biased Competition Theory”, which suggests that visual processing can be biased by other mental processes (e.g., topdown influence) to prioritize certain features that are most relevant to current task. Fig. 1 illustrates the flow of computation used by our SSC and EB-SSC building blocks compared to a standard feed-forward layer.
Our energy based approach for combining top-down and bottom-up information is closely tied to the ideas of Larochelle & Bengio (2008); Ji et al. (2011); Zhang et al. (2013); Li & Guo (2014)— although the model details are substantially different (e.g., Ji et al. (2011) and Zhang et al. (2013) use sigmoid non-linearities while Li & Guo (2014) use separate representations for top-down and bottom-up information). The energy function of Larochelle & Bengio (2008) is also similar but includes an extra classification term and is trained as a restricted Boltzmann machine.
1.1 NOTATION
Matrices are denoted as uppercase bold (e.g., A), vectors are lowercase bold (e.g., a), and scalars are lowercase (e.g., a). We denote the transpose operator with ᵀ, the element-wise multiplication operator with , the convolution operator with ∗, and the cross-correlation operator with ?. For vectors where we dropped the subscript k (e.g., d and z), we refer to a super vector with K components stacked together (e.g., z = [zᵀ1 , . . . , z ᵀ K ] ᵀ).
2 ENERGY-BASED SPHERICAL SPARSE CODING
Energy-based models capture dependencies between variables using an energy function that measure the compatibility of the configuration of variables (LeCun et al., 2006). To measure the compatibility between the top-down and bottom-up information, we define the energy function of EB-SSC to be the sum of bottom-up coding term and a top-down classification term:
E(x, y, z) = Ecode(x, z) + Eclass(y, z). (1) The bottom-up information (input signal x) and the top-down information (class label y) are tied together by a latent feature map z.
2.1 BOTTOM-UP RECONSTRUCTION
To measure the compatibility between the input signal x and the latent feature maps z, we introduce a novel variant of sparse coding that is amenable to efficient feed-forward optimization. While the idea behind this variant can be applied to either patch-based or convolutional sparse coding, we specifically use the convolutional variant that shares the burden of coding an image among nearby overlapping dictionary elements. Using such a shift-invariant approach avoids the need to learn dictionary elements which are simply translated copies of each other, freeing up resources to discover more diverse and specific filters (see Kavukcuoglu et al. (2010)).
Convolutional sparse coding (CSC) attempts to find a set of dictionary elements {d1, . . . ,dK} and corresponding sparse codes {z1, . . . , zK} so that the resulting reconstruction, r = ∑K k=1 dk ∗ zk accurately represents the input signal x. This is traditionally framed as a least-squares minimization with a sparsity inducing prior on z:
arg min z ‖x− K∑ k=1 dk ∗ zk‖22 + β‖z‖1. (2)
Unlike standard feed-forward CNN models that convolve the input signal x with the filters, this energy function corresponds to a generative model where the latent feature maps {z1, . . . , zK} are convolved with the filters and compared to the input signal (Bristow et al., 2013; Heide et al., 2015; Zeiler et al., 2010).
To motivate our novel variant of CSC, consider expanding the squared reconstruction error ‖x − r‖22 = ‖x‖22 − 2xᵀr + ‖r‖22. If we constrain the reconstruction r to have unit norm, the reconstruction error depends entirely on the inner product between x and r and is equivalent to the cosine similarity (up to additive and multiplicative constants). This suggests the closely related unit-length reconstruction problem:
arg max z xᵀ ( K∑ k=1 dk ∗ zk ) − β‖z‖1 (3)
s.t. ∥∥ K∑ k=1 dk ∗ zk ∥∥ 2 ≤ 1
In Appendix A we show that, given an optimal unit length reconstruction r̄∗ with corresponding codes z̄∗, the solution to the least squares reconstruction problem (Eq. 2) can be computed by a simple scaling r∗ = (xᵀr̄∗ − β2 ‖z̄
∗‖1)r̄∗. The unit-length reconstruction problem is no easier than the original least-squares optimization due to the constraint on the reconstruction which couples the codes for different filters. Instead consider a simplified constraint on z which we refer to as spherical sparse coding (SSC):
arg max ‖z‖2≤1 Ecode(x, z) = arg max ‖z‖2≤1 xᵀ ( K∑ k=1 dk ∗ zk ) − β‖z‖1. (4)
In 2.3 below, we show that the solution to this problem can be found very efficiently without requiring iterative optimization.
This problem is a relaxation of convolutional sparse coding since it ignores non-orthogonal interactions between the dictionary elements1. Alternately, assuming unit norm dictionary elements, the code norm constraint can be used to upper-bound the reconstruction length. We have by the triangle and Young’s inequality that:∥∥∑
k
dk ∗ zk ∥∥ 2 ≤ ∑ k ‖dk ∗ zk‖2 ≤ ∑ k ‖dk‖1‖zk‖1 ≤ D ∑ k ‖zk‖2 (5)
where the factor D is the dimension of zk and arises from switching from the 1-norm to the 2-norm. Since D ∑ k ‖zk‖2 ≤ 1 is a tighter constraint we have
max ‖ ∑ k dk∗zk‖2≤1 Ecode(x, z) ≥ max∑ k ‖zk‖2≤ 1 D Ecode(x, z) (6)
However, this relaxation is very loose, primarily due to the triangle inequality. Except in special cases (e.g., if the dictionary elements have disjoint spectra) the SSC codes will be quite different from the standard least-squares reconstruction.
1We note that our formulation is also closely related to the dynamical model suggested by Rozell et al. (2008), but without the dictionary-dependent lateral inhibition between feature maps. Lateral inhibition can solve the unit-length reconstruction formulation of standard sparse coding but requires iterative optimization.
2.2 TOP-DOWN CLASSIFICATION
To measure the compatibility between the class label y and the latent feature maps z, we use a set of one-vs-all linear classifiers. To provide more flexibility, we generalize this by splitting the code vector into positive and negative components:
zk = z + k + z − k z + k ≥ 0 z − k ≤ 0
and allow the linear classifier to operate on each component separately. We express the classifier score for a hypothesized class label y by:
Eclass(y, z) = K∑ k=1 w+ᵀy z + k + K∑ k=1 w−ᵀy z − k . (7)
The classifier thus is parameterized by a pair of weight vectors (w+yk and w − yk) for each class label y and k-th channel of the latent feature map.
This splitting, sometimes referred to as full-wave rectification, is useful since a dictionary element and its negative do not necessarily have opposite visual semantics. This splitting also allows the classifier the flexibility to assign distinct meanings or alternately be completely invariant to contrast reversal depending on the problem domain. For example, Shang et al. (2016) found CNN models with ReLU non-linearities which discard the negative activations tend to learn pairs of filters which are related by negation. Keeping both positive and negative responses allowed them to halve the number of dictionary elements.
We note that it is also straightforward to introduce spatial average pooling prior to classification by introducing a fixed linear operator P used to pool the codes (e.g., w+ᵀy Pz + k ). This is motivated by a variety of hand-engineered feature extractors and sparse coding models, such as Ren & Ramanan (2013), which use spatially pooled histograms of sparse codes for classification. This fixed pooling can be viewed as a form of regularization on the linear classifier which enforces shared weights over spatial blocks of the latent feature map. Splitting is also quite important to prevent information loss when performing additive pooling since positive and negative components of zk can cancel each other out.
2.3 CODING
Bottom-up reconstruction and top-down classification each provide half of the story, coupled by the latent feature maps. For a given input x and hypothesized class y, we would like to find the optimal activations z that maximize the joint energy function E(x, y, z). This requires solving the following optimization:
arg max ‖z‖2≤1 xᵀ ( K∑ k=1 dk ∗ zk ) − β‖z‖1 + K∑ k=1 w+ᵀyk z + k + K∑ k=1 w−ᵀyk z − k , (8)
where x ∈ RD is an image and y ∈ Y is a class hypothesis. zk ∈ RF is the k-th component latent variable being inferred; z+k and z − k are the positive and negative coefficients of zk, such that zk = z + k + z − k . The parameters dk ∈ RM , w + yk ∈ RF , and w − yk ∈ RF are the dictionary filter, positive coefficient classifier, and negative coefficient classifier for the k-th component respectively. A key aspect of our formulation is that the optimal codes can be found very efficiently in closedform—in a feed-forward manner (see Appendix B for a detailed argument).
2.3.1 ASYMMETRIC SHRINKAGE
To describe the coding processes, let us first define a generalized version of the shrinkage function commonly used in sparse coding. Our asymmetric shrinkage is parameterized by upper and lower thresholds −β− ≤ β+
shrink(β+,β−)(v) =  v − β + if v − β+ > 0
0 otherwise v + β− if v + β− < 0
(9)
Fig. 2 shows a visualization of this function which generalizes the standard shrinkage proximal operator by allowing for the positive and negative thresholds. In particular, it corresponds to the proximal operator for a version of the `1-norm that penalizes the positive and negative components with different weights |v|asym = β+‖v+‖1 +β−‖v−‖1. The standard shrink operator corresponds to shrink(β,−β)(v) while the rectified linear unit common in CNNs is given by a limiting case shrink(0,−∞)(v). We note that −β− ≤ β+ is required for shrink(β+,β−) to be a proper function (see Fig. 2).
2.3.2 FEED-FORWARD CODING
We now describe how codes can be computed in a simple feed-forward pass. Let
β+yk = β −w + yk, β − yk = β −w − yk (10)
be vectors of positive and negative biases whose entries are associated with a spatial location in the feature map k for class y. The optimal code z can be computed in three sequential steps:
1. Cross-correlate the data with the filterbank dk ? x
2. Apply an asymmetric version of the standard shrinkage operator
z̃k = shrink(β+yk,β − yk) (dk ? x) (11)
where, with abuse of notation, we allow the shrinkage function (Eq. 9) to apply entries in the vectors of threshold parameter pairs β+yk,β − yk to the corresponding elements of the argument.
3. Project onto the feasible set of unit length codes
z∗ = z̃
‖z̃‖2 . (12)
2.3.3 RELATIONSHIP TO CNNS:
We note that this formulation of coding has a close connection to single layer convolutional neural network (CNN). A typical CNN layer consists of convolution with a filterbank followed by a nonlinear activation such as a rectified linear unit (ReLU). ReLUs can be viewed as another way of inducing sparsity, but rather than coring the values around zero like the shrink function, ReLU truncates negative values. On the other hand, the asymmetric shrink function can be viewed as the sum of two ReLUs applied to appropriately biased inputs:
shrink(β+,β−)(x) = ReLU(x− β+)− ReLU(−(x+ β−)),
SSC coding can thus be seen as a CNN in which the ReLU activation has been replaced with shrinkage followed by a global normalization.
3 LEARNING
We formulate supervised learning using the softmax log-loss that maximizes the energy for the true class label yi while minimizing energy of incorrect labels ȳ.
arg min d,w+,w−,β≥0
α 2 (‖w+‖22 + ‖w−‖22 + ‖d‖22)
+ 1
N N∑ i=1 [− max ‖z‖2≤1 E(xi, yi, z) + log ∑ ȳ∈Y max ‖z̄‖2≤1 eE(xi,ȳ,z̄)]
s.t. − (β −w−yk) ≤ (β −w + yk) ∀y, k
, (13)
where α is the hyperparameter regularizing w+y , w − y , and d. We constrain the relationship between β and the entries of w+y and w − y in order for the asymmetric shrinkage to be a proper function (see Sec. 2.3.1 and Appendix B for details).
In classical sparse coding, it is typical to constrain the `2-norm of each dictionary filter to unit length. Our spherical coding objective behaves similarly. For any optimal code z∗, there is a 1-dimensional subspace of parameters for which z∗ is optimal given by scaling d inversely to w, β. For simplicity of the implementation, we opt to regularize d to assure a unique solution. However, as Tygert et al. (2015) point out, it may be advantageous from the perspective of optimization to explicitly constrain the norm of the filter bank.
Note that unlike classical sparse coding, where β is a hyperparameter that is usually set using crossvalidation, we treat it as a parameter of the model that is learned to maximize performance.
3.1 OPTIMIZATION
In order to solve Eq. 13, we explicitly formulate our model as a directed-acyclic-graph (DAG) neural network with shared weights, where the forward-pass computes the sparse code vectors and the backward-pass updates the parameter weights. We optimize the objective using stochastic gradient descent (SGD).
As mentioned in Sec. 2.3 shrinkage function is assymetric with parameters β+yk or β − yk as defined in Eq. 10. However, the inequality constraint on their relationship to keep the shrinkage function a proper function is difficult to enforce when optimizing with SGD. Instead, we introduce a central offset parameter and reduce the ordering constraint to pair of positivity constraints. Let
ŵ+yk = β + yk − bk ŵ − yk = β − yk + bk (14)
be the modified linear “classifiers” relative to the central offset bk. It is straightforward to see that if β+yk and β − yk that satisfy the constrain in Eq. 13, then adding the same value to both sides of the inequality will not change that. However, taking bk to be a midpoint between them, then both β+yk − bk and β − yk + bk will be strictly non-negative.
Using this variable substitution, we rewrite the energy function (Eq. 1) as
E′(x, y, z) = xᵀ ( K∑ k=1 dk ∗ zk ) + K∑ k=1 bk1 ᵀzk − K∑ k=1 ŵ+ᵀyk z + k + K∑ k=1 ŵ−ᵀyk z − k . (15)
where b is constant offset for each code channel. The modified linear “classification” terms now take on a dual role of inducing sparsity and measuring the compatibility between z and y.
This yields a modified learning objective that can easily be solved with existing implementations for learning convolutional neural nets:
arg min d,ŵ+,ŵ−,b
α 2 (‖ŵ+‖22 + ‖ŵ−‖22 + ‖d‖22)
+ 1
N N∑ i=1 [− max ‖z‖2≤1 E′(xi, yi, z) + log ∑ ȳ∈Y max ‖z̄‖2≤1 eE ′(xi,ȳ,z̄)]
s.t. ŵ+yk, ŵ − yk 0 ∀y, k
, (16)
where ŵ+ and ŵ− are the new sparsity inducing classifiers, and b are the arbitrary origin points. In particular, adding the K origin points allows us to enforce the constraint by simply projecting ŵ+ and ŵ− onto the positive orthant during SGD.
3.1.1 STACKING BLOCKS
We also examine stacking multiple blocks of our energy function in order to build a hierarchical representation. As mentioned in Sec. 3.1.1, the optimal codes can be computed in a simple feedforward pass—this applies to shallow versions of our model. When stacking multiple blocks of our energy-based model, solving for the optimal codes cannot be done in a feed-forward pass since the codes for different blocks are coupled (bilinearly) in the joint objective. Instead, we can proceed in an iterative manner, performing block-coordinate descent by repeatedly passing up and down the hierarchy updating the codes. In this section we investigate the trade-off between the number of passes used to find the optimal codes for the stacked model and classification performance.
For this purpose, we train multiple instances of a 2-block version of our energy-based model that differ in the number of iterations used when solving for the codes. For recurrent networks such as this, inference is commonly implemented by “unrolling” the network, where the parts of the network structure are repeated with parameters shared across these repeated parts to mimic an iterative algorithm that stops at a fixed number of iterations rather than at some convergence criteria.
In Fig. 3, we compare the performance between models that were unrolled zero to four times. We see that there is a difference in performance based on how many sweeps of the variables are made. In terms of the training objective, more unrolling produces models that have lower objective values with convergence after only a few passes. In terms of testing error, however, we see that full code inference is not necessarily better, as unrolling once or twice has lower errors than unrolling three or four times. The biggest difference was between not unrolling and unrolling once, where both the training objective and testing error goes down. The testing error decreases from 0.0131 to 0.0074. While there is a clear benefit in terms of performance for unrolling at least once, there is also a trade-off between performance and computational resource, especially for deeper models.
4 EXPERIMENTS
We evaluate the benefits of combining top-down and bottom-up information to produce classspecific features on the CIFAR-10 (Krizhevsky & Hinton, 2009) dataset using a deep version of our EB-SSC. All experiments were performed using MatConvNet (Vedaldi & Lenc, 2015) framework with the ADAM optimizer (Kingma & Ba, 2014). The data was preprocessed and augmented following the procedure in Goodfellow et al. (2013). Specifically, the data was made zero mean and whitened, augmented with horizontal flips (with a 0.5 probability) and random cropping. No weight decay was used, but we used a dropout rate of 0.3 before every convolution layer except for the first. For these experiments we consider a single forward pass (no unrolling).
4.1 CLASSIFICATION
We compare our proposed EB-SSC model to that of Springenberg et al. (2015), which uses rectified linear units (ReLU) as its non-linearity. This model can be viewed as a basic feed-forward version of our proposed model which we take as a baseline. We also consider variants of the baseline model that utilize a subset of architectural features of our proposed model (e.g., concatenated rectified linear units (CReLU) and spherical normalization (SN)) to understand how subtle design changes of the network architecture affects performance.
We describe the model architecture in terms of the feature extractor and classifier. Table 1 shows the overall network architecture of feature extractors, which consist of seven convolution blocks and two pooling layers. We test two possible classifiers: a simple linear classifier (LC) and our energy-based classifier (EBC), and use softmax-loss for all models. For linear classifiers, a numerical subscript indicates which of the seven conv blocks of the feature extractor is used for classification (e.g., LC7 indicates the activations out of the last conv block is fed into the linear classifier). For energy-based classifiers, a numerical subscript indicates which conv blocks of the feature extractor are replace with a energy-based classifier (e.g., EBC6−7 indicates the activations out of conv5 is fed into the energy-based classifier and the energy-based classifier has a similar architecture to the conv blocks it replaces). The notation differ because for energy-based classifiers, the optimal activations are a function of the hypothesized class label, whereas for linear classifiers, they are not.
The results shown in Table 2 compare our proposed model to the baselines ReLU+LC7 (Springenberg et al., 2015) and CReLU+LC7 (Shang et al., 2016), and to intermediate variants. The baseline models all perform very similarly with some small reductions in error rates over the baseline CReLU+LC7. However, CReLU+LC7 reduces the error rate over ReLU+LC7 by more than one percent (from 11.40% to 10.17%), which confirms the claims by Shang et al. (2016) and demonstrates the benefits of splitting positive and negative activations. Likewise, we see further decrease in the error rate (to 9.74%) from using spherical normalization. Though normalizing the activations doesn’t add any capacity to the model, this improved performance is likely because scale-invariant activations makes training easier. On the other hand, further sparsifying the activations yielded no
benefit. We tested values β = {0.001, 0.01} and found 0.001 to perform better. Replacing the linear classifier with our energy-based classifier further decreases the error rate by another half percent (to 9.23%).
4.2 DECODING CLASS-SPECIFIC CODES
A unique aspect of our model is that it is generative in the sense that each layer is explicitly trying to encode the activation pattern in the prior layer. Similar to the work on deconvolutional networks built on least-squares sparse coding (Zeiler et al., 2010), we can synthesize input images from activations in our spherical coding network by performing repeated deconvolutions (transposed convolutions) back through the network. Since our model is energy based, we can further examine how the topdown information of a hypothesized class effects the intermediate activations.
The first column in Fig. 4 visualizes reconstructions of a given input image based on activations from different layers of the model by convolution transpose. In this case we put in zeros for class biases (i.e., no top-down) and are able to recover high fidelity reconstructions of the input. In the remaining columns, we use the same deconvolution pass to construct input space representations of the learned classifier biases. At low levels of the feature hierarchy, these biases are spatially smooth since the receptive fields are small and there is little spatial invariance capture in the activations. At higher levels these class-conditional bias fields become more tightly localized.
Finally, in Fig. 5 we shows decodings from the conv2 and conv5 layer of the EB-SSC model for a given input under different class hypotheses. Here we subtract out the contribution of the top-down bias term in order to isolate the effect of the class conditioning on the encoding of input features. As visible in the figure, the modulation of the activations focused around particular regions of the image and the differences across class hypotheses becomes more pronounced at higher layers of the network.
5 CONCLUSION
We presented an energy-based sparse coding method that efficiently combines cosine similarity, convolutional sparse coding, and linear classification. Our model shows a clear mathematical connection between the activation functions used in CNNs to introduce sparsity and our cosine similarity convolutional sparse coding formulation. Our proposed model outperforms the baseline model and we show which attributes of our model contributes most to the increase in performance. We also demonstrate that our proposed model provides an interesting framework to probe the effects of class-specific coding.
APPENDIX A
Here we show that spherical sparse coding (SSC) with a norm constraint on the reconstruction is equivalent to standard convolutional sparse coding (CSC). Expanding the least squares reconstruction error and dropping the constant term ‖x‖2 gives the CSC problem:
max z 2xᵀ ( K∑ k=1 dk ∗ zk ) − ‖ K∑ k=1 dk ∗ zk‖22 − β K∑ k=1 ‖zk‖1.
Let = ‖ ∑K k=1 dk ∗ zk‖2 be the norm of the reconstruction for some code z and let u be the reconstruction scaled to have unit norm so that:
u =
∑K k=1 dk ∗ zk
‖ ∑K k=1 dk ∗ zk‖2 = K∑ k=1 dk ∗ z̄k with z̄ = 1 z
We rewrite the least-squares objective in terms of these new variables:
max z̄, >0 g(z̄, ) = max z̄, >0
2xᵀ ( u ) − ‖ u‖22 − β‖ z̄‖1
= max z̄, >0
2 ( xᵀu− β
2 ‖z̄‖1
) − 2
Taking the derivative of g w.r.t. yields the optimal scaling ∗ as a function of z̄:
(z̄)∗ = xᵀu− β 2 ‖z̄‖1.
Plugging (z̄)∗ back into g yields:
max z̄, >0 g(z̄, ) = max z̄,‖u‖2=1
( xᵀu− β
2 ‖z̄‖1
)2 .
Discarding solutions with < 0 can be achieved by simply dropping the square which results in the final constrained problem:
arg max z̄ xᵀ ( K∑ k=1 dk ∗ z̄k ) − β 2 K∑ k=1 ‖z̄k‖1
s.t. ‖ K∑ k=1 dk ∗ z̄k‖2 ≤ 1.
APPENDIX B
We show in this section that coding in the EB-SSC model can be solved efficiently by a combination of convolution, shrinkage and projection, steps which can be implemented with standard libraries on a GPU. For convenience, we first rewrite the objective in terms of cross-correlation rather than convolution (i.e., , xᵀ(dk ∗ zk) = (dk ? x)ᵀzk). For ease of understanding, we first consider the coding problem when there is no classification term.
z∗ = arg max ‖z‖22≤1 vᵀz− β‖z‖1,
where v = [(d1 ? x)ᵀ, . . . , (dK ? x)ᵀ]ᵀ. Pulling the constraint into the objective, we get its Lagrangian function: L(z, λ) = vᵀz− β‖z‖1 + λ ( 1− ‖z‖22 ) .
From the partial subderivative of the Lagrangian w.r.t. zi we derive the optimal solution as a function of λ; and from that find the conditions in which the solutions hold, giving us:
zi(λ) ∗ =
1
2λ ·
{ vi − β vi > β
0 otherwise vi + β vi < β . (17)
This can also be compactly written as:
z(λ)∗ = 1
2λ z̃, (18)
z̃ = s2 v − βs
where s = sign(z∗) ∈ {−1, 0, 1}|z| and s2 = s s ∈ {0, 1}|z|. The sign vector of z∗ can be determined without knowing λ, as λ is a Lagrangian multiplier for an inequality it must be nonnegative and therefore does not change the sign of the optimal solution. Lastly, we define the squared `2-norm of z̃, a result that will be used later:
‖z̃‖22 = z̃ᵀ(s2 v)− βz̃ᵀs = z̃ᵀv − β‖z̃‖1. (19)
Substituting z(λ)∗ back into the Lagrangian we get:
L(z(λ)∗, λ) = 1 2λ vᵀz̃− β 2λ ‖z̃‖1 + λ
( 1− 1
4λ2 ‖z̃‖22
) ,
and the derivative w.r.t. λ is:
∂L(z(λ)∗
∂λ = − 1 2λ2 vᵀz̃ + β 2λ2 ‖z̃‖1 + 1 + 1 4λ2 ‖z̃‖22.
Setting the derivative equal to zero and using the result from Eq. 19, we can find the optimal solution to λ:
λ2 = 1 2 z̃ᵀv − β 2 ‖z̃‖1 − 1 4 ‖z̃‖22 = 1 2 ‖z̃‖22 − 1 4 ‖z̃‖22
=⇒ λ∗ = 1 2 ‖z̃‖2.
Finally, plugging λ∗ into Eq. 18 we find the optimal solution
z∗ = z̃
‖z̃‖2 . (20)

Review:
First, I'd like to thank the authors for their answers and clarifications.
I find, the presentation of the multi-stage version of the model much clearer now.
Pros:
+ The paper states a sparse coding problem using cosine loss, which allows to solve the problem in a single pass.
+ The energy-based formulation allows bi-directional coding that incorporates top-down and bottom-up information in the feature extraction process. 
Cons:
+ The cost of running the evaluation could be large in the  multi-class setting, rendering the approach less attractive and the computational cost comparable to recurrent architectures.
+ While the model is competitive and improves over the baseline, the paper would be more convincing with other comparisons (see text). The experimental evaluation is limited (a single database and a single baseline)
------
The motivation of the sparse coding scheme is to perform inference in a feed forward manner. This property does not hold in the multi stage setting, thus optimization would be required (as clarified by the authors).
Having an efficient way of performing a bi-directional coding scheme is very interesting. As the authors clarified, this could not necessarily be the case, as the model needs to be evaluated many times for performing a classification.
Maybe an interesting combination would be to run the model without any class-specific bias, and evaluation only the top K predictions with the energy-based setting.
Having said this, it would be good to include a discussion (if not direct comparisons) of the trade-offs of using a model as the one proposed by Cao et al. Eg. computational costs, performance.
Using the bidirectional coding only on the top layers seems reasonable: one can get a good low level representation in a class agnostic way. This, however could be studied in more detail, for instance showing empirically the trade offs. If I understand correctly, now only one setting is being reported.
Finally, the authors mention that one benefit of using the architecture derived from the proposed coding method is the spherical normalization scheme, which can lead to smoother optimization dynamics. Does the baseline (or model) use batch-normalization? If not, seems relevant to test.
Minor comments:
I find figure 2 (d) confusing. I would not plot this setting as it does not lead to a function (as the authors state in the text).