E FFECT OF TOP - DOWN CONNECTIONS IN H IERARCHICAL S PARSE C ODING
Anonymous authors
Paper under double-blind review
A BSTRACT
Hierarchical Sparse Coding (HSC) is a powerful model to efﬁciently represent multi-dimensional, structured data such as images.
The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems.
However, neuroscientiﬁc evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers.
In this study, a new model called 2Layers Sparse Predictive Coding (2L-SPC) is introduced to assess the impact of this inter-layer feedback connection.
In particular, the 2L-SPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers.
2L-SPC and a 2-layers Hi-La networks are trained on 4 different databases and with different sparsity parameters on each layer.
First, we show that the overall prediction error generated by 2L-SPC is lower thanks to the feedback mechanism as it transfers prediction error between layers.
Second, we demonstrate that the inference stage of the 2L-SPC is faster to converge than for the Hi-La model.
Third, we show that the 2L-SPC also accelerates the learning process.
Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the 2L-SPC features are more generic and informative.
I NTRODUCTION
Finding a “efﬁcient” representation to model a given signal in a concise and efﬁcient manner is an inverse problem that has always been central to the machine learning community.
Sparse Coding (SC) has proven to be one of the most successful methods to achieve this goal.
SC holds the idea that signals (e.g. images) can be encoded as a linear combination of few features (called atoms) drawn from a bigger set called the dictionary (Elad, 2010).
The pursuit of optimal coding is usually decomposed into two complementary subproblems: inference (coding) and dictionary learning.
Inference consists in ﬁnding an accurate sparse representation of the input data considering the dictionaries are ﬁxed, it could be performed using algorithms like ISTA & FISTA (Beck & Teboulle, 2009), Matching Pursuit (Mallat & Zhang, 1993), Coordinate Descent (Li & Osher, 2009), or ADMM (Heide et al., 2015).
Once the representation is inferred, one can learn the atoms from the data using methods like gradient descent (Rubinstein et al., 2010; Kreutz-Delgado et al., 2003; Sulam et al., 2018), or online dictionary learning (Mairal et al., 2009a).
Consequently, SC offers an unsupervised framework to learn simultaneously basis vectors (e.g. atoms) and the corresponding input representation.
SC has been applied with success to image restoration (Mairal et al., 2009b), feature extraction (Szlam et al., 2010) and classiﬁcation (Yang et al., 2011; Perrinet & Bednar, 2015).
Interestingly, SC is also a ﬁeld of interest for computational neuroscientists.
Olshausen & Field (1997) ﬁrst demonstrated that adding a sparse prior to a shallow neural network was sufﬁcient to account for the emergence of neurons whose Receptive Fields (RFs) are spatially localized, band-pass and oriented ﬁlters, analogous to those found in the primary visual cortex (V1) of mammals (Hubel &
Wiesel, 1962).
Because most of the SC algorithms are limited to single-layer network, they cannot model the hierarchical structure of the visual cortex.
However, few solutions have been proposed to tackle Hierarchical Sparse Coding (HSC) as a global optimization problem (Sulam et al., 2018;
Makhzani & Frey, 2013; 2015; Aberdam et al., 2019; Sulam et al., 2019).
These methods are looking for an optimal solution of HSC without considering their plausibility in term of neuronal implementation.
Consequently, the quest for reliable HSC formulation that is compatible with a neural implementation remains open.
Rao & Ballard (1999) introduce the Predictive Coding (PC) to model the effect of the interaction of cortical areas in the visual cortex.
PC intends to solve the inverse problem of vision by combining feedforward and feedback connections.
In PC, feedback connection carries prediction of the neural activity of the lower cortical area while feedforward pass prediction error to the higher cortical area.
In such a framework, neural population are updated to minimize the unexpected component of the neural signal (Friston, 2010).
PC has been applied for supervised object recognition (Wen et al., 2018; Han et al., 2018; Spratling, 2017) or unsupervised prediction of future video frames (Lotter et al., 2016).
Interestingly, PC is ﬂexible enough to introduce a sparse prior to each layer.
Therefore, one can consider PC as a bio-plausible formulation of the HSC problem.
This formulation is to confront with the other bio-plausible HSC formulation that consists of a stack of independent Lasso problems (Sun et al., 2017).
To the best of our knowledge, no study has compared these two mathematically different formulations of the same problem of optimizing the Hierarchical Sparse Coding of images.
What is the effect of top-down connection of PC?
What are the consequences in term of computations and convergence?
What are the qualitative differences concerning the learned atoms?
The objective of this study is to experimentally answer these questions and to show that the PC framework could be successfully used for improving solutions to HSC problems.
We start our study by deﬁning the two different mathematical formulations to solve the HSC problem: the Hierarchical
Lasso (Hi-La) that consists in stacking Lasso sub-problems, and the 2-Layers Sparse Predictive Coding (2L-SPC) that leverages PC into a deep and sparse network of bi-directionally connected layers.
To experimentally compare both models, we train the 2L-SPC and Hi-La networks ondifferent databases and we vary the sparsity of each layer.
First, we compare the overall prediction error of the two models and we break it down to understand its distribution among layers.
Second, we analyze the number of iterations needed for the state variables of each network to reach their stability.
Third, we compare the convergence of both models during the dictionary learning stage.
Finally, we discuss the qualitative differences between the features learned by both networks in light of their activation probability.
M ETHODS 2. B ACKGROUND
In our mathematical description, italic letters are used as symbols for scalars , bold lower case letters for column vectors , bold uppercase letters for MATRICES and ∇ x L denotes the gradient of
L w.r.t.
to x .
The core objective of Hierarchical Sparse Coding (HSC) is to infer the internal state variables { γ ( k ) i } Li =1 (also called sparse map) for each input image x ( k ) and to learn the parameters { D i } Li =1 that solved the inverse problem formulated in Eq. 1.
[Equation]
L is the number of layers and (cid:15) ( k ) i is the layer i prediction-error corresponding to the input x ( k ) .
The map γ ( k ) i could be viewed as the projection of γ ( k ) i − 1 in the basis described by the atoms (denoted also basis vector) composing D i .
(cid:15) ( k ) i is historically called ’prediction error’
but it is actually quantifying the local reconstruction error between D Ti γ ( k ) i and γ ( k ) i − 1 .
The sparsity of the internal state variables, speciﬁed by the (cid:96) 0 pseudo-norm, is constrained by the scalar α i .
In practice we use 4-dimensional tensors to represent both vectors and matrices.
x ( k ) is a tensor of size [1 , c x , w x , h x ] with c x being the number of channels of the image, w x and h x the width and height of the image respectively.
In our mathematical description we raveled x ( k ) as a vector of size [ c x × w x × h x ] .
Furthermore, we impose a 2-dimensional convolutional structure to the parameters { D i } L i =1 .
If one describes the dictionary as a 4-dimensional tensor of size [ n d , c d , w d , h d ] , one can derive D i as a matrix of n d local features (size: c d × w d × h d ) that cover every possible location of the input x ( k ) (Sulam et al., 2018).
In other words, D i is a Toeplitz matrix.
For the sake of concision in our mathematical descriptions, we use matrix/vector multiplication in place of convolution as it is mathematically strictly equivalent.
Replaced in a biological context D i could be interpreted as the synaptic weights between two neural populations whose activity is represented by γ i − 1 and γ i respectively.
2. F ROM H IERARCHICAL L ASSO ...
One possibility to solve Eq.
1 while keeping the locality of the processing required by neural implementation, is to minimize a loss for each layer corresponding to the addition of the squared (cid:96) 2 -norm of the prediction error with a sparsity penalty.
To obtain a convex cost, we relax the (cid:96) 0 constraint into a (cid:96) 1 -penalty.
It deﬁnes, therefore, a loss function for each layer in the form of a standard Lasso problem (Eq. 2), that could be minimized using gradient-based methods:
[Equation]
In particular, we use the Iterative Shrinkage Thresholding Algorithm (ISTA) to minimize F w.r.t.
γ ( k ) i (Eq. 3) as it is proven to be computationally cheap (Beck & Teboulle, 2009).
In practice, we use an accelerated version of the ISTA algorithm called FISTA. In a convolutional case in which the proximal operator has a closed-form, FISTA has the advantage to converge faster than other sparse coding algorithms (e.g. Coordinate Descent) (Chalasani et al., 2013). Note that in Eq. 3 we have removed image indexation to keep a concise notation.
[Equation]
In Eq.
3, T α ( · ) denotes the non-negative soft-thresholding operator,
η c i is the learning rate of the inference process and γ ti is the state variable γ
i at time t .
Interestingly, one can interpret Eq.
3 as one loop of a recurrent layer that we will call the Lasso layer (Gregor & LeCun, 2010).
Following
Eq. 3, D Ti is a decoding dictionary that back-projects γ i into the space of the ( i − 1) -th layer.
This back-projection is used to elicit an error with γ i − 1 that will be encoded by D i to update the state variables γ i .
Finally, Lasso layers can be stacked together to form a Hierarchical Lasso (Hi-La) network (see Fig. 1 without blue arrow).
The inference of the overall Hi-La network consists in updating recursively all the sparse maps until they have reached a stable point.
2. ... TO H IERARCHICAL P REDICTIVE C ODING
Another alternative to solve Eq.
1 is to use the Predictive Coding (PC) theory.
Unlike the Lasso loss function, PC is not only minimizing the bottom-up prediction error, but it also adds a top-down prediction error that takes into consideration the inﬂuence of the upper-layer on the current layer (see Eq. 4).
In other words, ﬁnding the γ i that minimizes L consists in ﬁnding a trade-off between a representation that best predicts the lower level activity and another one that is best predicted by the upper-layer.
[Equation]
For consistency, we also use the ISTA algorithm to minimize L w.r.t γ i .
The update scheme is described in Eq.
5 (without image indexation for concision):
[Equation]
Fig.
1 shows how we can interpret this update scheme as a loop of a recurrent layer.
This recurrent layer, called Sparse Predictive Coding (SPC) layer, forms the building block of the 2 -Layers Sparse
Predictive Coding (2L-SPC) network (see Algorithm 2 in Appendix for the detailed implementation of the 2L-SPC inference).
The only difference with the Hi-La architecture is that the 2L-SPC includes an inter-layer feedback connection to materialize the inﬂuence coming from upper-layers (see the blue arrow in Fig. 1).
2. C ODING STOPPING CRITERION AND UNSUPERVISED LEARNING
For both networks, the inference process is ﬁnalized once the relative variation of γ ti w.r.t to γ t − 1 i is below a threshold denoted T stab .
In practice, the number of iterations needed to reach the stopping criterion is between 30 to 100 (see Fig.4 for more details).
Once the convergence is achieved, we update the dictionaries using gradient descent (see Algorithm. 1).
It was demonstrated by Sulam et al.
(2018) that this alternation of inference and learning offers reasonable convergence guarantee.
The learning of both Hi-La and 2L-SPC consists in minimizing the problem deﬁned in Eq. 6 in which N is the number of images in the dataset.
The learning occurs during the training phase only.
Conversely, the inference process is the same during both training and testing phases. [Equation] For both models, dictionaries are randomly initialized using the standard normal distribution (mean 0 and variance 1 ) and all the sparse maps are initialized to zero at the beginning of the inference process.
After every dictionary update, we (cid:96) 2 -normalize each atom of the dictionary to avoid any redundant solution.
Interestingly, although the inference update scheme is different for the two models, the dictionary learning loss is the same in both cases since the top-down prediction error term in L does not depend on D i (see Eq. 6).
This loss is then a good evaluation point to assess the impact of both 2L-SPC and Hi-La inference process on the layer prediction error (cid:15) i .
We used PyTorch 1.0 to implement, train, and test all the models described above.
The code of the two models and the simulations of this paper are available at www.github.com/XXX/XXX.
Algorithm do ∀ i, γ ( k ) i = # initialization while convergence not reached do for i = 1 to L do γ ( k ) i ← T η c
i λ i (cid:0) γ ( k ) i − η c i ∇ γ ( k ) i
L (cid:1) # inference for i = 1 to L do
D i ← D
i − η i ∇ D i L # learning (only for training)
E XPERIMENTAL SETTINGS : DATASETS AND PARAMETERS
We use 4 different databases to train and test both networks.
STL-10 .
The STL-10 database (Coates et al., 2011) is made of 100000 colored images of size 96 × 96 pixels (px) representing 10 classes of objects (airplane, bird...).
STL-10 presents a high diversity of objects view-points and background.
This set is partitioned into a training set composed of 90000 images, and a testing set of 10000 images.
AT&T .
The AT&T database (ATT, 1994) is made of 400 grayscale images of size 92 × 112 pixels (px) representing faces of 40 distinct subjects with different lighting conditions, facial expressions, and details.
This set is partitioned into batches of 20 images.
The training set composed ofimages (33 subjects) and the testing set id composed of 70 images (7 subjects).
CFD .
The Chicago Face Database (CFD) (Ma et al., 2015) consists of 1 , 804 high-resolution ( 2 , 444 × 1 , 718 px), color, standardized photographs of male and female faces of varying ethnicity between the ages of 18 and 40 years.
We re-sized the pictures to 170 × 120 px to keep reasonable computational time.
The CFD database is partitioned into batches of 10 images.
This dataset is split into a training set composed of 721 images and a testing set of 486 images.
MNIST .
MNIST (LeCun, 1998) is composed of 28 × 28 px, 70 , 000 grayscale images representing handwritten digits.
We decomposed this dataset into batches of 32 images.
This dataset is split into a training set composed of 60 , 000 digits and a testing set of 10 , 000 digits.
All these databases are pre-processed using Local Contrast Normalization (LCN) and whitening.
LCN is inspired by neuroscience and consists in a local subtractive and divisive normalization (Jarrett et al., 2009).
In addition, we use whitening to reduce dependency between pixels.
To draw a fair comparison between the 2L-SPC and Hi-La models, we train both models using the same set of parameters.
All these parameters are summarized in Table 1 for STL-10, MNIST and
CFD databases and in Appendix C for ATT database. Note that the parameter η c i is omitted in the table because it is computed as the inverse of the largest eigenvalue of D Ti D i (Beck & Teboulle, 2009).
To learn the dictionary D i , we use stochastic gradient descent on the training set only, with a learning rate η L i and a momentum equal to 0 .
In this study, we consider only 2-layered networks and we vary the sparsity parameters of each layer ( λ 1 and λ 2 ) to assess their effect on both 2L-SPC and Hi-La networks. To describe the range of explored parameters during simulations, we use the format [ 0 . 3 : 0 . 7 :: 0 . 1 , 0 . 5 ] ,which means that we vary λ 1 from 0 . 3 to 0 .
7 by step of 0 .
1 while λ 2 is ﬁxed to 0 .
DataBases STLCFD MNIST network param. D 1 size [64, 1, 8, 8] (2) [64, 3, 9, 9] (3) [32, 1, 5, 5] (2)
D 2 size [128, 64, 8, 8] (1) [128, 64, 9, 9] (1) [64, 32, 5, 5] (1)
T stab 1e5e- 5etraining param.
# epochs   η L1e1e- 5eη L5e5e- 1esimu.
param.
λ 1 range [ 0 . 2 : 0 . 3 ] λ 2 range [ 0 . 4 , 1 . 4 : 1 . 8 :: 0 .  R ESULTS For cross-validation, we run 7 times all the simulations presented in this section, each time with a different random dictionary initialization.
We deﬁne the central tendency of our curves by the median of the runs, and its variation by the Median Absolute Deviation (MAD) (Pham-Gia & Hung, 2001).
We prefer this measure to the classical mean ± standard deviation because a few measures did not exhibit a normal distribution.
All presented curved are obtained on the testing set.
4. 2L-SPC CONVERGES TO A LOWER PREDICTION ERROR
As a ﬁrst analysis we report the global prediction error, as computed as the sum of the prediction error ( (cid:15) i ) over all layers (see Fig. 3), and its decomposition among layer (see Fig.
2 and Appendix D fro AT&T database) for different value of sparsity.
For scaling reasons, and because the error bars are small we cannot display them on Fig.
2, we thus include them in Appendix Fig.
For all the simulations shown in Fig. 2 and Fig.
3, we observed that the global prediction error is lower for the 2L-SPC than for the Hi-La model.
As expected, in both models the prediction errors increase similarly when we increase λ 1 or λ 2 .
For all databases and sparsity parameters, Fig.
2 shows that the ﬁrst layer prediction error of the 2L-SPC is always higher, and the second layer prediction error [Figure] is always lower than the corresponding Hi-La prediction error.
This is expected: while the Hi-La ﬁrst layer is fully specialized in minimizing the prediction error with the lower level, the 2L-SPC ﬁnds a trade-off between lower and higher level prediction errors.
In addition, when λ 1 is increased, the Hi-La ﬁrst layer prediction error increases faster ( +139% for STL-10, +105% for CFD, +157% for MNIST and +93% for AT&T ) than the 2L-SPC ﬁrst layer prediction error ( +103% for STL10, +90% for CFD, +117% for MNIST and +76% for AT&T).
This suggests that a part of the penalty induced by the increase of λ 1 is absorbed by the second layer in the case of the 2L-SPC.
This is supported by the fact that the second layer prediction error of the 2L-SPC is in general increasing slightly faster than the Hi-La second layer error.
When λ 2 is increased, the prediction error of the ﬁrst layer of the Hi-La model is not varying whereas the 2L-SPC ﬁrst layer prediction error is slightly increasing ( +1% for STL-10, +3% for CFD, +7 .
1% for MNIST and +3 . 3% for AT&T).
The explanation here is straightforward: while the ﬁrst-layer loss of the 2L-SPC includes the inﬂuence of the upper-layer, the Hi-La doesn’t have such a mechanism.
It suggests that the inter-layer feedback connection of the 2L-SPC transfers a part of the extra-penalty coming from the increase of λ 2 in the ﬁrst layer.
Fig.3 i) and ii) show the mapping of the global prediction error when we vary the sparsity of each layer for the 2L-SPC and Hi-La, respectively.
These heatmaps conﬁrm what has been observed in Fig.2 and extend it to a larger range of sparsity values: both models’ losses are more sensitive to a variation of λ 1 than to a change in λ 2 .
Fig.3 iii) is a heatmap of the relative difference between the 2L-SPC and the Hi-La global losses.
It shows that the minimum relative difference between 2L-SPC and Hi-La (10.6%) is reached when λ 1 is maximal and λ 2 is minimal, and the maximum relative difference (19.99%) is reached when both λ 1 and λ 2 are minimal.
It suggests that the previously observed mitigation mechanism originated by the feedback connection is more efﬁcient when the sparsity of the ﬁrst layer is lower.
All these observations point in the same direction: the 2L-SPC framework mitigates the global prediction error thanks to a better distribution of the prediction error among layers.
This mechanism is even more pronounced when the sparsity of the ﬁrst layer is lower.
Surprisingly, while the inter-layer feedback connection of the 2L-SPC imposes more constraints on the state variables, it also happens to generate less global prediction error.
4. 2L-SPC HAS A FASTER INFERENCE PROCESS
One may wonder if this lower prediction error is not achieved at the cost of a slower inference process.
To address this concern, we report for both models the number of iterations needed by the inference process to converge towards a stable state on the testing set.
Fig.
4 shows the evolution of this quantity, for STL-10, CFD and MNIST databases (see Appendix E for AT&T database), when varying both layers’ sparsity.
For all the simulations, the 2L-SPC needs less iteration than the Hi model to converge towards a stable state.
We also observe that the data dispersion is, in addition to converging
[Figure]
4. 2L-SPC LEARNS FASTER
Fig.
5 shows the evolution of the global prediction error during the dictionary learning stage and evaluated on the testing set (see Appendix.
F for AT&T database). For the all databases, the 2L-SPC model reaches its minimal prediction error before the Hi-La model.
The convergence rate of both models is comparable, but the 2L-SPC has a much lower prediction error in the very ﬁrst epochs.
The inter-layer feedback connection of the 2L-SPC pushes the network towards lower prediction error since the very beginning of the learning. 4. Q UALITATIVE ANALYSIS OF THE FEATURES
[Figure]
Another way to grasp the impact of the inter-layer feedback connection is to visualize its effect on the dictionaries.
To make human-readable visualizations of the learned dictionaries we back-project them into the image space using a cascade of transposed convolution (see Appendix Fig.11).
These back-projection are called Receptive Fields (RFs).
Fig.
6 shows some of the RFs of the 2 layers and the second layer activation probability histogram for both models when they are trained on the
CFD database.
In general, ﬁrst layer RFs are oriented Gabor-like ﬁlters and second layer RFs are more speciﬁc and represent more abstract concepts (curvatures, eyes, mouth, nose...).
Second layer
RFs present longer curvatures in the 2L-SPC than in the Hi-La model: they cover a bigger part of the input image, and include more contextual and informative details.
In some extreme cases, the
Hi-La second layer RFs are over-ﬁtted to speciﬁc faces and do not describe the generality of the concept of face.
The red-framed RFs highlights one of these cases: the corresponding activation probabilities are 0 .
25% and 0 .
69% for Hi-La and 2L-SPC respectively.
This is supported by the fact that the lowest activation probability of the second layer’s atoms is higher for the 2L-SPC than for the Hi-La ( 0 .
30% versus 0 .
This phenomenon is even more striking when we sort all the features by activation probabilities in descending order (see Appendix Figures 13).
We ﬁlter out the highest activation probability (corresponding to the low-frequency ﬁlters highlighted by black square) of both Hi-La and 2L-SPC to keep good readability of the histograms.
All the ﬁlters are displayed in Appendix Fig.
12, Fig. 13, Fig.
14 and Fig.
15, for STL-10, CFD, MNIST and AT&T RFs respectively.
The atoms’ activation probability conﬁrm the qualitative analysis of the RFs: the features learned by the 2L-SPC are more generic and informative as they describe a wider range of images.
C ONCLUSION
What are the computational advantages of inter-layer feedback connections in hierarchical sparse coding algorithms?
We answered this question by comparing the Hierarchical Lasso (Hi-La) and the 2 -Layers Sparse Predictive Coding (2L-SPC) models.
Both are identical in every respect, except that the 2L-SPC brings inter-layer feedback connections.
This extra-connection forces the internal state variables of the 2L-SPC to converge toward a trade-off between on one hand an accurate prediction passed by the lower-layer and on the other hand a facilitated predictability by the upperlayer.
Experimentally, we demonstrated on 4 different databases and for a 2-layered network that the inter-layer feedback top-down connection (i) mitigates the overall prediction error by distributing it among layers, (ii) accelerates the convergence towards a stable internal state and (iii) accelerates the learning process.
Besides, we qualitatively observed that top-down connections bring contextual information that helps to extract more informative and less over-ﬁtted features.
The 2 L-SPC holds the novelty to consider Hierarchical Sparse Coding as a combination of local sub-problems that are tightly related. This a crucial difference with CNNs that are trained by backpropagating gradients from a global loss. To the best of our knowledge the 2 L-SPC is the ﬁrst one that leverage local sparse coding into a hierarchical and unsupervised algorithms (the ML-CSC from (Sulam et al., 2018) is equivalent to a one layer sparse coding algorithm (Aberdam et al., 2019), and the ML-ISTA from (Sulam et al., 2019) is trained using supervised learning). Moreover, even if our results are robust as they hold for 4 different databases and with a large spectrum of ﬁrst and second layer sparsity, further work will be conducted to generalize our results to deeper networks and different sparse coding algorithms such as Coordinate Descent or ADMM.
Further studies will show that our 2L-SPC framework could be used for practical applications like image inpainting, denoising, or image super-resolution.
A 2L-SPC PSEUDO CODE
Algorithm 2: input : image: x , dictionaries: { D i } Li =1 , (cid:96) 1 -penalty parameters: { λ i } Li =1 , stability threshold:
T stab γ t
= x { γ 0 i } Li =1 = 0 , { γ 1 m i } Li =1 =
# Initializing layer state variables and FISTA momentum
α 1 = 1 # Initializing momentum strength η c i = max ( eigen value ( D Ti D i ))
Stable = False # Initializing the stability criterion t
=while Stable ==
False do t +=α t +
= 1 + (cid:112)
+ 4( α t )2 for i = 1 to L do # Update lower-layer error (cid:15) LL =
γ tm i − 1 − D Ti γ tm i # Update the upper-layer error if i (cid:54) =
L then (cid:15) UL =
γ tm i − D
Ti +1 γ
tm i + else (cid:15) UL
=γ ti = T η ci
λ
i (cid:0) γ tm i + η c i D i (cid:15) LL − η c i (cid:15) UL (cid:1)
# Update layer state variables γ t
+1 m i =
T 0 (cid:16) γ ti + (cid:0) α t − 1 α t
+
(cid:1)(cid:0) γ ti − γ t − 1 i (cid:1)(cid:17)
# Update the FISTA momentum if
L (cid:94) i = (cid:16) (cid:107) γ ti − γ t − 1 i (cid:107)(cid:107) γ ti (cid:107)<
T stab (cid:17) then
Stable = True
# Update the stability creterion return { γ ti } Li =
Note: T α ( · ) denotes the element-wise non-negative soft-thresholding operator.
A fortiori, T 0 ( · )
[Table]
B E VOLUTION OF THE GLOBAL PREDICTION ERROR WITH ERROR BAR
[Figure]
C 2L-SPC P ARAMETERS ON ATT
[Table]
D P REDICTION ERRORS DISTRIBUTION ON AT&T
[Figure]
E NUMBER OF ITERATION OF THE INFERENCE ON AT&T
[Figure] F E VOLUTION G I LLUSTRATION OF THE BACK - PROJECTION MECHANISM
In a general case, the effective dictionary at layer i is computed as follow: D eff ,T i
[Equation]
H 2L-SPC AND H I -L A RF S ON STL
[Figure]
I 2L-SPC AND H I -L A RF S ON CFD
[Figure]
J 2L-SPC AND H I -L A RF S ON MNIST
[Figure]
The visualization shown here is the projection of the dictionaries into the input effective dictionaries have a size of 5 × 5 px ( a & and have a size of 14 × 14 ( c & d ) px respectively
K 2L-SPC AND H I -L A RF S ON AT&T
[Figure]