Original: G RADIENT - BASED T RAINING OF S LOW F EATURE A NALYSIS
Revised: G RADIENT - BASED T RAINING OF S LOW F EATURE A NALYSIS

Original: BY
Revised: BY

Original: D IFFERENTIABLE
Revised: D IFFERENTIABLE

Original: A PPROXIMATE
Revised: A PPROXIMATE

Original: W HITENING
Revised: W HITENING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Power Slow Feature Analysis, a gradient-based method to extract temporally slow features from a high-dimensional input stream that varies on a faster time-scale, as a variant of Slow Feature Analysis (SFA).
Revised: We propose Power Slow Feature Analysis, a gradient-based method to extract temporally slow features from a high-dimensional input stream that varies on a faster time-scale, as a variant of Slow Feature Analysis (SFA).

Original: While displaying performance comparable to hierarchical extensions to the SFA algorithm, such as
Revised: While displaying performance comparable to hierarchical extensions to the SFA algorithm, such as

Original: Hierarchical Slow Feature Analysis, for a small number of output-features, our algorithm allows fully differentiable end-to-end training of arbitrary differentiable approximators (e g., deep neural networks).
Revised: Hierarchical Slow Feature Analysis, for a small number of output-features, our algorithm allows fully differentiable end-to-end training of arbitrary differentiable approximators (e g., deep neural networks).

Original: We provide experimental evidence that PowerSFA is able to extract meaningful and informative low-dimensional features in the case of (a) synthetic low-dimensional data, (b) visual data and also for (c) a general dataset for which symmetric non-temporal relations between points can be deﬁned
Revised: We provide experimental evidence that PowerSFA is able to extract meaningful and informative low-dimensional features in the case of (a) synthetic low-dimensional data, (b) visual data and also for (c) a general dataset for which symmetric non-temporal relations between points can be deﬁned

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Finding meaningful representations in data is a core challenge in modern machine learning as the performance in many goal-directed frameworks such as reinforcement learning or supervised learning is strongly inﬂuenced by the quality of the former.
Revised: Finding meaningful representations in data is a core challenge in modern machine learning as the performance in many goal-directed frameworks, such as reinforcement learning or supervised learning, is strongly inﬂuenced by the quality of the former.

Original: Usually, features are either domain speciﬁc or acquired through learning.
Revised: Usually, features are either domain speciﬁc or acquired through learning.

Original: Most currently successful approaches for either deep supervised learning (Goodfellow2016) or reinforcement learning (Sutton & Barto (1998)) rely on a training signal, i.e., a classiﬁcation label or reward, to provide sufﬁcient indication which features of the input data should be extracted to increase performance.
Revised: Most currently successful approaches for either deep supervised learning (Goodfellow et al. or reinforcement learning (Sutton & Barto (1998)) rely on a training signal, i.e., a classiﬁcation label or reward, to provide sufﬁcient indication which features of the input data should be extracted to increase performance.

Original: However, in most real-world scenarios labels have to be provided by expert knowledge and reward signals are sparse.
Revised: However, in most real-world scenarios labels have to be provided by expert knowledge and reward signals are sparse.

Original: In unsupervised representation learning one tries to ﬁnd and apply a principle by which to extract meaning from data without assuming the availability of any goal-driven metrics.
Revised: In unsupervised representation learning, one tries to ﬁnd and apply a principle by which to extract meaning from data without assuming the availability of any goal-driven metrics.

Original: Examples for such principles are based on reconstruction error in principal component analysis or autoencoder networks (Bourlard & Kamp (1988)), statistical dependence of extracted features in independent component analysis (Comon (1994)), indistinguishability of synthetically generated data from samples of the input distribution in generative adversarial nets (Goodfellow et al.
Revised: Examples for such principles are based on reconstruction error in principal component analysis or autoencoder networks (Bourlard & Kamp (1988)), statistical dependence of extracted features in independent component analysis (Comon (1994)), indistinguishability of synthetically generated data from samples of the input distribution in generative adversarial nets (Goodfellow et al.

Original: (2014)), ﬁtting the probability distribution of input data with variational autoencoders (Kingma & Welling (2013)), neighborhood preservation used in locally linear embedding (LLE, Roweis & Saul (2000)), Laplacian eigenmaps (LEM, Belkin & Niyogi (2003)), as well as temporal coherence as used in slow feature analysis, (SFA, Wiskott & Sejnowski (2002)) or regularized slowness optimization (Bengio & Bergstra (2009)).
Revised: Further examples include: ﬁtting the probability distribution of input data with variational autoencoders (Kingma & Welling (2013)), neighborhood preservation used in locally linear embedding (LLE, Roweis & Saul (2000)), Laplacian eigenmaps (LEM, Belkin & Niyogi (2003)), as well as temporal coherence as used in slow feature analysis, (SFA, Wiskott & Sejnowski (2002)) or regularized slowness optimization (Bengio & Bergstra (2009)).

Original: The latter principle is the focus of this work and has been shown to provide a useful proxy for extracting underlying causes from time-series data such as position, head direction, identity of spatial view similar to those observed in rodent brains (Franzius et al.
Revised: Temporal coherence is the focus of this work and has been shown to provide a useful proxy for extracting underlying causes from time-series data such as position, head direction, identity of spatial view similar to those observed in rodent brains (Franzius et al.

Original: from ego-visual data, or object conﬁguration (identity, position, angle) from a tabletop view of moving objects (Franzius et al.
Revised: from ego-visual data. The principle has also been successfully applied to determining object conﬁguration (identity, position, angle) from a tabletop view of moving objects (Franzius et al.

Original: (2011))when applied in the form of SFA.
Revised: (2011)) in the form of SFA.

Original: A graph-based generalization of SFA has been used to achieve (at that time) state-of-the-art age estimation results on the MORPH dataset in Escalante-B.(2017).
Revised: A graph-based generalization of SFA has been used to achieve (at that time) state-of-the-art age estimation results on the MORPH dataset in Escalante-B.

Original: We propose a variant of SFA that approximately enforces the same constraints while being differentiable and thus allows training by gradient-descent-derived methods.
Revised: We propose a variant of SFA that approximately enforces the same constraints while being differentiable and thus allows training by gradient-descent-derived methods.

Original: Thisvariant makes it possible to leverage the representational power of complex models such as deep neural networks and useful ideas from that domain (dropout, batch normalization, activation regularization) to extract slow and informative features from data.
Revised: This makes it possible to leverage the representational power of complex models, such as deep neural networks, and useful ideas from that domain (dropout, batch normalization, activation regularization) to extract slow and informative features from data.

Original: To demonstrate the applicability of this approach, we provide three distinct experimental evaluations.
Revised: To demonstrate the applicability of this approach, we provide three distinct experimental evaluations.

Original:  R ELATED WORK 2. S LOWNESS - BASED M ETHODS
Revised:  R ELATED WORK 2. S LOWNESS - BASED M ETHODS

Original: While the original proposal of SFA by Wiskott & Sejnowski (2002) uses non-linear basis functions as a method to introduce non-linearity to the otherwise linear model, this classical approach has two limitations: a) The basis functions are ﬁxed and thus have to be chosen beforehand by expert knowledge or trial-and-error, and b) as the resulting model is shallow, its expressivity tends to scale unfavorably in the dimension of the expansion (Raghu et al.
Revised: While the original proposal of SFA by Wiskott & Sejnowski (2002) uses non-linear basis functions as a method to introduce non-linearity to the otherwise linear model, this classical approach has two limitations: (a) The basis functions are ﬁxed and thus have to be chosen beforehand by expert knowledge or trial-and-error, and (b) as the resulting model is shallow, its expressivity tends to scale unfavorably in the dimension of the expansion (Raghu et al.

Original: compared to hierarchical extensions discussed later.
Revised: compared to hierarchical extensions discussed later.

Original: In the case of polynomial expansion, expanding to degree d on e -dimensional input data results in (cid:0) d + ed (cid:1) -dimensional expanded data, e.g., quadratically expanding grayscale images of 180 × 90 pixels results in an output dimension > 131 · 10 6 .
Revised: In the case of polynomial expansion, expanding to degree d on e -dimensional input data results in (cid:0) d + ed (cid:1) -dimensional expanded data, e.g., quadratically expanding grayscale images of 180 × 90 pixels results in an output dimension > 131 · 10 6 .

Original: Extracting a single feature with a linear model would thus require 5 .
Revised: Extracting a single feature with a linear model would thus require 5 .

Original: 7 -times more parameters than the modern and powerful Xception network (Chollet (2016)).
Revised: 7 -times more parameters than the modern and powerful Xception network (Chollet (2016)).

Original: An alternative approach to increase expressivity using expansions is to apply low-degree nonlinearities repeatedly in a receptive-ﬁeld fashion, interlacing them with projection steps.
Revised: An alternative approach to increase expressivity using expansions is to apply low-degree nonlinearities repeatedly in a receptive-ﬁeld fashion, interlacing them with projection steps.

Original: This has been done in Hierarchical SFA (HSFA, Franzius et al.
Revised: This has been done in Hierarchical SFA (HSFA, Franzius et al.

Original: and Escalante-B. & Wiskott (2016)) and deep architectures in general.
Revised: and Escalante-B. & Wiskott (2016)) and deep architectures in general.

Original: But while the latter are typically trained in an end-to-end fashion by variants of stochastic gradient-descent, HSFA is trained in a layer-wise procedure, solving the linear SFA problem consecutively for each layer as closed-form solution and thereby assuming that globally slow features can be composed of decreasingly local slow features.
Revised: But while the latter are typically trained in an end-to-end fashion by variants of stochastic gradient-descent, HSFA is trained in a layer-wise procedure, solving the linear SFA problem consecutively for each layer as closed-form solution and thereby assuming that globally slow features can be composed of decreasingly local slow features.

Original: Escalante-B. & Wiskott (2016) shows that this assumption can be partially relaxed by adding information by-passes to the model.
Revised: Escalante-B. & Wiskott (2016) shows that this assumption can be partially relaxed by adding information by-passes to the model.

Original: Compared to HSFA, our method allows directly modifying parameters in different layers to optimize a global slowness objective.
Revised: Compared to HSFA, our method allows directly modifying parameters in different layers to optimize a global slowness objective.

Original: Our algorithm is in line with recent work harnessing the temporal coherence prior by Bengio et al.
Revised: Our algorithm is in line with recent work harnessing the temporal coherence prior by Bengio et al.

Original: (2013) in deep, self-supervised feature learning.
Revised: (2013) in deep, self-supervised feature learning.

Original: This can be done by having a slowness term in the loss function.
Revised: This can be done by having a slowness term in the loss function.

Original: To avoid the trivial, constant solution, another term is usually added.
Revised: To avoid the trivial, constant solution, another term is usually added.

Original: For example, a reconstruction loss in an auto-encoder (Goroshin et al.
Revised: For example, a reconstruction loss in an auto-encoder (Goroshin et al.

Original: (2015a)) or one-step latent code prediction (
Revised: (2015a)) or one-step latent code prediction (

Original: Goroshin et al.
Revised: Goroshin et al.

Original: (2015b)).
Revised: (2015b)).

Original: Deep temporal coherence has also been considered via the lens of similarity metric learning, for example by optimizing a contrastive loss (Jayaraman & Grauman (2016),
Revised: Deep temporal coherence has also been considered via the lens of similarity metric learning, for example by optimizing a contrastive loss (Jayaraman & Grauman (2016),

Original: Mobahi et al.
Revised: Mobahi et al.

Original: (2009)) or a triplet loss (Jansen et al.
Revised: (2009)) or a triplet loss (Jansen et al.

Original: (2017), Wang & Gupta (2015)).
Revised: (2017), Wang & Gupta (2015)).

Original: These metric learning approaches manage to avoid degenerate solutions by pushing points away from each other (in feature space) that are not temporal neighbors.
Revised: These metric learning approaches manage to avoid degenerate solutions by pushing points away from each other (in feature space) that are not temporal neighbors.

Original: Our work differs from these approaches as we seek to directly approximate the optimization problem as originally stated by Wiskott & Sejnowski (2002) in a deep learning setting, automatically handling the constant solution and ensuring that different features code for different information.
Revised: Our work differs from these approaches as we seek to directly approximate the optimization problem as originally stated by Wiskott & Sejnowski (2002) in a deep learning setting, automatically handling the constant solution and ensuring that different features code for different information.

Original: 2. G RAPH - BASED M ETHODS
Revised: 2. G RAPH - BASED M ETHODS

Original: SFA has been generalized to graph-based SFA (GSFA, Escalante-B. & Wiskott (2013)) and generalized SFA (Sprekeler (2011)).
Revised: SFA has been generalized to graph-based SFA (GSFA, Escalante-B. & Wiskott (2013)) and generalized SFA (Sprekeler (2011)).

Original: The former adapts only the objective function, while the latter additionally generalizes the constraints to D -orthogonality ( D being the degree matrix of an underlying graph).
Revised: The former adapts only the objective function, while the latter additionally generalizes the constraints to D -orthogonality ( D being the degree matrix of an underlying graph).

Original: Following generalized SFA, standard SFA can be shown to be a special case of
Revised: Following generalized SFA, standard SFA can be shown to be a special case of

Original: Laplacian Eigenmaps.
Revised: Laplacian Eigenmaps.

Original: Spectral Inference Networks (SpIN, Pfau et al.
Revised: Spectral Inference Networks (SpIN, Pfau et al.

Original: (2018)) utilize this connection to successfully derive a gradient-based SFA training as a special case and have been developed simultaneously to this work.
Revised: (2018)) utilize this connection to successfully derive a gradient-based SFA training as a special case.

Original: SpIN are based on correcting a biased gradient when directly optimizing the Rayleigh-Quotient with respect to the parameters that deﬁne the embedding.
Revised: They are based on correcting a biased gradient when directly optimizing the Rayleigh-Quotient with respect to the model’s parameters. The constraints are implicitly enforced through the loss function as opposed to directly whitening the output.

Original: Similar to PowerSFA, this procedure allows for employing anydifferentiable architecture to ﬁnd these embeddings.
Revised: Similar to our work, SpINs allow for employing any architecture to ﬁnd these embeddings.

Revised: However, the whitening proposed in this paper is applicable to any loss function as it is part of the model architecture and not inherently a part of the optimized objective.

Original: The generalization of PowerSFA proposed in section 6.3 is close to GSFA, assuming regularity (or rather: ignoring non
Revised: The generalization of PowerSFA proposed in section 6.3 is close to GSFA, assuming regularity (or rather: ignoring non-regularity) in the graph for the orthogonality constraint.

Original: SpectralNets (SN, Shaham et al.
Revised: SpectralNets (SN, Shaham et al.

Original: (2018)) are another closely related approach in which a differentiable approximator is trained to learn spectral embeddings used in subsequent k -means clustering.
Revised: (2018)) are another closely related approach in which a differentiable approximator is trained to learn spectral embeddings used in subsequent k -means clustering.

Original: However, opposed to PowerSFA and SpIN, they split a single optimization step into two parts: an ortho-normalization step based on explicitly calculating the Cholesky decomposition of the batch covariance matrix to set and freeze the weights of a linear output layer followed by a stochastic optimization step.
Revised: Opposed to PowerSFA and SpINs, they split a single optimization step into two parts: an ortho-normalization step based on explicitly calculating the Cholesky decomposition of the batch covariance matrix to set and freeze the weights of a linear output layer, followed by a stochastic optimization step.

Original: While this can be considered end-to-end depending on the paradigm, Shaham et al.
Revised: While this can be considered end-to-end depending on the paradigm, Shaham et al.

Original: (2018) do not indicate if and how this can be implemented as fully differentiable architecture.
Revised: (2018) do not indicate if and how this can be implemented as fully differentiable architecture.

Original: 2. O THER R ELATED A PPROACHES
Revised: 2. O THER R ELATED A PPROACHES

Original: In section 6.3 we consider a generalization of PowerSFA’s loss similar to GSFA and apply it to the NORB dataset (LeCun et al.
Revised: In section 6.3 we consider a generalization of PowerSFA’s loss similar to GSFA and apply it to the NORB dataset (LeCun et al.

Original: We thereby loosely follow the experimental procedure in Hadsell et al.
Revised: We thereby loosely follow the experimental procedure in Hadsell et al.

Original: The authors use a siamese neural network architecture (Bromley et al.
Revised: The authors use a siamese neural network architecture (Bromley et al.

Original: for optimizing pair-wise distances of embedded points to reﬂect similarity and dissimilarity structure of the data.
Revised: for optimizing pair-wise distances of embedded points to reﬂect similarity and dissimilarity structure of the data.

Original: In particular, they do not enforce orthogonality of the embeddings but rely on the optimization procedure to maximize informativeness.
Revised: In particular, they do not enforce orthogonality of the embeddings but rely on the optimization procedure to maximize informativeness.

Original:  S LOW F EATURE A NALYSIS
Revised:  S LOW F EATURE A NALYSIS

Original: Slow Feature Analysis (SFA) is based on the hypothesis that interesting high-dimensional streams of data that vary quickly in time are typically caused by a low number of underlying factors that vary comparably slow.
Revised: Slow Feature Analysis (SFA) is based on the hypothesis that interesting high-dimensional streams of data, that vary quickly in time, are typically caused by a low number of underlying factors that vary comparably slow.

Original: Therefore, slowness can be used as a proxy criterion by which to extract meaningful representations of these low-dimensional underlying causes even in the absence of labels.
Revised: Therefore, slowness can be used as a proxy criterion by which to extract meaningful representations of these low-dimensional underlying causes, even in the absence of labels.

Original: There is strong evidence in favor of this hypothesis as it has been shown that features extracted by SFA tend to encode highly relevant information about the data-generating environments, e.g. slow features encode and disentangle object identity, rotation, and position in visual tasks (Franzius et al.
Revised: There is strong evidence in favor of this hypothesis, as it has been shown that features extracted by SFA tend to encode highly relevant information about the data-generating environments, e.g., slow features encode and disentangle object identity, rotation, and position in visual tasks (Franzius et al.

Original: (2011)) as well as agent position and orientation from visual ﬁrst-person recordings of random movement similar to place cells in rodents or head-direction cells in primates (Franzius et al.
Revised: (2011)) as well as agent position and orientation from visual ﬁrst-person recordings of random movement similar to place cells in rodents, or head-direction cells in primates (Franzius et al.

Original: The notion of extracting slow features from a time-series dataset has been formalized as a sequential optimization problem.
Revised: The notion of extracting slow features from a time-series dataset has been formalized as a sequential optimization problem.

Original: Given a time-series { x t } t =0 ...
Revised: Given a time-series { x t } t =0 ...

Original: τ with x t ∈ R d , sequentially ﬁnd continuous functions g i : R d → R with:[Equation] 1a) [Equation]
Revised: τ with x t ∈ R d , sequentially ﬁnd continuous functions g i : R d → R with:

Original: 1b)

Original: [Equation]
Revised: [Equation]

Original: where (cid:10) · (cid:11) t is the time-average.
Revised: where (cid:10) · (cid:11) t is the time-average.

Original: The constraints ensure that each of the extracted features is informative (decorrelated to all others, equation (1d)) and non-trivial (unit variance, equation (1c)).
Revised: The constraints ensure that each of the extracted features is informative (decorrelated to all others, equation (1d)) and non-trivial (unit variance, equation (1c)).

Original: Originally, solutions to SFA directly were only proposed for the space of afﬁne functions g i ∈ G , for which a closed-form solution exists, and in a kernelized version that requires strong regularization.
Revised: Originally, solutions to SFA directly were only proposed for the space of afﬁne functions g i ∈ G , for which a closed-form solution exists, and in a kernelized version that requires strong regularization.

Original:  (A PPROXIMATE ) W HITENING
Revised:  (A PPROXIMATE ) W HITENING

Original: While the standard implementation of linear SFA (Zito et al.
Revised: The standard implementation of linear SFA (Zito et al.

Original: (2008)) is based on computing the closed-form solution to a generalized eigenvalue problem (Berkes & Wiskott (2005)), another approach is to ﬁrst whiten the time-series data followed a projection onto the minor components of the difference time-series { ˙ x t = x t +1 − x t } t =0 ...
Revised: (2008)) is based on computing the closedform solution to a generalized eigenvalue problem (Berkes & Wiskott (2005)). Another approach is to ﬁrst whiten the time-series data followed a projection onto the minor components of the difference time-series { ˙ x t = x t +1 − x t } t =0 ...

Original: τ − 1 .
Revised: τ − 1 .

Original: Whitened data has three important properties: a) it is mean-free (constraint (1b)), b) has unit variance if projected onto an arbitrary unit vector (constraint (1c)), and c) projections onto orthonormal vectors are decorrelated (constraint (1d)).
Revised: Whitened data has three important properties: (a) it is mean-free (constraint (1b)), (b) has unit variance if projected onto an arbitrary unit vector (constraint (1c)), and (c) projections onto orthonormal vectors are decorrelated (constraint (1d)).

Original: For a dataset ˜ X ∈ R d × N with N and d being size of dimension of the dataset, respectively, the corresponding whitened dataset is deﬁned as
Revised: For a dataset ˜ X ∈ R d × N with N and d being size and dimension of the dataset, respectively, the corresponding whitened dataset is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with W = D − 12 U T and C
Revised: with W = D − 12 U T and C

Original: = UDU T ∈ R d × d being the whitening matrix and the canonical eigendecomposition of the covariance matrix, respectively.
Revised: = UDU T ∈ R d × d being the whitening matrix and the canonical eigendecomposition of the covariance matrix, respectively.

Original: U ’s columns contain the eigenvectors of
Revised: U ’s columns contain the eigenvectors of

Original: C and D contains the corresponding eigenvalues on its diagonal.
Revised: C and D contains the corresponding eigenvalues on its diagonal.

Original: As C is typically assumed to be positive deﬁnite, D − 1 2 is well-deﬁned.
Revised: As C is typically assumed to be positive deﬁnite, D − 12 is well-deﬁned.

Original: One widely used method to extract eigenvector/-value pairs is power iteration .
Revised: One widely used method to extract eigenvector/-value pairs is power iteration .

Original: Starting from a random vector u 0 ∈ R R d , repeatedly applying:
Revised: Starting from a random vector u 0 ∈ R R d , repeatedly applying:

Original: [Equation]
Revised: [Equation]

Original: converges to the eigenvector u corresponding to the largest (absolute) eigenvalue λ .
Revised: converges to the eigenvector u corresponding to the largest (absolute) eigenvalue λ .

Original: The eigenvalue can then be extracted as
Revised: The eigenvalue can then be extracted as

Original: [Equation]
Revised: [Equation]

Original: and the spectral component corresponding to this eigenvector can be removed as
Revised: and the spectral component corresponding to this eigenvector can be removed as

Original: C

Original: [Equation]
Revised: [Equation]

Original: Repeating the procedure converges to the eigenvector corresponding to the next largest eigenvalue and so on.
Revised: Repeating the procedure converges to the eigenvector corresponding to the next largest eigenvalue and so on.

Original: We use a previously ﬁxed number of iterations for this method as experiments have shown that approximate whitening is enough to take meaningful optimization steps.
Revised: We use a previously ﬁxed number of iterations for this method as experiments have shown that approximate whitening is enough to take meaningful optimization steps.

Original: In practice, a relatively small number of iterations results in acceptable whitening for most non-degenerate cases.
Revised: In practice, a relatively small number of iterations results in acceptable whitening for most non-degenerate cases.

Original: Data: covariance matrix C , number of iterations N iter , data dimension d
Revised: Data: covariance matrix C , number of iterations N iter , data dimension d

Original: Result: whitening matrix W
Revised: Result: whitening matrix W

Original: W ← { 0 } d × d for i = 0; i < d ; i ++ do
Revised: W ← { 0 } d × d for i = 0; i < d ; i ++ do

Original: Sample r ∼ U [ − 1 , 1] d for j = 0; j < N iter ; j ++ do r ← Cr (cid:107) Cr (cid:107) end λ ← (cid:107) Cr (cid:107) 
Revised: Sample r ∼ U [ − 1 , 1] d for j = 0; j < N iter ; j ++ do r ← Cr (cid:107) Cr (cid:107) end λ ← (cid:107) Cr (cid:107) C [Equation]

Original: C ← C − λ rr T w i · = 1 √ λ r T endreturn W
Revised: endreturn W

Original: Note that in algorithm 1 each operation is differentiable with respect to C .
Revised: Note that in algorithm 1 each operation is differentiable with respect to C .

Original:  G RADIENT - BASED S LOW F EATURE A NALYSIS
Revised:  G RADIENT - BASED S LOW F EATURE A NALYSIS

Original: The key idea for gradient-based SFA is that a whitening layer can be applied subsequently to any differentiable architecture (such as deep neural networks) to enforce outputs that approximately obey the SFA constraints while still being a differentiable architecture.
Revised: The key idea for gradient-based SFA is that a whitening layer can be applied subsequently to any differentiable architecture (such as deep neural networks) to enforce outputs that approximately obey the SFA constraints while still being a differentiable architecture.

Original: As such, it can be trained using gradient-descent-like training procedures allowing for hierarchical architectures where every parameter is modiﬁed iteratively towards optimizing a global slowness objective as opposed to assuming a local-to-global slowness as in HSFA.
Revised: As such, it can be trained using gradient-descent-like training procedures, allowing for hierarchical architectures where every parameter is modiﬁed iteratively towards optimizing a global slowness objective, as opposed to assuming a local-to-global slowness as in HSFA.

Original: To formalize, if ˜ g θ :
Revised: To formalize, if [Equation]

Original: R N × d → R N × e is a differentiable function approximator parameterized by θ such as a neural network and
Revised: is a differentiable function approximator parameterized by θ such as a neural network and

Original: [Equation]
Revised: [Equation]

Original: denotes the approximate whitening procedure, then
Revised: denotes the approximate whitening procedure, then

Original: [Equation]
Revised: [Equation]

Original: is an approximator whose outputs approximately obey the SFA constraints.
Revised: is an approximator whose outputs approximately obey the SFA constraints.

Original: As a loss function, we deﬁne a general loss as
Revised: As a loss function, we deﬁne a general loss as

Original: [Equation]
Revised: [Equation]

Original: with g i being the i -th row of g θ ( X ) and w ij being the strength of the connection between two points x i and x j similar to weights in spectral graph embeddings.
Revised: with g i being the i -th row of g θ ( X ) and w ij being the strength of the connection between two points x i and x j similar to weights in spectral graph embeddings.

Original: For optimizing slowness, we deﬁne the weight as
Revised: For optimizing slowness, we deﬁne the weight as

Original: [Equation]
Revised: [Equation]

Original: with δ being the Kronecker delta.
Revised: with δ being the Kronecker delta.

Original: This connects consecutive steps in the time-series and disconnects the others 1 .
Revised: This connects consecutive steps in the time-series and disconnects the others 1 .

Original: The approximator can then be trained to minimize eq. 2 by following the negative gradient estimate of L with respect to θ , −∇ θ L for (mini-)batches of data as in any gradient-descent procedure.
Revised: The approximator can then be trained to minimize eq. 2 by following the negative gradient estimate of L with respect to θ , −∇ θ L for (mini-)batches of data as in any gradient-descent procedure.

Original: In our experiments, we used the ADAM optimizer (Kingma & Ba (2014)) with Nesterov-accelerated momentum (Dozat (2015)) to train the approximator.
Revised: In our experiments, we used the ADAM optimizer (Kingma & Ba (2014)) with Nesterov-accelerated momentum (Dozat (2015)) to train the approximator.

Original: Learning rate and additional hyperparameters were left at default values as implemented in the popular Keras -package
Revised: Learning rate and additional hyperparameters were left at default values as implemented in the popular Keras -package

Original: (Chollet et al.
Revised: (Chollet et al.

Original:  E XPERIMENTS 6. S YNTHETIC T RIGONOMETRIC D ATA
Revised:  E XPERIMENTS 6. S YNTHETIC T RIGONOMETRIC D ATA

Original: To show the general feasibility of this approach, we ﬁrst show that PowerSFA ﬁnds near-optimal solutions in the linear case for synthetic data.
Revised: To show the general feasibility of this approach, we ﬁrst show that PowerSFA ﬁnds near-optimal solutions in the linear case for synthetic data.

Original: Since standard SFA is a linear method that relies on (a) non-linear basis function expansion and (b) hierarchical processing to induce non-linearity, this means that PowerSFA could hypothetically be used a similar fashion (even though a gradient-based approach allows for more complex models in a natural way).
Revised: Standard SFA is a linear method that relies on (a) non-linear basis function expansion and (b) hierarchical processing to induce non-linearity. This means that PowerSFA could hypothetically be used a similar fashion (even though a gradient-based approach allows for more complex models in a natural way).

Original: The data is generated by trigonometric polynomials of degree N as:
Revised: The data is generated by trigonometric polynomials of degree N as:

Original: [Equation]
Revised: [Equation]

Original: with x , ε , α , β ∈ R D , coefﬁcients α in , β in ∼ N (0 , 1) .
Revised: with x , ε , α , β ∈ R D , coefﬁcients α in , β in ∼ N (0 , 1) .

Original: A noise term ε it ∼ N (0 , 0 . 01) is added for numerical stability when computing the closed form solution.
Revised: A noise term ε it ∼ N (0 , 0 . 01) is added to avoid numerical instabilities in the implementation of closed-form SFA, as singular covariance matrices can cause the underlying eigendecomposition to break.

Original: We implemented a temporal step-size of 2 π 100 , and generate T = 5000 steps with maximum degree
Revised: We implemented a temporal step-size of 2 π 100 , and generate T = 5000 steps with maximum degree

Original: N = 30 and dimension D = 100 .
Revised: N = 30 and dimension D = 100 .

Original: The data is whitened with N iter = 60 .
Revised: The data is whitened with N iter = 60 .

Original: Figure 1 shows the extracted features for different variants of SFA.
Revised: Figure 1 shows the extracted features for different variants of SFA.

Original: If the slowness loss is optimized without any constraints on the output features, themodel ﬁnds the optimal solution  to collapse all signals to a constant ( ∆ = 0 ), while if unit variance is enforced only the features with slowness very close to the smallest ∆ are extracted multiple times and thus the representation becomes highly redundant.
Revised: If the slowness loss is optimized without any constraints on the output features, the optimal solution is to collapse all signals to a constant ( ∆ = 0 ), while if unit variance is enforced only the features with slowness very close to the smallest ∆ are extracted multiple times and thus the representation becomes highly redundant.

Original: When using the approximate whitening, the quality of the solutions if comparable to the closed form solution gained by solving a generalized eigenvalue problem.
Revised: When using the approximate whitening, the quality of the solutions is comparable to the closed form solution gained by solving a generalized eigenvalue problem.

Original: The ordering was achieved for representational purposes by weighting the output features with a monotonically decreasing weight.
Revised: The ordering was achieved for representational purposes by weighting the output features with a monotonically decreasing weight.

Original: In Figure 2, the covariance matrices of the extracted signals are visualized.
Revised: In Figure 2, the covariance matrices of the extracted signals are visualized.

Original: Note that it is not a sensible approach to use PowerSFA to optimize a linear model for such lowdimensional data as the closed-form solution is easily attainable.
Revised: Note that it is not a sensible approach to use PowerSFA to optimize a linear model for such lowdimensional data as the closed-form solution is easily attainable.

Original: For this reason, this experiment should be understood as a proof-of-concept and a general demonstration of applicability rather than a recommendation for a use-case.
Revised: For this reason, this experiment should be understood as a proof-of-concept and a general demonstration of applicability rather than a recommendation for a use-case.

Revised: A very low number of iterations will render the optimization unstable resulting in fast output signals.

Revised: There seems to be no continuous trade-off between correlation and slowness mediated by the number of iterations.

Revised: Thus, it is recommended to ﬁnd a setting for N iter that allows for stable optimization.

Revised: Appendix A contains a small hyper-parameter study, illustrating that behavior for the linear case.

Original: [Figure]
Revised: To provide evidence on how gradient-based SFA can improve on solutions found by closed-form SFA, we encapsulate the original signal in a non-linear distortion: [Equation] This makes it impossible to extract slow signals by linearly unmixing the original components.

Revised: When applying closed-form SFA to non-linear problems, it is common to apply multiple low-degree polynomial expansion (e.g. quadratic) interlaced with linear closed-form SFA (or, more generally, a dimensionality reduction) in a greedy layer-wise training.

Revised: To make the extraction more difﬁcult, the maximum degree was increased to N = 300 and the temporal step-size was decreased to 2 π 10 5 to reduce temporal aliasing of very fast input components.

Revised: Consequently, we increased T to 10 5 and

Revised: N iter to 100 .

Revised: We use an architecture with three expansion layers (including a normalization to unit norm to avoid exploding gradients), and then apply (a) closed-form training and (b) gradient-based training.

Revised: Additionally, we compare the training methods on a multi-layer perceptron with tanh activation, since polynomial expansion functions are uncommon in gradient-trained models.

Revised: Both architectures are provided in Appendix B. Table 1 shows the average output slowness, once for the raw output from the network (with the possibility of very small correlations in the features) and once with closedform whitening applied to precisely enforce the constraints for a more meaningful comparison

Revised: [Table]

Revised: Quadratic expansion

Revised: [Table]

Revised: The results show that gradient-based training can improve slowness in multi-layer architectures compared to greedy layer-wise SFA.

Revised: Note that increasing the output dimension of the dimensionality reduction steps (or dropping them altogether) in the quadratic expansion network will, unsurprisingly, lead to improved performance and ultimately to convergence to similar minima in both networks.

Revised: However, this performance increase comes at the cost of high memory requirements and is usually not applicable in high-dimensional non-synthetic problems.

Original:  SFA can be used to extract slowly varying underlying causes from high-dimensional data, such as object position, identity, and rotation from visual simulations.
Revised: [Figure] SFA can be used to extract slowly varying underlying causes from high-dimensional data, such as object position, identity, and rotation from visual simulations.

Original: In Franzius et al.
Revised: In Franzius et al.

Original: (2011), textured three-dimensional ﬁsh-objects that change in x - and y -position as well as in-depth rotation φ have been used and it has been shown that HSFA features encode position and angle well.
Revised: (2011), textured three-dimensional ﬁsh-objects that change in x - and y -position as well as in-depth rotation φ have been used and it has been shown that HSFA features encode position and angle well.

Original: Since the code used to generate the stimulus data was outdated and could not be executed anymore, we reimplemented a similar scenario with a 3D model of an anvil (Figure 3, Pino4et (2016)) using the random walk procedure described in Franzius et al.
Revised: Since the code used to generate the stimulus data was outdated and could not be executed anymore, we reimplemented a similar scenario with a 3D model of an anvil (Figure 3, Pino4et (2016)) using the random walk procedure described in Franzius et al.

Original: (2011) to generate the conﬁgurations.
Revised: (2011) to generate the conﬁgurations.

Original: A current limitation of our model (cf.
Revised: A current limitation of our model (cf.

Original: section 7) is that it does not scale well in the number of output features, rendering the relatively high number of 512 output features used by Franzius et al.[Figure]
Revised: section 7) is that it does not scale well in the number of output features, rendering the relatively high number of 512 output features used by Franzius et al.

Original: (2011) infeasible.
Revised: (2011) infeasible.

Original: We used an output dimension of 10 with a comparable architecture that replaces  quadratic expansion by ELU-nonlinearities (a Keras -summary is given in Appendix A).
Revised: We used an output dimension of 10 with a comparable architecture that replaces [Figure] quadratic expansion by ELU-nonlinearities (a Keras -summary is given in Appendix C).

Original: Due to the low number of output features, we were not able to successfully extract the rotational angle Φ .
Revised: Due to the low number of output features, we were not able to successfully extract the rotational angle Φ .

Original: In line with the original publication, we computed linear regression to predict cos ( πx ) and cos ( πy ) and used the inverse transformation to extract the object position.
Revised: In line with the original publication, we computed linear regression to predict cos ( πx ) and cos ( πy ) and used the inverse transformation to extract the object position.

Original: The results are shown in Figure 4 and are comparable to those presented by Franzius et al.
Revised: The results are shown in Figure 4 and are comparable to those presented by Franzius et al.

Original: (2011) as they were able to achieve
Revised: (2011) as they were able to achieve

Original: a
Revised: a

Original: RMSE of 9% for x -position and 7% for y -position.
Revised: RMSE of 9% for x -position and 7% for y -position.

Original: 6. NORB
Revised: 6. NORB

Original: While gradient-based SFA is the main contribution of this work, previous work on generalizing SFA (Sprekeler (2011)) has shown that the SFA optimization problem is strongly related to a special case of the problem solved by a more general spectral embedding method, i.e., Laplacian Eigenmaps (Belkin & Niyogi (2003)), or, with small differences, graph-based SFA (Escalante-B. & Wiskott (2013)).
Revised: While gradient-based SFA is the main contribution of this work, previous work on generalizing SFA (Sprekeler (2011)) has shown that the SFA optimization problem is strongly related to a special case of the problem solved by a more general spectral embedding method, i.e., Laplacian Eigenmaps (Belkin & Niyogi (2003)), or, with small differences, graph-based SFA (Escalante-B. & Wiskott (2013)).

Original: For this reason, we deﬁned a more general loss function in equation 2.
Revised: For this reason, we deﬁned a more general loss function in equation 2.

Original: In fact, weights w ij can be deﬁned between any two points x i and x j of a given dataset, not just consecutive ones, thus allowing to optimize neighborhood-respecting embeddings for general graphs.
Revised: In fact, weights w ij can be deﬁned between any two points x i and x j of a given dataset, not just consecutive ones, thus allowing to optimize neighborhood-respecting embeddings for general graphs.

Original: This is similar (but not fully equivalent) to spectral embeddings on graph data, as used in algorithms such as Laplacian eigenmaps.
Revised: This is similar (but not fully equivalent) to spectral embeddings on graph data, as used in algorithms such as Laplacian eigenmaps.

Original: Our approach exhibits four signiﬁcant differences:
Revised: Our approach exhibits four signiﬁcant differences:

Original: It does not ﬁnd ordered features, but a rotation of an optimal embedding, 2.
Revised: It does not ﬁnd ordered features, but a rotation of an optimal embedding, 2.

Original: the found solution should be assumed to be only locally optimal (or a saddle point), 3.
Revised: the found solution should be assumed to be only locally optimal (or a saddle point), 3.

Original: it does ignore the graph (ir)regularity when enforcing the orthogonality constraint, and 4.
Revised: it does ignore the graph (ir)regularity when enforcing the orthogonality constraint, and 4.

Original: it allows a natural and scalable way to embed unseen points.
Revised: it allows a natural and scalable way to embed unseen points.

Original: Computing an out-of-sample embedding requires the same forward-pass through the differentiable architecture as the training embeddings.
Revised: Computing an out-of-sample embedding requires the same forward-pass through the differentiable architecture as the training embeddings.

Original: In particular, its complexity does not scale with the number of observed points used for training as is usually the case in other approximations, such as Nyström approximation Williams & Seeger (2001).
Revised: In particular, its complexity does not scale with the number of observed points used for training as is usually the case in other approximations, such as Nyström approximation Williams & Seeger (2001).

Original: We demonstrate the usefulness of such an approach by embedding an object of the NORB dataset, a collection of photographs of toys taken at different elevations, azimuths, and under different lighting conditions with the MobileNet architecture (Sandler et al.
Revised: We demonstrate the usefulness of such an approach by embedding an object of the NORB dataset, a collection of photographs of toys taken at different elevations, azimuths, and under different lighting conditions with the MobileNet architecture (Sandler et al.

Original: scaled with α = 0 .
Revised: scaled with α = 0 .

Revised: 5 and a depth multiplier of 2 in the Keras implementation.

Original: Following Hadsell et al.
Revised: Following Hadsell et al.

Original: (2006), we embedded photographs of a toy plane (Figure 5) in 972 conﬁgurations ( azimuths × 6 lighting conditions × 9 elevations angles) that were randomly split into a train- and test-set of sizes 660 and 312 respectively and the connection weights w ij were chosen as 1 if x i and x j differed only in one step either in rotation (i.e., one azimuth) or elevation (i.e., one level) andotherwise.
Revised: (2006), we embedded photographs of a toy plane (Figure 5) in 972 conﬁgurations (18 azimuths × 6 lighting conditions × 9 elevations angles) that were randomly split into a train- and test-set of sizes 660 and 312 respectively and the connection weights w ij were chosen as 1 if x i and x j differed only in one step either in rotation (i.e., one azimuth) or elevation (i.e., one level) and 0 otherwise.

Original: The weights were independent of lighting condition.
Revised: The weights were independent of lighting condition.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figures 6 and 7 show the three-dimensional embedding that was found in this setting.
Revised: Figures 6 and 7 show the three-dimensional embedding that was found in this setting.

Original: The data was embedded in a cylindrical shape in which the circumference encodes the rotation angle of the embedded object and the length along the cylinder encodes the elevation conﬁguration of the object for the train-set and in the out-of-sample case of the test-set.
Revised: The data was embedded in a cylindrical shape in which the circumference encodes the rotation angle of the embedded object and the length along the cylinder encodes the elevation conﬁguration of the object for the train-set and in the out-of-sample case of the test-set.

Original: Hadsell et al.
Revised: Hadsell et al.

Original: (2006) found a similar cylindrical encoding, however, their results exhibit a more clean-cut embedding.
Revised: (2006) found a similar cylindrical encoding, however, their results exhibit a more clean-cut embedding.

Original: We assume that this is due to DrLIM’s maximization of distance for dissimilar samples that our model does not implement.
Revised: We assume that this is due to DrLIM’s maximization of distance for dissimilar samples that our model does not implement.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We propose a new way of extracting informative slow features from quickly varying inputs based on differentiable whitening of processed batches of the input data.
Revised: We propose a new way of extracting informative slow features from quickly varying inputs based on differentiable whitening of processed batches of the input data.

Original: To experimentally show the feasibility of the method, we trained a linear model to extract slowly varying output signals from synthetic time-series by gradient-descent and showed that the differentiable whitening ensures informative- ness of the extracted features when optimizing a slowness loss function.
Revised: To experimentally show the feasibility of the method, we trained a linear model to extract slowly varying output signals from synthetic time-series by gradient-descent, and ﬁnd that the differentiable whitening ensures informativeness of the extracted features when optimizing a slowness loss function.

Original: Furthermore, the features corresponded to those found by closed-form SFA.
Revised: Furthermore, the features corresponded to those found by closed-form SFA.

Original: To show applicability to visual time-series data, we trained a convolutional neural network on an input stream of an object randomly rotating in-depth and moving in a 2-dimensional plane.
Revised: To show applicability to visual time-series data, we trained a convolutional neural network on an input stream of an object randomly rotating in-depth and moving in a 2-dimensional plane.

Original: Gradientbased SFA preserves the position of the object in a low-dimensional representation as does a hierarchical version of closed-form SFA.
Revised: Gradientbased SFA preserves the position of the object in a low-dimensional representation as does a hierarchical version of closed-form SFA.

Original: The rotation was not extracted successfully due to computational constraints in the naive implementation for a large number of output features (cf. 7).
Revised: The rotation was not extracted successfully due to computational constraints in the naive implementation for a large number of output features (cf. 7).

Original: In an experiment  on a non-time-series image dataset on which symmetric similarity relations between the data can be deﬁned, we show that a generalization of gradient-based SFA in the spirit of graph-based SFA is able to ﬁnd a low-dimensional representation that preserves and disentangles the conﬁguration parameters used to deﬁne the similarity, in this case, azimuth and elevation of a photographed toy.
Revised: One experiment was conducted on a non-time-series image dataset on which symmetric similarity relations between the data are deﬁned. We show that a generalization of gradient-based SFA in the spirit of graph-based SFA is able to extract a low-dimensional representation that preserves and disentangles the conﬁguration parameters used to deﬁne the similarity, in this case, azimuth and elevation of a photographed toy.

Original: This representation generalizes well to previously unseen conﬁgurations of the object.
Revised: This representation generalizes well to previously unseen conﬁgurations of the object.

Original: While the algorithm is still in a prototypical state, the proof-of-concept results presented in this paper show promise for gradient-based SFA by differentiable whitening to extract meaningful representations for goal-oriented learning while leveraging the expressive power of modern architectures, such as convolutional neural networks.
Revised: While the algorithm is still in a prototypical state, the proof-of-concept results presented in this paper show promise for gradient-based SFA by differentiable whitening to extract meaningful representations for goal-oriented learning while leveraging the expressive power of modern architectures, such as convolutional neural networks.

Original: In particular, differentiable whitening ensures non-redundancy of the output features.
Revised: In particular, differentiable whitening ensures non-redundancy of the output features.

Original: More research has to be dedicated to explore the computational limitations of this method and possibly lower the complexity for a larger number of output features.
Revised: More research has to be dedicated to explore the computational limitations of this method and possibly lower the complexity for a larger number of output features.

Original: Furthermore, closed-form SFA has a strong theoretical framework describing the optimal responses in an idealized setting.
Revised: Furthermore, closed-form SFA has a strong theoretical framework describing the optimal responses in an idealized setting.

Original: At this point, we are unclear how well this framework translates to the proposed method of slowness extraction as the used models typically suffer from local optima when being iteratively trained.
Revised: At this point, we are unclear how well this framework translates to the proposed method of slowness extraction as the used models typically suffer from local optima when being iteratively trained.

Original: One limitation of PowerSFA is that it currently does not scale favorably in the number of output features e .
Revised: One limitation of PowerSFA is that it currently does not scale favorably in the number of output features e .

Original: We see two main reasons for this: the necessary batch size to get a meaningful estimate of the batch-covariance estimate and its calculation.
Revised: We see two main reasons for this: the necessary batch size to get a meaningful estimate of the batch-covariance estimate and its calculation.

Original: The latter is due to the complexity of a naive R e × N batch
Revised: The latter is due to the complexity of a naive R e × N batch

Original: R N batch × e matrix-multiplication being O
Revised: R N batch × e matrix-multiplication being O

Original: ( N batch e 2 ) , while the former is due to the whitening procedure expecting a covariance matrix of full rank and thus N batch ≥ e samples.
Revised: ( N batch e 2 ) , while the former is due to the whitening procedure expecting a covariance matrix of full rank and thus N batch ≥ e samples.

Original: To reduce the lower bound on the batch-size at batch t a convex mixture with the covariance matrix of the previous batch might be applicable:
Revised: To reduce the lower bound on the batch-size at batch t a convex mixture with the covariance matrix of the previous batch might be applicable:

Original: [Equation]
Revised: [Equation]

Original: Note, that only the current batch’s covariance matrix C θ is considered parameter-dependent and allows to propagate a gradient for training.
Revised: Note, that only the current batch’s covariance matrix C θ is considered parameter-dependent and allows to propagate a gradient for training.

Original: Thus, large values for γ might cause a signiﬁcant bias in the gradient-estimate.
Revised: Thus, large values for γ might cause a signiﬁcant bias in the gradient-estimate.

Original: This has not been implemented in the proof-of-concept algorithm for this paper.
Revised: This has not been used to generate the proof-of-concept results for this paper

Original: Appendices A K
Revised: Appendices A S MALL PARAMETER - STUDY : NUMBER OF POWER ITERATIONS

Original: ERAS

Original: D ESCRIPTION (HSFA- LIKE A RCHITECTURE )
Revised: B C OMPARISON ARCHITECTURES ( SYNTHETIC DATA )

Original: [Table]
Revised: This appendix presents the architectures used for comparing greedy layer-wise with gradient-based end-to-end training of SFA.

Original: _________________________________________________________________ conv2d [Table] [Table] [Table]
Revised: Both networks have been trained with both training methods.

Original: _________________________________________________________________ conv2d
Revised: In the gradient-based setting, the Keras ’s Nadam optimizer has been used with default hyperparameters and a whitening layer has been applied to the network output.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] 102432_________________________________________________________________ activation_
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (None [Table] [Table] [Table]{+___________________________________ activation_2(Activation) (None,28,28,32) 
+}
Revised: {+___________________________________ activation_2(Activation) (None,28,28,32) 
+}

Original: [Table] (None [Table] [Table]{+_________________________________________________________________ dropout_2(Dropout) (None,28,28,32) 
+}
Revised: {+_________________________________________________________________ dropout_2(Dropout) (None,28,28,32) 
+}

Original: (None [Table] 16416_________________________________________________________________ activation_ [Table]{+_________________________________________________________________ conv2d_3(Conv2D) (None,25,25,32) 16416 _________________________________________________________________ activation_3 (Activation) (None, 25, 25, 32) 
+}
Revised: {+_________________________________________________________________ conv2d_3(Conv2D) (None,25,25,32) 16416 _________________________________________________________________ activation_3 (Activation) (None, 25, 25, 32) 
+}

Original: (None [Table] [Table] [Table]{+_________________________________________________________________ dropout_3(Dropout) (None,25,25,32) 
+}
Revised: {+_________________________________________________________________ dropout_3(Dropout) (None,25,25,32) 
+}

Original: [Table] [Table] (None [Table]{+_________________________________________________________________ conv2d_4(Conv2D) (None,11,11,32) 
+}
Revised: {+_________________________________________________________________ conv2d_4(Conv2D) (None,11,11,32) 
+}

Original: [Table] (None [Table] 16416_________________________________________________________________ activation_4(Activation) (None,11,11,32) 

Revised: _________________________________________________________________ activation_4(Activation) (None,11,11,32)

Original: _________________________________________________________________ dropout_4(Dropout) (None,11,11,32) 

Revised: _________________________________________________________________ dropout_4(Dropout) (None,11,11,32)

Original: _________________________________________________________________ conv2d_5(Conv2D) (None,8,8,32) 16416_________________________________________________________________ activation_5(Activation) (None,8,8,32) 

Revised: _________________________________________________________________ conv2d_5(Conv2D) (None,8,8,32) 16416_________________________________________________________________ activation_5(Activation) (None,8,8,32)

Original: _________________________________________________________________ dropout_5(Dropout) (None,8,8,32) 

Revised: _________________________________________________________________ dropout_5(Dropout) (None,8,8,32)

Original: _________________________________________________________________ conv2d_6(Conv2D) (None,3,3,32) 16416_________________________________________________________________ activation_6(Activation) (None,3,3,32) 

Revised: _________________________________________________________________ conv2d_6(Conv2D) (None,3,3,32) 16416_________________________________________________________________ activation_6(Activation) (None,3,3,32)

Original: _________________________________________________________________ dropout_6(Dropout) (None,3,3,32) 

Revised: _________________________________________________________________ dropout_6(Dropout) (None,3,3,32)

Original: _________________________________________________________________ flatten_ [Table] Flatten) (None, 288) 

Revised: _________________________________________________________________ flatten_1(Flatten) (None,288)

Original: _________________________________________________________________ dense_1(Dense) (None,10) 2890_________________________________________________________________ power_whitening_1(PowerWhit(None,10) 0================================================================={+2890 _________________________________________________________________ power_whitening_1 (PowerWhit (None, 10) 
 == == ==+}
Revised: _________________________________________________________________ dense_1(Dense) (None,10) {+2890 _________________________________________________________________ power_whitening_1 (PowerWhit (None, 10) 
 == == ==+}

Original: Totalparams:174,218Trainableparams:174,
Revised: == == == ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ==

Revised: ===

Revised: Totalparams:174,218Trainableparams:174,

Original: Non-trainableparams:0_________________________________________________________________
Revised: Non-trainableparams:0_________________________________________________________________