Biological Learning of Irreducible Representations of Commuting Transformations
Anonymous Author(s)
AffiliationAddress
Abstract
A challenge in neuroscience is to understand the neural mechanisms underlyingthe brain’s remarkable ability to learn and detect transformations of objects.
Cohenand Welling proposed a mathematical framework for learning efficient representations of transformations based on commutative Lie groups.
In particular, theyused a decomposition of a commutative Lie group into 2-dimensional irreduciblerepresentations.
We explore the possibility that the brain uses a similar learningparadigm.
Specifically, we propose bio-inspired mechanisms for learning theseirreducible representations.
We derive two algorithms for learning commutativegroups from sequences of transformed images.
The two approaches differ in theiralgorithmic foundations — one is based on SVD of the anti-symmetrized outerproduct of consecutive frames and the other based on PCA of the difference between consecutive frames.
The two algorithms lead to different neural networkrealizations — the former uses a one layer network while the latter uses a twolayer network.
Both networks work in an online setting and are able to recover theresults from Cohen and Welling, which focus on rotations, as well as learn moregeneric transformation groups.
Our circuits suggest patterns that can be searchedfor in nascent connectomics and physiology datasets.
Introduction
The brains of humans and other animals effortlessly learn and detect transformations of stimuli due totheir physical movement and changes of perspective.
For example, we can identify faces regardlessof their location and orientation in the visual field.
Similarly, flies, to maintain stable flight, detectrelative translational and rotational motion of their surroundings.
Thus, an important problem inneuroscience is to understand how the brain decomposes complex transformations into elementaryones, e.g., translations and rotations and measures their speed or magnitude.
An influential approach to motion detection and learning is due to Rao and Ruderman [10] and isbuilt around the idea of infinitesimal transformations, also known as Lie group generators.
Thisapproach separates the transformation identity (e.g., rotation around a given center, translation in agiven direction) from its velocity.
One can imagine a neural mechanism implementing a generator, which then will be able to detect a transformation of this kind and measure its velocity [1]
While a single generator describes a one parameter subgroup of transformations
Cohen and Welling
[4] considered multi-parameter groups, which in the case of rotations are so-called toroidal subgroups
Toroidal groups arise from decomposing every rotation into ‘irreducible representations’, which are
2D rotations that commute.
Irreducible components can then be combined to describe more complex rotations.
Such decomposition was earlier used by Bethge et al.
[2] albeit without accounting for varying velocity.
The ideas of irreducible representations and toroidal groups jointly open a theoretical possibility for neural mechanisms implementing a larger class of transformations consistently
Irreducible representations also offer another advantage for neural implementation compared to anapproach based on individual generators.
Mathematically, a generator is a matrix where each entrycorresponds to a pair of pixels of the input image or patch; in a neural implementation, this wouldcorrespond to a synapse multiplying the inputs from such a pair, one of them with a delay.
Also, aneuron implementing a generator would be required to have a number of multiplicative units thatscales quadratically with the number of pixels in the patch, which can be prohibitive for efficient datadriven learning.
(For the case of translations in a 1D image the latter issue was addressed in [1]).
When using the irreducible-representation approach, we will show that the need for multiplicativeunits does not arise, and only a linear number of synaptic connections will be needed per neuron.
An algorithm for learning toroidal groups from sequences of transformed images was developedby Cohen and Welling [4], but without a focus on biological plausibility.
Here, we propose twobiological solutions to learning toroidal groups and their irreducible representations.
Using the setupfrom [4], these neural networks receive pairs of transformed images, extract 2D subspaces, and outputthe estimate of the velocity of rotations within each subspace.
Both implementations operate inthe online setting, therefore satisfying a basic constraint of biological systems.
The first proposalis based on averaging of bivectors [2] and uses singular value decomposition (SVD) to extract thesubspace of maximal average rotation.
We derive a novel algorithm for online SVD: cross-poweriteration.
The network finds subsequent rotation subspaces using deflation.
The deflation part requiresconnections between neurons that are not biologically plausible.
Our second proposal, based on PCAof time difference vectors, uses local learning rules, which makes it biologically plausible [3].
Thisapproach introduces a two-layer network that decomposes the transformations into their irreducible2D rotations in the first layer, and outputs the angle of each 2D rotation in the second layer.
Havingsuch concrete circuit proposals should allow searching for them in the nascent connectomics andphysiology datasets.
Setup
[Figure]
: Pairs of consecutive vectorized video frames ( x t − 1 , x t ) .
Consider a stream of consecutive video frames (Fig. 1) vectorized into d -dimensional pixel-intensity vectors { x t } satisfyinglinear relations
[Equation]
Provided the time-step is small, the norm of x t is approximately constant between consecutive frames, so we can model
A t as an orthogonal matrix.
In this case, we can decompose
A t into independent rotations within orthogonal 2D subspaces,as follows
[Equation]
Here Q t is a d × d orthogonal matrix, Γ t is a d × d block-diagonal matrix whose blocks are 2 × 2rotation matrices indexed by θ t, 1 , . .
. , θ t,k , and we have assumed, for simplicity, that d is even andk := d/ 2 .
Pairs of consecutive column vectors ( q t, 2 i − 1 , q t, 2 i ) of Q t define orthogonal 2D subspacesand θ t,i represents the angle of rotation in that 2D subspace.
As desired, this canonically separatesthe transformation identity (the 2D subspace) from the velocity (the angle of rotation).
Following Cohen and Welling [4], we consider the problem of learning a representation for thek -dimensional commutative subgroup of rotations of the form A t = QΓ t Q ⊤ , where Q is a fixedorthogonal matrix and Γ t is any block-diagonal matrix of the form in equation (2).
The subgroupis commonly referred to as a toroidal subgroup because it is parameterized by the k -dimensionaltorus { ( θ 1 , . .
, θ k ) : θ j ∈ [0 , 2 π ) } .
With this simplification, the k mutually orthogonal 2D subspacesconstitute an irreducible representation of the subgroup [4].
The focus of this work is to derive a neurally plausible algorithm for (A) learning the irreduciblerepresentation from a stream of pairs of vectorized video frames { ( x t − 1 , x t ) } , and (B) outputting theangle of rotation θ t,i within each 2D subspace.
Learning the irreducible representation from data
We consider learning the irreducible representations from data in the offline setting using 2 approaches.
The first approach is based on an SVD of the average bivector between consecutive frames.
Thesecond appraoch is based on PCA of the covariance of the difference between consecutive frames.
The analysis here will be the starting point for the derivations of the online algorithms.
Throughoutthis section, we assume the sequence { x t } is centered and whitened so that ⟨ x t x ⊤ t ⟩ t = I d .
Inaddition, we assume that A t is independent of x t .
3.1 Learning on SVD of bivectors of consecutive frames
Given a pair of consecutive frames ( x t − 1 , x t ) , define the bivector
[Equation]
where we have used the relation x t =
A t x t − 1 .
The bivector is an anti-symmetric matrix, which isalso commonly referred to as the ‘wedge product’ or ‘exterior product’.
Under our assumptions thatx t and A t are independent and x t is whitened, we have
[Equation] [Equation]
where
[Equation]
Note that g i can be zero even if θ t,i is not identically zero.
For example, because sine is an oddfunction, if the distribution of θ t,i is symmetric about zero, then g i = 0 .
Therefore, to use thisapproach, we assume that g i ̸
= 0 for i = 1 , . . .
, k .
To learn the column vectors of Q , it will be convenient to write the SVD of B in terms of Q .
To thisend, we assume, without loss of generality, that g 1 ≥ · · · ≥ g k ≥ 0 .
Then we have the followinglemma, whose proof is deferred to the appendix.
Lemma 1.
The SVD of B can be written as B = UΣV ⊤ , where
U := Q , Σ :=
diag( g 1 , g 1 , . .
. , g k , g k ) ,
V :=
[ V 1 , . .
. , V d ] , and ( V 2 i − 1 , V 2 i ) =
( Q 2 i , − Q 2 i − 1 ) for i = 1 , . . .
, k .
In other words, each pair of left- and right-singular vectors of B , ( U i , V i ) , spans one of the 2Dmutually orthogonal subspaces.
Therefore, to learn the irreducible representation, it suffices to learnpairs of left- and right-singular vectors of B .
Note that the first pair of left- and right-singular vectors span the same subspace as the second pair ofleft- and right-singular vectors (assuming g 1 > g 2 ), and same with the third and fourth pairs, andso on.
Therefore, with this approach, care needs to be taken to ensure one is learning distinct 2Dsubspaces.
3.2 Learning on PCA of differences between consecutive frames
We now present an alternative method for learning the column vectors of Q , by performing PCA onthe covariance of the discrete time derivatives defined by ˙ x t := x t − x t − 1 .
Under our assumptionsthat A t is independent of x t and the vectors x t are whitened, we have
[Equation]
(5) where ˆ Γ = diag ( γ 1 , γ 1 , . .
. , γ k , γ k ) and γ i := 2 − 2 ⟨ cos θ t,i ⟩ t .
Importantly, the column vectors of
Q are eigenvectors of ⟨ ˙ x t ˙ x ⊤ t ⟩ t with associated eigenvalues γ i and, assuming that γ 1 > · · · > γ k ,each γ i corresponds to a unique 2D subspace
Therefore the transformations with the largest anglescan be found by performing PCA on ⟨ ˙ x t ˙ x ⊤ t ⟩ t γ i
Biological algorithms based on SVD of the average bivector
To compute the SVD of B , we consider the following optimization problem:
[Equation] [Equation]
whose solution is the matrices of left- and right-singular vectors of B .
4.1 Finding the subspace with largest rotation velocity
We start with an optimization problem for finding the top pair of singular vectors:
[Equation]
To optimize in the online setting, we perform projected stochastic gradient descent, which we refer toas cross-power iteration :
[Equation]
where η t > 0 is the learning rate and we have used the anti-symmetric property B ⊤ = − B .
Normalization steps are hard to map onto a biological circuit, so we follow the idea from Oja’salgorithm [7]: assuming small enough learning rates, update equations can be expanded using a
Taylor series in η t .
In this case, the iteration takes the form:
[Equation]
Convergence proof for this algorithm is given in the appendix.
Since the matrix B is antisymmetric, the singular vectors resulting from this optimization, u andv , are orthogonal, and define the 2D subspace with the largest average sine of the angle of rotation.
Now each input vector x t can be projected on the subspace, and we denote the projected coordinatesas a t := u ⊤ x t , b t :=
v ⊤ x t .
Using the definition of B t from Eq.
(3) and substituting a t , b t into theobjective:
[Equation]
where y t := a t b t − 1 − a t − 1 b t .
This suggests two linear neurons with common input and weights u and v generating output streams a t and b t , followed by a circuit known as a Hassenstein-Reichardt detector [11]; see Fig. 2(a).
Now the angle of rotation θ t can be recovered:
[Equation]
4.2 Full dimension: finding all subspaces
Subsequent singular vector pairs can be found by deflation.
Recall the identities for anti-symmetric
SVD: BV i =
σ i
[Equation]
, BU
[Equation] [Equation]
σ
[Equation]
=
[Equation] [Equation]
= g i as 141 in Lemma 1.
We can thus rewrite the SVD using odd indexes only:
[Equation]
Then U j , V j for j odd, which is all we need, can be found as solution of an optimization problemsimilar to Eq.
(7), but for the “deflated” matrix:
[Equation]
A straightforward deflation algorithm would find singular vector pairs one by one, correspondingto decreasing singular values.
Our goal however is to find all of them in parallel, in an online andneurally plausible manner.
The idea is, on every time step, to perform one step of the iterations from
Eqns.
(9a) and (9b) in a loop on sequentially deflated matrices, using current estimates of singularvectors.
This yields Algorithm 1:
[Figure]
Algorithm for i = 1 to k do u i , v i ← random, ∥ u i ∥ =
∥ v i ∥ = 1 {Initialize} end for for t = 1 , 2 , . . .
do
D ← 0 {current deflation estimate} for i = 1 to k do u i ←
[Equation]
←
[Equation]
D ← D
[Equation]
u
[Equation]
u
[Equation]
v
[Equation]
end for end for
The algorithm bears similarity to Generalized Hebbian Algorithm (GHA) for PCA [12], which is alsobased on deflation.
Proof of convergence can be found in the Appendix.
4.3 Biological SVD
The circuit in Fig. 2(a) above is not realizable inside a neuron because of the crossing pathways.
Italso cannot be realized with two neurons, as the original Hassenstein-Reichardt detector, becausethen the synaptic weight updates in one neuron would require information only available in the otherneuron, violating the principle of locality.
To overcome these difficulties we propose a modificationof the bivector expression:
[Equation]
(13) which is similar to what was suggested by Bahroun et al.
Individual matrices are no longeranti-symmetric; however, the expectation is anti-symmetric, and is the same as before under ourassumptions.
Projected on the 2D subspace of interest are now the difference and sum of vectors:α t := u ⊤ ( x t − x t − 1 ) and β t := v ⊤ ( x t + x t − 1 ) .
Then substituting in the objective we can rewrite:u ⊤ ˜ B t v = α t β t .
This suggests a neuron with two dendritic branches both connecting to the same set of inputs,
Fig. 2(b).
The first has synapses with weights u that also calculate temporal differences (respondto time derivative); the second one has synapses with weights v that also integrate over short timeperiods.
The neuron performs division in the soma and outputs y t = α t /β t
= tan θ t , from which theangle can be read.
We assume the activity is suppressed when β t is too small for reliable calculation of the ratio.
That means the neuron is going to have a “blind spot” when ( x t + x t − 1 ) is close toorthogonal to v .
The neural dynamics are described by the equations:
[Equation]
14) and synaptic plasticity rules are derived from the cross-power iteration algorithm using a Taylorapproximation in the learning rate, as in Eqns.
(9a)–(9b):
[Equation]
Biological algorithm based on PCA of time differences
We construct a two-layer network for outputting the angles θ t,i .
The first layer of our network projectsthe inputs onto the columns of Q , and the second layer computes the angles.
We adopt the similaritymatching approach [8] to derive a network that projects the inputs onto the 2D subspaces defined bypairs of column vectors of Q .
The ‘usual’ similarity matching approach [9] projects the inputs ontothe principal subspace.
To ensure that the inputs are projected onto the correct 2D subspace, we adoptthe approach in [6] to construct the first layer of our two-layer network, which projects the inputsonto the 2D subspaces spanned by pairs of column vectors of Q associated with the same eigenvalue.
5.1 Derivation of the first layer
We assume the first layer of the network has d neurons whose activities at time t are encoded in thevector y t .
Define the data matrices X :=
[ x 1 , . .
, x T ] , Y :=
[ y 1 , . . .
, y T ] , ˙ X :=
[ ˙ x 1 , . .
, ˙ x T ] and˙ Y :
=
[ ˙ y 1 , . .
. , ˙ y T ] , where recall that the dot represents a discrete time derivative, e.g., ˙ x t = x t − x t − 1 .
Let ˙ X =
QΣP ⊤ denote the SVD of the data matrix.
We start with the PCA objective function [6]
[Equation]
16) where Λ = diag ( λ 1 , λ 1 , . . .
, λ k , λ k ) is any fixed diagonal matrix with λ 1 > · · · > λ k .
Suppose thatthe diagonal elements γ i of the matrix ˆ Γ (see Eq.
(16)) obey γ 1 > γ 2 > · · · > γ k .
Then the optimalsolutions to (16) are given by
[Equation]
where R is any block-diagonal matrix with 2 × 2 blocks that are orthogonal matrices, see [6, Lemma1].
In particular, the optimal solution is the projection of the data matrix X onto the 2D subspacesdefined by the pairs of column vectors of Q .
Directly optimizing (16) will not result in an online algorithm due to
sample covariance term1 T ˙ X ˙ Y ⊤ and 1 T ˙
Y ˙ Y ⊤ .
Therefore, to obtain an online algorithm, we encode the sample covariancesin feedforward and recurrent synaptic weight matrices W , M ∈ R d × d by substituting in with thefollowing Legendre transforms
[Equation]
to obtain
[Equation]
18) where
[Equation] [Equation]
After interchanging the order of optimization, we arrive at the objective
[Equation]
In the offline setting, we first optimize over ˙ Y and find that ˙
Y =
M − 1 W ˙ X , or equivalently,
Y =
M − 1 W .
We then take a gradient descent ascent step with respect to ( W , M ) :
[Equation]
In the online setting, at each time step t , the network receives the input x t and computes the outputy t by running the recurrent neural dynamics to equilibrium:
[Equation]
The network then computes ˙
y t :=
y t − y t − 1 and the synaptic weights are updated according to thelocal learning rules
[Equation]
5.2 Derivation of the second layer
The first layer of the network consists of d neurons and it transforms the inputs so that the 2Dsubspaces represented by pairs of neurons.
The second layer of the network consists of k neuronswith each neuron computing an angle θ i,t .
The i th neuron of the second layer receives inputs( y t − 1 , 2 i − 1 , y t, 2 i − 1 , y t − 1 , 2 i , y t, 2 i ) from the (2 i − 1) st and
i th neurons of the first layer, which satisfythe relation
[Equation]
Therefore,
[Equation]
and so
[Equation]
which we define to be the output of the i th neuron in the second layer.
initialize W , M positive definite, x 0 = 0 , y
=for t = 1 , 2 . . .
do y t =
M − 1 Wx t
W ← W + η (( y t − y t − 1 )
( x t − x t − 1 ) ⊤ − W )
M ← M + η (( y t − y t − 1 )( y t − y t − 1 ) ⊤ − ΛMΛ ) for i = 1 , . .
, k do θ t,i = arctan (cid:16) y
i − 1 ,t − 1 y 2 i,t − y
i,t − 1 y 2 i − 1 ,t y 2 i − 1 ,t − 1 y 2 i,t + y 2 i,t − 1 y 2 i − 1 ,t (cid:17) end for end for
Simulations
Simulations here are intended to demonstrate the ability of our algorithms to learn arbitrary toroidalgroups from data, i.e. recover 2D subspaces and estimate rotation angles, and also generate reasonablefilters.
For that purpose we created a toroidal group with three 2D subspaces in 10-dimensionalspace.
An orthogonal matrix Q was generated randomly, then rotations angles in each subspacewere generated with independent normal distributions with means 0 . 3 , 0 . 2 , 0 . 1 , and correspondingstandard deviations 0 . 4 , 0 . 3 , 0 . 2 .
These rotations were applied to vectors generated from a standardnormal distribution in 10 dimensions.
Simulation results are shown in Fig.
In total we generated10 6 samples and rotations that were fed to both algorithms.
Learning rates were manually selected tobe 5 · 10 − 4 for both algorithms.
This was repeated 10 times with random initialization, which took14 minutes total on a MacBook Pro with 3.5 GHz Dual-Core Intel Core i7 processor.
Subspace fitloss is evaluated as one minus cosine between the true and estimated planes.
Angle estimates aredepicted for the latest 1000 iterations and pooled randomly from all 10 runs.
[Figure]
Next we look at the filters generated by our algorithms, i.e., pairs of vectors found for each 2Dsubspace.
Rotations of images around the center is an example that both [2] and [4] looked at.
Weused random images of size 16 × 16 and rotated each by random angles around the center.
Rotationswere performed using bi-linear approximation, ignoring pixels outside the central circle.
We ran by
SVD algorithm on a total of 2 × 10 6 images and rotations with the learning rate equal to 2 ·
10 − 4 .
Figure 5(a) shows the 40 top filters obtained; corresponding pairs of vectors defining subspaces areon the upper and lower half of the chart.
The results are similar to those obtained from theory in[2], also to those obtained from random images in [4].
Running time was 40 min on MacBook Prowith 3.5 GHz Dual-Core Intel Core i7 processor.
Similar experiments on naturalistic images andtheoretical evaluation are presented in Appendix C.
[Figure]
Sub-pixel translations of 1D images can be approximated by cyclic shifts, which are rotations of theimage vector.
We generated 25 × 10 5 15-pixel 1D images from the standard normal distribution.
Top 4 filter pairs obtained by PCA algorithm are shown in Fig. 5(b); they look like Fourier pairs, asexpected, and power spectrum of each pair in Fig. 5(c) confirms that.
Execution took 5 min.
All code for these experiments is included in the Supplementary material.
Conclusion
In this work, we developed two biologically plausible neural algorithms that can learn toroidalgroups of transformations, detect transformations and measure their velocity.
Our approach is basedon irreducible representations of rotations, as used in [2; 4].
This allowed us to suggest neuralimplementations that avoid unrealistic multiplicative units encountered in the designs based on Liegroup generators.
Our first algorithm is based on SVD of the average bivector, derived from two images under transformation.
For that we have developed a novel online SVD algorithm.
The second algorithm isbased on PCA of the difference between vectorized images under transformation.
Arising two-layernetwork fully adheres to the principles of biological plausibility outlined in [3].
Thus, our workgenerates experimentally testable hypotheses that can be searched for in nascent connectomics andphysiological datasets from various species and brain regions.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section Simulations
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section Simulations
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section Simulations
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](
b
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](
c
) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Proof of Lemma 1
Define the d × d orthogonal matrix P by
[Equation]
Since P is orthogonal, we have
[Equation]
where U := Q , Σ :=
diag( g 1 , g 1 , . .
. , g k , g k ) and V :=
QP .
We assume, without loss of generality,that g 1 ≥ · · · ≥ g k ≥ 0 .
By definition, the (2 i − 1) st column vectors of U and V and the 2 i th columnvectors of U and V satisfy the relations( u 2 i − 1 , v 2 i − 1 ) =
( q 2 i − 1 , − q 2 i ) , ( u 2 i , v 2 i )
=
( q 2 i , q 2 i +1 ) , i = 1 , . .
, k.
B Convergence proof for cross-power iteration with deflation
Assume B = UΣV ⊤ is SVD, singular values in Σ are non-increasing, and B is full rank.
Singularvalues of anti-symmetric matrix come in pairs, but we will assume for simplicity that there are notwo equal pairs.
We prove the convergence by induction, similar to the proof for GHA [12], so westart from the top singular vector pair.
Differential equations for the iterations in Eqns.8 are
[Equation]
[Equation]
The variables u , v can be expanded in the column spaces of orthogonal matrices U and V : u =
Up , v = Vq .
Substituting into Eqn.27, obtain differential equations for the new variables:
[Equation]
[Equation]
For vector p we follow the evolution of the ratio of the component p 1 corresponding to the topsingular value σ 1 and some other p k such that σ k < σ 1 .
From the equations above:
[Equation] [Equation]
For every k -th component of vectors p , q we can calculate:
[Equation]
The solution is
[Equation]
and converges to 1 when t goes to infinity since σ k > 0 .
Using this, the expression in parenthesis in
Eq.30 converges to σ k − σ 1 , which is negative, so p k / p 1 goes to zero for every k where σ k < σ 1 .
Weare left to show that the norm of the vector is maintained at 1.
For that we follow the evolution ofz :=
u ⊤ u :
[Equation]
since we initialized z = 1 .
We conclude that u converges to U 1 , top left singular vector.
Similarly, vconverges to V 1 , top right singular vector.
Having the base for the induction, for the induction step i we assume that for all k <
i
the variablesu k , v k converge correspondingly to U 2 k − 1 , V 2 k − 1 , columns of U and V .
Now need to prove thatu i , v i converge to U 2 i − 1 , V 2 i − 1 .
Differential equations corresponding to Algorithm 1 are:
[Equation]
Substituting as before u i =
Up i , v i =
Vq i and multiplying from the left the first equation by U ⊤and the second – by V ⊤ gives:
[Equation] [Equation]
where J :=
U ⊤ V is a block-diagonal matrix with blocks: (cid:18)
−
[Equation]
The task now is to prove thatp i and q i converge to unit vector with 1 at 2 i − 1 -st component, the rest being zeros.
By the assumption of the induction step, p k , q k for k < i converge to corresponding unit vectors.
Then simple matrix calculation show, that we have the following convergence, as fast as the slowestof p k , q k :
[Equation]
where Γ is diagonal matrix, and on the diagonal values 1 occupy first 2( i − 1) positions, the rest arezeros.
Differential equations now take form:
[Equation]
[Equation]
These are somewhat similar to 28, 29, and we proceed similarly to observe:
[Equation] [Equation]
From the first of these equations we conclude that q ik / p
ik converges to 1 when k ≥
i − 1 , i.e. γ k = 0 .
Then for k > 2 i − 1 from the second equation:
[Equation] [Equation]
which converges to zero because of the ordering of singular values.
For k < 2 i − 1 we have:
[Equation] [Equation]
which converges to zero because singular values are positive.
As before, we can show that the normof vector p
i
C Experiments on naturalistic images and evaluation
We performed an experiment on natural images from the Van Hateren database [5].
Patches of size16 × 16 were randomly selected from image frames, then rotated each by random angles aroundthe center.
Rotations were performed using a bi-linear approximation, ignoring pixels outside thecentral circle.
SVD algorithms was performed on 10 6 patches, first for 40 top pairs of filters, then for20.
The resulting filter images are presented in Figure 6, (b) and (c), top row, along with results onrandom images in (a) (the latter is a duplicate of Figure 5(a), included here for visual comparison).
As expected, results on natural images look less clean that those on random images, but visuallysimilar.
Also, filters obtained when only 20 of them were sought for in the algorithm run, are betterthan top 20 from the run where 40 filters were sought.
[Figure]
Next we are providing quantitative evaluation of the obtained filters versus theoretical predictions.
The subtlety here is that the left and right singular vectors are not uniquely determined—there is adegeneracy due to the fact that there are multiple modes corresponding to the same singular value.
Indeed, the rotation operator in two dimensions has eigenvectors that factorize between a radial partg ( r ) , that is unconstrained, and an angular part which needs to have a sinusoidal shape of a singlefrequency commensurate with the 2 π periodicity of the angle θ :
[Equation]
where k ∈ Z .
[Equation]
The singular vectors that our algorithm learns are the real and imaginary parts of ψ .
To assess to what extent our filters are of the form (42), we need to separate the radial and angularparts.
To do so, we first convert them from Cartesian to polar coordinates, obtaining a matrix Ψwhere the row index corresponds to the radial direction and the column index corresponds to theangle.
The factorization from eq.
(42) implies that Ψ should be rank-1, and thus expressible as aproduct Ψ = R Θ of a purely radial component R and a purely angular component Θ .
To assess whether our filters approximate the expected factorization, we perform an SVD of Ψ andcalculate the fraction of the variance explained by the top component (that is, the ratio betweenthe square of the top singular value and the sum of the squares of all singular values).
A perfectfactorization as in eq.
(42) would lead to a fraction of explained variance equal to 1, corresponding toa rank-1 Ψ .
We show these fractions for the learned filters in the bottom row of Figure 6.
Finally, we check whether the angular component Θ indeed corresponds to a single sinusoid bytaking the Fourier transform of this angular part.
We plot the power spectrum obtained from the
Fourier transform in the bottom row of Figure 6.
A perfect filter would have a sharp peak at a singlefrequency in this spectrum.
Not surprisingly, the top filters, corresponding to higher singular values, have sharper spectra andhigher explained-variance numbers.
Overall, these results confirm what was observed by visualinspection of the filters: the quality of the filters from natural images is somewhat weaker than thosefrom random images, and using a smaller number of filters in the simulation improves the quality ofthe top filters.
D Reconstruction of transformed image using the network outputs
In this section, we verify that the learned representation of our algorithm is indeed informative ofthe transformation by performing reconstruction of the transformed image.
That is, given x t − 1 andthe output ˆ θ t of Algorithm 1 (with output dimension set to k = 20 so that we only recover a strictsubset of the filters), we want to reconstruct x t .
To this end, we train a decoder network on the samenaturalistic images training data that was used to train our algorithm in Appendix C. Specifically, wetrain an MLP with a single hidden layer of width 512.
We take the input to be the concatenation ofx t − 1 and the output ˆ θ t of Algorithm 1.
We then minimize the mean-squared error
[Equation]
= (cid:88)
[Equation]
Having trained this decoder network, we verify that the transformations are correctly learned.
Todo so, we apply the transformations A t to new images that were not seen during training (we use
MNIST images for easy visualization of the transformation) to obtain a sequence ˜ x t .
We then use thedecoder network to reconstruct the transformed images.
The results of this experiment can be seenin Fig.
Here, the top row represents f (˜ x t − 1 , ˆ θ t ) and the bottom row represents the ground truth˜ x t .
Even though we do not use the full set of filters, we