Original: Learning a Convolutional Bilinear Sparse Code for Natural Videos
Revised: Learning a Convolutional Bilinear Sparse Code for Natural Videos

Original: Anonymous Author(s

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In contrast to the monolithic deep architectures used in deep learning today forcomputer vision, the visual cortex processes retinal images via two functionallydistinct but interconnected networks: the ventral pathway for processing objectrelated information and the dorsal pathway for processing motion and transformations [8].
Revised: In contrast to the monolithic deep architectures used in deep learning today forcomputer vision, the visual cortex processes retinal images via two functionallydistinct but interconnected networks: the ventral pathway for processing objectrelated information and the dorsal pathway for processing motion and transformations [8].

Original: Inspired by this cortical division of labor and properties of the magnoand parvocellular systems [5], we explore an unsupervised approach to featurelearning that jointly learns object features and their transformations from naturalvideos.
Revised: Inspired by this cortical division of labor and properties of the magnoand parvocellular systems [5], we explore an unsupervised approach to featurelearning that jointly learns object features and their transformations from natural videos.

Original: We propose a new convolutional bilinear sparse coding model that (1)allows independent feature transformations and (2) is capable of processing largeimages.
Revised: We propose a new convolutional bilinear sparse coding model that(1) allows independent feature transformations and (2) is capable of processinglarge images.

Original: We also introduce a new biologically-plausible inference technique anda learning procedurethat leverages smooth motion in natural videos.
Revised: Our learning procedure leverages smooth motion in natural videos.

Original: Our resultsshow that our model can learn groups of features and their transformations directlyfrom natural videos in a completely unsupervised manner.
Revised: Our results show that our model can learn groups of features and their transformations directly from natural videos in a completely unsupervised manner.

Original: The learned "dynamicﬁlters" exhibit certain equivariance properties, resemble cortical spatiotemporalﬁlters, and capture the statistics of transitions between video frames.
Revised: Thelearned "dynamic ﬁlters" exhibit certain equivariance properties, resemble cortical spatiotemporal ﬁlters, and capture the statistics of transitions between videoframes.

Original: Our modelcan be viewed as one of the ﬁrst approaches to demonstrate unsupervised learningof primary "capsules" (proposed by Hinton and colleagues for supervised learning) and has strong connections to the Lie group approach to visual perception.
Revised: Our model can be viewed as one of the ﬁrst approaches to demonstrateunsupervised learning of primary "capsules" (proposed by Hinton and colleaguesfor supervised learning) and has strong connections to the Lie group approach tovisual perception.

Original:  Motivation
Revised:  Motivation

Original: During early development, the brain learns a general-purpose internal representation of objects fromunlabeled image sequences.
Revised: During early development the brain learns a general-purpose internal representation of objects fromunlabeled image sequences.

Original: This representation is compositional and leverages the decompositionof objects into parts, sub-parts, and features, along with their relative transformations.
Revised: This representation is compositional and leverages the decompositionof objects into parts, sub-parts, and features, along with their relative transformations.

Original: In contrast,modern object recognition systems based on deep learning require thousands of labeled examplesand typically discard information about transformations (via pooling) in order to achieve invariance.
Revised: In contrast,modern object recognition systems based on deep learning require thousands of labeled examplesand typically discard information about transformations (via pooling) in order to achieve invariance.

Original: Information about transformations is critical for tasks such as movement planning and spatialreasoning.
Revised: Information about transformations is critical for tasks such as movement planning and spatialreasoning.

Original: Current unsupervised models produce representations that either lack interpretability or hierarchicaldepth.
Revised: Current unsupervised models produce representations that either lack interpretability or hierarchicaldepth.

Original: Variational autoencoders and generative adversarial networks (GANs) typically produce non- interpretable features that do not match the object/parts hierarchy inherent in natural visual scenes
Revised: Variational autoencoders and generative adversarial networks (GANs) typically produce noninterpretable features that do not match the object/parts hierarchy inherent in natural visual scenes.

Original: Because they do not explicitly model transformations, these models have difﬁculty generalizing to the vast range of viewing conditions that objects can appear in.
Revised: Because they do not explicitly model transformations, these models have difﬁculty generalizing tothe vast range of viewing conditions that objects can appear in.

Original: Sparse coding and its variants can learn interpretable features from unlabeled images: these features resemble the localized oriented (Gabor) receptive ﬁelds found in the primary visual cortex.
Revised: Sparse coding and its variants canlearn interpretable features from unlabeled images: these features resemble the localized oriented(Gabor) receptive ﬁelds found in the primary visual cortex.

Original: However, these models again do not model transformations and have been difﬁcult to generalize to deeper hierarchies due to the combi- natorial explosion of possible features
Revised: However, these models again do notmodel transformations and have been difﬁcult to generalize to deeper hierarchies due to the combinatorial explosion of possible features.

Original: We propose a new model for unsupervised learning motivated by the idea that the combinatorialexplosion problem can be mitigated by a neural architecture that processes the identity (“what”) andthe pose (“where”) of objects and their parts separately.
Revised: We propose a new model for unsupervised learning motivated by the idea that the combinatorialexplosion problem can be mitigated by a neural architecture that processes the identity (“what”) andthe pose (“where”) of objects and their parts separately.

Original: Such an architecture acknowledges the ventral/dorsal processing dichotomy in the visual cortex: the ﬁrst is mostly responsible for processingcontent and identity of objects while the latter is responsible for processing motion and transformations.
Revised: Such an architecture acknowledges the ventral/dorsal processing dichotomy in the visual cortex: the ﬁrst is mostly responsible for processingcontent and identity of objects while the latter is responsible for processing motion and transformations.

Original: We introduce a new bilinear sparse coding model that builds on previous bilinear generative modelsby (1) allowing each feature to have its own transformation and (2) accommodating large imagesvia convolution.
Revised: We introduce a new bilinear sparse coding model that builds on previous bilinear generative modelsby (1) allowing each feature to have its own transformation and (2) accommodating large imagesvia transposed convolutions.

Original: Furthermore, emulating the slower response times of the parvo pathway comparedto the magno pathway, we assume that at short time scales, object identities at each location willremain the same, modeling any fast changes as changes in object transformation values.
Revised: Furthermore, emulating the slower response times of the parvo pathway compared to the magno pathway, we assume that at short time scales, object identities at eachlocation will remain the same, modeling any fast changes as changes in object transformation values.

Original: We demonstrate our model by using short natural video sequences to learn features and their transformations.
Revised: We demonstrate our model by using short natural video sequences to learn features and theirtransformations.

Original: The resulting collection of “steerable” ﬁlters can be viewed as dynamic features resembling the spatiotemporal receptive ﬁelds reported in  primary visual cortex.
Revised: The resulting collection of “steerable” ﬁlters can be viewed as dynamic featuresresembling the spatiotemporal receptive ﬁelds reported in the primary visual cortex.

Original: Our model is also one of the ﬁrst toapply ideas from sparse coding to solve the problem of unsupervised learning of "primary capsules" 1previously proposed by Hinton and colleagues for supervised learning [4].
Revised: Our model isalso one of the ﬁrst to apply ideas from sparse coding to solve the problem of unsupervised learningof "primary capsules" 1 previously proposed by Hinton and colleagues for supervised learning [4].

Original:  Model2.1 Independent Bilinear Sparse Coding
Revised:  Model

Revised: 2.1 Independent Bilinear Sparse Coding

Original: In bilinear sparse coding [3, 1], an image patch is modeled as a combination of features B ij withtwo sets of coefﬁcients r i (object coefﬁcients) and x j (transformation coefﬁcients) that interactmultiplicatively:
Revised: In bilinear sparse coding [3, 1], an image patch is modeled as a combination of features B ij withtwo sets of coefﬁcients r i (object coefﬁcients) and x j (transformation coefﬁcients) that interactmultiplicatively:

Original: [Equation]
Revised: [Equation]

Original: Let (cid:80) j x j B ij =
Revised: Let (cid:80) j x j B ij =

Original: B i ( x ) where x represents the transformation vector consisting of x j ’s. Then
Revised: B i ( x ) where x represents the transformation vector consisting of x j ’s. Then

Original: I (cid:39) (cid:80) i r i B i ( x ) , which is the standard linear generative model used in sparse coding, PCA, ICAetc.
Revised: I (cid:39) (cid:80) i r i B i ( x ) , which is the standard linear generative model used in sparse coding, PCA, ICAetc.

Original: The r i coefﬁcients correspond to the degree to which each feature exists in the input.
Revised: The r i coefﬁcients correspond to the degree to which each feature exists in the input.

Original: Thex j coefﬁcients linearly combine a set of similar features to produce a dynamic “steerable” feature
Revised: Thex j coefﬁcients linearly combine a set of similar features to produce a dynamic “steerable” feature

Original: B i ( x ) .
Revised: B i ( x ) .

Original: The goal is for these dynamic features to capture an equivariance class centered aroundan underlying feature B i .
Revised: The goal is for these dynamic features to capture an equivariance class centered aroundan underlying feature B i .

Original: As a result, the r coefﬁcients remain invariant regardless of the speciﬁcinstantiation of the features, the variation being accounted for by x .
Revised: As a result, the r coefﬁcients remain invariant regardless of the speciﬁcinstantiation of the features, the variation being accounted for by x .

Original: To learn sparse part-like featuresof objects, sparsity is enforced on either r or both r and x via some appropriate sparsity penalty.
Revised: To learn sparse part-like featuresof objects, sparsity is enforced on either r or both r and x via some appropriate sparsity penalty.

Original: Typically bilinear sparse coding models are trained using pairs of video frames I t
Revised: Typically bilinear sparse coding models are trained using pairs of video frames I t

Original: +

Revised: +

Original: and I t , with rﬁxed and x inferred separately to account for the difference between frames:
Revised: and I t , with rﬁxed and x inferred separately to account for the difference between frames:

Original: [Equation]
Revised: [Equation]

Original: There is a strong connection to the Lie group approach to vision [2] where two consecutive framesare modelled as I t +1 = T (∆ x ) I t where T is a transformation operator.
Revised: There is a strong connection to the Lie group approach to vision [2] where two consecutive framesare modelled as I t +1 = T (∆ x ) I t where T is a transformation operator.

Original: The ﬁrst-order Taylorseries approximation of the Lie model [7, 6] is given by: I t +

Revised: The ﬁrst-order Taylorseries approximation of the Lie model [7, 6] is given by: I t +

Original: =
Revised: =

Original: I t + (cid:80) j ∆ x t,j ∇ x j I t whichmeans that
Revised: I t + (cid:80) j ∆ x t,j ∇ x j I t whichmeans that

Original: ∆ I = (cid:80) j ∆ x t,j ∇ x j I t .
Revised: ∆ I = (cid:80) j ∆ x t,j ∇ x j I t .

Original: Suppose I t (cid:39) (cid:80) i r
Revised: Suppose I t (cid:39) (cid:80) i r

Original: i U i where U i ∈ R
Revised: i U i where U i ∈ R

Original: d × 1 form an underlying feature set.
Revised: d × 1 form an underlying feature set.

Original: Replacing ∇ x j with the transformation matrix G j ∈ R d × d , we obtain:∆ I (cid:39) (cid:80) j ∆ x t,j G j (cid:80) i r
Revised: Replacing ∇ x j with the transformation matrix G j ∈ R d × d , we obtain:∆ I (cid:39) (cid:80) j ∆ x t,j G j (cid:80) i r

Original: i U
Revised: i U

Original: i = 
Revised: i = (cid:80) [Equation]

Original: (cid:80) i r i (cid:80) 
Revised: r i (cid:80) [Equation]

Original: j ∆ x t,j G j U i .
Revised: ∆ x t,j G j U i .

Original: Comparing with Equation 2 above, we seethat B ij =
Revised: Comparing with Equation 2 above, we seethat B ij =

Original: G j U i .
Revised: G j U i .

Original: We build on this model by allowing features to have independent pose parameters x ij so that featurescan transform independently from frame to frame.
Revised: We build on this model by allowing features to have independent pose parameters x ij so that featurescan transform independently from frame to frame.

Original: We also go beyond image patches to modelinglarge images by using transpose convolutions ( ∗ T ), resulting in a new bilinear model for images:
Revised: We also go beyond image patches to modelinglarge images by using transposed convolutions ( ∗ T ), resulting in a new bilinear model for images:

Original: [Equation]
Revised: [Equation]

Original: To distinguish our model from past models, we refer to traditional bilinear sparse coding as BSCand our independent bilinear sparse coding model as IBSC.
Revised: To distinguish our model from past models, we refer to traditional bilinear sparse coding as BSCand our independent bilinear sparse coding model as IBSC.

Original: 2.2 Inference
Revised: 2.2 Inference

Original: The reconstruction-based loss function for consecutive frames of a video is given by:
Revised: The reconstruction-based loss function for consecutive frames of a video is given by:

Original: L ( r,x
Revised: L ( r,

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: with r, x ≥ 0 .
Revised: with r, x ≥ 0 .

Original: The ﬁrst term is the mean-squared reconstruction error.
Revised: The ﬁrst term is the mean-squared reconstruction error.

Original: The other terms include asparsity penalty on r and weight decay for G and U .
Revised: The other terms include asparsity penalty on r and weight decay for G and U .

Original: To stabilize learning we project each B ij =
Revised: To stabilize learning we project each B ij =

Original: G j U i to unit (cid:96) 2 norm ( P (cid:96) 2 , 1 . 0 ) .
Revised: G j U i to unit (cid:96) 2 norm ( P (cid:96) 2 , 1 . 0 ) .

Original: Inference for BSC is typically performed by initializing x to some canonical vector and then alternatively optimizing r and x [3].
Revised: Inference for BSC is typically performed by initializing x to some canonical vector and then alternatively optimizing r and x [3].

Original: One of the issues with this approach is that the canonical vector mightbe a poor approximation to the true underlying pose parameters, especially in the case of independent features as in our model.
Revised: One of the issues with this approach is that the canonical vectormight be a poor approximation to the true underlying pose parameters, especially in the case ofindependent features as in our model.

Original: We address this issue by convolving each feature B ij with the imageto produce a feature map α ijt 
Revised: We convolve each feature B ij with the image to produce afeature map α ijt =

Original: = B ij ∗ I t .
Revised: B ij ∗ I t .

Original: We then project onto some appropriately chosen normball to compute x ijt = P (cid:96),ρ ( α ijt ) .
Revised: We then project onto some appropriately chosen norm ball to computex ijt = P (cid:96),ρ ( α ijt ) .

Original: 2 Inference proceeds by keeping x ﬁxed while r is optimizedto reconstruct the input.
Revised: 2 Inference proceeds by alternatively optimizing r and x until convergence.

Original: To optimize r , we use iterative thresholding.
Revised: Tooptimize r , we use iterative thresholding, while x is optimized by projected gradient descent.

Original: Both sets of coefﬁcients areforced to be non-negative, using a rectiﬁer for r and projecting on the positive part of the norm ballfor x .
Revised: Bothsets of coefﬁcients are forced to be non-negative, using a rectiﬁer for r and projecting on the positivepart of the norm ball for x .

Original: Note that we do not jointly optimize both r and x .

Original: We found in our experiments that optimizing xafter its initialization did not have any signiﬁcant effect in the quality of the learned features or onthe reconstruction.

Original:  Experiments
Revised:  Experiments

Original: For our experiments, we used 1920 ×
Revised: For our experiments, we used 1920 ×

Original: 1080 resolution YouTube videos converted to gray scaleand scaled down to 236 × 176 pixels per frame.
Revised: 1080 resolution YouTube videos converted to gray scaleand scaled down to 236 × 176 pixels per frame.

Original: The frames were normalized using subtractivenormalization 3 .
Revised: The frames were normalized using subtractivenormalization 3 .

Original: We extracted triplets of  frames, with r assumed to be constant for each tripletduring training.
Revised: We extracted sequences of 5 consecutive frames, with r assumed to be constantfor each sequence during training.

Original: We excluded triplets in the largest 5% of Euclidean norm difference between framesto exclude sudden camera changes or changes between scenes.
Revised: We excluded sequences in the largest 5% of Euclidean normdifference between frames to exclude sudden camera changes or changes between scenes.

Original: We used a stride of half the size ofthe kernel for transpose convolutions.
Revised: We useda stride of half the size of the kernel for transposed convolutions.

Original: Our model learns localized oriented Gabor-like features similar to those seen in sparse coding.
Revised: Our model learns localized oriented Gabor-like features similar to those seen in sparse coding.

Original: Figure 1(a) shows a subset of the learned 12 × 12 pixel features: each row shows B i : corresponding todifferent transformed versions of the same underlying feature.
Revised: Figure 1 shows a subset of the learned 12 × 12 pixel features: each column shows B i : correspondingto different transformed versions of the same underlying feature.

Original: Subsampling the video by a factor of4 results in features corresponding to larger transformations, as shown in Figures 1(b) and 1(c).

Original: Notethat not only translations but other transformations are learned as well, e.g., rotations and warping.
Revised: Note that not only translations butother transformations are learned as well, e.g., rotations and warping.

Original: The learned bilinear features allow accurate reconstruction, as seen for an example input in Figures1(d) and 1(e).
Revised: The learned bilinear featuresallow accurate reconstruction, as seen for an example input in Figures 2(a) and 2(b).

Original: All feature sets were 2 × overcomplete.
Revised: All feature setswere 2 × overcomplete.

Original: To test whether each B i ( x i ) corresponds to a “steerable” ﬁlter, we visualize in Figures 2(a-d) asubset of the different instantiations (with different x
Revised: To test whether each B i ( x i ) corresponds to a “steerable” ﬁlter, we visualize in Figures 3(a-e) asubset of the different instantiations (with different x

Original: i ’s) of each feature across different inputsand image locations from our natural videos.
Revised: i ’s) of each feature across different inputsand image locations from our natural videos.

Original: Note that the model captures a wide range of suchinstantiations.
Revised: Note that the model captures a wide range of suchinstantiations.

Original: To determine whether each B i ( x i ) captures the progression of a single underlyingfeature across frames, we visualized the evolution of features across triplets of frames.
Revised: To determine whether each B i ( x i ) captures the progression of a single underlyingfeature across frames, we visualized the evolution of features across sequences of frames.

Original: As seen in 
Revised: As seenin Figure 4, the learned features evolve across frames in a manner similar to spatiotemporal ﬁltersin the visual cortex, e.g., direction-selective Gabor ﬁlters moving in a particular direction.  Conclusion & Future Work We extend the bilinear sparse coding model to handle large images and independent feature transformations.

Original: Figure 2(e), the learned features evolve across frames in a manner similar to spatiotemporal ﬁltersin the visual cortex, e.g., direction-selective Gabor ﬁlters moving in a particular direction.
Revised: Our model learns to group similar features together, leveraging the smoothness of naturalvideos. Perhaps the most interesting direction for future work is that of extending this approachhierarchically. [Figure]  Acknowledgments

Revised: This work was supported by NSF grant no.

Revised: EEC-1028725, CRCNS/NIMH grantno.

Revised: 1R01MH112166- and a grant from the Templeton World Charity Foundation(TWCF).