Unsupervised Learning of Algebraic Structure from Stationary Time Sequences
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recently, there is a surge of interest in the data-driven learning of underlying simple relations and symmetry in the dataset.
However, ﬁnding such a symmetryoften requires structural assumptions about either a dataset or a model, or both.
Inthis study, we investigate the symmetry that can be learned from time sequencesof length at least three.
We show that, if each time sequence in the dataset hasa certain stationary property (e.g. constant velocity, constant acceleration), thealgebraic structure that helps us extrapolate far into the future can be discoveredwithout supervision.
We will showcase our method from both empirical and theoretical perspectives.
We will also show that, when the representation is trainedso that the future observation can be predicted well by a linear transition in thelatent space, the hidden disentangled structure of the dataset naturally emergesas a by-product through simultaneous block-diagonalization.
Our result suggeststhat ﬁnding a simple structured relation and learning a model with extrapolationcapability are two sides of the same coin.
Introduction
The recent evolution and successes of neural networks in machine learning ﬁelds have shown theimportance of symmetry-aware neural network models [17, 39, 33, 54].
In particular, symmetries inthe form of geometric/algebraic constraints have been proven useful in various applications involving high-dimensional, highly-structured observations.
For example, recent literature of robotics andreinforcement learning has succeeded in exploiting the knowledge of geometrical symmetries to improve the sample efﬁciency [53, 56] or to train a model that generalizes to unseen observations [49].
However, building an inductive bias that matches the given dataset of interest is challenging and recent studies have been exploring the ways to learn symmetries itself from observational sequences.
Many of these approaches consider settings with relatively restrictive assumptions or weak supervision.
For example, [47] allows the trainer to use the knowledge of the identities of the actions usedin making the transition.
Meanwhile, [3, 31, 30] essentially assume that the transition velocity of allsequences in the datasets are the same.
These studies indicate that there is still much room left for the question of “what is required forthe sequential dataset/model to enable the unsupervised learning of the equivariance relation corresponding to the underlying symmetry.”
This paper advances this investigation by showing that ifthe sequential dataset consists of time series with a certain stationary property (constant velocity,constant acceleration), we can learn the underlying symmetries by simply training the model to beable to predict the future with linear transition in the latent space.
We achieve this feat by training anencoder-decoder model in a framework of meta-learning, which consists of (1) the internal trainingstep to compute the prediction loss of linear transition in the latent space and (2) the external trainingstep to update the encoder and decoder.
We train our model by splitting each observation sequence into two components that are respectively dedicated to the internal step and the external step.
Ourexperimental results on multiple sequential datasets indicate that our method can capture the hiddenequivariance relation.
We will also show that, in alignment with group representation theory [37],the learned linear transitions in our framework can be simultaneously block-diagonalized over allsequences in the dataset, and that each block corresponds to a disentangled factor of variation.
Related works
Recently, numerous studies have explored the ways to learn symmetry in a data-driven manner.
There is rich literature in unsupervised/weakly supervised approaches that use sequential datasetsto exploit the structure that is shared across time, and they all differ by the types of inductive bias.
For example, the object-centric approach introduces inductive bias in the form of architecture, andequips the model with pre-deﬁned slots to be allocated to objects [34, 29, 35].
Meanwhile, [20, 1]assumes that the symmetry to be found takes the form of the energy conservation law, and learnseach variable in the law as a function of observations.
While this approach assumes that some energyis preserved in each observation, we assume that the transition parameters like velocity and acceleration are preserved within each observation.
Other more indirect forms of inductive bias includethose relevant to distributional sparseness and symmetry deﬁned through algebraic constraints.
[27]for instance assumed that every stationary component of a given time series is generated by a ﬁniteand independent latent time series.
[36] proposed to sparsely model the transition with a distribution of large kurtosis.
Our work belongs to a family of unsupervised learning that seeks to ﬁnd theunderlying symmetry of the dataset based on an algebraic inductive bias that the transitions can berepresented linearly in some latent space.
In this sense, our inductive bias also has a connection to Koopman operator [38, 23, 2].
We are different from these studies in that we are aiming to learn acommon encoding function (i.e. lifting function) under which the set of sequences following different dynamics can be described linearly.
Also [59] applied Koopman operator on pedestrian walkingsequences, and [21] used Koopman operator to separate the foreground from the background.
However, [59] does not set out their model to solve the extrapolation, and neither [59] nor [21] discussesthe natural algebraic decomposition of the latent space that results solely from the objective to predict the unseen future.
Unsupervised learning with algebraic/geometrical constraints Many studies impose algebraicconstraints that reﬂect some form of geometrical assumptions.
[15] uses a known coordinate mapparametrization of a Lie group family to construct a posterior distribution on the manifold.
[46]assumes that the observations are dense enough on the data-manifold to describe its tangent space,and exploits a property of random walks on the product manifold to decompose the data space.
Inthe analysis of sequential datasets, [12, 50, 9, 47, 11] make some Lie group type assumptions aboutthe transition.
[47] also assumes that the identity of the actions in the sequences is known.
As for theapproaches with less explicitly geometrical touch, [31] uses capsule structure in their probabilisticframework to model a ﬁnitely cyclic structure while retaining the computability of posterior distribution.
By design, [31] assumes that all sequences in the dataset transition with the same cyclicvelocity.
[58] enforces the underlying transition action to be commutative.
Some of these studieslearn the representation so that the linear transition in the latent space can be explicitly computed[47, 11, 3].
In particular, [3] presents a theory that suggests that a representation without this featurewould have topological defects, such as discontinuity.
Our approach shares a similar philosophy withthese works except that, instead of imposing a strong assumption about the underlying symmetry,we only make a relatively weak stationarity assumption about the dataset; although we assume eachsequence to be transitioning with constant velocity/acceleration, we allow the velocity/accelerationto vary across different sequences.
Disentangled Representation Learning Disentangled structure [26] is a form of symmetry thathas also been actively studied.
It is known that, under the i.i.d assumption of examples, unsupervised learning of disentanglement representation is not achievable without some inductive biasesencoded in models and datasets [43].
In response to this work, subsequent works have exploreddifferent frameworks such as weakly-/semi-supervised settings [45, 44] and learning on sequentialexamples [36] to learn disentangled representations.
For example, PhyDNet [22] disentangles theknown physical dynamics from the unknown factors by preparing an explicit module called PhyCell.
ICA [28] and recent works [60, 55] also discuss the identiﬁability property of learned representa- tions.
Classical methods like [26, 5, 32] take an approach of incorporating the inductive bias in theform of a probabilistic model.
We are different from many previous methods in that we do not equip our model with an explicitdisentanglement framework.
Our method achieves disentanglement as a by-product of training amodel that can predict the future linearly in the latent space.
The set of latent linear transformationsestimated by our method for different time sequences can be simultaneously block-diagonalized,and the latent space each block corresponds to a disentangled feature.
Our data assumption aboutconstant velocity/acceleration might be similar in taste to the setting used by [28], in which theobserved time series can be split into the ﬁnite number of stationary components.
Learning of equivariant structure from stationary time sequences
Our goal is to learn the underlying symmetry structure of a dataset in an unsupervised way thathelps us predict the future.
What do us humans require for the dataset when we are tasked to, forexample, predict where a thrown ball would be in the next second?
We hypothesize that we solvesuch a prediction task by analyzing a short, past time-frame with a certain stationary property (e.g.,constant velocity/acceleration).
Indeed, people with good dynamic visual acuity can chase a fastmoving object, because they can identify such a short stationary time-frame and use it to predict thenear future linearly in their latent space.
Based on this intuition, we propose to provide the trainerwith a dataset consisting of constant velocity/acceleration sequences.
We formalize this idea below.
Our dataset S consists of sequences in some ambient space X , so that eachmember s ∈ S takes the form s =
[ s t ∈ X ; t = 1 , ..., T ] ∈ S .
Because we want s to be describing 110 a sequence that transitions with constant velocity, we assume that all s t in a given instance of s ∈ S 111 are related by a ﬁxed transition operation g ∈ G so that s t +1 =
g ◦ s t for all t , where G is the set oftransition operators on X and each g ∈ G acts on x ∈ X by sending x to g ◦ x . We assume that X is 113 closed under G ; that is, g ◦ x ∈ X for all x ∈ X , g ∈ G .
We allow G to be continuous as well, so that 114 g might not have a ﬁnite order (For instance, if g is a rotation with speed 2 π r with irrational r , anyﬁnite repetition of g would not agree with identity mapping).
This way, our setting is different fromthose used in [31] that explores a cyclic structure using the capsules of same size.
We emphasize thatthe transition action g is generally assumed to differ across the different members of S .
For example,if G is a set of rotations and X a set of images, then the rotational speed, direction and the initialimage may all be different for any two distinct sequences, s and s ! .
Because each instance of S ischaracterized by s 1 and g , we may write s ( s 1 , g ) to denote a sequence that begins with initial frames 1 and transitions with g .
Summarizing, S is a subset of { [ g t ◦ s 1 ; t = 0 , ..., T − 1]; s 1 ∈ X , g ∈ G} .
Prediction framework through equivariance Our strategy is to exploit the stationary propertyof each s ∈ S to seek an invertible continuous function Φ : X → R a × m such that there exists some 124 M :
G → R a × a satisfying
[Equation]
This relation is known as equivariance [10]. In other words, we seek a model in which X and
R a × m are invertibly related by an equivariance relation with respect to G , where g ∈ G acts on 127 Φ ( x ) ∈ R a × m via the map Φ ( x ) %→ M g Φ ( x ) with M g ∈ R a × a .
We assume m > 1 in our study 1 .
In this framework, we predict the sequence s ( s 1 , g ) with the relation Φ − 1 ( M g Φ ( s t − 1 )) =
˜ s t .
When
G is a group, (1) would imply M gh Φ ( x ) =
Φ ( gh ◦ x ) =
M g Φ ( h ◦ x ) =
M g M h
Φ ( x ) for all x , andthe map g %→ M g is called a representation of G [8, 57, 9].
Because we are aiming to establish the framework in which the representation of each action gcan be explicitly computed, our philosophy has much in common with the proposition of [3].
Thisapproach is in contrast to [31], which encodes a predeﬁned cyclic structure in the model.
3.1 Learning equivariance relation from stationary sequential dataset
However, training the model satisfying (1) with just the constant velocity assumption is not a trivialtask, because this assumption only assures that, for each sequence s ( s 1 , g ) , there is a sequencespeciﬁc operator M ( g, s 1 ) that determines the transition. In order to satisfy the full equivariance (1),
M ( g, s 1 ) shall not depend on s 1 (i.e. homogeneous with respect to s 1 ).
At the same time, becausethe constant velocity assumption applies to each sequence over all time intervals, it at least assuresthat the latent transition M is well deﬁned within each sequence; that is, M ( g, x ) =
M ( g, g ◦ s 1 ) =
M ( g, g 2 ◦ s 1 ) · · · and so on.
It turns out that, with some regularity assumptions on the model andthe choice of G , we can extend this observation to say that M satisﬁes intra-orbital homogeneity ;that is, M ( g, x ) is constant on the orbit G ◦ x
= { g ◦ x ; g ∈ G} for each x .[-
-]
Proposition 3.1.
Suppose that Φ ( s t ) =
M ( g, s 1 )
Φ ( s t − 1 ) for all s and t .
If m > a and if G is acompact commutative Lie group, then M satisﬁes intra-orbital homogeneity.
Also, if M satisﬁes intra-orbital homogeneity, M ( g, x ) and M ( g, x ! ) for any pair ( x, x ! ) in differentorbits G ◦ x & =
G ◦ x ! can be shown to be at least similar.
Proposition 3.2.
Suppose that M ( g, x ) satisﬁes intra-orbital homogeneity, and suppose that G is acompact connected group.
If M ( g, x ) is continuous with respect x , then for all ( x, x ! ) , there existssome P such that PM ( g, x ) P − 1 =
M ( g, x ! ) .
Thus, much of the equivariance property can be satisﬁed automatically by training the representationon the set of stationary sequences.
Interestingly, as we experimentally demonstrate later, our trainingmethod in the next section successfully learns a fully equivariant Φ without explicitly enforcing thechange of basis P to be I .
Encoding Algebraic Structure by Meta-optimization
We propose a meta-learning way to learn a homeomorphic function Φ : X → R a × m with equivariance property by seeking an injective Φ such that M ( g, s 1 ) Φ ( s t ) =
Φ ( s t +1 ) for all g ∈ G , s 1 ∈ X .
We learn such Φ by casting this problem as a meta-learning problem in which M ( g, s 1 ) is to beinternally estimated for each s .
In other words, we seek a pair of an encoder Φ and a decoder Ψ suchthat L ( Φ , Ψ ) =
min M ’ Ψ ( M Φ ( s t ))
− s t +1 ’ 22 is optimized at all time t for each s .
We conduct this optimization by splitting each s
= { s 1 , ..., s T } into conditional time sequence 162 s c = { s 1 , ..., s T c } and validation time sequence s p = { s T c
+1 , ..., s T } , while using the former for 163 the internal optimization of M to force the linear algebraic relation in the latent space and using thelatter for the prediction loss.
More precisely, we solve the following optimization problem about Φand Ψ :
[Equation]
where
[Equation]
Figure 1 describes our overall framework when T c = 2 and T p = 1 .
It might appear as if we canalso set s = s c and optimize the following reconstruction version of Eq.(2):
[Equation]
However, as we will see in the experiment section, the use of the validation sequence s p makes asubstantial difference in the learned representation.
This is most likely because the minimization ofthe validation error of M ∗ on s p would encourage Φ to exclude the s c -speciﬁc information from thetransition M ∗ .
We will illustrate this effect in the experiment section.
We shall note that we canalso parameterize M as M := exp( A ) , where A ∈ R a × a is a Lie algebra element to be internallyoptimized.
This type of approach was used in [13] for building Lie group convolutional network andin [50, 11] for predicting a sequence that is not necessarily stationary.
In order to train their model,[50] used additional parameters to diagonalize each algebra element as well as hyperparameters tostabilize the training.
We also experimented with a Lie-algebra style representation of M ∗ and used
SGD to internally optimize the exponent parameters, but we needed to carefully tune the hyperparameter for the norm regularization term to stabilize the training and never really succeeded totrain the model without collapsing.
Our internal optimization procedure is free of such a parametertuning.
Φ ( s 1 ) , Φ ( s 2 ) , the method solves the least square problem: M ∗ = arg min M ’ M Φ ( s 1 ) − Φ ( s 2 ) ’ 2 F .
The model then predicts the future observations by ˜ s t +2 =
Ψ (( M ∗ ) t Φ ( s 2 )) .
These processes (including the linear problem) are all differentiable.
Internal Optimization of M ∗
Because theinternal optimization in Eq.(2) is a linear problem, it can be solved analytically as
[Equation]
where H + =
[Equation]
Φ ( [Equation] [Equation] ; ... ; Φ ( s T c
[Equation]
[Equation]
∈ 185 R a × ( T c − 1) m and H +
=
[ Φ ( s 2 ); ... ; Φ ( s T c ) ∈
R a × ( T c − 1) m are the horizontal concatenationsof the encoded frames and H † +0 is the MoorePenrose pseudo inverse of H +0 .
Because M ∗ ( s c | Φ ) is a closed form with respect to Φ ,the loss (2) can be directly optimized by differentiating it with respect to the parameters ofboth Φ and Ψ .
Thus, the training is done in anend-to-end manner.
Figure 1 summarizes theoverall procedure to make prediction on a givensequence when T c = 2 , T p = t .
We note that,although we have assumed the dataset to consistof constant-velocity sequences, we can readilyextend our method to the dataset consisting of the time series with higher-order stationarity, such asconstant acceleration.
See Section 5.4 for the detailed explanation of the model extension and theexperimental results.
Experiments
We conducted several experiments to investigate the efﬁcacy of our framework.
In this section,we brieﬂy explain the experimental settings.
For more details, please see Appendix D. We testedour framework on Sequential MNIST, 3DShapes [4], and SmallNORB [42].
Sequential MNISTis created from MNIST dataset [41].
For all experiments, we used a ResNet [24]-based encoderdecoder architecture and we set a = 16 and m = 256 so that the latent space lives in R 16 × 256 .
For Sequential MNIST, we chose our G to be the set of all combinations of three types of transformations: shape rotation, hue rotation, and translation, and randomly sampled a single instanceof g ∈ G for each sequence(See Appendix D for the examples of sequences).
To create each sequence, we ﬁrst resized the MNIST image to 24 × 24, applied repetitions of a randomly sampled,ﬁxed member of g ∈ G and embedded the results to 32 × 32 images.
For shape and hue rotations,we randomly sampled the velocity of angles from uniform distribution on the interval [ − π / 2 , π / 2)for each sequence.
For translation, we randomly sampled the start point and end point in the rangeof then moved the digit images on a straight line between the sampled points.
Wealso experimented on sequential MNIST with background (Sequential MNIST-bg).
For Sequantial MNIST-bg, we used the same generation rule as Sequential MNIST but we added backgroundimages behind the moving digits.
For the background, we used a randomly sampled images from ImageNet [48], which were all resized to 32 × 32.
Also, we only used the images of digit 4 for mostof the experiments on Sequential MNIST/MNIST-bg.
Unless otherwise noted, all evaluations in thispaper for the Sequential MNIST are based on training with only digit 4 .3DShapes and SmallNORB are datasets with multiple factors of variation.
We created a setof constant-velocity sequences from these datasets by varying a ﬁxed combination of factors foreach sequence.
That is, on these datasets, we chose our G to be the set of variations of factors, andsampled each g as # i g !
i i , where g i represents the increase of i -th factor by one unit and " i ∈ Z .
Thus, the value of " i represents the velocity in the direction of the i -th factor on the grid.
For3DShapes, we chose wall hue , ﬂoor hue , object hue , scale , and orientation as the factors to vary.
We varied elevation and azimuth for SmallNORB.
For the split of each sequence into ( s c , s p ) , weset T c = 2 and T p = 1 on all of the constant velocity experiments.
We also conducted experiments for the sequences with constant acceleration on Sequential MNIST.
To create a sequence with constant acceleration, we chose a pair g a , g v ∈ G for each sequence, andgenerated s by setting s t
+1 =
g ta g v s t .
We elaborate on the detail of this extension in 5.4.
[Figure] [Figure] As ablations, we tested several variants of our method: ﬁxed [-2x2blocks (When we perform internal optimization, we seperated the latent tensor H ∈ R 
-]2x2 blocks (abbreviated as ﬁxedblocks), Neural M ∗ , Reconstruction model (abbreviated as Rec.
Model), and Neural transition . For the method of ﬁxed 2x2 blocks , we separated the latent tensor Φ ( s ) ∈ R 16 × 256 into 8 subtensors{ Φ ( k ) ( s ) ∈ R 2 × 256 } 8 k
=
and calculated pseudo inverse for each k to compute the transition in each R 2 × 256 dimensional space. This variant yields M ∗ as a direct sum of eight 2 × 2 matrices.
We testedthis variant to see the effect of introducing a predetermined representation theoretic structure as in[9]. For Rec. model, we trained Φ and Ψ based on L r in Eq. 3 with T c = 3 . We tested this variantto see the effect of our use of T p .
For Neural M ∗ , we trained an additional network M θ that mapss c to a transition matrix, replaced M ∗ with M θ in (4), and optimized θ and ( Φ , Ψ ) simultaneously.
We may see Neural M ∗ as a variant in which the meta part of the internal and external training isremoved from our method.
For Neural transition, we trained 1x1 1D-convolutional networks tobe applied to latent sequences in the past to produce the latent tensor in the next time step; forinstance, ˜ s t +1 =
Ψ
(1DCNN( Φ (s t ) , Φ (s t − 1 ))) when T c = 2 2 .
The 1DCNN was applied along themultiplicity dimension m .
In this variant, the relation between Φ ( s t ) and Φ ( s t +1 ) is not necessarilylinear.
Section D.1 in Appendix describes each of the comparison methods more in detail.
In testingall of these variants, we used the same pair of encoder and decoder architecture as the proposedmethod.
5.1 Qualitative and quantitative results on the prediction
Figure 2 shows the example sequences generated by the proposed model and comparative models.
Figure 3 presents the prediction performance at T c + T p when T c = 2 .
To produce this result, weback-propagated the prediction error at T p = 1 to the encoder during the training, and the predictionat T p > 1 was used to evaluate the extrapolation performance.
Our method successfully predicts theimages for T p ≥ 1 .
Neural transition and Neural M ∗ had almost the same prediction performance at T p = 1 , but they both failed in extrapolation.
Our ﬁxed 2 × 2 blocks variant failed in extrapolation as
MNIST
MNIST-bg 3DShapes
SmallNORB
Method
[Table]
L
[Table] L [Table]
Rec.
Model 38.
Neural M ∗ 7.
Ours 5.
well. This might be because the over-regularized structure of 2x2 block hindered with the trainingof the SGD optimization [16].
To evaluate how our learned representation relates to the structural features in the dataset, we also regressed the factors of transition from M ∗ and regressed the class of the digits from Φ ( s 1 ) (Figures 8and 9 in Appendix A).
SimCLR [7] and contrastive predictive coding (CPC) [52, 25] are tested asbaselines.
Please see Appendix D for the detailed experimental settings for SimCLR and CPC.
Ourmethod yields the representation with better prediction performance than the comparative methodson the test datasets.
5.2 Equivariance performance
As we have described through Section 3.1 and 4, the equivariance is achieved when M ∗ ( s ( s 1 , g ) | Φ )in (2) does not depend on s 1 , where we recall that s ( s 1 , g ) represents the sequence that begins withs 1 and transitions with g .
To see how much the trained model is equivariant to the transformations inthe sequential dataset, we therefore calculated the equivariance error , which is the prediction errorfrom applying M ∗ ( s | Φ ) to Φ ( s !
T c ) for a pair s & = s !
that transitions with the same g .
In other words,when T c = 2 , we compute the following;
[Equation]
where S ( g ) represents the set of all sequences that transition with g .
For each pair of s & = s !
weset T c = 2 and T p = 1 as done in the experiments in the previous sections.
Figure 4b comparesthe Neural M ∗ method against our method in terms of the equivariance error.
Figure 4a shows theresult of applying M ∗ ( s | Φ ) on Φ ( s ! 2 ) and applying M ∗ ( s ! | Φ ) on Φ ( s 2 ) .
We see that when we swap M ∗ this way, Neural M ∗ also swaps the digits; this implies that M ∗ learned by Neural M ∗ encodesthe sequence speciﬁc information together with the transition.
On the other hand, swapping of M ∗does not affect the prediction for our method, suggesting that our method is succeeding to learnan equivariant model.
This is somewhat surprising, because our model does not have the explicitmechanism to enforce the full equivariance ( P
= I in Section 3.1).
5.3 Structures found by simultaneous block-diagonalization of M ∗ s
So far, we have observed that our model has an ability to learn a G -equivariant model.
Representation theory guarantees that, if G is a compact connected group, any representation D : G → R a ×
a
[Figure]
can be simultaneously block-diagonalized; that is, there is a common change of basis U suchthat UD g U T = $ j V ( j ) g , where V ( j ) g is called irreducible representation that cannot be blockdiagonalized any further [37, 9, 57].
The equivariance of our Φ suggests that M ∗ ( s | Φ ) may besimultaneously block-diagonalizable as well.
This block-diagonalization sometimes reveals disentanglement structure because any irreducible representation of G 1 × G 2 is of form V (1) ⊗ V (2) , where
V ( k ) is an irreducible representation of G k .
In particular, if M ’s irreducible representations have theform V (1) ⊗ 1 or 1 ⊗ V (2) , then each block would either corresponds to the action of G 1 or of G 2 .
To ﬁnd U that simultaneously block-diagonalizes all M ∗ ( s | Φ ) , we optimized U based on the following objective function that measures the block-ness of ˆ M ∗ ( s ) :
= UM ∗ ( s | Φ ) U − 1 based on thenormalized graph Laplacian operator ∆ :
[Equation] [Equation]
where A ( ˆ M ∗ ( s )) =
abs ( ˆ M ∗ ( s ))
abs ( ˆ M ∗ ( s )) T with abs ( ˆ M ∗ ( s )) represents the matrix suchthat abs ( ˆ M ∗ ( s )) ij =
| ˆ M ∗ ij | .
Our objective function is based on the fact that, if we are given anadjacency matrix A of a graph, then the number of connected components in the graph can be identiﬁed by looking at the rank of the graph Laplacian.
For the derivation, please see Appendix E. Figure 5a shows the block structure revealed by simultaneous block-diagonalization through U trainedby minimizing the average of L bd over all s .
Figure 5b shows the results of applying transformationof only one block.
We can see that each block only alters one factor of variation.
Our results suggest that the learned M ∗ captures the hidden disentangled structure of the group actions behind thedatasets.
5.4 Extension to the sequences with constant acceleration
We have seen that our method successfully learns an equivariant structure from the set of constantvelocity sequences.
In this section, we show that we can extend our concept to the set of sequences sharing the stationarity of higher order (constant acceleration).
By deﬁnition, the pairof Φ ( s t ) and Φ ( s t − 1 ) encodes the information about the velocity at t .
When the multiplicitym is sufﬁciently large 4 , the velocity can be estimated by: 1 M t =
Φ ( s t ) Φ ( s t − 1 ) † .
Because thiswould yield a sequence of velocities, we can simply apply our method again to estimate the constant acceleration by 2 M ∗ = 1 M +11 M † +0 where 1 M +0 =
[ 1 M 2 ; ... ; 1 M T c − 1 ] ∈ R a × ( T c − 2) a and3 V : G → { 1 } [Figure] 1 M + [Equation]
=
[Equation] M [Equation] ; ... ; 1 M T c ] ∈ R a × ( T c − 2) a .
We can then predict the future representation ˜ s t fort =
T c + 1 , ..., T p by
[Equation]
where
[Equation]
7) where # represents multiplications from left.
We train Φ and Ψ by minimizing the mean squared error between ˜ s t and s t for t =
T c + 1 , ..., T as in Eq.(2).
To create a sequence of constant acceleration from MNIST dataset, we only used shape and color rotations.
We chose the initial velocity forthese rotations randomly on the interval [ − π / 5 , π / 5) for each sequence, and chose the accelerationon the interval [ − π / 40 , π / 40) .
The results are shown in Figure 6.
The Neural transition and theconstant-velocity version of our method failed to predict the accelerated sequence, while the 2ndorder model succeeded in predicting the sequence even after t p > 5 .
Discussion & Limitations
How is full equivariance achieved in our method?
The theoretical results we provided in section3.1 only assure that M ( g, x ) and M ( g, x ! ) are similar when the underlying group is commutative,compact and connected.
However, as we have shown experimentally, our method seems to be learning Φ for which the estimators of M satisfy M ∗ ( s ( g, x ) | Φ ) ∼ =
M ∗ ( s ( g, x ! ) | Φ ) .
This can be happening because our framework and the training method based on the internal optimization in the latentspace is somehow encouraging M ∗ to be orthogonal (See the loss curve of orthogonality of M ∗ in Appendix A).
Maybe this is forcing the change of matrix P such that PM ( g, x ) P −
=
M ( g, x ! ) tobe also rotations as well, which commutes with M ( g, x ) itself.
Also, Figure 2b and 3 show that, asreported in [30], the models trained with reconstruction loss like (3) does not well capture the grouptransformation behind the sequences: the encoder representation was found to be signiﬁcantly worsethan that of the model trained with (2).
We hypothesize that (3) fails to remove the sequence-speciﬁcinformation from M ∗ , while (3) succeeds to do so by training the model to be able to predict theunseen images.
Towards learning symmetries from more realistic observations As we are making a connectionbetween our prediction framework and group equivariance, we are essentially assuming that thetransitions are always invertible, because group is closed under inversion.
However, this might notbe always the case in real world applications; for instance, if the image sequences are the sequentialrenderings of a rotating 3D object, the transitions are generally not invertible because only a partof the object is visible at each time step.
We experimented Sequential ShapeNet, which is createdfrom ShapeNet [6] dataset.
A series of rendered images is generated by sequentially applying 3Drotations of different speeds for each axis.
Generated results on Sequential ShapeNet (See 23 in Appendix B show that actually our current method was not able to generate the images on 3D rotateddatasets.
If the transitions are not invertible, some measures must be taken in order to resolve theindeterminacy, such as probabilistic modeling or additional structural inductive bias.
Broader impact Because our study generally contributes to predictions and extrapolation, it hasas much potential to negatively affect the society as most other prediction methods.
In particular,applications of our method to image sequence can be potentially integrated into weapon systems, forexample.
At the same time, our unsupervised learning of the symmetrical structure from sequentialdatasets may also contribute to new discoveries in the systems of ﬁnance, medical science, physicsand other ﬁelds of ML such as reinforcement learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] The abstract and introduction reﬂects the contributions of thispaper.(b)
Did you describe the limitations of your work?
[Yes] In the Discussion & Limitations section, we discuss the type of dataset to which our work cannot be directly applied, as well asthe theoretical problem that has not been solved.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We included the broader impact discussion in Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read the guidelines and we conﬁrmed that the contents in our paper conformed to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We included the fullset of assumptions.
We provide the details of the assumptions in the supplementary material.
(b) Did you include complete proofs of all theoretical results?
[Yes] We provide complete proofsof theoretical results in Supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provided the code for
reproducing the experimental results as a supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] We explained how we chose the hyperparameters, the data splits in section and appendix section D.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We include the standard deviation values of equivariance performance in Appendix A (We omitted them in the main submission because of the space limitation) and the linear classiﬁcation performance of the transition parameters in Appendix A. [Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of [Table] GPUs, internal cluster, or cloud provider)?
[Yes] We explained the number of GPUs to train
[Table]
the models and approximated costs for reproducing the full results of experiments in D.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... [Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited [41] for MNIST,[6] for ShapeNet, [32] for 3Dshapes and [42] for SmallNORB. [Table]
(b) Did you mention the license of the assets?
[Yes] For each dataaset we used, we included theURL to the license information in the reference section. [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A](d)
Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[Yes] For ShapeNet dataset, we are approved to download the assets. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable in-
[Table]
formation or offensive content?
[Yes] We don’t ﬁnd any personally identiﬁable informationin the images used in our experiments. [Table]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review Board(IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amount spenton participant compensation?
[N/A]
[Table]