Unsupervised Learning of Algebraic Structure from Stationary Time Sequences
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recently, there is a surge of interest in the data-driven learning of underlying simple relations and symmetry in the dataset.
However, ﬁnding such a symmetryoften requires structural assumptions either about a dataset or a model.
In thisstudy, we investigate the symmetry that can be learned from time sequences oflengths at least three.
We show that, if each time sequence in the dataset has acertain stationary property (e.g. constant velocity, constant acceleration), the algebraic structure that helps us extrapolate far into the future can be discoveredwithout supervision.
We will showcase our method from both empirical and theoretical perspectives.
We will also show that, when the representation is trainedso that the future observation can be predicted well by a linear transition in thelatent space, the hidden disentangled structure of the dataset naturally emergesas a by-product through simultaneous block-diagonalization.
Our result suggeststhat ﬁnding a simple structured relation and learning a model with extrapolationcapability are two sides of the same coin.
Introduction
The recent evolution and successes of neural networks in machine learning ﬁelds has shown theimportance of symmetry-aware neural network models [16, 35, 32, 50].
In particular, symmetriesin the form of geometric/algebraic constraints have been proven useful in various applications involving high-dimensional, highly-structured observations.
For example, recent literature of roboticsand reinforcement learning have succeeded in exploiting the knowledge of geometrical symmetriesto improve the sample efﬁciency [49, 52] or to train a model that generalizes to unseen observations [46].
However, building an inductive bias that matches the given dataset of interest is challenging, and recent studies have been exploring ways of learning symmetries itself from observational sequences.
Many of these approaches consider settings with relatively restrictive assumption or weak supervision.
For example, [44] allows the trainer to use the knowledge of the identities of the actions usedin making the transition.
Meanwhile, [3, 27, 28] essentially assume that the transition-velocity of allsequences in the datasets are the same.
These studies indicate that there is still much room left for the question of “what is required forthe sequential dataset/model to enable the unsupervised learning of the equivariance relation corresponding to the underlying symmetry?”
This paper advances this investigation by showing that if thesequential dataset consists of time series with certain stationary property (constant velocity, constantacceleration), we can learn the underlying symmetries by simply training the encoder model to beable to predict the future with linear transition in the latent space.
We achieve this feat in the way ofmeta-learning, which consists of (1) the internal training step to compute the prediction loss of lineartransition in the latent space and (2) the external training step to update the encoder. We train our model by splitting each observation sequence into two components that are respectively dedicated tothe internal and external step. Our experimental results on synthetic and benchmark dataset indicatethat our method can capture the equivariance relation, thereby succeeding to extrapolate the unseensequences far into the future.
We will also show that, in alignment to group representation theory,the learned linear transitions in our framework can be simultaneously block-diagonalized over allsequences in the dataset, and that each block corresponds to a disentangled factor of variation.
Related works
Recently, numerous studies have explored the ways to learn symmetry in data-driven manner.
Thereare rich literatures in unsupervised/weakly supervised approaches that use sequential datasets toexploit the structure that is shared across time, and they all differ by the types of inductive bias.
For example, object centric approaches introduces inductive bias in the form of architecture, andequips the model with pre-deﬁned slots to be allocated to objects [30, 26, 31].
Meanwhile, [19, 1]assumes that the symmetry to be found takes the form of the energy conservation law, and learnseach variables in the law as a function of observations.
While this approach assumes that someenergy is preserved in each observation, we assume that velocity/acceleration is preserved withineach observation.
Other more indirect forms of inductive bias include those relevant to distributionalsparseness and symmetry deﬁned through algebraic constraints.
[24] for instance assumed that everystationary component of a given time series is generated by a ﬁnite and independent latent timeseries.
[33] proposed to model the transition sparsely with a distribution of large kurtosis.
Ourwork belongs to a family of unsupervised learning that seeks to ﬁnd the underlying symmetry of thedataset based on an algebraic inductive bias that the transitions can be represented linearly in somelatent space.
In this sense, our inductive bias also has a connection to Koopman operator [34, 20, 2].
However, we are different from these studies in that we are aiming to learn a common encodingfunction(i.e. lifting function) under which the set of sequences following different dynamics can bedescribed linearly.
Unsupervised learning with algebraic/geometrical constraints Many studies impose algebraicconstraints that reﬂects some form of geometrical assumption.
[14] uses a known coordinate mapparametrization of a Lie group family to construct a posterior distribution on the manifold.
[43]assumes that the observations are dense enough on the data-manifold to describe its tangent space,and exploits a property of random walks on the product manifold to decompose the data space.
Inthe analysis of sequential datasets, [10, 47, 8, 44, 9] make some Lie group type assumption about thetransition.
[44] also assume that the identity of the actions in the sequences are known.
As for theapproaches with less explicitly geometrical touch, [27] uses capsule structure in their probabilisticframework to model a ﬁnitely cyclic structure while retaining the computability of posterior distribution.
By design, [27] assumes that all sequences in the dataset transitions with the same cyclicvelocity.
[54] enforces the underlying transition action to be commutative.
Some of these studieslearn the representation so that the linear transition in the latent space can be explicitly computed[44, 9, 3].
In particular, [3] presents a theory that suggests that a representation without this featurewould have topological defects, such as discontinuity.
Our approach shares a similar philosophy withthese works except that, instead of imposing a strong assumption about the underlying symmetry,we only make a relatively weak stationarity assumption about the dataset; although we assume eachsequence to be transitioning with constant velocity/acceleration, we allow the velocity/accelerationto vary across different sequences.
Disentangled Representation Learning Disentangled structure [23] is a form of symmetry thathas also been actively studied.
It is known that, under the i.i.d assumption of examples, unsupervisedlearning of disentanglement representation is not achievable without some inductive biases encodedin models and datasets [39].
In response to this work, subsequent works have explored differentframeworks such as weakly-/semi-supervised settings [41, 40] and learning on sequential examples [33] to provably learn disentangled representations.
ICA [25] and recent works [55, 51] alsodiscuss the identiﬁebility property of learned representations.
Classical methods like [23, 13, 29]take an approach of incorporating the inductive bias in the form of probabilistic model.
Our method achieves disentanglement as a by-product of training a predictor model with linearalgebraic constraint.
As we mentioned in the introduction, the set of latent linear transformationestimated by our method for different time sequences can be simultaneously block-diagonalized, and the latent space each block corresponds to a disentangled feature.
We achieve this disentanglementwith a data assumption that each sequence transitions with constant velocity/acceleration. Our dataassumption might be similar in taste to the setting used by [25], in which the observed time seriescan be split into ﬁnite number of stationary components.
Learning of Data structure from stationary time sequences
Our goal is to learn the underlying symmetry structure of a dataset in an unsupervised way thathelps us predict the future.
What do us humans require for the dataset when we are tasked to, forexample, predict where a thrown ball would be in the next second?
We hypothesize that we areenabling such a feat by analyzing a short, past time-frame with a certain stationary property (e.g.constant velocity/accelerataion).
Indeed, only a handful of people with good dynamic visual acuitycan chase a fast moving object, because this ability allows them to identify such a stationary timeframe and use it to predict the near future linearly in their latent space.
Based on this intuition, wepropose to provide the trainer with a dataset consisting of constant velocity/acceleration sequences.
More precisely, we suggest to learn the structure of a given dataset X by using a set of time-seriesin X that are transitioning in the same way at each time step.
We formalize this idea below.
Dataset structure Our dataset S consists of time series, so that each member s ∈ S takes form 106 s =
[ s k ∈ X ; k = 1 , ..., T ] ∈ S .
Because we want s to be describing a time-series that transitions atconstant velocity, we assume that all s k in a given instance of s ∈ S are related by a ﬁxed transitionoperator g : X → X so that s t +1 =
g ◦ s t for all t . We use G to denote the set of transition operators,and we assume that X is closed under G ; that is, g ◦ x ∈ X for all x ∈ X , g ∈ G .
We would allow 110 G to be continuous as well, so that g might not have a ﬁnite order (For instance, if g is a rotationwith speed 2 π r with irrational r , any ﬁnite repetition of g would not agree with identity mapping).
This way, our setting is different from those used in [27] that explores a cyclic structure using thecapsules of same size.
We emphasize that the transition action g generally differs across the differentmembers of s .
For example if G is a set of rotations and X a set of images, then the rotational speed, direction and the initial image may be all different for two distinct instances of s and s ! .
Becauseeach instance of S is characterized by s 1 and g , we may write s ( s 1 , g ) to denote a sequence thatbegins with inital frame s 1 and transitions with g .
We use G to denote the set of operators that areused to generate S .
Summarizing, S is a subset of { [ g k ◦ s 1 ; k = 1 , ..., T − 1]; s 1 ∈ X , g ∈ G} .
Prediction framework through equivariance Our strategy is to exploit the stationary propertyof each s ∈ S to seek an invertible continuous function Φ : X → R a × m such that there exists some 121 M :
G → R a × a satisfying
[Equation]
for all x ∈ X and g ∈ G .
In other words, X and R a × m are invertibly related by an equivariance 123 relation with respect to G , where G acts on H ∈ R a × s via the action g ◦ latent H =
M g H .
Inthis framework, we predict the sequence s ( s 1 , g ) with the relation Φ − 1 ( M g Φ ( s t − 1 )) =
ˆ s t .
Notethat when Φ is a linear function and G is a group, this implies that X is a representation spaceof G and M is a representation of G ; in other words, M is a map such that M g M h = M gh for allg, h ∈ G [7, 53, 8].
This relation is automatically forced from (1) because M gh Φ ( x ) =
Φ ( gh ◦ x ) =
M g Φ ( hx ) =
M g M h
Φ ( x ) for all x . We also note that, when m > 1 , the action of M g on R a × m canbe realized by applying the same M g to m -copies of R a . In other words, our prediction frameworkassume that there are m number of subspaces that react to g in the same way. We assume m > 1in our study, as considered in [3].
The case in which there are no copies of subspaces that actin the same way is called multiplicity-free in the literature of representation theory [7, 17], and isknown to be a special case that happens only under a restrictive condition on the dimension of theobservations space [36].
A similar idea has been used in model architecture as well [12].
Becausewe are proposing the framework in which the representation of each action g can be explicitlycomputed, our philosophy has much in common with the proposition of [3].
This approach is incontrast to [27], which encodes a predeﬁned cyclic structure in the model.
Figure 1 describes ouroverall framework.
3.1 Learning equivariance relation from stationary sequential dataset
How does the stationary property (constant velocity assumption) help us ﬁnd a representation satisfying the equivariance relation (1)? As we elaborate in the next section, we train our Φ by requiringthat, for each time sequence s ( s 1 , g ) , there exists some "sequence"-common matrix M ( g, s 1 ) suchthat Φ ( s t ) =
M ( g, s 1 ) Φ ( s t − 1 ) .
On its own, this requirement seems to only enforce that the latenttransition M in a given sequence is well deﬁned; that is, M ( g, x ) =
M ( g, g ◦ s 1 ) =
M ( g, g 2 ◦ s 1 ) · · ·and so on.
Indeed, this is insufﬁcient for the full equivariance (1), because we want M ( g, x ) to beindependent of x , thereby achieving Φ ( g ◦ x ) =
M ( g, x ) Φ ( x ) = M g Φ ( x ) with well-deﬁned M g . It turns out that, with continuity assumption on M and an assumption on the choice of G , thisproperties can be partially satisﬁed under certain regularity conditions. We say that M satisﬁesintra-orbital homogeneity if M ( g, x ) is constant on the orbit G ◦ x
= { g ◦ x ; g ∈ G } for each x . {+
+}
The next proposition shows that the intra-orbital homogeneity is satisﬁed if G is sufﬁciently regular.
Proposition 3.1.
Suppose that Φ ( s t ) =
M ( g, s 1 )
Φ ( s t − 1 ) for all s and t .
If m > a and if G is acompact commutative Lie group, then M satisﬁes intra-orbital homogeneity.
Also, if M satisﬁes intra-orbital homogeneity, M ( g, x ) and M ( g, x ! ) for any pair ( x, x ! ) in differentorbits G ◦ x % =
G ◦ x ! can be shown to be at least similar.
Proposition 3.2.
Suppose that M ( g, x ) satisﬁes intra-orbital homogeneity, and suppose that G is acompact connected group.
If M ( g, x ) is continuous with respect x , then for all ( x, x ! ) , there existssome P such that PM ( g, x ) P − 1 =
M ( g, x ! ) .
Thus, much of the homogeneity properties can be satisﬁed automatically by training the reresentation on the set of stationary sequences.
Interestingly, as we experimentally demonstrate later, ourtraining method that we will elabote in the next section successfully learns a fully equivariant Φwithout explicitly enforcing the change of basis P to be I , suggesting that our method’s scope isbeyond these theoretical results.
Encoding Algebraic Structure by Meta-optimization
We would propose a meta-learning way to learn a homeomorphic function Φ : X → R a × m withequivariance property by seeking an injective Φ such that M ( g, s 1 ) Φ ( s t ) =
Φ ( s t +1 ) for all g ∈ G ,s 1 ∈ X .
We propose to learn such Φ by casting this problem as a meta-learning problem in which 167 M ( g, s 1 ) is to be internally estimated for each s .
In other words, we seek a pair of encoder Φ and adecoder Ψ such that L ( Φ , Ψ ) =
min M & Ψ ( M Φ ( s t ))
− s t +1 & 22 is optimized for all s at all time t .
We propose to conduct this optimization by splitting each s
= { s 1 , ..., s T } into conditional timesequence s c = { s 1 , ..., s T c } and validation time sequence s p = { s T c
+1 , ..., s T } , while using the 171 former for the internal optimization of M to force the linear algebraic relation in the latent spaceand using the latter for the prediction loss.
More precisely, we solve the following optimizationproblem about Φ and Ψ :
[Equation]
M
[Equation] [Equation] where M ∗ ( s c | Φ ) = arg
[Equation]
M
[Equation] It might appear as if we can also set s = s c and optimize the following reconstruction version of Eq.(2):
[Equation]
However, as we will see in the experiment section, the use of the validation sequence s p makes asubstantial difference in the training process.
This is most likely because the minimization of thevalidation error of M ∗ optimized on s c would encourage Φ to exclude the s c -speciﬁc informationfrom the transition M ∗ .
We will illustrate this effect in the experimental section.
We shall notethat we can also parametrize M as M := exp( A ) , where A ∈ R a × a is a Lie algebra element to beinternally optimized.
This type of approach was used in [11] for building Lie group convolutionalnetwork and for [47, 9] in predicting a sequence that is not necessarily stationary.
In order to traintheir model, [47] used additional parameters to diagonalize each algebra element as well as hyperparameters to stabilize the training.
We also experimented with Lie-algebra style representation of
M ∗ and used SGD to internally optimize the exponent parameters, but we needed to carefully tunethe hyper-parameter for the norm regularization term.
Our internal optimization procedure is free ofsuch a parameter tuning.
Internal Optimization of M ∗
Because the internal optimization in Eq.(2) is a linear problem, itcan be solved analytically as
[Equation]
where H +0 =
[ Φ ( s 1 ); ... ; Φ ( s T c − 1 )]
∈ R a × ( T c − 1) s and H +
=
[ Φ ( s 2 ); ... ; Φ ( s T c )
∈ R a × ( T c − 1) sare the horizontal concatenation of the encoded frames and H † +0 is the Moore-Penrose pseudo inverse of H +0 .
Because M ∗ ( s c | Φ ) is a closed form with respect to Φ , the loss (2) can be directlyoptimized by differentiating with respect to the parameters of both Φ and Ψ .
Thus, the training isdone in an end-to-end manner.
Figure 1 and summarizes the overall procedure to make prediction ona given sequence.
We note that, although we have assumed the dataset to consist of constant-velocitysequences, we can readily extend our method to the dataset consisting of the time series with higherorder stationarity, such as constant-acceleration.
See Section 5.4 for the detailed explanation of themodel extension and the experimental results.
Experiments
[Figure]
We conducted several experiments to investigate the efﬁcacy of our framework.
In this section,we brieﬂy explain the experimental settings.
For more details, please see the appendix section D. We tested our framework on Sequential MNIST, 3DShapes [4], and SmallNORB [38].
Sequential MNIST is created from the MNIST [37] dataset.
For all experiments, we used a ResNet [21]-basedencoder-decoder architecture and we set a = 16 and m = 256 so that the latent space lives in R 16 × 256 .
For sequential MNIST, we chose our G to be the set of all combinations of three types of transformations: shape rotation, hue rotation, and translation (See Appendix D for the examples of sequences),and randomly sampled one instance of g ∈ G for each sequence.
To create each sequence, we ﬁrstresized the MNIST image to 24 × 24, applied repetitions of a randomly sampled, ﬁxed member ofg ∈ G and embedded the results to 32 × 32 images.
For shape and hue rotations, we randomly sampled the velocity of angles from uniform distribution on the interval [ − π / 2 , π / 2) for each sequence.
For translation, we randomly sampled the start point and end point in the range of thenmoved the digit images on the straight line between the sampled points.
We also experimented onsequenaital MNIST with background (We call it Sequential MNIST-bg).
For sequantial MNIST-bg,we used the same generation rule as sequential MNIST but we added background images behindthe moving digits.
For the background, we used a randomly sampled images from ImageNet [45]dataset, which were all resized to 32 × 32.
Also, we only used the images of digit 4 for most of theexperiments on sequential MNIST/MNIST-bg.
Unless otherwise noted, all evaluations in this paperfor the sequential MNIST are based on digit4 only training.3DShapes and SmallNORB are datasets with multiple factors of variation.
We created a setof constant-velocity sequences from these datasets by varying a ﬁxed combination of factors foreach sequence.
That is, on these datasets, we chose our G to be the set of variations of factors, andsampled each g as # i g !
i i , where g i represents the increase of i -th factor by one unit and " i ∈ Z .
Thus, the value of " i represents the velocity in i -th factor direction in the grid.
For 3DShapes, wechose wall hue, ﬂoor hue, object hue, scale, and orientation as the factors to vary.
We varied elevation and azimuth for SmallNORB.
For the split of the sequence, we set T c = 2 and T p = 1 on all of [Figure] [Figure] the constant velocity experiments.
We also conducted experiments for the sequences with constantacceleration on sequential MNIST.
To create a sequence with constant acceleration, we chose a pairg a , g v ∈ G for each sequence, and generated s by setting s t
+1 =
g ta g v s t .
We elaborate the detail ofthis extension in 5.4.
As ablations, we tested several variants of our method: Proposed method with ﬁxed {+2x2blocks (When we perform internal optimization, we seperated the latent tensor H ∈ R 
+}
× 256into 8 subtensors { H k ∈ R 2 × m } 8 k =1 and calculated pseudo inverse for each k ), Reconstructionmodel (Training Φ and Ψ with L r with T = 3 ), Neural M ∗ (Abbreviated as Rec.
Model.
Uses an additional neural network to map s c to M ∗ ), and Neural transition (Autoregressive model on Φ ( s t ) ) 1 .
In testing these variants, we used the same pair of encoder and decoder architecture.
For Neural M ∗ ,we trained an additional network M θ that takes a pair of observations to output the transition matrix, replaced M θ with M ∗ in (4), and optimized θ and Φ , Ψ simultaneously.
For Neural transitionmodel, we trained 1x1 1D-convolutional networks on the Φ ( s ) s to produce the latent tensor in thenext time step: For instance, ˆ H t +1 =
1DCNN( Φ ( s t ) , Φ ( s t − 1 )) when T c = 2 .
The 1DCNN wasapplied along the multiplicity dimension ( m ).
5.1 Qualitative and quantitative results on the prediction
Figure 2 shows the example sequences generated by the proposed model and comparative models.
The graph value at t p represents the prediction performance at T c + t p where T c = 2 .
Thus,during the training, the prediction of the error at t p = 1 is back-propageted to the encoder, and theprediction at t p > 1 is an extrapolation.
Our method successfully predicts the images for t p ≥ 1 .
Neural transition and Neural M ∗ models had almost the same prediction performance at t p = 1 , butthey both failed in extrapolation.
Also, our method failed in extrapolation when we parametrized Mwith ﬁxed block structure. This might be because over-regularization hindered with the training of SGD optimization [15]. The training however succeeded when we left more freedom for M .
MNIST
MNIST-bg 3DShapes
SmallNORB
Method
L p L p equiv L p L p equiv
L p
L p equiv
L p L p equiv
Rec.
Model 38.
Neural M ∗ 7.
Ours 5.
[Figure]
To evaluate the wellness of the representation learned by our method, we also predicted the factors of transition from M ∗ (Figure 8 in the appendix A).
SimCLR [6] and Contrastive predictivecoding (CPC) [48, 22] are tested as baselines.
Please see the appendix D for the detailed experimental settings for SimCLR and CPC.
Our method yields better representation than all comparativemethods on both MNIST and MNIST-bg datasets.
5.2 Equivariance performance
As we mentioned in the 3.1 section, by the construction of our framework, the equivarianceis achieved when M ( g, x ) does not depend on x . We therefore do not want its estimator M ∗ ( s ( s 1 , g ) | Φ ) to depend on s 1 , where we recall that s ( s 1 , g ) represents the sequence that beginswith s 1 and transitions with g .
To see how much the trained model is equivariant to the transformation underlying the sequence, we therefore calculated equivariance error as the prediction errorfrom applying M ∗ ( s | Φ ) to Φ ( s ! 2 ) for a pair of s %
= s !
that transitions with common g .
In otherwords, we compute the following;
[Equation]
where S ( g ) represents the set of all sequences that transitions with g .
For each of s , s !
we set T c = 2 and T p = 1 as done in the experiments in the previous sections.
Figure 4 compares the Neural M ∗ methods against our method in terms of equivariance error.
Figure 4a shows the resultof applying M ∗ ( s | Φ ) on Φ ( s ! 2 ) and applying M ∗ ( s ! | Φ ) on Φ ( s 2 ) .
We see that when we swap M ∗this way, Neural M ∗ also swap the images; this implies that M ∗ learned by Neural M ∗ encodessequence speciﬁc information together with the transition.
On the other hand, swapping of M ∗learned by our method does not affect the prediction, suggesting that our method is succeedingto learn an equivariant model.
This is somewhat surprising, because our model does not have theexplicit mechanism to enforce the inter-homogeneity ( P
= I in Section 3.1).
5.3 Measuring the block-ness by Simultaneous diagonalization of M ∗ s
So far, we have observed that our model has an ability to learn G -equivariant model.
Representationtheory gurantees that, if G is a compact connected group, any representation D : G → R a × acan be simultanesouly block-diagonalized; that is, there is a common change of basis U suchthat
UD g U T = $ j V ( j ) g , where V ( j ) g is called irreducible representation that cannot be blockdiagonalized any further. We may have V ( j ) g = 1 ∈ R 1 × 1 for all g for some ( j ) , because
[Figure]
V : G → { 1 } is a valid irreducible representation for any G .
The equivariance of our Φ suggests that 280 M ∗ ( s | Φ ) may be simultaneously block-diagonalizable as well.
This block-diagonalization sometimes reveals disentanglement structure because any irreducible representation of G 1 × G 2 is of form
V (1) ⊗ V (2) , where V ( k ) is an irreducible representation of G k .
In particular, if M ’s irreducible representations have the form V (1) ⊗ 1 or 1 ⊗ V (2) , then each block would either corresponds to theaction of G 1 or of G 2 .
To ﬁnd U that simultaneously block-diagonalizes all M ∗ ( s | Φ ) , we optimized U based on the following objective function that measures the block-ness of ˆ M ∗ ( s ) :
= U − 1 M ∗ ( s | Φ ) U based on thenormalized graph Laplacian operator ∆ :
[Equation] [Equation]
where ˆ M ∗ ( s ) =
abs ( ˆ M ∗ ( s ))
abs ( ˆ M ∗ ( s )) T with abs ( ˆ M ∗ ( s )) representing the matrix suchthat abs
( M ∗ ( s )) ij = | M ∗ ij | .
Our objective function is based on the fact that, if we are given anadjacency matrix A of a graph, then the number of connected components in the graph can be identiﬁed by looking at the rank of graph Laplacian.
For the derivation, please see the appendix E. Figure 5a shows the block structure revealed by simultaneous block-diagonalization through U trainedby minimizing U with respect to the the average of L bd over all s .
Figure 5b shows the resultsof applying just transformation of one block.
We can see that each block only alters one factor ofvariation.
Our results suggest that the learned M ∗ captures the disentangled structure of the groupactions behind the datasets.
5.4 Extension to the sequences with constant acceleration
We have seen that our method successfully learns an equivariant structure from the set of constantvelocity sequences.
In this section, we show that we can extend our concept to the set of sequencessharing the stationarity of higher order (constant acceleration).
By deﬁnition, the pair of Φ ( s t ) and Φ ( s t − 1 ) encodes the information about the velocity at t .
Whenthe multiplicity m is sufﬁciently large, the velocity can be estimated by: 1 M t =
Φ ( s t ) Φ ( s t − 1 ) † .
Because this would yield a sequence of velocities, we can simply apply our method again to estimatethe constant acceleration by 2 M ∗ = 1 M +11 M † +0 where 1 M +0 =
[ 1 M 2 ; ... ; 1 M T c − 1 ] ∈ R a × ( T c − 1) sand 1 M +1 =
[ 1 M 3 ; ... ; 1 M T c ] ∈ R a × ( T c − 1) s .
We can then predict the future representation ˜ s t fort =
T c + 1 , ..., T p by
[Equation]
& where
[Equation] [Figure]
Neural transition overﬁtted and collapsed from t p = 5 (beyond the training horizon).
where # represents multiplications from left.
We train Ψ and Ψ by minimizing the mean squarederror between ˜ s t and s t for t =
T c + 1 , ..., T p as in Eq.(2).
To create a sequence of constantacceleration from MNIST dataset, we only used shape and color rotations.
We chose the initialvelocity for these rotations randomly on the interval [ − π / 5 , π / 5) for each sequence, and chose theacceleration on the interval [ − π / 40 , π / 40) .
The results are shown in Figure 6.
The Neural transitionand the constant-velocity version of our method failed to predict he accelerated sequence, while the2nd order model succeeded in predicting the sequence even after t p > 5 .
Discussion & Limitations
[Figure]
How is full equivariance achieved in our method?
The theoretical results we provided in section3.1 only assures that M ( g, x ) and M ( g, x ! ) are similar when the underlying group is commutative,compact and connected.
However, as we have shown in experiment, our method seems to be learningΦ for which the estimators of M satisﬁes M ∗ ( s ( g, x ) | Φ ) ∼ =
M ∗ ( s ( g, x ! ) | Φ ) .
This can be happeningbecause our framework and the training method based on the internal optimization in the latent spaceis somehow encouraging M ∗ to be orthogonal (See the loss curve of orthogonality of M ∗ in theappendix A).
Maybe this is forcing the change of matrix P such that PM ( g, x ) P −
=
M ( g, x ! ) tobe also rotations as well, which commutes with M ( g, x ) itself.
Also, Figure 2b and 3 show that, asreported in [28], the models trained with reconstruction loss like (3) does not well capture the grouptransformation behind the sequences: the encoder representation was found to be signiﬁcantly worsethan that of the model trained with (2).
We hypothesize that (3) fails to remove the sequence-speciﬁcinformation from M ∗ , while (3) succeeds to do so by training the model to be able to predict theunseen images.
Towards learning symmetries from more realistic observations As we are making a connectionbetween our prediction framework and group equivariance, we are essentially assuming that thetransitions are always invertible, because group is closed under inversion.
However, this might notbe always the case in real world applications; for instance, if the image sequences are the sequentialrenderings of a rotating 3D object, the transitions are generally not invertible because only a partof the object is visible at each time step.
We experimented Sequential ShapeNet, which is createdfrom ShapeNet [5] dataset.
A series of rendered images is generated by sequentially applying 3Drotations of different speeds for each axis.
Generated results on sequential ShapeNet (See 13 inthe appendix B shows that actually the our current method was not able to generate the images on3D rotation datasets.
If the transitions are not invertible, some measures must be taken in order toresolve the indeterminacy, such as probabilistic modeling or additional structural inductive bias.
Because our study generally contributes to predictions and extrapolation, it has as much potentialto negatively affect the society as most other prediction methods.
In particular, applications of ourmethod to image sequence can be potentially integrated into weapon systems, for example.
At thesame time, our unsupervised learning of the symmetrical structure from sequential dataset may alsocontribute to new discoveries in the systems of ﬁnance, medical science, physics and other ﬁelds of ML such as reinforcement learning.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] . You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s con- tributions and scope?
[Yes] The abstract and introduction reﬂects the contributions of thispaper.(b)
Did you describe the limitations of your work?
[Yes] In the Discussion & Limitations section, we discuss the type of dataset to which our work cannot be directly applied, as well asthe theoretical problem that has not been solved.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We includedthe broader impact discussion in the section 7(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes] We2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We included the fullset of assumptions.
We provide the details of the assumptions in the supplementary material.(b)
Did you include complete proofs of all theoretical results?
[Yes] We provide complete proofsof theoretical results in Supplementary material.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] We provided the code for
reproducing the experimental results as a supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We explained how we chose the hyper parameters, the data splits in thesection 5 and the appendix section D.
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] We include the standard deviation values of equivariance perfor- mance in the appendix section A (We omitted them in main submission because of the spacelimitation) and the linear classiﬁcation performance of the transition parameters in the appendix A.
(d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[Yes] We explained the number of GPUs to train
[Table]
the models and approximated costs for reproducing the full results of experiments in D.
[Table] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited [37] for MNIST,[5] for ShapeNet, [29] for 3Dshapes and [38] for SmallNORB.
(b) Did you mention the license of the assets?
[Yes] For each dataaset we used, we included theURL to the license information in the reference section.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A](d)
Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] For ShapeNet dataset, we are approved to download the assets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable in-
formation or offensive content?
[Yes] We don’t ﬁnd any personally identiﬁable information in the images used in our experiments.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] [Table]
(b) Did you describe any potential participant risks, with links to Institutional Review Board [Table] (IRB) approvals, if applicable?
[N/A] [Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount spenton participant compensation?
[N/A]
[Table]