Linear Transformations in Autoencoder Latent Space Predict Time Translations in Active Matter System
Abstract
Machine Learning (ML) approaches are promising for deriving dynamical predictions of physical systems from data.
ML approaches are relevant in active matter, a field that spans scales and studies dynamics of far-from-equilibrium systems where there are significant challenges in predicting macroscopic behavior from microscopic interactions of active particles.
A major challenge in applying ML to active systems is encoding a continuous representation of time within a neural network.
In this work, we develop a framework for predicting the dynamics of active networks of protein filaments and motors by combining a low-dimensional latent representation inferred through an autoencoder with a linear shift neural network that encodes time translation as a linear transformation within the latent space.
Our method enables predicting the contraction and boundary deformations of active networks with various geometries.
Although our method is trained to predict 20 time steps into the future, it can generalize to periods of 60 time steps and recapitulate the past 30 frames of a single given observation with less than 10% error.
Finally, we derive an approximate analytic expression for the linear transformation in the latent space that captures the dynamics.
Broadly, our study reveals that neural networks are powerful for forecasting the behavior of active matter systems in the complete absence of knowledge of the microscopic dynamics.
Introduction
Active matter systems are made of many autonomous agents that consume energy to generate motion.
Examples of living and non-living active systems across scales include the cellular cytoskeleton, bacterial colonies, schools of fish and drone swarms, to name a few [1].
To engineer active systems at all scales and in different settings, we need to predict how active systems self-organize into collective spatiotemporal patterns [2].
A fundamental limitation for mechanistic models of active matter arises from its out-of-equilibrium nature, which breaks detailed balance, and time-reversal symmetry [3].
Furthermore, the vast diversity of active agents makes it difficult to find a comprehensive theoretical framework of active matter.
An alternative approach to forecast the dynamics of active systems that bears great potential is Machine Learning.
Specifically, Deep Neural Networks (DNNs) offer data-driven model-free predictions of complex systems [4, 5].
The multilayered hierarchical structure of DNNs provides an automated way to identify the most relevant degrees of freedom of a system by iteratively coarse-graining structured data without any prior knowledge of the underlying complexity [6].
However, the designing principles of DNNs’ architectures that model active systems, as well as the physical interpretations of the representations learned by DNNs, are still largely unexplored.
In this work, we use DNNs to investigate the dynamics of an optically-controlled active matter system composed of microtubule filaments and light-switchable kinesin motor proteins [7].
In the active system under study, light patterns can be used to determine when and where motors reorganize microtubules to create non-equilibrium structures.
We predict the dynamics of macroscopic patterns that emerge from light-induced microtubule reorganization using what we refer to as the Translational
Convolutional Autoencoder (TCAE).
TCAE learns a low-dimensional representation of microscopy data, constrained by a learned translation function.
The translation function makes it possible to map input time differences to translation vectors in latent space.
By feeding an initial image observation and varying scalar time differences to the TCAE, we can evolve the single observations through time and achieve low-error predictions.
Although our method is trained to predict a fixed future time interval, it generalizes to prolongated periods.
Surprisingly it can also infer the past dynamics if negative scalar time differences are provided to the TCAE.
We hypothesize that the physical properties of the active system are embedded in the latent space our DNN constructs.
This work shows the potential DNNs have for extracting physical properties and predicting the behavior of complex non-linear out-of-equilibrium systems.
Methodology 2.1 Data Description
The data consists of 16 active matter experiments as described in [8].
The light patterns include 4 different shapes (circle, hexagon, rectangle, and triangle), with four initial sizes ( 900 , 750 , 600 , 450 µ m ).
The raw videos have a resolution of 2048 × 2048 , and a total video length of 170 frames.
We down-sampled the videos to a resolution of 112 × 112 pixels, and normalized to have a range of values between 0 and 1.
We chose 15 videos for training and left 1 for test evaluation.
2.2 Approach
We extend the input reconstruction capabilities of the Convolutional Autoencoder [9] for time propagation prediction.
Our proposed method takes as input the image at time t and a change in time ∆ t to output a prediction for the image at time t + ∆ t .
First, an encoder function E : R m × m (cid:55)→ R n maps an image X st to a latent representation z st , where s indexes the list of movies, and t is time.
Then, a linear time translation operator translates the image embedding z st by a learned time-translation Φ , scaled by time shift ∆ t ; namely z st (cid:55)→ z
st + Φ ∆ t .
Finally, the decoder D : R n (cid:55)→ R m × m takes the translated image embedding and predicts the frame at time t + ∆ t ; i.e. D ( z st + Φ ∆ t ) = (cid:101)
X st +∆ t .
We restrict our training task to predict for ∆ t = { 1 , . . .
, 20 } frames into the future.
With training dataset
X = { X st } (cid:12)(cid:12) s ∈ [1 , 15] t ∈ [1 , 170] ; and a dissimilarity (i.e. loss) function L , we find the network operators:
[Equation]
We choose L to be a summation over all pixels of the pixel-wise binary cross-entropy function:
L ( (cid:101) X st , X st ) =
(cid:80) i,j H ( p ij , ˜ p ij ) where ( i, j ) labels the pixels, and p ij ,
˜ p ij are the values of pixels in
X st and (cid:101) X st , respectively.
The cross entropy reads H ( p, ˜ p ) =
− ( p log(˜ p ) + (1 − p ) log(
− ˜ p )) .
Results 3.1 Interpretation of Time Translation in the Latent Space
Calculating the mismatch between the ground truth and predicted images exhibits up to ∼ 95% accuracy of prediction; see Fig.
Such a high accuracy is suggestive of a dynamical mechanism that can be recapitulated by uniform translation of the latent space.
In the 2D latent space spanned by coordinates ( z 1 , z 2 ) .
We expand the argument of the function D , i.e. the latent space coordinates z + Φ ∆ t ; and demand that L ( D ( z ( t ) + Φ ∆ t ) − D ( z ( t + ∆ t ))) is minimized.
Note that cross entropy is always non-negative; hence zero lower bound.
For the first term we have: D ( z ( t ) + Φ ∆ t ) =
D ( z ( t ))
+ ( Φ ∆ t ) ∇ z ( t ) D
( z ( t ))
+ O ( Φ 2 ∆ t 2 ) , and for the second term: D ( z ( t + ∆ t )) =
D ( z ( t ))
+ ( ∂ t z ( t )∆ t ) ∇ z ( t ) D
( z ( t ))
+ O ( ∂ t z ( t ) 2 ∆ t 2 ) .
Since the loss function is minimized over the entire data set, we claim that ⟨ Φ − ∂ t z ( t ) ⟩ s,t, ∆ t = 0 ; thus Φ = ⟨ ∂ t z ( t ) ⟩ s,t, ∆ t .
Here ⟨•⟩ s,t, ∆ t denotes averaging over all the data sets, time points, and time translations.
Therefore, to the first order approximation, Φ can be thought of as the average over ∂ t z ( t ) , namely the slopes of the lines connecting the points z ( t ) and z ( t + ∆ t ) for all the points along the ∆ t = 0 curves.
The average over t , is equivalent to connecting the two ends of the curves: ∆ z = z ( t max ) − z
( t min ) .
Averaging these vectors over all datasets we obtain the translation Φ that minimizes the loss function.
Finally we shall mention that the qualitative similarity between the curves corresponding to different geometries suggests that they would possibly collapse onto a single curve upon performing rotation and scaling transformations in the latent space.
Therefore, while the latent space carries information about the geometry and size, its 2D embedding predominantly captures the features that are independent of specific shapes, namely the contraction of the networks.
3.2 Active Matter Dynamics Prediction Analysis
An example of predictions on the test dataset made by the TCAE is shown in Fig.
(3C).
Perceptually, the predictions made by taking the test frame at t = 50 as starting point agree with unseen experimental data up to ∆ t = 60.
We can observe the contraction and shape preservation of the microtubule network.
Although in the training phase, we constrained ∆ t to take positive values between 0 and 20, the time embedding Φ can be scaled by negative values, namely time translation in the opposite direction.
Given that we can move in either time direction in the latent space, TCAE can perform whole video predictions from a single given frame.
An example is shown in Fig.
(3B).
We see that
TCAE predictions follow closely the microtubule network area dynamics observed in experimental data at least for 90 percent of the predicted frames from t = 33 to t = 150 .
Finally, we define a simple metric to evaluate the shape mismatch between the ground truth and the predicted segmented (binary) images, X TCAE and X GT .
The normalized mismatch index ε is defined as deviation of the overlapping area from unity.
The overlapping area is in turn defined as the area of the bit-wise product of the two binary images, normalized by the area of the ground truth image.
Using the double-sum notation A : B = (cid:80) i,j A ij B ij , we get: ε = 1 − ( X TCAE :
X GT ) /
( X GT : X GT ) .
This metric is used to quantify the error of whole video prediction across all frames.
Results are shown in
Fig.
(3A).
For 90% of frames, it is possible to predict on average 38.5 frames into the future with less than 10% error.
A maximum of 57 future frames (with less than 10% error) can be predicted if we fix t = 66 , 67 .
If we consider predicting past dynamics, we can take 90% of frames to predict on average 25.6 frames with less than 10% error.
If we set t = 79 , 80 , we can reach a maximum offrames in the past (with less than 10% error).
Summary
We propose TCAE, an autoencoder-based framework that incorporates a linear shift neural network that encodes time translation as a linear transformation within the latent space.
Using our method, we construct a latent representation of an active matter system that predominantly captures the contraction of its microtubule networks.
By leveraging the constructed latent representation, we can make whole video predictions from single observations.
We obtained high accuracy (less than 10% error) in our predictions when compared to unseen ground truth data.
Acknowledgments and Disclosure of Funding
The authors are grateful to Emanuel Flores Bautista and Guruprasad Raghavan for scientific discussions.
We acknowledge funding through the Packard Foundation, the Moore Foundation and the
Heritage medical research institute.
The authors declare no competing financial interests.