Original: L EARNING S TATE R EPRESENTATIONS IN C OMPLEX S YSTEMS WITH M ULTIMODAL D ATA Anonymous authors Paper under double-blind review
Revised: P3acani

Original: A BSTRACT
Revised: A BSTRACT

Original: Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors.
Revised: Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors.

Original: Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the ﬁeld still lacks a large-scale standard dataset for uniﬁed comparison.
Revised: Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the ﬁeld still lacks a large-scale standard dataset for uniﬁed comparison.

Original: In this work, we present a large-scale dataset and evaluation framework for representation learning for the complex task of landing an airplane.
Revised: In this work, we present a large-scale dataset and evaluation framework for representation learning for the complex task of landing an airplane.

Original: We implement and compare several approaches to representation learning on this dataset in terms of the quality of simple supervised learning tasks and disentanglement scores.
Revised: We implement and compare several approaches to representation learning on this dataset in terms of the quality of simple supervised learning tasks and disentanglement scores.

Original: The resulting representations can be used for further tasks such as anomaly detection, optimal control, model-based reinforcement learning, and other applications
Revised: The resulting representations can be used for further tasks such as anomaly detection, optimal control, model-based reinforcement learning, and other applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In order to be able to act in real world scenarios and control a complex system such as an airplane, a car, or an industrial facility, an automated agent needs to process very complex high-dimensional data coming from different domains: video feeds from different cameras, LIDAR sensors on a car, altitude and speed sensors on an airplane, various sensors related to the internal state of the system, and so on.
Revised: In order to be able to act in real world scenarios and control a complex system such as an airplane, a car, or an industrial facility, an automated agent needs to process very complex high-dimensional data coming from different domains: video feeds from different cameras, LIDAR sensors on a car, altitude and speed sensors on an airplane, various sensors related to the internal state of the system, and so on.

Original: An important problem in this regard would be to map this rich stream of multimodal information into a lower-dimensional space that would compress all modalities into a uniform latent representation (embedding); the agent could then use this embedding to learn or otherwise construct control algorithms.
Revised: An important problem in this regard would be to map this rich stream of multimodal information into a lower-dimensional space that would compress all modalities into a uniform latent representation (embedding); the agent could then use this embedding to learn or otherwise construct control algorithms.

Original: Thus, representation learning lies at the heart of optimal control for complex systems with multimodal unstructured features.
Revised: Thus, representation learning lies at the heart of optimal control for complex systems with multimodal unstructured features.

Original: Over the last decade, deep neural networks have surpassed other methods in processing nearly all modalities of high-dimensional unstructured data, including images, natural language texts, sounds, and time series.
Revised: Over the last decade, deep neural networks have surpassed other methods in processing nearly all modalities of high-dimensional unstructured data, including images, natural language texts, sounds, and time series.

Original: One of the most important properties of neural networks that has made the deep learning revolution possible is their ability to extract meaningful low-dimensional representations of raw unstructured input data.
Revised: One of the most important properties of neural networks that has made the deep learning revolution possible is their ability to extract meaningful low-dimensional representations of raw unstructured input data.

Original: Representation learning with deep neural networks is a large and wellestablished area of research (Kingma & Welling, 2013; Hinton & Salakhutdinov, 2006; Radford et al., 2015).
Revised: Representation learning with deep neural networks is a large and wellestablished area of research (Kingma & Welling, 2013; Hinton & Salakhutdinov, 2006; Radford et al., 2015).

Original: Latent features learned by deep neural networks ﬁnd applications in various domains, including reinforcement learning for complex systems (Zhang et al., 2018; Watter et al., 2015; van
Revised: Latent features learned by deep neural networks ﬁnd applications in various domains, including reinforcement learning for complex systems (Zhang et al., 2018; Watter et al., 2015; van

Original: Hoof et al., 2016).
Revised: Hoof et al., 2016).

Original: In these works, the authors often propose special techniques and architectures to design the latent space in different ways suitable for further use: make the latent space locally linear, capture the dynamics, and so on.
Revised: In these works, the authors often propose special techniques and architectures to design the latent space in different ways suitable for further use: make the latent space locally linear, capture the dynamics, and so on.

Original: Designing such feature extractors is a complex task, usually done by hand.
Revised: Designing such feature extractors is a complex task, usually done by hand.

Original: Moreover, it is hard to compare different architectures, to a large extent because it is far from obvious how to measure the quality of resulting representations.
Revised: Moreover, it is hard to compare different architectures, to a large extent because it is far from obvious how to measure the quality of resulting representations.

Original: One reason for that is that the best metric for the quality of learned representations would be the quality of the ﬁnal task in question, which is hard to obtain in reinforcement learning due to the sheer scale of this ﬁnal task.
Revised: One reason for that is that the best metric for the quality of learned representations would be the quality of the ﬁnal task in question, which is hard to obtain in reinforcement learning due to the sheer scale of this ﬁnal task.

Original: A common way to deal with such problems is to use a uniﬁed dataset and a uniﬁed set of benchmarks, such as, for example, ImageNet and the ILSVRC benchmark in computer vision (Russakovsky et al., 2015).
Revised: A common way to deal with such problems is to use a uniﬁed dataset and a uniﬁed set of benchmarks, such as, for example, ImageNet and the ILSVRC benchmark in computer vision (Russakovsky et al., 2015).

Original: Such uniﬁed datasets might also prove useful for transfer learning tasks.
Revised: Such uniﬁed datasets might also prove useful for transfer learning tasks.

Original: However, the ﬁeld of reinforcement learning for complex systems is yet to agree on a common representative large-scale dataset.
Revised: However, the ﬁeld of reinforcement learning for complex systems is yet to agree on a common representative large-scale dataset.

Original: In this work, we present such a multimodal dataset for the representation learning problem together with a uniﬁed benchmark framework for feature extractors suitable for a comprehensive comparative evaluation of different feature extractors.
Revised: In this work, we present such a multimodal dataset for the representation learning problem together with a uniﬁed benchmark framework for feature extractors suitable for a comprehensive comparative evaluation of different feature extractors.

Original: This dataset has been gathered with large- scale computer simulations based on the X-Plane simulator and consists of data streams from various sensors along with images taken from the frontal camera of the plane.
Revised: This dataset has been gathered with largescale computer simulations based on the X-Plane simulator and consists of data streams from various sensors along with images taken from the frontal camera of the plane.

Original: We propose and implement several different metrics for comparison between extracted features.
Revised: We propose and implement several different metrics for comparison between extracted features.

Original: We also survey, implement, and compare different neural architectures for learning multimodal state representations.
Revised: We also survey, implement, and compare different neural architectures for learning multimodal state representations.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: In Section 2 we survey related work on representation learning and evaluation of representations.
Revised: In Section 2 we survey related work on representation learning and evaluation of representations.

Original: Section 3 presents the X-Plane dataset and explains its characteristic features.
Revised: Section 3 presents the X-Plane dataset and explains its characteristic features.

Original: In Section 4, we present the various representation learning models that we have implemented and compared on this dataset.
Revised: In Section 4, we present the various representation learning models that we have implemented and compared on this dataset.

Original: Section 5 shows the evaluation metrics and presents a large-scale comparison across different models, and Section 6 concludes the paper.
Revised: Section 5 shows the evaluation metrics and presents a large-scale comparison across different models, and Section 6 concludes the paper.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Modern agent control methods commonly use techniques based on deep learning as feature extractors to deal with complex multimodal data, either explicitly (Zhang et al., 2018;
Revised: Modern agent control methods commonly use techniques based on deep learning as feature extractors to deal with complex multimodal data, either explicitly (Zhang et al., 2018;

Original: Watter et al., 2015) or implicitly (Mnih et al., 2015).
Revised: Watter et al., 2015) or implicitly (Mnih et al., 2015).

Original: Explicit techniques separate representations of the learning environment from the agents operating in this enviroment; the environment representation can be learned either independently (Zhang et al., 2018) or in a common end-to-end architecture (Parisi et al., 2017).
Revised: Explicit techniques separate representations of the learning environment from the agents operating in this enviroment; the environment representation can be learned either independently (Zhang et al., 2018) or in a common end-to-end architecture (Parisi et al., 2017).

Original: There are several major approaches to constructing such models (Lesort et al., 2018): (1) autoencoders that reconstruct the input observation data, producing the latent representation between encoder and decoder; (2) forward models that predict the next state either in the latent space or in the raw data, basing the prediction on the current latent representation; (3) inverse models that use two states to predict the actions between them; this approach can be combined with forward models; (4) models with prior knowledge of the system that impose additional constraints on the latent space according to some fundamental system properties such as causality, temporal continuity, or some more speciﬁc knowledge about the system (Jonschkowski & Brock, 2015).
Revised: There are several major approaches to constructing such models (Lesort et al., 2018): (1) autoencoders that reconstruct the input observation data, producing the latent representation between encoder and decoder; (2) forward models that predict the next state either in the latent space or in the raw data, basing the prediction on the current latent representation; (3) inverse models that use two states to predict the actions between them; this approach can be combined with forward models; (4) models with prior knowledge of the system that impose additional constraints on the latent space according to some fundamental system properties such as causality, temporal continuity, or some more speciﬁc knowledge about the system (Jonschkowski & Brock, 2015).

Original: If there is no additional data labeling available such as, e.g., a list of factors, the most direct way to measure the quality of representations would be by evaluating the ﬁnal quality of solving the main task that a model can achieve based on this representation.
Revised: If there is no additional data labeling available such as, e.g., a list of factors, the most direct way to measure the quality of representations would be by evaluating the ﬁnal quality of solving the main task that a model can achieve based on this representation.

Original: However, in real world cases, in particular in reinforcement learning, the main task is often hard to solve and unstable to train, so it cannot be consistently used to evaluate latent representations.
Revised: However, in real world cases, in particular in reinforcement learning, the main task is often hard to solve and unstable to train, so it cannot be consistently used to evaluate latent representations.

Original: Therefore, many indirect ways have been proposed to measure the quality of representations that introduce proxies that can be expected to lead to better solutions of the ﬁnal control problem.
Revised: Therefore, many indirect ways have been proposed to measure the quality of representations that introduce proxies that can be expected to lead to better solutions of the ﬁnal control problem.

Original: The most common indirect approaches include (see also a comprehensive survey by Lesort et al.
Revised: The most common indirect approaches include (see also a comprehensive survey by Lesort et al.

Original: (2018)): (1) task performance , the most intuitive metric, where representation quality is measured by the quality of performing some other relatively simple task, e.g., by predicting some available target variables with simple models that take the latent representation as input (Higgins et al., 2016; van Hoof et al., 2016); (2) KNN-MSE , proposed by Lesort et al.
Revised: (2018)): (1) task performance , the most intuitive metric, where representation quality is measured by the quality of performing some other relatively simple task, e.g., by predicting some available target variables with simple models that take the latent representation as input (Higgins et al., 2016; van Hoof et al., 2016); (2) KNN-MSE , proposed by Lesort et al.

Original: (2017), measures the degree of preservation of the same neighbors between the latent space and the ground truth:
Revised: (2017), measures the degree of preservation of the same neighbors between the latent space and the ground truth:

Original: KNN [Equation]
Revised: KNN-MSE ( I ) =

Revised: 1 k (cid:80) I (cid:48) ∈ KNN( I,k ) (cid:107) φ

Original: ( I ) [Equation] [Equation]
Revised: ( I ) − φ ( I (cid:48) )

Original: (cid:107) , where I is the initial raw input, I (cid:48) is a neighbour of I , and φ is the feature extractor; KNN-MSE is a good metric in situations where the distance in the original input space is well-deﬁned but becomes hard to apply for highly variable multimodal data; (3) a similar approach with humans in the loop is to evaluate whether similar input states according to human evaluation do indeed map to close representations in the latent space (Sermanet et al., 2018) ; however, for complex multimodal inputs this is again inapplicable since a human would not have an intuitive notion of similarity between two sets of several hundred sensor readings; (4) disentanglement scores (Eastwood & Williams, 2018; Higgins et al., 2016) measure the disentanglement (mutual independence) of extracted features; if there are some known generative factors, these metrics assess whether individual elements of the latent representation capture individual generative factors independently; we will consider such metrics in detail in Section 5.1.
Revised: (cid:107) , where I is the initial raw input, I (cid:48) is a neighbour of I , and φ is the feature extractor; KNN-MSE is a good metric in situations where the distance in the original input space is well-deﬁned but becomes hard to apply for highly variable multimodal data; (3) a similar approach with humans in the loop is to evaluate whether similar input states according to human evaluation do indeed map to close representations in the latent space (Sermanet et al., 2018); however, for complex multimodal inputs this is again inapplicable since a human would not have an intuitive notion of similarity between two sets of several hundred sensor readings; (4) disentanglement scores (Eastwood & Williams, 2018; Higgins et al., 2016) measure the disentanglement (mutual independence) of extracted features; if there are some known generative factors, these metrics assess whether individual elements of the latent representation capture individual generative factors independently; we will consider such metrics in detail in Section 5.1.

Original: As a representative multimodal dataset for a complex system, we have used the X-Plane ﬂight simulator (Laminar Research, 2018), well-known for its faithful simulation of all systems of an aircraft.
Revised: As a representative multimodal dataset for a complex system, we have used the X-Plane ﬂight simulator (Laminar Research, 2018), well-known for its faithful simulation of all systems of an aircraft.

Original: It has already been used to solve optimal control problems for aircraft; e.g., Bittar et al.
Revised: It has already been used to solve optimal control problems for aircraft; e.g., Bittar et al.

Original: (2014) develop separate control laws for stable and maneuvering ﬂight, and Garcia & Barnes (2009) use X-Plane to simulate a system of several unmanned aerial vehicles.
Revised: (2014) develop separate control laws for stable and maneuvering ﬂight, and Garcia & Barnes (2009) use X-Plane to simulate a system of several unmanned aerial vehicles.

Original: However, to the best of our knowledge this is the ﬁrst attempt to produce a large-scale dataset for representation learning from X-Plane or any other ﬂight simulator.
Revised: However, to the best of our knowledge this is the ﬁrst attempt to produce a large-scale dataset for representation learning from X-Plane or any other ﬂight simulator.

Original:  T HE X-P LANE D ATASET
Revised:  T HE X-P LANE D ATASET

Original: We present the X-Plane Dataset (Dataset, 2018; Code, 2018) as a benchmark for evaluating the quality of state representations (embeddings), where the main problem is to encode the state of some complex system, learn a mapping from high-dimensional multimodal data to a low-dimensional latent representation.
Revised: We present the X-Plane Dataset (Dataset, 2018; Code, 2018) as a benchmark for evaluating the quality of state representations (embeddings), where the main problem is to encode the state of some complex system, learn a mapping from high-dimensional multimodal data to a low-dimensional latent representation.

Original: We have used the X-Plane simulation environment because it is extremely accurate and can produce a lot of useful and different sensor data.
Revised: We have used the X-Plane simulation environment because it is extremely accurate and can produce a lot of useful and different sensor data.

Original: In total, we have recordedlandings, with mean duration of a landing of 115 seconds.
Revised: In total, we have recordedlandings, with mean duration of a landing of 115 seconds.

Original: For every landing, the dataset contains the readings of 1090 sensors arranged in time series recorded with with a frequency of 5 frames per second, together with the corresponding image taken from the camera located at the front of the airplane; the images are recorded at the same frequency.
Revised: For every landing, the dataset contains the readings of 1090 sensors arranged in time series recorded with with a frequency of 5 frames per second, together with the corresponding image taken from the camera located at the front of the airplane; the images are recorded at the same frequency.

Original: Table 1 summarizes the various groups of sensors recorded in the dataset, showing the number of sensors in a group together with a brief description.
Revised: Table 1 summarizes the various groups of sensors recorded in the dataset, showing the number of sensors in a group together with a brief description.

Original: The total dataset size is 93GB uncompressed; it contains sensor readings and about 7 M 256 × 256 images, joined into per-ﬂight videos for better compression.
Revised: The total dataset size is 93GB uncompressed; it contains sensor readings and about 7 M 256 × 256 images, joined into per-ﬂight videos for better compression.

Original: To ensure diversity in the dataset, we recorded landings with random perturbations and in different environments: with different airports and runways, time of day, weather conditions etc.
Revised: To ensure diversity in the dataset, we recorded landings with random perturbations and in different environments: with different airports and runways, time of day, weather conditions etc.

Original: We have used at most 12 perturbations (abrupt changes in the environment such as, e.g., wind gusts, malfunction of various systems in the aircraft, and so on) applied during each ﬂight; Table 2 summarizes various failures and perturbations.
Revised: We have used at most 12 perturbations (abrupt changes in the environment such as, e.g., wind gusts, malfunction of various systems in the aircraft, and so on) applied during each ﬂight; Table 2 summarizes various failures and perturbations.

Original: We have used a custom control unit based on the Boeing 737 guide (Brady, 2014) in order to make automatic landings stable.
Revised: We have used a custom control unit based on the Boeing 737 guide (Brady, 2014) in order to make automatic landings stable.

Original: During dataset collection, for every landing we chose a random airport and a random runway and spawned a plane in a random position, usually at a distance of 3-5 miles from the runway.
Revised: During dataset collection, for every landing we chose a random airport and a random runway and spawned a plane in a random position, usually at a distance of 3-5 miles from the runway.

Original: There are, in total, 114 different runways in 70 airports used in the dataset.
Revised: There are, in total, 114 different runways in 70 airports used in the dataset.

Original: Then we applied different landing conditions—landing speed, ﬂaps position, time of day, visibility, precipitation and so on— and enabled the autopilot that lowered the plane on the glide slope path.
Revised: Then we applied different landing conditions—landing speed, ﬂaps position, time of day, visibility, precipitation and so on— and enabled the autopilot that lowered the plane on the glide slope path.

Original: To achieve a better landing, we disable the autopilot before touchdown, increase the pitch of the plane, and after touchdown we decrease the pitch and enable reverse thrust.
Revised: To achieve a better landing, we disable the autopilot before touchdown, increase the pitch of the plane, and after touchdown we decrease the pitch and enable reverse thrust.

Original: At the end of the ﬂight, there are three possible situations: successful landing, aircraft crash, or time limit reached.
Revised: At the end of the ﬂight, there are three possible situations: successful landing, aircraft crash, or time limit reached.

Original: We set a strict time limit ofseconds for every ﬂight.
Revised: We set a strict time limit ofseconds for every ﬂight.

Original: We have also done an airport-stratiﬁed split of the dataset into 4 parts: training set for feature extractors, validation set used for early stopping, training set for benchmark supervised learning models, and test set for scoring the benchmark models.
Revised: We have also done an airport-stratiﬁed split of the dataset into 4 parts: training set for feature extractors, validation set used for early stopping, training set for benchmark supervised learning models, and test set for scoring the benchmark models.

Original: We have made the dataset available at (Dataset, 2018).
Revised: We have made the dataset available at (Dataset, 2018).

Original:  M ODELS
Revised:  M ODELS

Original: For experimental evaluation of different representation learning models, we used a wide variety of different models, from basic autoencoders to dynamic actions-aware encoders.
Revised: For experimental evaluation of different representation learning models, we used a wide variety of different models, from basic autoencoders to dynamic actions-aware encoders.

Original: In this section, we present these models, from simple to more complex ones.
Revised: In this section, we present these models, from simple to more complex ones.

Original: Our models were constructed with four core building blocks: standard recurrent LSTM cell (Hochreiter & Schmidhuber, 1997), onedimensional convolutions (Conv1D), ConvLSTM (Shi et al., 2015), and an attention cell (Bahdanau et al., 2014).
Revised: Our models were constructed with four core building blocks: standard recurrent LSTM cell (Hochreiter & Schmidhuber, 1997), onedimensional convolutions (Conv1D), ConvLSTM (Shi et al., 2015), and an attention cell (Bahdanau et al., 2014).

Original: The attention cell A ( x ) operates as follows (see Fig. 2):
Revised: The attention cell A ( x ) operates as follows (see Fig. 2):

Original: [Equation]
Revised: [Equation]

Original: where x is the input, W is a d × N matrix for embedding size d and input dimension N , V is the matrix with columns v i , c is a vector of size d , s are attention weights.
Revised: where x is the input, W is a d × N matrix for embedding size d and input dimension N , V is the matrix with columns v i , c is a vector of size d , s are attention weights.

Original: We considered the following speciﬁc models (Table 4 lists all models with brief descriptions and numerical results).
Revised: We considered the following speciﬁc models (Table 4 lists all models with brief descriptions and numerical results).

Original: One-dimensional autoencoders.
Revised: One-dimensional autoencoders.

Original: A simple autoencoder that trains to reconstruct a given set of timesteps (Fig. 2a).
Revised: A simple autoencoder that trains to reconstruct a given set of timesteps (Fig. 2a).

Original: We used four types of autoencoders (AE) in the comparison: autoencoder with 1-layer LSTM cells for encoder and decoder, with 2-layer LSTM cells, with a 6-layer convolutional autoencoder with kernel size 7.
Revised: We used four types of autoencoders (AE) in the comparison: autoencoder with 1-layer LSTM cells for encoder and decoder, with 2-layer LSTM cells, with a 6-layer convolutional autoencoder with kernel size 7.

Original: In LSTM autoencoders, after the encoder block we used simple averaging of vectors from all timestamps in the time series, using the result as embedding.
Revised: In LSTM autoencoders, after the encoder block we used simple averaging of vectors from all timestamps in the time series, using the result as embedding.

Original: In
Revised: In

Original: Conv1D, after the encoder block we used max-pooling to get a single vector from the time series; in the decoder, it is copied the necessary number of times and fed to another Conv1D block.
Revised: Conv1D, after the encoder block we used max-pooling to get a single vector from the time series; in the decoder, it is copied the necessary number of times and fed to another Conv1D block.

Original: In our experiments, we used 6 layers of Conv1D with kernel size 7.
Revised: In our experiments, we used 6 layers of Conv1D with kernel size 7.

Original: Image Autoencoders.
Revised: Image Autoencoders.

Original: We trained two image autoencoders for reconstructing individual pictures from the ﬂight time series.
Revised: We trained two image autoencoders for reconstructing individual pictures from the ﬂight time series.

Original: The ﬁrst autoencoder uses a PCA encoder and a PCA decoder.
Revised: The ﬁrst autoencoder uses a PCA encoder and a PCA decoder.

Original: To obtain
Revised: To obtain

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the features from a ﬂight sample, we apply the encoder for each image in the sequence and compute the average of all resulting feature vectors.
Revised: the features from a ﬂight sample, we apply the encoder for each image in the sequence and compute the average of all resulting feature vectors.

Original: The second architecture is a bit more complicated: its encoder contains the ResNet34 (He et al., 2016) architecture with an additional fully connected layer whose size varies depending on the embedding size d , and whose decoder contains ﬁfteen two-dimensional convolutional layers.
Revised: The second architecture is a bit more complicated: its encoder contains the ResNet34 (He et al., 2016) architecture with an additional fully connected layer whose size varies depending on the embedding size d , and whose decoder contains ﬁfteen two-dimensional convolutional layers.

Original: In contrast to the PCA model, instead of averaging the feature vectors for each ﬂight we used the standard deviation (this class of models worked better empirically in our experiments).
Revised: In contrast to the PCA model, instead of averaging the feature vectors for each ﬂight we used the standard deviation (this class of models worked better empirically in our experiments).

Original: Autoregression Models.
Revised: Autoregression Models.

Original: These models are trained to predict a state at time t + k given some subset of states up to time t .
Revised: These models are trained to predict a state at time t + k given some subset of states up to time t .

Original: In our experiments, we trained autoregression models to predict with k = 1 (next time step) and k = 30 .
Revised: In our experiments, we trained autoregression models to predict with k = 1 (next time step) and k = 30 .

Original: In this task, as an encoder we have compared LSTM, Conv1D,
Revised: In this task, as an encoder we have compared LSTM, Conv1D,

Original: ConvLSTM, and attention-based architectures.
Revised: ConvLSTM, and attention-based architectures.

Original: After each architecture, we applied a simple linear layer.
Revised: After each architecture, we applied a simple linear layer.

Original: Context Models.
Revised: Context Models.

Original: These models employ ideas similar to the word2vec CBOW model (Mikolov et al., 2013).
Revised: These models employ ideas similar to the word2vec CBOW model (Mikolov et al., 2013).

Original: Given a short sequence of timesteps, the model trains to predict several timesteps before and after the sequence; in our experiments, we used window size of 10 before and 10 after given states.
Revised: Given a short sequence of timesteps, the model trains to predict several timesteps before and after the sequence; in our experiments, we used window size of 10 before and 10 after given states.

Original: To train context models, we have used LSTM, attention cells, and Conv1D blocks.
Revised: To train context models, we have used LSTM, attention cells, and Conv1D blocks.

Original: After an encoder block, we repeated an embedding 2 C times, where C is the context size, and then applied a simple linear layer to each vector with different weights for each of the C timestamps.
Revised: After an encoder block, we repeated an embedding 2 C times, where C is the context size, and then applied a simple linear layer to each vector with different weights for each of the C timestamps.

Original: Multimodal Temporal Encoder.
Revised: Multimodal Temporal Encoder.

Original: The multimodal temporal encoder (MTE), introduced by Yang et al.
Revised: The multimodal temporal encoder (MTE), introduced by Yang et al.

Original: (2017), is a state of the art model intended to fuse inputs from different modalities (see
Revised: (2017), is a state of the art model intended to fuse inputs from different modalities (see

Original: Fig. 3).
Revised: Fig. 3).

Original: MTE uses an LSTM unit for each modality and shares hidden states across these units.
Revised: MTE uses an LSTM unit for each modality and shares hidden states across these units.

Original: By doing this, MTE forces the model to learn a fused representation across modalities; formally, it adds a correlation loss that computes the correlation between projections of differ- ent modalities L corr ( H 1 t , H
Revised: By doing this, MTE forces the model to learn a fused representation across modalities; formally, it adds a correlation loss that computes the correlation between projections of differ- ent modalities L corr ( H 1 t , H

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal loss is calculated as
Revised: The ﬁnal loss is calculated as

Original: L = (cid:80) Mi
Revised: L = (cid:80) Mi

Original: =1 L recon ,i − λL corr , where M is the batch size, h jt is the representation of modality j at timestep t , H jt
Revised: =1 L recon ,i − λL corr , where M is the batch size, h jt is the representation of modality j at timestep t , H jt

Original: =
Revised: =

Original: h jti } Mi =1 , H jt = 1 M (cid:80)
Revised: h jti } Mi =1 , H jt = 1 M (cid:80)

Original: Mi h jti , λ is a constant hyperparameter, and h t is the ﬁnal embedding (see Fig. 3).
Revised: Mi h jti , λ is a constant hyperparameter, and h t is the ﬁnal embedding (see Fig. 3).

Original: The sensors data is encoded with a three-layer fully connected neural network with dropout and ReLU activations, and a pretrained ResNet18 architecture for the images.
Revised: The sensors data is encoded with a three-layer fully connected neural network with dropout and ReLU activations, and a pretrained ResNet18 architecture for the images.

Original: The Dynamics Model.
Revised: The Dynamics Model.

Original: This approach, proposed by Zhang et al.
Revised: This approach, proposed by Zhang et al.

Original: (2018), introduces a way to decouple the training process (for a reinforcement learning task) into learning the dynamics model and the reward model.
Revised: (2018), introduces a way to decouple the training process (for a reinforcement learning task) into learning the dynamics model and the reward model.

Original: The dynamics model (shown in Fig. 4) is trained using a combination of four loss functions: reconstruction loss L t, recon ( θ enc , θ dec ) =
Revised: The dynamics model (shown in Fig. 4) is trained using a combination of four loss functions: reconstruction loss L t, recon ( θ enc , θ dec ) =

Original: ( s t − ˆ s t ) 2 ,
Revised: ( s t − ˆ s t ) 2 ,

Original: state loss
Revised: state loss

Original: L t, state ( θ for , θ dec ) =
Revised: L t, state ( θ for , θ dec ) =

Original: ( s t +1 − ˆ s t +1 ) 2 , forward loss L t, for ( θ for , θ enc ) =
Revised: ( s t +1 − ˆ s t +1 ) 2 , forward loss L t, for ( θ for , θ enc ) =

Original: ( z t +1 − ˆ z t +1 ) 2 , and inverse loss (with a trainable LSTM unit)
Revised: ( z t +1 − ˆ z t +1 ) 2 , and inverse loss (with a trainable LSTM unit)

Original: L t, inv ( θ inv ) =
Revised: L t, inv ( θ inv ) =

Original: ( a t − ˆ a t ) 2 .
Revised: ( a t − ˆ a t ) 2 .

Original: The ﬁnal loss is computed as L ( θ dynamics ) =
Revised: The ﬁnal loss is computed as L ( θ dynamics ) =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: ( λ dec ( L t, recon + L t, state ) + λ for L t, for + λ inv L t, inv ) , where ˆ z t +1 , h t = f for ( z t , a t , h t − 1 ; θ for ) , ˆ a t = f inv ( z t , z t +1 ; θ inv ) , ˆ s t +1 = f
Revised: ( λ dec ( L t, recon + L t, state ) + λ for L t, for + λ inv L t, inv ) , where ˆ z t +1 , h t = f for ( z t , a t , h t − 1 ; θ for ) , ˆ a t = f inv ( z t , z t +1 ; θ inv ) , ˆ s t +1 = f

Original: dec (ˆ z t +1 ; θ dec ) , s t is the state at time t , a t is the action at time t , h t is the hidden state, and z t is the latent representation at time t ; λ dec , λ for , and λ inv are (constant) hyperparameters.
Revised: dec (ˆ z t +1 ; θ dec ) , s t is the state at time t , a t is the action at time t , h t is the hidden state, and z t is the latent representation at time t ; λ dec , λ for , and λ inv are (constant) hyperparameters.

Original: If the dynamics change, we need only to re-train the LSTM unit and can keep the encoder and decoder unchanged, with the assumption that already learned representation contains all the necessary information about the new dynamics.
Revised: If the dynamics change, we need only to re-train the LSTM unit and can keep the encoder and decoder unchanged, with the assumption that already learned representation contains all the necessary information about the new dynamics.

Original: Since the original approach was presented only for inputs with a single modality, we expanded this idea for the multimodal case.
Revised: Since the original approach was presented only for inputs with a single modality, we expanded this idea for the multimodal case.

Original: To encode data from the sensors, we used a three-layer fully connected neural network with dropouts and ReLU activations.
Revised: To encode data from the sensors, we used a three-layer fully connected neural network with dropouts and ReLU activations.

Original: For the images, we used a pretrained ResNet18 architecture.
Revised: For the images, we used a pretrained ResNet18 architecture.

Original: After that, a linear layer was used to concatenate outputs of encoders for each modality into the embeddings, and the ﬁnal embedding results by averaging over the time steps.
Revised: After that, a linear layer was used to concatenate outputs of encoders for each modality into the embeddings, and the ﬁnal embedding results by averaging over the time steps.

Original:  E XPERIMENTAL EVALUATION 5. E VALUATION F RAMEWORK
Revised:  E XPERIMENTAL EVALUATION 5. E VALUATION F RAMEWORK

Original: As part of the dataset package, we have implemented and made available the framework which is designed to make a comprehensive evaluation of learned representation based on a number of ﬁxed predeﬁned tests.
Revised: As part of the dataset package, we have implemented and made available the framework which is designed to make a comprehensive evaluation of learned representation based on a number of ﬁxed predeﬁned tests.

Original: The main idea behind our framework is to combine the two main approaches to measuring representation quality.
Revised: The main idea behind our framework is to combine the two main approaches to measuring representation quality.

Original: First, we measure quality of representations by using them as features for a number of simple tasks.
Revised: First, we measure quality of representations by using them as features for a number of simple tasks.

Original: This approach was used, in particular, in the Black Box Learning
Revised: This approach was used, in particular, in the Black Box Learning

Original: Challenge (Goodfellow et al., 2013) and Unsupervised and Transfer Learning Challenge (Guyon et al., 2011), whose main objective was to learn a good representation from rich unlabeled data, and representations were evaluated on supervised learning tasks that were not known to the participants.
Revised: Challenge (Goodfellow et al., 2013) and Unsupervised and Transfer Learning Challenge (Guyon et al., 2011), whose main objective was to learn a good representation from rich unlabeled data, and representations were evaluated on supervised learning tasks that were not known to the participants.

Original: The second approach is to evaluate the disentanglement in representations.
Revised: The second approach is to evaluate the disentanglement in representations.

Original: For this we use a QEDR (Quantitative Evaluation of Disentangled Representations) framework proposed by Eastwood &
Revised: For this we use a QEDR (Quantitative Evaluation of Disentangled Representations) framework proposed by Eastwood &

Original: Williams (2018).
Revised: Williams (2018).

Original: The idea is that the ideal representation of data is a vector of separated and independent generative factors for the data (perhaps scaled and permuted).
Revised: The idea is that the ideal representation of data is a vector of separated and independent generative factors for the data (perhaps scaled and permuted).

Original: The matrix R , where R ij is the relative importance of code (representation) variable c
Revised: The matrix R , where R ij is the relative importance of code (representation) variable c

Original: i in predicting generative factor z j , is used to compute three evaluation metrics for the quality of a representation.
Revised: i in predicting generative factor z j , is used to compute three evaluation metrics for the quality of a representation.

Original: Disentanglement is a measure of the degree to which a representation factorizes the factors of variation in original data; for a code variable
Revised: Disentanglement is a measure of the degree to which a representation factorizes the factors of variation in original data; for a code variable

Original: c
Revised: c

Original: i it is deﬁned as D i = 1 − H
Revised: i it is deﬁned as D i = 1 − H

Original: ( P i. ) where H ( P i. ) is the entropy of the pseudodistribution P i. ; P ij =
Revised: ( P i. ) where H ( P i. ) is the entropy of the pseudodistribution P i. ; P ij =

Original: R ij / (cid:80)
Revised: R ij / (cid:80)

Original: k R ik denotes the “probability” of c i being important for predicting generative factor z j , and total disentanglement is computed as the weighted average (cid:80) i ρ
Revised: k R ik denotes the “probability” of c i being important for predicting generative factor z j , and total disentanglement is computed as the weighted average (cid:80) i ρ

Original: i D i , where ρ i
Revised: i D i , where ρ i

Original: =
Revised: =

Original: (cid:80) j R ij / (cid:80) kj R kj is the relative code variable importance.
Revised: (cid:80) j R ij / (cid:80) kj R kj is the relative code variable importance.

Original: Completeness is a measure of the degree to which a factor of variation in the original data is captured by a single code variable; for a factor
Revised: Completeness is a measure of the degree to which a factor of variation in the original data is captured by a single code variable; for a factor

Original: z j it is deﬁned as C j = 1 − H ( ˜ P .j )
Revised: z j it is deﬁned as C j = 1 − H ( ˜ P .j )

Original: where H ( ˜ P .j ) is the entropy of the pseudo-distribution ˜ P .j ; total completeness is the average of C j .
Revised: where H ( ˜ P .j ) is the entropy of the pseudo-distribution ˜ P .j ; total completeness is the average of C j .

Original: Informativeness measures the amount of information that a representation captures about the underlying factors of variation; following Eastwood & Williams (2018), we use normalised root-mean-square ( NRMSE ) as the informativeness metric.
Revised: Informativeness measures the amount of information that a representation captures about the underlying factors of variation; following Eastwood & Williams (2018), we use normalised root-mean-square ( NRMSE ) as the informativeness metric.

Original: We use failure scores as generative factors since we know the ground truth for them and they are mutually independent.
Revised: We use failure scores as generative factors since we know the ground truth for them and they are mutually independent.

Original: We deﬁne R ij =
Revised: We deﬁne R ij =

Original: | W ij | , where W is the weight matrix of lasso regression learned on the representation vector to predict the vector of factors.
Revised: | W ij | , where W is the weight matrix of lasso regression learned on the representation vector to predict the vector of factors.

Original: Our evaluation framework is shown in Fig.
Revised: Our evaluation framework is shown in Fig.

Original: We assume that a feature extractor constructs a single vector representation for a time series of sensor readings.
Revised: We assume that a feature extractor constructs a single vector representation for a time series of sensor readings.

Original: We have implemented disentanglement, completeness, and informativeness scores computed on 12 failures scores as generative factors and 5 supervised learning targets, including: wind speed, turbulence power, vertical acceleration at the time of landing (a quality of touchdown), autoregression (given t states, predict state t + 1 ), failure classiﬁcation (in dataset generation we randomly applied various failures to the airplane, and the task is to ﬁnd out whether a failure is present in a given time series).
Revised: We have implemented disentanglement, completeness, and informativeness scores computed on 12 failures scores as generative factors and 5 supervised learning targets, including: wind speed, turbulence power, vertical acceleration at the time of landing (a quality of touchdown), autoregression (given t states, predict state t + 1 ), failure classiﬁcation (in dataset generation we randomly applied various failures to the airplane, and the task is to ﬁnd out whether a failure is present in a given time series).

Original: To evaluate supervised learning, we ﬁrst apply the provided feature extractors for every training example, then train simple models (namely, a three-layer fully connected neural network with ReLU activations) on these features for each target, and ﬁnally compute the accuracy on the hold-out set; these accuracies are the ﬁnal performance metrics.
Revised: To evaluate supervised learning, we ﬁrst apply the provided feature extractors for every training example, then train simple models (namely, a three-layer fully connected neural network with ReLU activations) on these features for each target, and ﬁnally compute the accuracy on the hold-out set; these accuracies are the ﬁnal performance metrics.

Original: We have made the code for the evaluation pipeline available at (Code, 2018).
Revised: We have made the code for the evaluation pipeline available at (Code, 2018).

Original: 5. E VALUATION RESULTS
Revised: 5. E VALUATION RESULTS

Original: Table 4 shows the main results of our evaluation study; in this section we interpret these quantitative results.
Revised: Table 4 shows the main results of our evaluation study; in this section we interpret these quantitative results.

Original: The models were trained on the X-Plane dataset with regression tasks evaluated on a holdout set; time series lengths varied from 25 to 75 during training.
Revised: The models were trained on the X-Plane dataset with regression tasks evaluated on a hold-out set; time series lengths varied from 25 to 75 during training.

Revised: To establish a simple baseline for our models we use mean target value over the training set as a prediction.

Revised: It is denoted as

Original:  First, we see that different regression tasks have very different models doing well on them; e.g., most models cannot outperform even the mean baseline for wind regression, and models that show good results on wind regression perform poorly on auto regression and vice versa.
Revised: MEAN baseline in Table 4. First, we see that different regression tasks have very different models doing well on them; e.g., most models cannot outperform even the mean baseline for wind regression, and models that show good results on wind regression perform poorly on auto regression and vice versa.

Original: The effect of images on the results is contradictory: we have trained TS Regression and Dynamics models on three types of data (sensors, images, and both) each, and for TS Regression adding images improves scores on benchmarks, while for the Dynamics model the results deteriorate.
Revised: The effect of images on the results is contradictory: we have trained TS Regression and Dynamics models on three types of data (sensors, images, and both) each, and for TS Regression adding images improves scores on benchmarks, while for the Dynamics model the results deteriorate.

Original: The results show that embedding size d needs to be tuned for each model separately; some models even “explode” and give unstable results (very large errors) for some values of d ; there is no general correlation between d and benchmark scores.
Revised: The results show that embedding size d needs to be tuned for each model separately; some models even “explode” and give unstable results (very large errors) for some values of d ; there is no general correlation between d and benchmark scores.

Original: In terms of inference time t , it appears that the Dynamics model trained on sensors only is the best tradeoff between efﬁciency and quality with a large margin.
Revised: In terms of inference time t , it appears that the Dynamics model trained on sensors only is the best tradeoff between efﬁciency and quality with a large margin.

Original: Both disentanglement and completeness scores are small, perhaps because we used only a small subset of generative factors to compute them.
Revised: Both disentanglement and completeness scores are small, perhaps because we used only a small subset of generative factors to compute them.

Original: The disentanglement score is better for smaller d , which shows that information is packed more efﬁciently in this case.
Revised: The disentanglement score is better for smaller d , which shows that information is packed more efﬁciently in this case.

Original: Performance of the models that use images strongly depends on the quality of the images themselves.
Revised: Performance of the models that use images strongly depends on the quality of the images themselves.

Original: For example, if the horizon and runway are clearly visible (as in Table 5a), models with images outperform models that do not use them (Table 5), while on night-time noisy images (Table 5b) models that use images lose.
Revised: For example, if the horizon and runway are clearly visible (as in Table 5a), models with images outperform models that do not use them (Table 5), while on night-time noisy images (Table 5b) models that use images lose.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have presented the dataset for learning state representation and a uniﬁed evaluation framework to measure the quality of multimodal joint representations produced by different encoders, through both secondary supervised learning tasks and quantitative metrics of disentanglement and informativeness quality.
Revised: In this work, we have presented the dataset for learning state representation and a uniﬁed evaluation framework to measure the quality of multimodal joint representations produced by different encoders, through both secondary supervised learning tasks and quantitative metrics of disentanglement and informativeness quality.

Original: This work represents an attempt to help advance the research in model-based reinforcement learning and state representation learning by providing a uniﬁed.
Revised: This work represents an attempt to help advance the research in model-based reinforcement learning and state representation learning by providing a uniﬁed.

Original: At the same time, the large-scale comparison between baseline and state of the art models that we perform in this work has produced some interesting results by itself.
Revised: At the same time, the large-scale comparison between baseline and state of the art models that we perform in this work has produced some interesting results by itself.

Original: In general, we believe that this dataset can become the new standard for evaluation in representation learning for complex systems, and wehope it will inspire many novel techniques to advance the state of the art that we have attempted to establish and quantify in our practical evaluation.
Revised: In general, we believe that this dataset can become the new standard for evaluation in representation learning for complex systems, and we

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: hope it will inspire many novel techniques to advance the state of the art that we have attempted to establish and quantify in our practical evaluation.