Original: D ECOUPLING FEATURE EXTRACTION FROM
Revised: D ECOUPLING FEATURE EXTRACTION FROM

Original: POLICY
Revised: POLICY

Original: LEARNING : ASSESSING BENEFITS OF STATE REPRESENTATION LEARNING IN GOAL BASED ROBOTICS
Revised: LEARNING : ASSESSING BENEFITS OF STATE REPRESENTATION LEARNING IN GOAL BASED ROBOTICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges, in particular in terms of sample efﬁciency.
Revised: Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges, in particular in terms of sample efﬁciency.

Original: Against end-to-end learning, state representation learning can help learn a compact, efﬁcient and relevant representation of states that speeds up policy learning, reducing the number of samples needed, and that is easier to interpret.
Revised: Against end-to-end learning, state representation learning can help learn a compact, efﬁcient and relevant representation of states that speeds up policy learning, reducing the number of samples needed, and that is easier to interpret.

Original: We evaluate several state representation learning methods on goal based robotics tasks and propose a new unsupervised model that stacks representations and combines strengths of several of these approaches.
Revised: We evaluate several state representation learning methods on goal based robotics tasks and propose a new unsupervised model that stacks representations and combines strengths of several of these approaches.

Original: This method encodes all the relevant features, performs on par or better than end-to-end learning, and is robust to hyper-parameters change
Revised: This method encodes all the relevant features, performs on par or better than end-to-end learning, and is robust to hyper-parameters change

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A common strategy to learn controllers in robotics is to design a reward function that deﬁnes the task and search for a policy that maximizes the collected rewards with a Reinforcement Learning (RL) approach.
Revised: A common strategy to learn controllers in robotics is to design a reward function that deﬁnes the task and search for a policy that maximizes the collected rewards with a Reinforcement Learning (RL) approach.

Original: In RL, the controlled system (environment and robot) is deﬁned by a state s t , i.e., the relevant variables for a controller, often of low dimension (e.g., positions of a robot and a target).
Revised: In RL, the controlled system (environment and robot) is deﬁned by a state s t , i.e., the relevant variables for a controller, often of low dimension (e.g., positions of a robot and a target).

Original: At a given state s t , the agent will receive an observation o t from the environment and a reward r t .
Revised: At a given state s t , the agent will receive an observation o t from the environment and a reward r t .

Original: In some applications, the observation may be directly the state, but in the general case, the observation is raw sensor data (e.g., images from the robot camera).
Revised: In some applications, the observation may be directly the state, but in the general case, the observation is raw sensor data (e.g., images from the robot camera).

Original: RL must then learn a policy that takes observations as input and returns the action a t that maximizes the reward r t .
Revised: RL must then learn a policy that takes observations as input and returns the action a t that maximizes expected return.

Original: When the state is not directly accessible, RL should recover it from the observation to learn a good control policy.
Revised: When the state is not directly accessible, RL should recover it from the observation to learn a good control policy.

Original: This could be learned implicitly by an end-to-end approach (cf Fig. 1), i.e. by learning a policy from observation to action, or explicitly by, at ﬁrst, extracting a representation of this state from the observation and then learning the policy from it.
Revised: This could be learned implicitly by an end-to-end approach (cf Fig. 1), i.e. by learning a policy from observation to action, or explicitly by, at ﬁrst, extracting a representation of this state from the observation and then learning the policy from it.

Original: State representation learning (SRL) (Lesort et al., 2018) aims at learning those states as a compact representation from raw observations and without explicit supervision.
Revised: State representation learning (SRL) (Lesort et al., 2018) aims at learning those states as a compact representation from raw observations and without explicit supervision.

Original: One key goal of learning state representation separately from learning the policy is to improve the sample efﬁciency of the full process by reducing the search space.
Revised: One key goal of learning state representation separately from learning the policy is to improve the sample efﬁciency of the full process by reducing the search space.

Original: Indeed, end-to-end approaches, even if adequate for simulation settings, are often not sample efﬁcient enough for real life learning as sampling observations from the environment is particularly costly and time consuming in robotics.
Revised: Indeed, end-to-end approaches, even if adequate for simulation settings, are often not sample efﬁcient enough for real life learning as sampling observations from the environment is particularly costly and time consuming in robotics.

Original: Another crucial advantage of reducing the search space is to improve stability of policy learning, a common issue in RL (Henderson et al., 2017).
Revised: Another crucial advantage of reducing the search space is to improve stability of policy learning, a common issue in RL (Henderson et al., 2017).

Original: Although SRL is not restricted to robotics, in this paper, we demonstrate its utility in goal-based robotics tasks, i.e. the controlled agent is a robot and sparse rewards are directly linked to a goal deﬁned in the environment.
Revised: Although SRL is not restricted to robotics, in this paper, we demonstrate its utility in goal-based robotics tasks, i.e. the controlled agent is a robot, the reward is sparse and only depends on the previous state and taken action, not on a succession of states (therefore excluding tasks like walking or running).

Original: Several approaches exist for SRL that differ in the information they can encode.
Revised: Several approaches exist for SRL that differ in the information they can encode.

Original: This paper aims at investigating the beneﬁt of different ways of combining state of the art SRL approaches on policy learning for various goal based robotics tasks.
Revised: This paper aims at investigating the beneﬁt of different ways of combining state of the art SRL approaches on policy learning for various goal based robotics tasks.

Original: The contributions of this paper are:
Revised: The contributions of this paper are:

Original: • we show the usefulness of decoupling feature extraction from policy learning (Section 5.4)  • we propose a new way of combining approaches by stacking state representations instead of mixing them, that allows a better disentanglement (Section 4.4) • we investigate the inﬂuence ofthe different hyper-parameters when learning a state representation (Section 5.5)
Revised: • we show the usefulness of decoupling feature extraction from policy learning (Section 5.4) and that random features provide a good baseline • we propose a new way of combining approaches by stacking state representations instead of mixing them, that allows to mitigate the problem of conﬂicting objectives and favors disentanglement (Section 4.4) • we investigate the inﬂuence of hyper-parameters when learning a state representation (Section 5.5)

Original: This paper is organized the following way: we ﬁrst introduce the state of the art in SRL for robotics (Section 2), and clarify how we deﬁne an appropriate state representation (Section 3.1) and a relevant method to learn it (Section 3.2).
Revised: This paper is organized the following way: we ﬁrst introduce the state of the art in SRL for robotics (Section 2), and clarify how we deﬁne an appropriate state representation (Section 3).

Original: Then, we explain how we designed our SRL combination approach (Section 4).
Revised: Then, we explain how we designed our SRL combination approach (Section 4).

Original: Finally, we justify and illustrate our approach with experiments in various simulated robotics tasks (Section 5).
Revised: Finally, we justify and illustrate our approach with experiments in various simulated robotics tasks (Section 5).

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In reinforcement learning, a classic preparatory approach is to design some features by hand, in order to facilitate policy learning.
Revised: In reinforcement learning, a classic preparatory approach is to design some features by hand, in order to facilitate policy learning.

Original: However the manual design may be difﬁcult, laborious and requires domain knowledge.
Revised: However the manual design may be difﬁcult, laborious and requires domain knowledge.

Original: Hence, this process can be automated using methods that are able to learn these features (also called representations) (Böhmer et al., 2015; Singh et al., 2012; Boots et al., 2011).
Revised: Hence, this process can be automated using methods that are able to learn these features (also called representations) (Böhmer et al., 2015; Singh et al., 2012; Boots et al., 2011).

Original: This problem is commonly called State Representation Learning (SRL).
Revised: This problem is commonly called State Representation Learning (SRL).

Original: We can deﬁne it more precisely as a particular kind of representation learning where the learned features are in low dimension, evolve through time, and are inﬂuenced by the actions of an agent (Lesort et al., 2018).
Revised: We can deﬁne it more precisely as a particular kind of representation learning where the learned features are in low dimension, evolve through time, and are inﬂuenced by the actions of an agent (Lesort et al., 2018).

Original: SRL is used as a preliminary step for learning a control policy.
Revised: SRL is used as a preliminary step for learning a control policy.

Original: The representation is learned on data gathered in the environment by an exploration policy.
Revised: The representation is learned on data gathered in the environment by an exploration policy.

Original: One particular advantage of this step is that it reduces the search space and gives to reinforcement learning an informative representation, instead of raw data (e.g. pixels).
Revised: One particular advantage of this step is that it reduces the search space and gives to reinforcement learning an informative representation, instead of raw data (e.g. pixels).

Original: This allows to solve tasks more efﬁciently (Munk et al., 2016).
Revised: This allows to solve tasks more efﬁciently (Lange et al., 2012; Wahlström et al., 2015; Munk et al., 2016).

Revised: We refer to (Lesort et al., 2018) for a complete review of SRL for control.

Original: In robotics, SRL is particularly interesting as the learning process is very slow and data hungry.
Revised: In robotics, SRL is particularly interesting as the learning process is very slow and data hungry.

Original: With real robots, learning happens in real time and cannot be accelerated or easily multiprocessed as in simulated environments.
Revised: With real robots, learning happens in real time and cannot be accelerated or easily multiprocessed as in simulated environments.

Original: However, since the learning process remains time consuming and the availability of robots is limited by cost and maintenance constraints, several approaches prefer to iterate at ﬁrst in simulation to learn robotics tasks (Jonschkowski & Brock, 2015;
Revised: However, since the learning process remains time consuming and the availability of robots is limited by cost and maintenance constraints, several approaches prefer to iterate at ﬁrst in simulation to learn robotics tasks (Jonschkowski & Brock, 2015;

Original: Watter et al., 2015; Curran et al., 2016; Lesort et al., 2017; Jonschkowski et al., 2017).
Revised: Watter et al., 2015; Curran et al., 2016; Lesort et al., 2017; Jonschkowski et al., 2017).

Original: Our proposal is based along this line.
Revised: Our proposal is based along this line.

Original: The robotics environment provides us with rewards, observations and actions, that can be used to deﬁne
Revised: The robotics environment provides us with rewards, observations and actions, that can be used to deﬁne

Original: SRL loss functions.
Revised: SRL loss functions.

Original: Forward models (Munk et al., 2016), inverse models (Shelhamer et al., 2017), datareconstruction models (Mattner et al., 2012; Curran et al., 2016) or priors knowledge (Jonschkowski &
Revised: Forward models (Munk et al., 2016), inverse models (Shelhamer et al., 2017), datareconstruction models (Mattner et al., 2012; Curran et al., 2016) or priors knowledge (Jonschkowski &

Original: Brock, 2015; Lesort et al., 2017) are several approaches that exploit those environments data to learn meaningful representations.
Revised: Brock, 2015; Lesort et al., 2017) are several approaches that exploit those environments data to learn meaningful representations.

Original: These methods can also be combined to improve the quality of the learned representations.
Revised: These methods can also be combined to improve the quality of the learned representations.

Original: Some examples include mixing a data-reconstruction objective and a forward model loss (Watter et al., 2015; Krishnan et al., 2015; Ha & Schmidhuber, 2018), coupling a forward model together with an inverse model (Pathak et al., 2017), and using both data-reconstruction and priors loss functions (Finn et al., 2015).
Revised: Some examples include mixing a data-reconstruction objective and a forward model loss (Watter et al., 2015; Krishnan et al., 2015; Ha & Schmidhuber, 2018), coupling a forward model together with an inverse model (Pathak et al., 2017), and using both data-reconstruction and priors loss functions (Finn et al., 2015).

Original: The goal of this paper is thus to compare decoupling feature extraction (SRL) from end-to-end policy learning, and to explore various possible combinations to learn these features.
Revised: The goal of this paper is thus to compare decoupling feature extraction (SRL) from end-to-end policy learning, and to explore various possible combinations to learn these features.

Original: Our setting is similar to the one used in Hindsight Experience Replay (Andrychowicz et al., 2017) that tackles the problem of solving goal-based robotics tasks with sparse reward.
Revised: Our setting is similar to the one used in Hindsight Experience Replay (Andrychowicz et al., 2017) that tackles the problem of solving goal-based robotics tasks with sparse reward.

Original: In their experiments, the agent has a direct access to the positions of the controlled robot and target.
Revised: In their experiments, the agent has a direct access to the positions of the controlled robot and target.

Original: Our work, on the contrary, uses the raw pixels as input.
Revised: Our work, on the contrary, uses the raw pixels as input.

Original: The extraction of relevant positions must be learned by the different methods.
Revised: The extraction of relevant positions must be learned by the different methods.

Original:  S TATE R EPRESENTATION R EQUIREMENTS
Revised:  S TATE R EPRESENTATION R EQUIREMENTS

Original: SRL aims at extracting relevant information from raw sensor data.
Revised: SRL aims at extracting relevant information from raw sensor data.

Original: This ability is not the only substantial characteristic of a SRL model.
Revised: This ability is not the only substantial characteristic of a SRL model.

Original: In this section, we provide additional important facets of a good state space and the aspects of an adequate method.
Revised: In this section, we provide additional important facets of a good state space.

Original: 3. C HARACTERISTICS OF A SUITABLE STATE REPRESENTATION

Original: From a high-level point of view, the state representation should retain useful information from the observation in order to solve the task and ﬁlter out irrelevant parts.
Revised: From a high-level point of view, the state representation should retain useful information from the observation in order to solve the task and ﬁlter out irrelevant parts.

Original: More precisely, the state space should be:
Revised: More precisely, the state space should be:

Original: Compact : a good state representation should have a low dimension compared to the raw sensor data.
Revised: Compact : a good state representation should have a low dimension compared to the raw sensor data.

Original: It should only keep relevant information, ignoring distractors (irrelevant parts of the observation).
Revised: It should only keep relevant information, ignoring distractors (irrelevant parts of the observation).

Original: This will reduce the search space for RL, leading to a more stable and sample-efﬁcient policy learning.
Revised: This will reduce the search space for RL, leading to a more stable and sample-efﬁcient policy learning.

Original: A lowdimensional space is also easier to interpret.
Revised: A lowdimensional space is also easier to interpret.

Original: Sufﬁcient : all the important information to solve the task should be encoded into the state space.
Revised: Sufﬁcient : all the important information to solve the task should be encoded into the state space.

Original: Otherwise, the agent will under-perform (cannot reach maximal performance) or even fail.
Revised: Otherwise, the agent will under-perform (cannot reach maximal performance) or even fail.

Original: Disentangled : the state representation should untangle factors of variation. Each dimension of the feature space should be independent, otherwise it encodes redundant information.
Revised: Disentangled : the state representation should untangle factors of variation independently, i.e., each dimension of the feature space should represent a particular factor of variation.

Original: A disentangled state representation shouldalso facilitate policy learning (because the policy network does not have to learn how to decipher the raw data).
Revised: A disentangled state representation should facilitate policy learning (because the policy network does not have to learn how to decipher the raw data).

Revised: The non-redundancy of representation is preferable for disentanglement but not necessary, as long as the representation dimension stays low.

Revised: Generalizable :

Revised: As for any machine learning algorithm, representations learned by SRL should be able to generalize to unseen situations.

Revised: In the SRL context, we can deﬁne 2 different degrees of generalization: 1) generalization to new environment states and 2) generalization to different environments.

Original: In the context of a goal-based robotics task, a sufﬁcient state representation should extract the position of the robot, and the position of the goal.
Revised: In the context of a goal-based robotics task, a sufﬁcient state representation should extract the position of the robot, and the position of the goal.

Original: If velocities are also needed, they can be approximated using ﬁnite differences between two consecutive positions, as in Jonschkowski et al.
Revised: If velocities are also needed, they can be approximated using ﬁnite differences between two consecutive positions, as in (Jonschkowski et al., 2017).

Original: A disentangled feature space should encode only one coordinate per dimension, i.e., one dimension should encode the x-coordinate of the robot position, another one the y-coordinate, etc.
Revised: A disentangled feature space should encode only one coordinate per dimension, i.e., one dimension should encode the x-coordinate of the robot position, another one the y-coordinate, etc.

Original: 3. A SPECTS OF AN ADEQUATE METHOD

Original: In the previous section, we detailed the aspects that should be fulﬁlled by a satisfactory state space.

Original: The solutions that meet these requirements are not unique.

Original: Therefore, we present additional characteristics that deﬁne an appropriate method and that may guide the construction of such model.

Original: A good solution should be as simple as possible, not sensitive to hyper-parameter changes, and applicable to many settings.

Original: Hence, we consider that an acceptable method should be:

Original: Simple : the method should be as simple as possible, i.e. have a minimum number of components, tricks and hyper-parameters.

Original: Robust : it should be robust to hyper-parameters change (i.e., minimal tuning needed).

Original: Versatile : it should adapt to various settings with only minor modiﬁcations.

Original:  I NCREMENTALLY BUILDING A POTENTIAL ADEQUATE METHOD
Revised:  I NCREMENTALLY BUILDING A POTENTIAL ADEQUATE METHOD

Original: Given the general objectives deﬁned in the previous section, we now propose a way to combine several approaches by tackling one objective at a time, using a particular context for a concrete illustration.
Revised: Given the general objectives deﬁned in the previous section, we now propose a way to combine several approaches by tackling one objective at a time, using a particular context for a concrete illustration.

Original: This part aims at giving insights on the different SRL methods, taking advantage of goal-based robotics tasks as an application example.
Revised: This part aims at giving insights on the different SRL methods, taking advantage of goal-based robotics tasks as an application example.

Original: 4. E NCODING S TATE OF THE A GENT : R OBOT P OSITION
Revised: 4. E NCODING S TATE OF THE A GENT : R OBOT P OSITION

Original: One important aspect to encode for RL is the state of the controlled agent.
Revised: One important aspect to encode for RL is the state of the controlled agent.

Original: In the context of goal-based robotics tasks, it corresponds to the robot position.
Revised: In the context of goal-based robotics tasks, it corresponds to the robot position.

Original: A simple method consists of using an inverse dynamics objective : given the current s t and next state s t +1 , the task is to predict the taken action a t .
Revised: A simple method consists of using an inverse dynamics objective : given the current s t and next state s t +1 , the task is to predict the taken action a t .

Original: The type of dynamics learned is constrained by the network architecture.
Revised: The type of dynamics learned is constrained by the network architecture.

Original: For instance, using a linear model imposes linear dynamics.
Revised: For instance, using a linear model imposes linear dynamics.

Original: The state representation learned encodes only controllable elements of the environment.
Revised: The state representation learned encodes only controllable elements of the environment.

Original: Here, the robot is part of them.
Revised: Here, the robot is part of them.

Original: However, the features extracted by an inverse model are not always sufﬁcient : in our case, they do not encode the position of the target since the agent cannot act on it.
Revised: However, the features extracted by an inverse model are not always sufﬁcient : in our case, they do not encode the position of the target since the agent cannot act on it.

Original: 4. E NCODING A DDITIONAL I NFORMATION : T ARGET P OSITION
Revised: 4. E NCODING A DDITIONAL I NFORMATION : T ARGET P OSITION

Original: Since learning to extract the robot position is not enough to solve goal-based tasks, we need to add extra objective functions in order to encode the position of the target object.
Revised: Since learning to extract the robot position is not enough to solve goal-based tasks, we need to add extra objective functions in order to encode the position of the target object.

Original: In this section, we consider two of them: minimizing a reconstruction error (auto-encoder model) or a reward prediction loss.
Revised: In this section, we consider two of them: minimizing a reconstruction error (auto-encoder model) or a reward prediction loss.

Original: Auto-encoder:
Revised: Auto-encoder:

Original: Thanks to their reconstruction objective, auto-encoders compress information in their latent space.
Revised: Thanks to their reconstruction objective, auto-encoders compress information in their latent space.

Original: Auto-encoders tend to encode only aspects of the environment that are salient in the input image.
Revised: Auto-encoders tend to encode only aspects of the environment that are salient in the input image.

Original: This means they are not task-speciﬁc: relevant elements can be ignored and distractors (unnecessary information) can be encoded into the state representation.
Revised: This means they are not task-speciﬁc: relevant elements can be ignored and distractors (unnecessary information) can be encoded into the state representation.

Original: They usually need more dimensions that apparently required to encode a scene (e.g. in our experiments, it requires more than 10 dimensions to encode a 2D position).
Revised: They usually need more dimensions that apparently required to encode a scene (e.g. in our experiments, it requires more than 10 dimensions to encode a 2D position).

Original: Reward prediction: The objective of a reward prediction module leads to state representations that are specialized in a task.
Revised: Reward prediction: The objective of a reward prediction module leads to state representations that are specialized in a task.

Original: However, this does not constrain the feature space to be disentangled (or to have a particular structure).
Revised: However, this does not constrain the feature space to be disentangled or to have any particular structure.

Original: Using the reward prediction objective alone yields state representations with one cluster per reward value.

Original: To enforce some structure, we give s t and s t +1 instead of s t and a t (the action should be implicitly encoded into the state representation).

Original: In the context of goal-based robotics, the task can be restricted to predicting if the reward is positive or not.

Original: 4. COMBINING APPROACHES
Revised: 4. COMBINING APPROACHES

Original: Combining objectives makes it possible to share the strengths of each model.
Revised: Combining objectives makes it possible to share the strengths of each model.

Original: In our application example, the previous sections suggest that we should mix objectives to encode both robot and target positions.
Revised: In our application example, the previous sections suggest that we should mix objectives to encode both robot and target positions.

Original: The simplest way to combine objectives is to minimize a weighted sum of the different loss functions, i.e. reconstruction, inverse dynamics and reward prediction losses:
Revised: The simplest way to combine objectives is to minimize a weighted sum of the different loss functions, i.e. reconstruction, inverse dynamics and reward prediction losses:

Original: [Equation]
Revised: [Equation]

Original: Each weight represents the relative importance we give to the different objectives.
Revised: Each weight represents the relative importance we give to the different objectives.

Original: Because we consider each objective to be relevant, we chose the weights such that they provide gradients with similar magnitudes.
Revised: Because we consider each objective to be relevant, we chose the weights such that they provide gradients with similar magnitudes.

Original: 4. S PLITTING INSTEAD OF 
Revised: 4. S PLITTING INSTEAD OF COMBINING STATE REPRESENTATIONS

Original: [Equation] [Figure] [Equation]

Original: [Figure]
Revised: [Figure]

Original: Combining objectives into a single embedding is not the only option to have features that are sufﬁcient to solve the tasks.
Revised: Combining objectives into a single embedding is not the only option to have features that are sufﬁcient to solve the tasks.

Original: Stacking representations, which also favors disentanglement , is another way of solving the problem.
Revised: Stacking representations, which also favors disentanglement , is another way of solving the problem.

Original: We use this idea in the SRL Splits model, where the state representation is split into several parts where each optimizes a fraction of the objectives.
Revised: We use this idea in the SRL Splits model, where the state representation is split into several parts where each optimizes a fraction of the objectives.

Original: This prevents objectives that can be opposed from cancelling out and allows a more stable optimization.
Revised: This prevents objectives that can be opposed from cancelling out and allows a more stable optimization.

Original: This process is similar to training several models but with a shared feature extractor, that projects the observations into the state representation.
Revised: This process is similar to training several models but with a shared feature extractor, that projects the observations into the state representation.

Original: In practice, as showed in Fig. 2, each loss is only applied to part of the state representation.
Revised: In practice, as showed in Fig. 2, each loss is only applied to part of the state representation.

Original: In the experiments, to encode both target and robot positions, we combine the strength of auto-encoders, reward and inverse losses using a state representation of dimension 200.
Revised: In the experiments, to encode both target and robot positions, we combine the strength of auto-encoders, reward and inverse losses using a state representation of dimension 200.

Revised: The reconstruction loss is the mean squared error between original and reconstructed image.

Revised: We used cross entropy loss for the reward prediction and inverse dynamics losses.

Original: The reconstruction and reward losses 1 are applied on a ﬁrst split of 198 dimensions and the inverse dynamics loss on the 2 remaining dimensions (encoding the robot position).
Revised: The reconstruction and reward losses 1 are applied on a ﬁrst split of 198 dimensions and the inverse dynamics loss on the 2 remaining dimensions (encoding the robot position).

Original: To have the same magnitude for each loss, we set w reconstruction = 1 , w reward = 1 and w inverse = 2 .
Revised: To have the same magnitude for each loss, we set w reconstruction = 1 , w reward = 1 and w inverse = 2 .

Original: The choice of the different hyper-parameters (losses, weights, state dimension, training-set-size) and the robustness to changes are explored and validated in the experiments section (Section 5) and
Revised: The choice of the different hyper-parameters (losses, weights, state dimension, training-set-size) and the robustness to changes are explored and validated in the experiments section (Section 5) and

Original: Appendix B.
Revised: Appendix B.

Original:  E XPERIMENTS AND R ESULTS 5. E NVIRONMENTS
Revised:  E XPERIMENTS AND R ESULTS 5. E NVIRONMENTS

Original: [Figure] In order to evaluate the methods, we use 4 environments proposed in S-RL Toolbox (Rafﬁn et al., 2018)
Revised: In order to evaluate the methods, we use 4 environments proposed in S-RL Toolbox (Rafﬁn et al., 2018)

Original: (Fig. 3).
Revised: (Fig. 3).

Original: These environments of incremental difﬁculty are specially designed for evaluating SRL methods in a robotics context.
Revised: These environments of incremental difﬁculty are specially designed for evaluating SRL methods in a robotics context.

Original: The environments are variations of two main settings: a 2D environment with a mobile robot and a 3D environment with a robotic arm.
Revised: The environments are variations of two main settings: a 2D environment with a mobile robot and a 3D environment with a robotic arm.

Original: In all settings, there is a controlled robot and one target that is randomly initialized.
Revised: In all settings, there is a controlled robot and one target that is randomly initialized.

Original: In the experiments, the robot is controlled using discrete actions (but the approaches we present are not limited to that domain) and the reward is sparse: +1 when reaching the goal, -1 when hitting an obstacle and 0 everywhere else.
Revised: In the experiments, the robot is controlled using discrete actions (but the approaches we present are not limited to that domain) and the reward is sparse: +1 when reaching the goal, -1 when hitting an obstacle and 0 everywhere else.

Original: The four environments used are: 1D/2D random target with mobile robot and random/moving target with robotic arm.
Revised: The four environments used are: 1D/2D random target with mobile robot and random/moving target with robotic arm.

Original: 1D/2D random target mobile navigation : This environment consists of a navigation task using a mobile robot, similar to the task in (Jonschkowski & Brock, 2015), with either a cylinder (2D target) or a horizontal band (1D target) on the ground as a goal, randomly initialized at the beginning of each episode. The mobile robot can move in four directions ( forward, backward, left, right ) and will get a +1 reward when reaching the target, -1 when hitting walls, and 0 otherwise. Episodes have a maximum length of 250 steps (hence, an upper bound max. reward of 250).
Revised: [Figure]

Original: Robotic arm with random/moving target : In this setting, a robotic arm, ﬁxed to a table, has to reach a randomly initialized target on the table.
Revised: Robotic arm with random/moving target : In this setting, a robotic arm, ﬁxed to a table, has to reach a randomly initialized target on the table.

Original: The target can be static during the episode or slowly moving back and forth along one axis.
Revised: The target can be static during the episode or slowly moving back and forth along one axis.

Original: The arm is controlled in the x , y and z position using inverse kinematics.
Revised: The arm is controlled in the x , y and z position using inverse kinematics.

Original: The agent received a +1 reward when it reaches the goal, -1 when hitting the table, and 0 otherwise.
Revised: The agent received a +1 reward when it reaches the goal, -1 when hitting the table, and 0 otherwise.

Original: The episode terminates either when the robot hits the table or when it touches 5 times the target (hence, the max.
Revised: The episode terminates either when the robot hits the table or when it touches 5 times the target (hence, the max.

Original: reward value is 5).
Revised: reward value is 5).

Original: Episodes have a maximum length of 1000/1500 steps in the random/moving target settings, respectively.
Revised: Episodes have a maximum length of 1000/1500 steps in the random/moving target settings, respectively.

Original: All environments correspond to a fully observable Markov Decision Process (MDP), i.e., target object and agent are always visible and the next observation
Revised: All environments correspond to a fully observable Markov Decision Process (MDP), i.e., target object and agent are always visible and the next observation

Original: o t +1 only depends on the previous couple ( o t , s t ) (except for the robot arm setting with moving target where there is small uncertainty for the position of the target).
Revised: o t +1 only depends on the previous couple ( o t , s t ) (except for the robot arm setting with moving target where there is small uncertainty for the position of the target).

Revised: We chose those speciﬁc tasks because they were i) designed for robotics, ii) appropriate for evaluating SRL models iii) of gradual difﬁculty.

Revised: Those environments cover basic goal-based robotics tasks: mobile navigation and reaching a 3D position.

Revised: Then, they are designed with simplicity in mind, making the extracted features easier to interpret.

Revised: It is also clear what a good state representation should encode because of the small number of relevant elements.

Original: 5. E VALUATION METRICS
Revised: 5. E VALUATION METRICS

Original: We use two methods to evaluate a learned state representation.
Revised: We use two methods to evaluate a learned state representation.

Original: First, since the main goal of extracting relevant features is to solve a task, we compare performance in Reinforcement Learning.
Revised: First, since the main goal of extracting relevant features is to solve a task, we compare performance in Reinforcement Learning.

Original: To have quantitative results, each RL experiment uses 10 different random seeds 2 .
Revised: To have quantitative results, each RL experiment uses 10 different random seeds 2 .

Original: We chose two metrics: mean reward over 100 episodes at the end of training and mean reward over 100 episodes for a given budget (a ﬁxed number of timesteps).
Revised: We chose two metrics: mean reward overepisodes at the end of training and mean reward over 100 episodes for a given budget (a ﬁxed number of timesteps).

Original: This last metric is particularly relevant when doing robotics: the budget is much more limited than in simulation and we want to reach an acceptable performance as soon as possible.
Revised: This last metric is particularly relevant when doing robotics: the budget is much more limited than in simulation and we want to reach an acceptable performance as soon as possible.

Original: Then, since we have access to the true positions, we can also compute the correlation between ground truth states and learned states.
Revised: Then, since we have access to the true positions, we can also compute the correlation between ground truth states and learned states.

Original: However, looking at a correlation matrix when the state dimension is large is impractical.
Revised: However, looking at a correlation matrix when the state dimension is large is impractical.

Original: Therefore, we use the measure GroundTruthCorrelation ( GTC ) described in Rafﬁn et al.
Revised: Therefore, we use the metric GroundTruthCorrelation ( GTC ) described in (Rafﬁn et al., 2018).

Original: It measures the maximum correlation (in absolute value) in the learned representation for each dimension of the ground truth  states.
Revised: It measures the maximum correlation (in absolute value) in the learned representation for each dimension of the ground truth (GT) states.

Original: GTC gives insights on the learned states: if they are sufﬁcient and disentangled , then each component of the GTC will be close to 1.
Revised: GTC is deﬁned as: [Equation] where ρ s, ˜ s is the Pearson correlation coefﬁcient for the pair ( s , ˜ s ), where ˜ s is the GT state, s the learned state, i ∈ (cid:74) 0 , | ˜ s | (cid:75) , j ∈ (cid:74) 0 , | s | (cid:75) , ˜ s

Original: By taking the average across components of  GTC , a metric can be derived, named GTC mean .
Revised: = [˜ s 1 ; ... ; ˜ s n ] , and ˜ s k being the k th dimension of the GT state vector. GTC produces a vector of a ﬁxed size for a given environment making it able to compare different approaches and different representation dimensions. This measure allows to have some understanding of what was learned.

Original: It allows to have a rough estimation of how much information  was encoded and make a comparison between SRL models.
Revised: More precisely, GTC can provide us with information on the sufﬁciency: it gives insights of which component of the Ground Truth state was encoded in the state representation. If in addition, the representation is disentangled, then each factor of variation will be encoded by a different component, leading to a GTC close to 1. that uses 5 random seeds However, this measure is only a proxy to minimally guarantee that the required information is encoded: having a good GTC is sufﬁcient to succeed in RL but not necessary.

Revised: For instance, as described more extensively in Appendix C, the correlation is not invariant to transformations, such as rotation of the frame.

Revised: Finally, this metric is not an estimate of the representation compactness since it is independent of the representation dimension.

Original: 5. I MPLEMENTED APPROACH AND BASELINES
Revised: 5. I MPLEMENTED APPROACH AND BASELINES

Original: We evaluate the two proposed combination methods:
Revised: We evaluate the two proposed combination methods:

Original: • SRL Combination
Revised: • SRL Combination

Original: The combination of reconstruction, reward and inverse losses is done by averaging them on a single embedding (Sec. 4.3).
Revised: The combination of reconstruction, reward and inverse losses is done by averaging them on a single embedding (Sec. 4.3).

Original: • SRL Splits
Revised: • SRL Splits

Original: The model described in Sec.
Revised: The model described in Sec.

Original: 4.4 and Fig. 2 that combines reconstruction, reward and inverse losses using splits of the state representation.
Revised: 4.4 and Fig. 2 that combines reconstruction, reward and inverse losses using splits of the state representation.

Original: and compare them with several baselines:
Revised: and compare them with several baselines:

Original: • Raw Pixels Learning a policy in an end-to-end manner, directly from pixels to actions.
Revised: • Raw Pixels Learning a policy in an end-to-end manner, directly from pixels to actions.

Original: • Ground Truth (GT)
Revised: • Ground Truth (GT)

Original: Hand engineered features: true robot and target object positions.
Revised: Hand engineered features: true robot and target object positions.

Original: • Supervised A model trained with Ground Truth states as targets in a supervised setting.
Revised: • Supervised A model trained with Ground Truth states as targets in a supervised setting.

Original: • Random Features
Revised: • Random Features

Original: The feature extractor, a convolutional network, is ﬁxed after random initialization.
Revised: The feature extractor, a convolutional network, is ﬁxed after random initialization.

Original: • Auto-encoder
Revised: • Auto-encoder

Original: We took the best model between auto-encoder (cf Sec. 4), denoising auto-encoder and Variational Auto-Encoder (VAE) (Kingma & Welling, 2013), which was in our case the vanilla one.
Revised: We took the best model between auto-encoder (cf. Sec. 4), denoising auto-encoder and Variational Auto-Encoder (VAE) (Kingma & Welling, 2013), which was in our case the vanilla one.

Original: • Robotic 
Revised: • Robotic Priors

Original: Priors The method described in (Jonschkowski et al., 2017) that encodes prior knowledge about the world as losses 3 .
Revised: The method described in (Jonschkowski et al., 2017) that encodes prior knowledge about the world as losses 3 .

Original: Each state representation has a dimension of 200 and is learned using 20 000 samples collected with a random policy.
Revised: Each state representation has a dimension of 200 and is learned using 20 000 samples collected with a random policy.

Original: The implementation and additional training details can be found in Appendix A.
Revised: The implementation and additional training details can be found in Appendix A.

Original: 5. E ND - TO -E ND VERSUS S TATE R EPRESENTATION L EARNING Environments
Revised: 5. E ND - TO -E ND VERSUS S TATE R EPRESENTATION L EARNING Environments

Original: [Table]
Revised: [Table]

Original: Ground [Table] Table 1 displays the mean reward, averaged on 100 episodes, for each environment after RL training.
Revised: Table 1 displays the mean reward, averaged on 100 episodes, for each environment after RL training.

Revised: Tableshows the RL performance for different budgets on the Robotic Arm with random target task.

Original: To compare SRL methods, GTC , GTC mean and associated RL performance are displayed in Table 2 for the navigation task with a 2D target.
Revised: To compare SRL methods, GTC , GTC mean and associated RL performance are displayed in Table 2 for the navigation task with a 2D target.

Original: Complete results for all the environments can be found in Appendix B.
Revised: Complete results for all the environments can be found in Appendix B [Table]

Original: For every environment, there is always a SRL method that reaches or exceeds the performance obtained using only the raw pixels as input.
Revised: For every environment, there is always a SRL method that reaches or exceeds the performance obtained using only the raw pixels as input.

Original: The gap is more striking on the robotic arm environments. [Table]
Revised: SRL methods do not necessary improve ﬁnal performance (see Table 1), however as shown in Table 3, it is useful to improve the learning speed of the policy. For instance, in the robotic arm task, much more samples are needed to attain similar levels of performance to those achieved by learning in an end-to-end manner.

Original: SRL Splits is the approach that performs on par or better than learning from raw pixels across all the tasks.
Revised: SRL Splits is the approach that performs on par or better than learning from raw pixels across all the tasks.

Original: Its counterpart, SRL Combination , that uses only a single embedding, gives also positive results, except for the navigation environment with a 2D random target where it under-performs.
Revised: Its counterpart, SRL Combination , that uses only a single embedding, gives also positive results, except for the navigation environment with a 2D random target where it under-performs.

Original: The GTC provides us with  some insights (see Table 2): both methods extract the robot position (absolute correlation close to 1), yet the target position is better encoded with the SRL splits method, which may explain the gap in performance.
Revised: The GTC provides us with [Table] some insights (see Table 2): both methods extract the robot position (absolute correlation close to 1), yet the target position is better encoded with the SRL splits method, which may explain the gap in performance.

Original: In the robotic arm setting, combining approaches does not seem to be of much beneﬁt.
Revised: In the robotic arm setting, combining approaches does not seem to be of much beneﬁt.

Original: Two possible reasons may explains that.
Revised: Two possible reasons may explains that.

Original: First, compared to the mobile robot, the robotic arm and the target are visually salient so an auto-encoder is sufﬁcient to solve the task.
Revised: First, compared to the mobile robot, the robotic arm and the target are visually salient so an auto-encoder is sufﬁcient to solve the task.

Original: Second, the actions magnitude is much smaller in the robotic arm environments, therefore learning an inverse model is much harder in this setting.
Revised: Second, the actions magnitude is much smaller in the robotic arm environments, therefore learning an inverse model is much harder in this setting [Table]

Original: Ground Truth states naturally outperform all the methods across all environments.
Revised: Ground Truth states naturally outperform all the methods across all environments.

Original: This highlights the importance of having a low dimensional and informative representation.
Revised: This highlights the importance of having a low dimensional and informative representation.

Original: The Supervised baseline allows to quickly attain an acceptable performance, but then reaches a plateau (e.g. Fig. 6).
Revised: The Supervised baseline allows to quickly attain an acceptable performance, but then reaches a plateau (e.g. Fig. 6).

Original: Compared to the unsupervised methods, it apparently generalizes less efﬁciently to data not present in the training set.
Revised: Compared to the unsupervised methods, it apparently generalizes less efﬁciently to data not present in the training set.

Original: As in Burda et al.
Revised: As in Burda et al.

Original: (2018), the Random Features model performs decently on all the environments and sometimes better (cf Table 7) than learned features.
Revised: (2018), the Random Features model performs decently on all the environments and sometimes better (cf. Table 3) than learned features.

Original: Looking at the GTC (Tables 2, 3, 6, 8), random features keep the useful information to solve the tasks.
Revised: Looking at the GTC (Tables 2, 5, 8, 9), random features keep the useful information to solve the tasks.

Revised: Hence, we hypothesize that random features should work in environments where visual observations are simple enough because they can preserve enough information.

Original: Despite good results in mobile robot navigation with a static target (Jonschkowski & Brock, 2015), Robotic
Revised: Despite good results in mobile robot navigation with a static target (Jonschkowski & Brock, 2015), Robotic

Original: Priors are not well suited when the target changes from episode to episode.
Revised: Priors are not well suited when the target changes from episode to episode.

Original: As described in Lesort et al.
Revised: As described in Lesort et al.

Original: (2017), robotics priors lead to a state representation that contains one cluster per episode, which prevent generalization and good performances in these RL tasks.
Revised: (2017), robotics priors lead to a state representation that contains one cluster per episode, which prevent generalization and good performances in these RL tasks.

Original: The auto-encoder has mixed results. It allows to solve all environments, yet it under-performs in the navigation tasks.
Revised: The auto-encoder has mixed results: it allows to solve all environments, yet it under-performs in the navigation tasks.

Original: When we explored the latent space using the S-RL Toolbox (Rafﬁn et al., 2018), we noticed that one dimension of the state space could act on both robot and target positions in the reconstructed image.
Revised: When we explored the latent space using the S-RL Toolbox (Rafﬁn et al., 2018), we noticed that one dimension of the state space could act on both robot and target positions in the reconstructed image.

Original: Our hypothesis, also supported by the GTC , is that the state space is not disentangled.
Revised: Our hypothesis, also supported by the GTC [Table] is that the state space is not disentangled.

Original: This approach does not make use of additional information that the environment provides, such as actions and rewards, leading to a latent space that may lack of informative structure.
Revised: This approach does not make use of additional information that the environment provides, such as actions and rewards, leading to a latent space that may lack of informative structure.

Original: 5. A BLATION  AND HYPERPARAMETERS INFLUENCE STUDY
Revised: 5. A BLATION STUDY AND HYPERPARAMETERS INFLUENCE STUDY

Original:  To better understand the inﬂuence of each hyper-parameter and study the robustness of SRL, we performed a thorough analysis of SRL Splits in the mobile robot navigation with 2D random target setting.
Revised: [Table] To better understand the inﬂuence of each hyper-parameter and study the robustness of SRL, we performed a thorough analysis of SRL Splits in the mobile robot navigation with 2D random target setting.

Original: Figure 4 (and Table 10 in the appendix) show the result of the ablation study performed on the SRL Splits model.
Revised: Figure 4 (and Table 11 in the appendix) show the result of the ablation study performed on the SRL Splits model.

Original: As expected, the inverse model allows to extract the position of the controllable object, which is the robot.
Revised: As expected, the inverse model allows to extract the position of the controllable object, which is the robot.

Original: This helps to solve the task and results in a performance boost.
Revised: This helps to solve the task and results in a performance boost.

Original: In the same vein, the addition of a reward loss favors the encoding of the target position.
Revised: In the same vein, the addition of a reward loss favors the encoding of the target position.

Original: It also does not seem necessary to separate the reconstruction and reward losses as they encode the same information.
Revised: It also does not seem necessary to separate the reconstruction and reward losses as they encode the same information.

Original: Table 11 displays the inﬂuence of the weights of the loss combination on the ﬁnal mean reward.
Revised: Table 4 displays the inﬂuence of the weights of the loss combination on the ﬁnal mean reward.

Original: It shows that the method works on a wide range of different weighting schemes, as long as the reconstruction and the inverse loss have similar magnitude.
Revised: It shows that the method works on a wide range of different weighting schemes, as long as the reconstruction and the inverse loss have similar magnitude.

Original: When the reconstruction weight is one order of magnitude greater, the model behaves like an auto-encoder (because the feature extractor is shared).
Revised: When the reconstruction weight is one order of magnitude greater, the model behaves like an auto-encoder (because the feature extractor is shared).

Original: In the appendix, extra results (Figs. 10, 11 and 12) exhibit the stability and robustness of SRL against additional hyper-parameter changes (random seed, training set size and dimensionality of the state learned).
Revised: In the appendix, extra results (Figs. 10, 11 and 12) exhibit the stability and robustness of SRL against additional hyper-parameter changes (random seed, training set size and dimensionality of the state learned).

Original: The state dimension needs to be large enough (at least 50 dimensions for the mobile navigation environment), but increasing it further has no incidence on the performance in RL.
Revised: The state dimension needs to be large enough (at least 50 dimensions for the mobile navigation environment), but increasing it further has no incidence on the performance in RL.

Original: In a similar way, a minimal number of training samples ( 10000 ) is required to efﬁciently solve the task.
Revised: In a similar way, a minimal number of training samples ( 10000 ) is required to efﬁciently solve the task.

Original: Over that limit, adding more samples does not affect the ﬁnal mean reward.
Revised: Over that limit, adding more samples does not affect the ﬁnal mean reward.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: During our experiments, we found that learning the policy end-to-end was more sensitive to hyperparameter changes.
Revised: During our experiments, we found that learning the policy end-to-end was more sensitive to hyperparameter changes.

Original: For instance, hyper-parameters tuning of A2C (Fig.7) was needed in order to have decent results for the pixels, whereas the performance was stable for the SRL methods.
Revised: For instance, hyper-parameters tuning of A2C (Fig.7) was needed in order to have decent results for the pixels, whereas the performance was stable for the SRL methods.

Original: This can be explained by the reduced search space: the task is simpler to solve when features are already extracted.
Revised: This can be explained by the reduced search space: the task is simpler to solve when features are already extracted.

Original: A more in-depth study would be interesting in the future.
Revised: A more in-depth study would be interesting in the future.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, wehave presented the advantages of decoupling feature extraction from policy learning in RL, on a set of goal-based robotics tasks.
Revised: In this work, we presented the advantages of decoupling feature extraction from policy learning in RL, on a set of goal-based robotics tasks.

Original: This decomposition reduces the search space, accelerates training, improves performancesin most settings and gives more easily interpretable representations with respect to the true state of the system.
Revised: This decomposition reduces the search space, accelerates training, does not degrade ﬁnal performances and gives more easily interpretable representations with respect to the true state of the system.

Revised: We show also that random features provide a good baseline versus end-to-end learning.

Original: We introduced a new way of effectively combining approaches by splitting the state representation.
Revised: We introduced a new way of effectively combining approaches by splitting the state representation.

Original: This method uses the strengths of different SRL models and reduces interference between opposed or conﬂicting objectives when learning a feature extractor.
Revised: This method uses the strengths of different SRL models and reduces interference between opposed or conﬂicting objectives when learning a feature extractor.

Original: Finally, we showed the inﬂuence of hyper-parameters on SRL models and the relative robustness of those models against perturbations.
Revised: Finally, we showed the inﬂuence of hyper-parameters on SRL Split model and the relative robustness of this model against perturbations.

Original: Future work should take advantage of the study done in simulation to experiment those methods on real robots.
Revised: Future work should take advantage of the study done in simulation to experiment those methods on real robots.

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: Each state representation is learned using 20 000 samples collected using a random policy.
Revised: Each state representation is learned using 20 000 samples collected using a random policy.

Original: We kept for each method the model with the lowest validation loss during the 30 training epochs.
Revised: We kept for each method the model with the lowest validation loss during the 30 training epochs.

Original: We used the same network architecture from (Rafﬁn 
Revised: We used the same network architecture from (Rafﬁn et al., 2018) for all the models. The input observations of all models are RGB images of size 224 × 224 × 3 . Navigation environments use 4 discrete actions ( right, left, forward, backward ); robotic arm environments use one more ( down ) action.

Original: We used PPO (the GPU version called PPO2) and A2C implementations from stable-baselines (Hill et al., 2018), a fork of OpenAI Baselines (Dhariwal et al., 2017).
Revised: We used PPO (the GPU version called PPO2) and A2C implementations from stable-baselines (Hill et al., 2018), a fork of OpenAI Baselines (Dhariwal et al., 2017).

Original: PPO was the RL algorithm that worked well across environments and methods without any hyper-parameter tuning, and therefore, the selected one for our experiments.
Revised: PPO was the RL algorithm that worked well across environments and methods without any hyper-parameter tuning, and therefore, the selected one for our experiments.

Original: Regarding the network learning the policies, the same architecture is used in all different methods.
Revised: Regarding the network learning the policies, the same architecture is used in all different methods.

Original: For the approaches that do not use pixels, it is a 2-layers MLP, whereas for learning from raw pixels, it is the CNN from (Mnih et al., 2015) present in OpenAI baselines.
Revised: For the approaches that do not use pixels, it is a 2-layers MLP, whereas for learning from raw pixels, it is the CNN from (Mnih et al., 2015) present in OpenAI baselines.

Original: Observations are normalized, either by dividing the input by 255 (for raw pixels) or by computing a running mean/std average (for SRL models).
Revised: Observations are normalized, either by dividing the input by 255 (for raw pixels) or by computing a running mean/std average (for SRL models).

Original: For the SRL Splits and SRL Combination methods, we used a linear model for the inverse dynamics, and a 2-layers MLP of 16 units each with ReLU activation for the reward prediction.
Revised: For the SRL Splits and SRL Combination methods, we used a linear model for the inverse dynamics, and a 2-layers MLP of 16 units each with ReLU activation for the reward prediction.

Original: Only one minor adjustment was made on the SRL Splits method for the robotic arm environments: because the controlled robot is more complex, the inverse model was allocated more dimensions (10 instead of 2) but keeping the total state dimension constant (equal to 200).
Revised: Only one minor adjustment was made on the SRL Splits method for the robotic arm environments: because the controlled robot is more complex, the inverse model was allocated more dimensions (10 instead of 2) but keeping the total state dimension constant (equal to 200).

Original: Code and data to reproduce our results will be released after the reviewing process.
Revised: Code and data to reproduce our results will be released after the reviewing process.

Original: B A DDITIONAL R ESULTSB. M OBILE N AVIGATION WITH 1D T ARGET
Revised: B A DDITIONAL R ESULTS

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Navigation Target
Revised: a r

Original: [Figure]
Revised: [Figure]

Original: Budget
Revised: s

Original:  [Table]Supervised
Revised: [Figure] [Figure] [Table]

Original: [Table]
Revised: [Table]

Revised: RL (using PPO) per episode (average on 100 episodes) for different budgets, with standard error in Navigation 1D target environment.

Original: B. M OBILE N AVIGATION WITH 2D T ARGET
Revised: B. M OBILE N AVIGATION WITH 2D T ARGET

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: R e w a r d s
Revised: R e w a r d s

Original: [Figure]
Revised: [Figure]

Original: splits supervised
Revised: splits supervised

Original: Ground Truth Correlation x robot y robot
Revised: Ground Truth Correlation x robot y robot

Original: z robot x target y target
Revised: z robot x target y target

Original: Mean
Revised: Mean

Original: Mean Reward
Revised: Mean Reward

Original: Ground Truth      4.2 ± 0.
Revised: Ground Truth      4.2 ± 0.

Original: Supervised 0.
Revised: Supervised 0.

Original: 3.1 ± 0.
Revised: 3.1 ± 0.

Original: Random Features 0.
Revised: Random Features 0.

Original: 4.1 ± 0.
Revised: 4.1 ± 0.

Original: Robotic Priors 0.
Revised: Robotic Priors 0.

Original: 2.6 ± 0.
Revised: 2.6 ± 0.

Original: Auto-Encoder 0.
Revised: Auto-Encoder 0.

Original: 3.4 ± 0.
Revised: 3.4 ± 0.

Original: SRL Combination 0.
Revised: SRL Combination 0.

Original: 2.9 ± 0.
Revised: 2.9 ± 0.

Original: SRL Splits 0.
Revised: SRL Splits 0.

Original: 3.7 ± 0.
Revised: 3.7 ± 0.

Original: Budget (in timesteps) 1 Million 2 Million 3 Million 5 Million

Original: Ground Truth 4.1 ± 0.

Original: 4.1 ± 0.

Original: 4.1 ± 0.

Original: 4.2 ±

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: Ground Truth Correlation x robot y robot
Revised: Ground Truth Correlation x robot y robot

Original: z robot x target y target
Revised: z robot x target y target

Original: Mean
Revised: Mean

Original: Mean Reward
Revised: Mean Reward

Original: Ground Truth      4.6 ± 0.
Revised: Ground Truth      4.6 ± 0.

Original: Supervised 0.
Revised: Supervised 0.

Original: 3.1 ± 0.
Revised: 3.1 ± 0.

Original: Random Features 0.
Revised: Random Features 0.

Original: 3.0 ± 0.
Revised: 3.0 ± 0.

Original: Robotic Priors 0.

Original: 2.6 ± 0.

Original: Auto-Encoder 0.
Revised: Auto-Encoder 0.

Original: 3.0 ± 0.
Revised: 3.0 ± 0.

Original: SRL Combination 0.
Revised: SRL Combination 0.

Original: 2.9 ± 0.
Revised: 2.9 ± 0.

Original: SRL Splits 0.
Revised: SRL Splits 0.

Original: 2.5 ± 0.
Revised: 2.5 ± 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 Million
Revised: Million

Original: [Table]
Revised: [Table]

Original: Million3 Million 5 Million Ground Truth 4.3 ± 0. 4.4 ± 0. 4.4 ± 0.
Revised: Million

Original: 4.6 ±

Original: [Table]
Revised: [Table]

Original: B. A BLATION STUDY
Revised: Million

Original: [Table]
Revised: [Table]

Original: B
Revised: Supervised

Original: w reconstruction
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B. I NFLUENCE OF THE RANDOM SEED
Revised: B. A BLATION STUDY

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Influence
Revised: d

Original: [Figure]
Revised: [Figure]

Original:  seed_ seed_ seed_ seed_ seed_
Revised: Influence seed_ seed_ seed_ seed_ seed_

Original: Figure 10 shows that the SRL Split method is stable and its performance does not depend on the random seed used.[Figure] [Figure]
Revised: Figure 10 shows that the SRL Split method is stable and its performance does not depend on the random seed used.

Original: B. I NFLUENCE OF THE STATE DIMENSION
Revised: B. I NFLUENCE OF THE STATE DIMENSION

Original: As shown in ﬁgure 11, the state dimension for the SRL model needs to be large enough in order to efﬁciently solve the task.
Revised: As shown in ﬁgure 11, the state dimension for the SRL model needs to be large enough in order to efﬁciently solve the task.

Original: However, over a threshold, increasing the state dimension does not affect (positively or negatively) the performance in RL.
Revised: However, over a threshold, increasing the state dimension does not affect (positively or negatively) the performance in RL.

Original: B. I NFLUENCE OF THE TRAINING SET SIZE

Original: [Figure]
Revised: [Figure]

Original: Influence of the Training Set Size
Revised: Influence of the State Dimension

Original: [Figure]
Revised: [Figure]

Original:  The inﬂuence of the training set size (Fig. 12) is somehow similar to the inﬂuence of the state dimension (Fig. 11).
Revised: [Figure] The inﬂuence of the training set size (Fig. 12) is somehow similar to the inﬂuence of the state dimension (Fig. 11).

Original: A minimal number of training samples is required to solve the task, but over a certain limit, increasing the training set size is not beneﬁcial anymore.
Revised: A minimal number of training samples is required to solve the task, but over a certain limit, increasing the training set size is not beneﬁcial anymore.

Revised: C I NFLUENCE OF R OTATION ON GTC

Revised: In this section, we discuss the inﬂuence of rotation on the GTC metric in order to answer the question: how would rotating the measurement frame of the ground truth state inﬂuence the results?

Revised: First, let’s start with an intuitive example of a 2D robot position.

Revised: In that case, if α is the frame rotation angle, the new x coordinate (we will write it x (cid:48) ) is x (cid:48)

Revised: = x × cos ( α ) + y × sin ( α ) .

Revised: The correlation between x and x (cid:48) should a priori decrease until α = π/ 4 ( cos ( π/ 4)

Revised: = sin ( π/ 4) ≈ 0 . 71 ), after which x (cid:48) should have a higher correlation with y (and y (cid:48) should be more correlated with x ).

Revised: This ﬁrst example uses in fact the additional assumption that σ y σ x ≈ 1 , as we will see in the more formal answer we present in the following paragraph.

Revised: We will keep the study in ]0 , π/ 2[ because the behavior for other values of alpha can be deduced by symmetry.

Revised: Under the assumption that the data are zero-centered 4 and that x and y are independent, we can derive these formulas for correlation between x , y and x (cid:48) :

Revised: [Equation]

Revised: Looking at the equations, the correlation between x and x (cid:48) ( ρ xx (cid:48) ) will decrease with α , whereas ρ yx (cid:48) will increase with it.

Revised: The two coefﬁcients are equal for α = tan − 1 ( σ 2 x σ 2 y ) (equal to π/ 4 when σ x = σ y ) with a value of

Revised: [Equation]

Revised: which is around 0 .

Revised: 71 when σ

Revised: [Equation]

Revised: Hence, the two important factors are the rotation angle α and the ratio σ y σ x .

Revised: The additional assumption (that is true in our experiments) that makes GTC less sensitive to rotation is σ y σ x ≈ 1 .

Revised: If this is not the case, a simple solution is to normalize ground truth data (zero mean, unit variance).