Paper: ASID: Active Exploration for System Identification in Robotic Manipulation (ICLR-2024)

Reviewer comment (Presentation):
Presentation/clarity can be improved: specifically, the abstract and introduction mostly describe the field of active exploration for system identification and adaptive control, as opposed to the specific method proposed, which appears to overstate the paper’s novelty. Furthermore, the approach warrants a better intuitive explanation. As I understand it, the Fisher Information objective attempts to quantify the sensitivity of model parameters to trajectories expected given some policy. Therefore, maximizing this objective yields a policy that, when executed, yields the maximum additional information about the model parameters.

Author response:
__”Presentation/clarity can be improved: specifically, the abstract and introduction mostly describe the field of active exploration for system identification and adaptive control, as opposed to the specific method proposed, which appears to overstate the paper’s novelty. Furthermore, the approach warrants a better intuitive explanation. As I understand it, the Fisher Information objective attempts to quantify the sensitivity of model parameters to trajectories expected given some policy. Therefore, maximizing this objective yields a policy that, when executed, yields the maximum additional information about the model parameters.”__

This is correct. As described in Section 3, the inverse Fisher information matrix serves as a fundamental lower bound on parameter estimation error, so exploring to maximize the Fisher information will collect data that will yield the most accurate estimate of the unknown parameters.

Intuitively, the Fisher information quantifies the sensitivity of the trajectory distribution to the unknown parameter. Note that the Fisher information is defined in terms of the gradient of the log-likelihood with respect to the unknown parameter. Thus, trajectory distributions for which the Fisher information is large correspond to trajectory distributions that are very sensitive to the unknown parameters, i.e., trajectory distributions that are significantly more likely under one set of parameters than another. By exploring to maximize the Fisher information, we, therefore, will collect trajectories that are maximally informative about the unknown parameters, since we will observe trajectories that will be significantly more likely under one set of parameters than another.

Reviewer comment (Theory):
For a paper proposing active exploration for the sake of system identification, I wanted to see more discussion of the following: a) regret minimization, i.e. can you prove that your method minimizes regret and achieves the best policy with the fewest interactions with the environment? b) identifiability, i.e. can you say anything about whether all system parameters will be uniquely identified with infinite interactions?

Author response:
__”For a paper proposing active exploration for the sake of system identification, I wanted to see more discussion of the following: a) regret minimization, i.e. can you prove that your method minimizes regret and achieves the best policy with the fewest interactions with the environment? b) identifiability, i.e. can you say anything about whether all system parameters will be uniquely identified with infinite interactions?”__

(a) We thank the reviewer for bringing up the regret of our approach, as this is an important feature of our method. It is known that there is a fundamental tradeoff between minimizing standard online regret (i.e. the amount of reward achieved in every online interaction compared to the best possible reward achieved) and exploring to learn the optimal policy: to learn the optimal policy as quickly as possible, one must often explore in a way that incurs large regret (see e.g. Proposition 4.3 of [1] below for a formal discussion of this). Our aim is to explore so as to learn the optimal policy as efficiently as possible and, as such, given the above dichotomy, our approach may not minimize regret (collect a large amount of reward) during exploration. Indeed, in tasks such as rod balancing this is clearly visible: our exploration policy does not aim to pick up the rod (which is required to collect reward), but instead seeks to push it, as this provides more informative data. This is in sharp contrast to the majority of existing sim2real methods we are aware of, which aim to minimize regret during exploration, typically by simply transferring a policy from sim which aims to solve the goal task, and is a key advantage of our approach over such methods.
On whether or not our approach can identify the best policy as quickly as possible, we do not have a formal proof. However, we remark that 1) our approach will collect data to optimally minimize parameter estimation error (see our reply to Reviewer zEL2) and 2) our approach will achieve a significantly faster rate for learning the best policy than existing sim2real approaches which do not explicitly explore as ours does, as is shown by Proposition 4.3 of [1]. There is in fact a subtle and interesting tradeoff between finding the best policy and estimating parameters accurately—certain parameters may be more relevant to finding the best policy than others, and finding the best policy at the optimal rate exploration should be focused particularly on these parameters (this is formalized in the context of linear dynamical systems in [1]). Incorporating this in our method is beyond the scope of this work but is an exciting direction for future work.

(b) Whether or not the parameter is identifiable depends on the properties of the given environment. It is possible that two different parameters induce identical distributions over trajectories, in which case the true parameter will not be identifiable. We make no assumptions on whether or not the necessary assumptions for identifiability are met. We do remark, however, that in cases where the true parameter is not identifiable, the optimal policy for the downstream task on these simulators would be identical (since for the optimal policies to differ there must be a difference in trajectory distributions). Thus, in cases when the true parameter is not identifiable, identifying it is irrelevant to solving the downstream task, and so will not affect our method’s ability to succeed on this task.

We will add further discussion on these points to the final version of the paper.

[1] Wagenmaker, Andrew J., Max Simchowitz, and Kevin Jamieson. Task-optimal exploration in linear dynamical systems. ICML, 2021

Reviewer comment (Theory):
More detail on why the Fisher Information is used vs other methods (observability Grammian, Kalman Filter covariance).

Author response:
__”More detail on why the Fisher Information is used vs other methods (observability Grammian, Kalman Filter covariance).”__

As described in Section 3, the inverse Fisher information matrix is a fundamental lower bound on the difficulty of parameter estimation and, in addition, serves as an upper bound when the estimator is, for example, the maximum likelihood estimate. Given this, to minimize parameter estimation error (which is the goal of our exploration procedure), we should aim to explore so as to maximize the Fisher information—doing this will collect data that produces the most accurate estimate of the unknown parameters. Thus, in short, maximizing the Fisher information is the mathematically optimal thing to do if the goal is to minimize estimation error. While criteria other than the Fisher information may be used to direct exploration as the reviewer suggests, they lack this optimality property, and would not necessarily collect data that yields the best possible parameter estimate.

We also remark that using the Fisher information to guide the data collection has a long history in the theory of statistics and experiment design. We refer the reviewer to works such as [1] and [2] below for further discussion of this, as well as formal justification of the argument we have presented above.

[1] Luc Pronzato and Andrej Pazman. Design of experiments in nonlinear models. Lecture notes in statistics, 212(1), 2013.

[2] Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.

Reviewer comment (Novelty):
The main weakness for this paper is that this pipeline is very similar to other exploration methods in model-based RL. For example: Shyam P, Jaśkowski W, Gomez F. Model-based active exploration. In International conference on machine learning 2019 May 24 (pp. 5779-5788). Pathak D, Gandhi D, Gupta A. Self-supervised exploration via disagreement. In International conference on machine learning 2019 May 24 (pp. 5062-5071). In fact, the pipeline is quite similar to Shyam et al. albeit the metrics and models used are different. However, due to the similarities in the process, those papers should be discussed and, ideally, included in the comparison.

Author response:
__”The main weakness for this paper is that this pipeline is very similar to other exploration methods model-based RL. For example: Shyam et al. Model-based active exploration. ICML, 2019. In fact, the pipeline is quite similar to Shyam et al. albeit the metrics and models used are different. However, due to the similarities in the process, those papers should be discussed and, ideally, included in the comparison.__

__I do not fully understand the reference to REPS as that is a model-free RL method. There is no transition model estimation.
”__

We would like to note that ASID is not quite in the same realm as a model-based RL algorithm. Instead, it considers exploring and collecting data so as to identify the parameters of a physics-based *simulator* (such as MuJoCo or PyBullet), rather than an end-to-end model as most model-based RL algorithms would do. In the following, we discuss 1) the conceptual difference of ASID from model-based exploration in terms of explorative behavior and model extrapolation, and 2) provide empirical evidence.

_1) Conceptual difference of ASID from model-based exploration_

ASID fundamentally differs from model-based exploration in the problem setting. Model-based exploration typically leverages fully learned models to guide exploration with the goal of achieving coverage in the state space. The exploration methods proposed by [1] [2] use a learned ensemble of forward models and their disagreement to learn multiple exploration policies that together cover the state space. The collected data is then used to train a downstream task policy. In contrast, our approach relies on the use of an existing physics-based simulator and explores to identify specific unknown parameters in the simulator (e.g. mass, friction coefficients), rather than learning a full model. Our exploration policy searches out state transitions that are most affected by a change in the underlying physics parameters [Appendix Fig. 10c, 10d]. We then roll out a single trajectory in the real world to identify the parameters of the simulator. We use a physics simulator because it extrapolates to unseen states in contrast to a fully learned model [Appendix Fig. 11]. 

This means we don't learn a dynamics model anywhere in our pipeline and use a model-free black box optimization method (REPS) to identify the physics parameters of our simulator. However, we'd like to note that using a simulator in the process can be seen as a very specific instance of a model-based approach.

_2) Empirical Evidence_

We construct an experiment to differentiate ASID from Model-based active exploration (MAX) [1]. The environment contains four spheres out of which the red sphere’s friction parameters are varied while keeping the parameters of the three blue spheres fixed. As in our previous experiments, the robot uses delta endeffector control (x,y) and is equipped with a peg instead of a gripper [Appendix Fig. 9].

Investigating the exploration behavior, we show that MAX aims to cover the state space through multiple policies throughout training. Since it uses the disagreement between fully learned dynamics models, it gets distracted by novel states induced by the movement of all spheres (red and blue) [Appendix Fig. 10a, 10b]. In contrast, our method based on the Fisher information yields a single policy that seeks out the sphere affected by the changing physics parameters (red) and ignores the irrelevant spheres (blue) even if they lead to novel states [Appendix Fig. 10c, 10d].

With the collected data of both exploration methods, we train a forward dynamics model $s_{t+1} = f_{theta}(s_t,a_t)$. We model $f_{theta}$ as a three-layer MLP and train using the MSE loss until convergence. When evaluated on out-of-distribution trajectories, i.e., trajectories not included in the training data, we find the model to be extremely inaccurate [Appendix Fig. 11]. While the simulator extrapolates to unseen states and correctly predicts the movement of the sphere, the model hallucinates movement even when the endeffector does not interact with it at all [Appendix Fig. 11c, 11d]! These findings make ASID preferable to a purely model-based approach.

[1] Pranav Shyam, Wojciech Jaśkowski, Faustino Gomez. Model-Based Active Exploration. ICML, 2018

[2] Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta. Self-supervised exploration via disagreement. ICML, 2019

Reviewer comment (Experiments):
While the experimental section is one of the strengths due to the evaluation in a realistic robotic scenario, the methods should also be evaluated on standard benchmarks for comparison, such as HalfCheetah.

Author response:
__”While the experimental section is one of the strengths due to the evaluation in a realistic robotic scenario, the methods should also be evaluated on standard benchmarks for comparison, such as HalfCheetah.”__

Standard benchmarks like HalfCheetah don’t include varying physics parameters and, therefore, do not cover the tasks we aim to solve. As shown in the follow-up experiments [Appendix Fig. 10d], our method yields directed exploration toward state transitions affected by the changing physics parameters.

Reviewer comment (Experiments):
The baseline used [Kumar2019] seems very weak (in Fig 4 it does not explore at all).

Author response:
__”The baseline used [Kumar2019] seems very weak (in Fig 4 it does not explore at all).”__
The reason [2] performs poorly on our tasks lies in the data used to train the estimator. [2] uses object-centric primitives to collect the data, e.g., the robot always pushing the rod. Every trajectory is, therefore, somewhat informative about the rod's center of mass.
Since we assume no access to object-centric primitives, we instead use random exploration to collect the initial data and train the estimator. The data now also contains uninformative trajectories from which the parameters cannot be estimated, e.g., because the robot did not interact with the rod. Due to the long horizon, large state and action space, however, the estimator overfits to all trajectories, being equally certain/uncertain about informative and uninformative trajectories. Now, training a policy using the estimator log-likelihood as a reward leads to the policy learning the initial uninformative random exploration behavior used to collect the data.

Reviewer comment (Novelty):
The work of Kumar2019 does not seem to be related to exploration with mutual information as stated in this work.

Author response:
__”Furthermore, the work of Kumar2019 does not seem to be related to exploration with mutual information as stated in this work.”__

The mutual information of a distribution over physics parameters $\Phi$ and corresponding trajectory distribution $\mathrm{T}$ is defined as:
$I(\Phi|\mathrm{T}) = H\[\Phi] - H[\Phi|\mathrm{T}]$ with $H[\cdot]$ the entropy. With $H[\Phi]$ a fixed quantity, we can approximate $H[\Phi|\mathrm{T}]$ by learning an estimator $q_{\theta}(\phi|\tau)$ from data parameterized by $\theta$ and predicting the physics parameters $\phi$ from a trajectory $\tau$ (c.f. [1]). Minimizing the prediction error of estimator $q$ over $\theta$ (c.f. [2]) then corresponds to maximizing the mutual information.

Reviewer comment (Theory):
I do not fully understand the reference to REPS as that is a model-free RL method. There is no transition model estimation.

Author response:
__”The main weakness for this paper is that this pipeline is very similar to other exploration methods model-based RL. For example: Shyam et al. Model-based active exploration. ICML, 2019. In fact, the pipeline is quite similar to Shyam et al. albeit the metrics and models used are different. However, due to the similarities in the process, those papers should be discussed and, ideally, included in the comparison.__

__I do not fully understand the reference to REPS as that is a model-free RL method. There is no transition model estimation.
”__

We would like to note that ASID is not quite in the same realm as a model-based RL algorithm. Instead, it considers exploring and collecting data so as to identify the parameters of a physics-based *simulator* (such as MuJoCo or PyBullet), rather than an end-to-end model as most model-based RL algorithms would do. In the following, we discuss 1) the conceptual difference of ASID from model-based exploration in terms of explorative behavior and model extrapolation, and 2) provide empirical evidence.

_1) Conceptual difference of ASID from model-based exploration_

ASID fundamentally differs from model-based exploration in the problem setting. Model-based exploration typically leverages fully learned models to guide exploration with the goal of achieving coverage in the state space. The exploration methods proposed by [1] [2] use a learned ensemble of forward models and their disagreement to learn multiple exploration policies that together cover the state space. The collected data is then used to train a downstream task policy. In contrast, our approach relies on the use of an existing physics-based simulator and explores to identify specific unknown parameters in the simulator (e.g. mass, friction coefficients), rather than learning a full model. Our exploration policy searches out state transitions that are most affected by a change in the underlying physics parameters [Appendix Fig. 10c, 10d]. We then roll out a single trajectory in the real world to identify the parameters of the simulator. We use a physics simulator because it extrapolates to unseen states in contrast to a fully learned model [Appendix Fig. 11]. 

This means we don't learn a dynamics model anywhere in our pipeline and use a model-free black box optimization method (REPS) to identify the physics parameters of our simulator. However, we'd like to note that using a simulator in the process can be seen as a very specific instance of a model-based approach.

_2) Empirical Evidence_

We construct an experiment to differentiate ASID from Model-based active exploration (MAX) [1]. The environment contains four spheres out of which the red sphere’s friction parameters are varied while keeping the parameters of the three blue spheres fixed. As in our previous experiments, the robot uses delta endeffector control (x,y) and is equipped with a peg instead of a gripper [Appendix Fig. 9].

Investigating the exploration behavior, we show that MAX aims to cover the state space through multiple policies throughout training. Since it uses the disagreement between fully learned dynamics models, it gets distracted by novel states induced by the movement of all spheres (red and blue) [Appendix Fig. 10a, 10b]. In contrast, our method based on the Fisher information yields a single policy that seeks out the sphere affected by the changing physics parameters (red) and ignores the irrelevant spheres (blue) even if they lead to novel states [Appendix Fig. 10c, 10d].

With the collected data of both exploration methods, we train a forward dynamics model $s_{t+1} = f_{theta}(s_t,a_t)$. We model $f_{theta}$ as a three-layer MLP and train using the MSE loss until convergence. When evaluated on out-of-distribution trajectories, i.e., trajectories not included in the training data, we find the model to be extremely inaccurate [Appendix Fig. 11]. While the simulator extrapolates to unseen states and correctly predicts the movement of the sphere, the model hallucinates movement even when the endeffector does not interact with it at all [Appendix Fig. 11c, 11d]! These findings make ASID preferable to a purely model-based approach.

[1] Pranav Shyam, Wojciech Jaśkowski, Faustino Gomez. Model-Based Active Exploration. ICML, 2018

[2] Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta. Self-supervised exploration via disagreement. ICML, 2019

Reviewer comment (Theory):
It seems that the system relies on the assumption that a learned simulator is able to generate accurate trajectories, but that is not the case for out of distribution trajectories. Exploration precisely minimizes that effect, but the probabilistic model should be able to capture the lack of information in out of distribution data. Currently, the only uncertainty comes from the noise if I understand correctly.

Author response:
__”It seems that the system relies on the assumption that a learned simulator is able to generate accurate trajectories, but that is not the case for out of distribution trajectories. I understand that exploration precisely minimizes that effect, but the probabilistic model should be able to capture the lack of information in out of distribution data. Currently, the only uncertainty comes from the noise if I understand correctly. ”__

We assume the simulator perfectly matches the real environment for some existing setting of unknown parameters (cf. Section 3), i.e., the true environment can be modeled by some setting of the simulator. As such, our estimation problem reduces to estimating this unknown parameter, and we do not consider out-of-distribution trajectories since, if we are able to identify the true parameter, the distribution of trajectories generated by the simulator will match the one generated by the real environment. This assumption underlies a variety of applications of sim2real transfer, which assume that the simulator (with appropriately chosen parameters or distributions of parameters) can effectively model the real world, allowing for policy transfer from sim to real [1, 2].

In cases where this does not hold, training a policy in sim is often still a valuable starting point for learning in real [3]. In such settings, learning a targeted distribution of simulator parameters to train a policy for deployment and finetuning the resulting policy in real often leads to improved performance [3, 4]. While such settings violate our assumptions, our exploration procedure would nonetheless generate informative data to learn such targeted distributions of simulator parameters.

[1] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. ICRA, 2019

[2] Bingjie Tang, Michael A. Lin, Iretiayo Akinola, Ankur Handa, Gaurav S. Sukhatme, Fabio Ramos, Dieter Fox, Yashraj Narang. IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Reality. arXiv preprint arXiv:2305.1711, 2023

[3] Laura Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. ICRA, 2022

[4] Allen Z. Ren, Hongkai Dai, Benjamin Burchfiel, Anirudha Majumdar. AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer. CoRL, 2023

Reviewer comment (Reproducibility):
How did you use RANSAC for tracking?

Author response:
__”How did you use RANSAC for tracking?”__
We use pyRANSAC-3D [1], an open-source implementation for fitting primitive shapes like spheres and cuboids to a pointcloud. The method decomposes a cuboid into 3 plane equations and estimates those by 1) sampling points, 2) fitting a model, and 3) evaluating the model under all available points (c.f. RANSAC). This process is repeated until a threshold criteria or a maximum number of iterations is met. From the plane equations, we can extract the position and orientation of the rod.

[1] https://github.com/leomariga/pyRANSAC-3D/

Reviewer comment (Theory):
The work mentions that it assumes the optimal policy can be found. That is a rather big assumption for RL as finding the optimum is not guaranteed. Another aspect is that often the reward function does not truly represent what we want to optimize for. It is unclear whether the proposed approach actually needs to find the optimum or if a "good enough" policy is also acceptable.

Author response:
__”The work mentions that it assumes the optimal policy can be found. That is a rather big assumption for RL as finding the optimum is not guaranteed and the other aspect is that often the reward function does not truly represent what we want to optimize for. Does the proposed approach actually need to find the optimum or is a "good enough" policy also acceptable?”__

We do not rely on finding an optimal exploration strategy. To showcase this, we evaluate different training checkpoints of our policy training in [Appendix Fig. 10d]. The results show that even a “good enough” policy that has not converged to the optimal policy is acceptable for running system identification as most of them perform a reasonable exploration strategy i.e., pushing the sphere that is affected by the parameter variations.

Reviewer comment (Evaluation):
What is the runtime of the entire system?

Author response:
__What is the runtime of the entire system?__

Training the exploration policy takes 2-3h on a 12th Gen Intel Core i9-12900K and one NVIDIA GeForce RTX 3090ti. SysID takes 10-20min and downstream task training task 10-20min (CEM and PPO with early stopping upon achieving 100% success on the training env) on a 12th Gen Intel(R) Core(TM) i7-12700KF and one NVIDIA GeForce RTX 3070 Ti. Note that all stages use multiple simulations in parallel and scale with the number of CPUs.

Reviewer comment (Experiments):
How hard is it to come up with a simulation for the first goal?

Author response:
__How hard is it to come up with a simulation for the first goal?__

Can you please clarify which environment you’re referring to?

Reviewer comment (Theory):
There are simplifying assumptions made for the exploration Fisher loss. How limiting are they?

Author response:
__”There are simplifying assumptions made for the exploration Fisher loss, how limiting are they?”__

The simplifying assumptions on the Fisher information loss are not majorly limiting under standard assumptions on the dynamics. In particular, if dynamics take the form $s_{h+1} = f(s_h,a_h)$, as is the case in the setting we consider, then it is often reasonable to assume that there is some small Gaussian perturbation of the state (this is a common assumption, for example, in much of the control theory literature, e.g. canonical settings such as LQG). Given this, our simplified expression for the Fisher Information in Section 4.1 is in fact not a simplification but is exact. In our experiments, we did not observe that using this approximation resulted in a performance loss, even when training a simulator that does not precisely exhibit Gaussian noise.

Reviewer comment (Theory):
Equation 4 states that an initial distribution of parameters is assumed. How is this obtained?

Author response:
__Equation 4 states that an initial distribution of parameters is assumed, how is this obtained?__

We use the initial parameters settings provided by MuJoCo as a starting point and adjust the mean and standard deviation such that it produces a stable simulation with reasonable-looking/realistic behavior. We find that the initial parameter settings do not matter too much for parameters such as mass, friction, and center of mass, and our method performs well across a range of initial parameter distributions.

Reviewer comment (Theory):
The text states that the system isn't using a differentiable physics engine. If one was used, what would this mean for the method?

Author response:
__“The text states that the system isn't using a differentiable physics engine. If one was used, what would this mean for the method?”__

Access to a differentiable physics engine allows one to compute the Fisher information directly. This would make our reward estimate less noisy and stabilize the training of the exploration policy.

Reviewer comment (Theory):
How many parameters can be estimated and what happens when parameters are coupled or jointly multi-modal?

Author response:
__How many parameters can be estimated and what happens when parameters are coupled or jointly multi-modal?__

Our exploration procedure naturally handles coupled parameters. Note that the Fisher Information encodes the interaction between parameters, and how jointly varying parameters affect the distribution. Thus, minimizing the inverse trace of the Fisher Information matrix naturally incentivizes exploration that jointly learns coupled parameters.

Estimating the parameters from a rollout trajectory then depends on the used SysID method and the number of compute budgets available for sampling the simulator. When parameters are coupled or jointly multi-modal black-box optimization methods do not converge to a Dirac delta distribution but instead indicate uncertainty by returning a distribution over parameters. In this case, the downstream task training stage of our method could be augmented by domain randomizing over the returned parameter distribution.