Paper: CCIL: Continuity-Based Data Augmentation for Corrective Imitation Learning (ICLR-2024)

Reviewer comment (Novelty):
While the writing was clear and easy to understand, the paper lacked substantial content. I didn't find any need to pause and think while reading and I skimmed through the paper rather quickly.

Author response:
> “While the writing was clear and easy to understand, the paper lacked substantial content”

Our proposal stems from a key observation - that the dynamics of robotic systems abides by physics laws and contains local continuities. While certainly simple, the literature in imitation learning has not investigated this form of structure for data augmentation and for alleviating the compounding errors. As seen from our empirical results, this data augmentation method is principled and makes a substantial difference in performance.

Reviewer comment (Experiments):
There are many Offline IL algorithms proposed recently in literature that have the same settings where they don't make any new interaction with the environment or the expert. Comparisons with them would be interesting to see.

Author response:
> “many Offline IL algorithms proposed recently in literature”

Our proposal is meant to only leverage the given expert demonstration data, without access to reward or large amounts of offline data. Addressing your request, we just added [MILO](https://arxiv.org/abs/2106.03207) and [MoREL](https://arxiv.org/abs/2005.05951) for comparison. We would be happy to take a look if you could point us to other Offline IL algorithms with minimal assumptions.

Reviewer comment (Experiments):
I would recommend the authors to also report results on Humanoid environment from Mujoco.

Author response:
> “report results on the Humanoid environment”

All our MuJoCo experiments used the expert data from the D4RL dataset which, unfortunately, did not provide demonstrations for the Humanoid. To keep the evaluations consistent and use an open-source dataset for MuJoCo, we did not include results on the Humanoid environment.

Reviewer comment (Theory):
The proposed method is formulated in a discrete-time state transition model, whereas the corresponding true system operates in continuous-time. Therefore, the proposed method implicitly applies a time discretization. In this case, the time interval is critical, and I think the Lipschitz constant depends on the time interval. How did the authors determine an appropriate Lipschitz constant? Or, are there any assumptions on the time discretization in the proposed method?

Author response:
Indeed the Lipschitz constant of the dynamics will depend on the time interval. Without assuming any prior knowledge about the system dynamics, we use empirical experiments to choose the desired local Lipschitz continuity for each task domain, as documented in Appendix D.3. Essentially, we tried constraining the Lipschitz continuity per each neural network layer to use different continuities (L=  2 / 3 / 5) and inspect the resulting training errors. Since the theoretical bound of our generated labels depends on both L and the training errors $\epsilon$ of the learned dynamics model (e.g., <= 0.001 · L + (1 + L)ϵ ), we choose the hyperparameter to yield the best theoretical bound.

Reviewer comment (Novelty):
I do not fully understand the major differences between the techniques of the proposed method and Data as Demonstrator (DaD) proposed by Venkatraman et al. (2015)? I think the core idea is similar; therefore, it is worth discussing the advantages of the proposed method.

Author response:
Similar to DaD, our method aims to generate “corrective” labels. However, our method is fundamentally different to DaD in (1) the assumptions and (2) the problem we are trying to solve. 

(1) We cited DaD in Section 2 paragraph 1 and explained that this method leveraged a form of **invariance** for data augmentation. Our method relies on the presence of local continuity instead. Our assumption is less strong and is applicable beyond position controlled domains. 

(2) DaD is a data augmentation method for training a dynamic model but not for policy. NoiseBC extends DaD to train a policy. It also leverages a form of known invariance (e.g., position control), injects noise disturbance and reuses the action label. Hence we include NoiseBC as a baseline, essentially reusing the ideas from DaD. We showed that our method could outperform NoiseBC in most domains we considered.

Reviewer comment (Experiments):
Two augmentation techniques are proposed, but I am unsure whether either would be equally useful. Is it possible to conduct an additional ablation study where one of the techniques is removed?

Author response:
In our empirical evaluations we found that different tasks might benefit more from different techniques. For simplicity, in all our experiments, we used one of the techniques proposed. For example, on MuJoCo and MetaWorld tasks, NoisyAction technique outperforms Backward technique on half of the tasks. The benefits of using which technique to generate labels depends on the ground truth system dynamics and the learned dynamics function. We attach an additional ablation comparing the effectiveness of two techniques on Metaworld.

**Metaworld**
| | | | | | 
|--------------------------------------|:------------------------:|:------------------------:|:------------------------:|:------------------------:|
|         | **CoffeePull** |**ButtonPress** | **CoffeePush**|**DrawerClose**
| NoisyAction | **4168.46 $\pm$192.98** | 3758.89 $\pm$ 67.39  |**2484.19 $\pm$ 976.03** |  3988.67 $\pm$ 168.27
| Backward | 3954.12$\pm$322.67 | **3775.22 $\pm$ 91.24** |1584.45$\pm$935.01 |**4145.45 $\pm$ 76.23**
| VanillaBC | 3552.59 $\pm$233.41 | **3693.02 $\pm$ 104.99** |1288.19 $\pm$ 746.37 | 2809.56 $\pm$ 439.70

Reviewer comment (Writing):
The first paragraph on page 5: $\hat{f}(s_t, a_t) \to s_{t+1}^* - s_t^*$ should be $\hat{f}(s_t^*, a_t^*) \to s_{t+1}^* - s_t^*$.

Author response:
Thank you for pointing out the typo. We will correct it in the updated manuscript.

Reviewer comment (Writing):
Is $\bar{\lambda}$ in Equation (3) is an average of $\{ \lambda_j \}_j$?

Author response:
$\bar{\lambda} is a chosen hyper-parameter.

Reviewer comment (Writing):
Please define $f'$ in Equation (3) explicitly.

Author response:
We apologize for the typo and confusion and will clarify the equation in the updated manuscript. In EQ3, we first estimates the local Lipschitz continuity around the expert data $(s_j^*, a_j^*)$ for the learned dynamic model $\hat{f}$ using a sampled noise $\delta_s$, such that  $\hat{f’}(s_j^*+\delta_s, a_j^*) \approx  \frac{ \hat{f}(s_j^*+\delta_s, a_j^*) - \hat{f}(s_j^*, a_j^*) } {|| \delta_s ||}$. We then record if the estimated Lipschitz continuity violates the given hyperparameter $L$ that enforces Lipschitz continuity:  $Lipschitz(s_j^*, a_j^*) = E_{\delta_s \sim \mathcal{N} } ReLU(\hat{f’}(s_j^*+\delta_s, a_j^*) - L) = E_{\delta_s \sim \mathcal{N} }  max(\hat{f’}(s_j^*+\delta_s, a_j^*) - L, 0)$

Reviewer comment (Writing):
Regarding the technique 1 (Backtrack label), what does "xlabel" mean?

Author response:
Thank you for correcting our typo. "xlabel" should be "label".

Reviewer comment (Experiments):
The proposed method is only compared to vanilla BC and noisy BC. The proposed method declares it constructs a dynamics model for policy learning and has used implementation with a model-based RL framework; therefore, it would be more robust to also include a comparison with other model-based RL methods. As model-based RL also constructs a dynamic model first before planning the most effective actions.

Author response:
Per your request, we compared our method against two additional baselines: an imitation learning method [MILO](https://arxiv.org/abs/2106.03207) and model-based offline reinforcement learning method [MoREL](https://arxiv.org/abs/2005.05951). 

MILO is a state-of-the-art IL method which also learns a dynamic model from data and uses the learned model to learn from demonstrations. However, MILO usually demands access to a large dataset of transitions (in order to train a good dynamic model). In our experiments, we train MILO with only the given expert data. Similarly, model-based Offline RL requests additional access to a reward function as an assumption. 

Our results indicate that our method, CCIL, consistently outperforms these baselines in various tasks:
- In the car domain, CCIL outperformed all other baselines.
- In the drone domain (given the limited time, we used a subset of data for training), CCIL outperformed all other baselines.
- In MuJoCo tasks, CCIL was the leading method on 3 out of 4 tasks.
- In MetaWorld tasks, CCIL was competitive, tying with other baselines in 3 out of 4 tasks and slightly underperforming only in one task against MILO.

Reviewer comment (Reproducibility):
There is a lack of clarity in important implementation details. The process of generating corrective labels is discussed in Section 4 but the paper does not make it clear how these labels are employed in later stages. The additional corrective labels could be used to train the imitation learning agent, presumably a neural network? However, if this is the case, further details on the network's implementation could be discussed.

Author response:
We augment the original training dataset with the newly generated labels. We then train imitation learning agents using these augmented data by feeding the data into a policy function. Our policy function predicts actions from input states. It is modeled by a neural network with 2 layers of 128 neurons each, using ReLU activation. Our implementation uses the BC algorithm from the open-source library [d3rlpy](https://takuseno.github.io/d3rlpy/).

Reviewer comment (Experiments):
The reason why noise BC underperforms compared to vanilla BC is not clear. If we reduce the noise added to the BC, noiseBC's performance should align more closely with that of vanilla BC. Nonetheless, in multiple tasks, noiseBC exhibits significantly poorer results. This could potentially be attributed to the fact that the added noise has not been not carefully chosen and thus, an excessive amount of noise has been injected into the system?

Author response:
There are two main reasons why Noise BC underperforms compared to vanilla BC. First, certain domains may not be well-suited for NoiseBC. For instance, in drone navigation tasks using RPM control and in MuJoCo locomotion tasks that use torque control, reusing the action label after state perturbation may not be effective. Second, as you correctly pointed out, the level of noise introduced is crucial. Inappropriate noise levels can significantly impact performance.

In our experiments, we experimented with the size of the noise (variance = 0.001, 0.0005, 0.0001). Our experiments currently applied a small noise level for NoiseBC across all tasks (noise variance = 0.0001). However, recognizing your concern, we experimented with a smaller noise level (variance = 0.00005) in two MuJoCo tasks (Hopper and Walker) where NoiseBC lagged behind vanilla BC. We observed that Noise BC's performance is indeed sensitive to the size of the noise. Reducing the noise slightly improved NoiseBC's performance, but it still did not reach the level of Vanilla BC. This suggests that the application of NoiseBC requires careful calibration of the noise level.

These observations highlight the limitations of NoiseBC, further emphasizing the need for our proposed method.

| | | | 
|--------------------------------------|:------------------------:|:------------------------:|
|          | **Hopper** | **Walker**
| NoiseBC (original, var=0.0001) | 1563.56 $\pm$1012.02 | 2893.21 $\pm$ 1076.89  
| NoiseBC (new, var=0.00005) | 2300.27 $\pm$823.85 | 2906.48 $\pm$ 326.78 
| VanillaBC | 2902.78 $\pm$689.64 | 3810.63 $\pm$ 828.23