From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper6216/Reviewer_SKVh
title: Re: Reply to Reviewer SKVh 
comment: I thank the authors for the feedback. I have read it together with the other reviews. After discussion, I agree with the other reviewers that the results shown in the paper appear to be somehow non-convincing, due to the possible presence of confounders in the experiments. Furthermore, the theoretical result shown in the main paper remains an incremental contribution, obtained by adaptations of existing results. For these reasons, I keep my borderline evaluation.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Ziniu Li
comment: Thank you for your interest in our work! We get valuable insights from the various works you recommended.

The estimation error in our paper is the difference between the true dynamics and MLE dynamics by the Bellman Operator (refer to our response A2 to Reviewer f8Vp), and this is equivalent to the model estimation error in [3]. Therefore, even if the transition function is deterministic, an estimation error due to insufficient dataset coverage can occurs.
In order to show how discount factor controls the estimation error, we express the second term of Theorem 1 in more detail as follows.


$
\frac{\hat{\gamma}\epsilon_P}{1-\hat{\gamma}}=\frac{\hat{\gamma}}{1-\hat{\gamma}}(E_{d^\pi_{\widehat{P},\hat{\gamma}}} [D_{TV}(\widehat{P}||P)] + E_{d^E_{\widehat{P},\hat{\gamma}}} [D_{TV}(\widehat{P}||P)])
$
$
=\frac{\hat{\gamma}}{1-\hat{\gamma}}(1-\hat{\gamma})\sum_{t=0}^\infty\hat{\gamma}^tD_{TV}(\widehat{P}(s_{t+1}\vert s_t,a_t)\Vert P(s_{t+1}\vert s_t,a_t)\vert s_0\sim p_0(\cdot),a_t\sim\pi(\cdot\vert s_t),s_{t+1}\sim\widehat{P}(s_{t+1}\vert s_t,a_t))
$
$
+\frac{\hat{\gamma}}{1-\hat{\gamma}}(1-\hat{\gamma})\sum_{t=0}^\infty\hat{\gamma}^tD_{TV}(\widehat{P}(s_{t+1}\vert s_t,a_t)\Vert P(s_{t+1}\vert s_t,a_t)\vert s_0\sim p_0(\cdot),a_t\sim\pi_E(\cdot\vert s_t),s_{t+1}\sim\widehat{P}(s_{t+1}\vert s_t,a_t))
$
$
=\sum_{t=1}^\infty\hat{\gamma}^tD_{TV}(\widehat{P}(s_{t+1}\vert s_t,a_t)\Vert P(s_{t+1}\vert s_t,a_t)\vert s_0\sim p_0(\cdot),a_t\sim\pi(\cdot\vert s_t),s_{t+1}\sim\widehat{P}(s_{t+1}\vert s_t,a_t))
$
$
+\sum_{t=1}^\infty\hat{\gamma}^tD_{TV}(\widehat{P}(s_{t+1}\vert s_t,a_t)\Vert P(s_{t+1}\vert s_t,a_t)\vert s_0\sim p_0(\cdot),a_t\sim\pi_E(\cdot\vert s_t),s_{t+1}\sim\widehat{P}(s_{t+1}\vert s_t,a_t))
$


Since the second term is expressed as a discounted sum of error, the lower the gamma, the lower the sum of error, which can be interpreted as the effect of reducing the effective planning horizon. Thus, even if error $D_{TV}(\widehat{P}\Vert P)$ is $\hat{\gamma}$-independent and data-dependent, by reducing the effective horizon, it is possible to reduce the discounted sum of errors reflected in the value function.

As you mentioned, our paper only tested in a deterministic Mujoco locomotion task. So we are planning to do a test with stochastic transition function as one of the future work.

From: ~Ziniu_Li1
title: Interesting Work on Offline Imitation
comment: Dear authors,

Thanks for your exciting work, which provides new insights from the perspective of controlling the discount factor. The theoretical analysis in Theorem 1 is really interesting, which suggests that there is a trade-off regarding the estimation error and approximation error. I would like to share several related works, which, together with your work, may provide a better picture of offline imitation. 

[1] studied the pure offline imitation problem and showed that BC is minimax optimal with the only expert dataset. [1] may suggest that BC is still good when the additional suboptimal dataset is limited. Meanwhile, [2] showed that the state-action distribution matching methods (with a small or large discount factor) could be reduced to BC in this scenario, and therefore such methods may work well without tuning the discount factor.

[3] studied the same offline imitation problem with your work and showed that with a pessimistic penalty design, state-action distribution matching methods could perform well. [3] may suggest that the pessimistic penalty can also address the ``estimation error" issue (see explanation below). 

Finally, may I ask what does the estimation error mean for MuJoco locomotion tasks considered in this paper? For me, a lower discount factor cannot minimize the estimation error here, because, for such tasks, the transition function is deterministic, and there is no randomness (except for initial states). Prior works [4, 5] considered the traditional RL problems and argued that a lower discount factor could reduce the variance by controlling the range of value functions (for stochastic transition tasks) to mitigate the estimation error issue. For the mentioned work [3], the ``estimation error" actually amounts to the dataset coverage for MuJoCo locomotion tasks. That is, if some expert state-action pairs are not well covered by the supplementary suboptimal dataset, there is a large epistemic uncertainty (even for deterministic transition tasks) in implementing the state-action distribution matching. This scheme seems data-dependent and $\gamma$-independent. I am sorry that I am a little unclear about the counterpart in this paper. I would appreciate it a lot if you could explain this point.

---

[1] Rajaraman, N., Yang, L., Jiao, J., & Ramchandran, K. (2020). Toward the fundamental limits of imitation learning. *Advances in Neural Information Processing Systems*, *33*, 2914-2924.

[2] Li, Z., Xu, T., Yu, Y., & Luo, Z. Q. (2022). Rethinking ValueDice: Does It Really Improve Performance?. *arXiv preprint arXiv:2202.02468*.

[3] Chang, Jonathan, et al. "Mitigating Covariate Shift in Imitation Learning via Offline Data With Partial Coverage." *Advances in Neural Information Processing Systems* 34 (2021): 965-979.

[4] Petrik, Marek, and Bruno Scherrer. "Biasing approximate dynamic programming with a lower discount factor." *Advances in neural information processing systems* 21 (2008).

[5] Jiang, Nan, et al. "The dependence of effective planning horizon on model accuracy." *Proceedings of the 2015 international conference on autonomous agents and multiagent systems*. 2015.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer f8Vp (1/3)
comment: Dear Reviewer f8Vp, 

We appreciate your constructive feedback and would like to address the question you asked.
For more clear explanation, we have posted a figure of the additional experiments on following link.

https://anonymous.4open.science/r/oil-iclr2023-F298/For_review/reviewer_f8Vp.md

**Q1: The proposed optimization objective and methodology is not novel and has been considered in several previous approaches. The only novelty is the study of the role of the “planning horizon” (discount factor) on algorithm performance. Given the multiple confounding factors, it would have been useful to perhaps consider other optimization routines as well. Several offline imitation learning approaches have been developed recently, for example [3]*. Some of the relevant citations are also missing.** (Note that reference [3]* is what you suggested in official review)

A1: The purpose of this paper is to solve the pathology that was occurring in the distribution matching problem in imitation learning. To clarify the purpose, we presented a process of deriving an optimization objective similar to the previous works, and we used it to show where the pathology occurred exactly and what erroneous objectives we have been optimizing.

Therefore, the main scope of our paper is to present a solution to pathology rather than optimization routine. In this paper, we present a way to adjust the effective horizon that has been overlooked in the previous works by applying a simple method. As a result, we confirmed through experiments that our method shows better and stable performance than DemoDICE without an explicit regularizer. Therefore, our work's strength lies in these areas.

We appreciate your recommendation about other references [3]\*. But, the problem setting of [3]\* is different from our paper. In the paper [3]*, the point that the discriminator is trained about uniform data distribution and used as an intrinsic reward is the same as our paper. However, the policy update objective is Q indicator-weighted behavioral cloning for uniform data distribution, which is not affected by discounted visitation distribution. Therefore, the paper [3]\* does not correspond to the scope covered by our paper.



**Q2: I am not sure that the derived performance bound is completely relevant to the proposed approach. This is a standard optimality gap derivation, seen in many model-based approaches. However, it usually refers to model-based policy optimization, hence the relevance of the dynamics model justification. In the model-free offline training setting, there really is no dynamic distribution shift, since all data available for training is from the true MDP.**

A2: In offline RL, it is well-known that the fixed point value function of the approximate bellman operator (based on offline dataset) in the Model-free RL is the same as the true value function of the MLE MDP model estimated from the offline dataset, e.g. [1].
Similarily, although this paper studies a model-free approach, the approximate transition dynamics (based on offline dataset) is reflected in the distribution matching objective through the Bellman flow constraint, and the fixed point performance can be analyzed by analyzing policy performance optimized under MLE MDP model instead.

**Q3: The main reason for studying the planning horizon (discount factor) effect is the tradeoff between policy training mismatch and dynamical shift, which is hard to justify in model-free methods, hence this makes the motivation dubious. Moreover the given performance gap really holds for any offline RL method and sa far as I’m aware the issue of planning horizon length has not been extensively considered in the much wider offline RL (including model-based RL) setting, It is not clear to me why this is more relevant to the imitation learning setting than regular offline RL setting.**

A3: As we argued above, we believe that the tradeoff exists even in model free methods. It can be also seen in previous studies on offline RL that demonstrated the effect of changing the discount factor [2]. Up to our knowledge, however, research on these points has not been conducted in the scope of IL. Furthermore, we believe that the idea of controlling the discount factor is also suited to offline IL, to extend the discourse about behavior cloning (no lookahead) and visitation distribution matching (infinite step lookahead) and to provide a better tradeoff.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer f8Vp (2/3)
comment: **Q4: The main issue is that degradation in performance is attributed to mismatch between $d^E(s,a)$ (which is the theoretical goal) and the empirical distribution $E(s,a)$. However virtually any adversarial imitation method samples from $E(s,a)$, why is this issue more pronounced in the offline cases. It would be helpful to see experiments with established online imitation learning methods that demonstrate similar degradation when varying discount factors.**

A4: Thank you for your insightful feedback. We conducted an experiment with online IL algorithms varying the discount factor, as you suggested. The results with DAC [3] algorithm is shown in link we posted (Figure C-1). The results show that those online IL algorithms are not significantly affected by lowering $\gamma$.

This difference comes from the different policy optimization processes of offline (DICE-based) IL algorithms and online IL algorithms. The weighted behavior cloning of offline (DICE-based) IL algorithms optimizes the policy that directly minimizes the visitation distribution matching objective, and suffers from the issue we showed in the paper. On the other hand, most of the online IL algorithms, e.g. GAIL [4] and DAC, obtain a reward function out of discriminator and run RL algorithms to optimize the policy. Note that we do not use the discounted visitation distribution when computing the policy gradient although we have to according to the derivation of policy gradients: strictly speaking, GAIL and DAC do not minimize the visitation distribution matching objective exactly, and it makes those algorithm free from the issue proposed in this paper (with a cost of unstable learning due to changing reward function).

**Q5: It is not clear why this issue is so pronounced here, such that performance of some of these algorithms will completely collapse. In fact Figure 1. shows good performance for DemoDICE with different values of the discount parameter before collapsing. This seems to be more indicative with issues of offline policy optimization, rather than density ratio issues, since that is pre-trained and can be fixed.**

A5: As we mentioned above, depending on the policy optimization algorithm we use, we may become free from the problem addressed in the paper (and the performance collapse in Figure 1). However, different policy optimization algorithms have their own strengths and weaknesses. We believe that offline DICE-based IL algorithms are one promising direction since it does not require joint training of discriminator and policy, which greatly stabilizes overall learning as changing reward functions can make RL algorithms very unstable. The pathology found in the paper and the solution proposed may be only applied to these DICE family of offline IL algorithms, but we believe that it is worth investigating the problem rather than just applying other completely different algorithm that works.

**Q6: The experiments don’t seem to support some of the claims of the paper. The authors use standard MuJoCo benchmarks, HalfCheetah, Walker, Hopper and Ant. I believe more convincing experiments should have significant differences between these distributions, for example long-range robot manipulation environments, such that the robot arm executes significantly different actions in different parts of the state space throughout hundreds of steps in the demonstration.**

A6: We appreciate your constructive advice for the experiment. For evaluation in a non-stationary environment, we conducted an experiment in the pen-v0 task of the Adroit domain. We created three suboptimal datasets with D1 / D2 / D3 configurations, which is consisted of expert and cloned data, and used them to measure the learning curve for 3-seed (Figure in link we posted (Figure C-2)). As a result, even in the Adroit domain, IGI shows robust learning performance against the discount factor compared to DemoDICE - similar to what we reported with MuJoCo locomotion tasks.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer f8Vp (3/3)
comment: **Q7: The experiments use a mix of expert and random data for the offline dataset from D4RL. This contains a significant amount of expert-level data, that is often enough to solve the tasks with simple BC. Experiments would be more convincing if ran on the full set of D4RL environments.**

A7: Results of this experiment are shown in the link we posted (Figure C-3). Figure shows the evaluations of learning 1 million steps according to the discount factor when the suboptimal dataset is composed of 850 medium trajectories, and the gray line represents DemoDICE's best performance. As a result, in an environment other than Hopper, it was possible to obtain a higher return than DemoDICE with $\gamma=0.99$, showing similar results to what shown in the paper. We will try to provide more empirical results as suggested.

[1] Lee, Byungjun, et al. "Batch reinforcement learning with hyperparameter gradients." International Conference on Machine Learning. PMLR, 2020.

[2] Hu, Hao, et al. "On the Role of Discount Factor in Offline Reinforcement Learning." arXiv preprint arXiv:2206.03383 (2022).

[3] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-actor-critic: Addressing sample inefﬁciency and reward bias in adversarial imitation learning. In International Conference on Learning Representations (ICLR), 2019.

[4] Ho, Jonathan, and Stefano Ermon. "Generative adversarial imitation learning." Advances in neural information processing systems 29 (2016).

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer a6oc - (1/2) 
comment: Dear Reviewer a6oc,

We really appreciate your constructive review. To address your concerns about our paper, we conducted an additional experiments.

The Figure of experiment results were uploaded to anonymous github.  

https://anonymous.4open.science/r/oil-iclr2023-F298/For_review/reviewer_a6oc.md

**Q1 : Theorem 1 provides a potential reason for why we may want to lower gamma, or at least there should be an optimal tradeoff. For the second term, is it guaranteed that if we decrease $\hat{\gamma}$, it will always decrease? It seems $\epsilon_P$ also depends on 
$\hat{\gamma}$ and may increase as $\hat{\gamma}$ decreases. This may deserves a more rigorous analysis and may also relates to later empirical observations.** 

A1 : In Theorem 1, 
$
\epsilon_P=E_{d^\pi_{\widehat{P},\hat{\gamma}}} [D_{TV}(\widehat{P}||P)] + E_{d^E_{\widehat{P},\hat{\gamma}}} [D_{TV}(\widehat{P}||P)] = \epsilon_{P,1}+\epsilon_{P,2} 
$

To check the second term's proportionality to $\hat{\gamma}$, we anlayze $E_{d^\pi_{\widehat{P},\hat{\gamma}}}\left[D_{TV}(\widehat{P}\Vert P)\right]$ first. It is expectation of Total variation distance between MLE dynamics and true dynamics over $\hat{\gamma}$-discounted state action distribution $d^\pi_{\widehat{P},\hat{\gamma}}$. Therefore, we can infer that the lower the $\hat{\gamma}$, the greater the expectation will reflect the initial timestep sample. If Total Variation distance is higher in initial timestep sample, then lowering $\hat{\gamma}$ can increase $\epsilon_{P,1}$. Considering these points, it cannot be guaranteed that $\epsilon_{P,1}$ is always proportional to $\hat{\gamma}$. However, the important point is actually whether the second term of Theorem 1, $\frac{\hat{\gamma}\epsilon_{P,1}}{1-\hat{\gamma}}$, is proportional to $\hat{\gamma}$. 

For analysis, we consider a worst situation where we have the maximum $D_{TV}(\widehat{P}\Vert P)$ only in timestep=0. Then,
$\epsilon_{P,1}$ is 

$
\epsilon_{P,1}=(1-\hat{\gamma})\sum_{t=0}^\infty\hat{\gamma}^tD_{TV}(\widehat{P}(s_{t+1}\vert s_t,a_t)\Vert P(s_{t+1}\vert s_t,a_t))\;\vert s_0\sim p_0(\cdot),\,a_t\sim\pi(\cdot\vert s_t),\,s_{t+1}\sim\widehat{P}(s_{t+1}\vert s_t,a_t)
$

$
=(1-\hat{\gamma})\sum_{t=0}^0 \hat{\gamma}^t \vert D_{TV}(\widehat{P}\Vert P)\vert_\infty = (1-\hat{\gamma})\vert D_{TV}(\widehat{P}\Vert P)\vert_\infty.
$

In this situation, $\epsilon_{P,1}$ is
inversely proportional to discount factor. However, by multiplying
$\frac{\hat{\gamma}}{1-\hat{\gamma}}$ to $\epsilon_{P,1}$, we can see
that $\frac{\hat{\gamma}\epsilon_{P,1}}{1-\hat{\gamma}}$ is equal to
$\hat{\gamma}\vert D_{TV}(\widehat{P}\Vert P)\vert_\infty$, which is
propotional to $\hat{\gamma}$. As for
$\frac{\hat{\gamma}\epsilon_{P,2}}{1-\hat{\gamma}}$, it can be inferred
that the same conclusion is reached. As a result, we can say that unless
$\widehat{P} = P$, $\frac{\hat{\gamma}\epsilon_P}{1-\hat{\gamma}}$ has
at least first-order proportionality to $\hat{\gamma}$.\
\
To verify that the conclusion is actually valid, we measured
$\frac{\hat{\gamma}\epsilon_P}{1-\hat{\gamma}}$ according to discount
factor and dataset configuration in the finite-descrete environment. We
compose the dataset configuration by increasing suboptimal trajectories.
Figure B-1 exactly shows the second term is exponentially proportional
to discount factor and inversely proportional to dataset size.

**Q2 : Implementing IGI on the expert data can be limited. Despite the better performance on the tested settings, will implementing IGI on the suboptimal dataset increase the bias according to the theory? For example, would it still work if the suboptimal dataset is not a mix of expert and random dataset, but only consists of some medium level data such as the replay buffer?** 

A2 : Following your suggestion, we conducted additional experiments to see the bias of IGI on non-expert data. We used D2-configuration (100/800, refer to paper for details), and applied the IGI only to the random-policy data within the suboptimal dataset.

Looking at the test results in Figure B-2, the result was similar to the previous one at $\gamma=0.99$, but the performance worsens as we lower the gamma. Since the gap between (IGI based on optimal dataset) v.s. (IGI based on random dataset) increases as gamma decreases, because they are similar in early timesteps, it can be understood as an effect of bias of IGI on suboptimal dataset. The two attached graphs in Figure B-3 show the timestep distribution of IGI when $\gamma$ is 0.99 and 0.8, indicating that the timestep=0 probability is 0.1 when $\gamma=0.99$ and 0.005 when $\gamma=0.8$.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer a6oc - (2/2)
comment: **Q3 : so what's the point of using a small gamma and trying to make it work?
Also what about gamma=0.999, does the difference between IGI and prior
works become even smaller?** 

A3 : We have conducted additional experiments to show the advantage of changing the discount factor more clearly. Please refer to the general response to see the details. Especially, we believe that the additional finite random MDP experiments clearly show the advantage of lowered discount factor.

On top of that, we conducted an experiment in HalfCheetah when $\gamma=0.999$ as you suggested. In Figure B-4, while the difference between IGI and prior works become even smaller since the discounted state visitation becomes very similar to the dataset distribution, $\gamma=0.999$ increases the effect of sampling error (of offline dataset), and the performance is worse than smaller $\gamma$ throughout all algorithms/datasets.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer SKVh
comment: Dear Reviewer SKVh,

Thank you for your thoughtful review! 
First of all, we uploaded the general response to all, please refer to it to give an answer about why one would choose to lower the discount factor. 

**Q1: It is not very clear why one would choose to lower the discount factor while having online trajectories from an expert that tries to maximize the undiscounted return. The problem definition is not very clear.**

A1: The experts that try to maximize the undiscounted return can be considered as an optimal policy where $\gamma_{eval}$ is close to 1. Theorem 1 suggests that although $\gamma_{eval}$ is close to 1, we can get better performance in offline IL by lowering $\gamma$. We have also provided more empirical evidence on lowered $\gamma$ giving better performance, so please refer to the general response and additional experiment results.

**Q2: The experiments are only done with 3 random seeds**

A2: We are currently experimenting with two more random seeds and will revise the experiment results in the paper.

**Q3: The algorithm definition is not clear**

A3: Including Appendix D.1 (ALGORITHM DETAILS), we modified our overall paper to be more clear to read. See our response to your Clarity and Quality concerns.

**Q4: The theoretical result is not particularly novel (see Novelty)**

A4: Thank you for pointing out the missing references. We added both citations accordingly.

**Q5 - Clarity and Quality: In my opinion, the most interesting parts are in the appendices C and D. In appendix C, the derivation of the IGI is clear, but its practical usage in the algorithm (Appendix D) is not at all. Specifically, it seems that three time series are sampled (first three lines of the while loop), then the results are used in the three following lines to sample some sequences of state/ state-action-state data from the dataset but it is not clear how all this is used in computing the various loss functions. The algorithm description is way too synthetic.**

A5: We explicitly specified where the IGI is used in Appendix D.1 (ALGORITHM DETAILS) to provide more clear usage. We also added a description of where to use each sequence of state/state-action-state data from the dataset among various loss functions. Moreover, we added supplementary explanations of the algorithm details to help to understand. See Appendix D in our paper. 

**Q6 - Clarity and Quality: As minor comment, I have note that some strange facts that are present don’t have an explanation: in Figure 1 it seems that the learning curves for DemoDICE gets worse with more training steps. Do you have a reason for this? The same things happen in the experiments of the Appendix E for many baselines.**

A6: The reason why DemoDICE gets worse with more training steps is because of the problem in the policy extraction stage due to the error distribution matching objective. DemoDICE optimizes Eq. (16) of our paper, and $\pi^*(a|s)$ will be learned differently from the $\pi_E(a|s)$ that was originally supposed to be matched at the end. However, in the early stage of learning, the importance weight being optimized can be a good indicator of weighting the experiences, and may result in a good performance temporarily (e.g. initial uniform importance weight leads to behavior cloning, and in early stages importance weights will assign high weights to state-actions that are similar to expert’s).

**Q7 - Reproducibility: About reproducibility, it seems to me that the section D.2 IMPLEMENTATION DETAIL misses details about the choice of the random seeds (this is not a major criticism, by the way)**

A7: We added our choice of random seeds in Section D.2 IMPLEMENTATION DETAIL to reproduce our works. We also uploaded our code at Supplementary Materials.

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: General Response
comment: We leave the general response to the questions that are commonly mentioned. We understand the concerns of all reviewers and are fully aware of the question of why we should use a low discount factor that is repeatedly mentioned. In this regard, we made additional experiments to clearly show the advantages of using lower discount factor instead of higher one. We found that under the experiment setting we used to report Figure 5 (refer to below link), the size of the dataset was large enough to make $\epsilon_P$ in the second term of Theorem 1 negligible, and the effect of controlling the discount factor is not clearly visible. Therefore, we made additional experiments by lowering the amount of dataset about 10 times less (40 expert trajectories and 80 random-policy trajectories) than previous dataset settings and experimented in the MuJoCo continuous control environments over 3 random seeds. As shown in the attached figure, we confirmed that the optimal discount factor appeared at lower than 0.99, except for the HalfCheetah-v2 environment. Evaluation of DemoDICE[1] shown in the figure is the best setting among different discount factors for each environment (Note the best discount factor of DemoDICE is 0.99 for all environments). 

To explain the reason why a different trend came out only in HalfCheetah environment, unlike the other environments, the HalfCheetah environment has the peculiarity that learning is stable in the offline RL problem, even when the policy is unregularized. For example, refer to the experiment result by varying the policy regularization factor in [2] (Figure 9), HalfCheetah environment shows good performance compared to other environments even when a low regularization factor $\alpha$ is applied. Combining these experimental results with the regularization effect on the discount factor in the offline RL referred to [3] (Section 3), we can expect that the performance of the learned policy can be improved when a high discount factor is used in the HalfCheetah environment. Eventually, the reason why the attached figure shows a different trend only in the HalfCheetah environment is interpreted as reflecting this environmental disposition, and the error bound that we proposed in Theorem 1 is valid under general situations. 

Additionally, we made supplementary experiments in finite and discrete MDP, and we added its result and detailed explanations to Appendix E.1. As a result, we showed that by controlling the effective planning horizon using IGI, it is possible to obtain a policy with better performance using a low discount factor, which was not available in previous works.

https://anonymous.4open.science/r/oil-iclr2023-F298/For_review/Figure_5_prev.md

[1] Kim, Geon-Hyeong, et al. "DemoDICE: Offline imitation learning with supplementary imperfect demonstrations." International Conference on Learning Representations. 2021.

[2] Wu, Yifan, George Tucker, and Ofir Nachum. "Behavior regularized offline reinforcement learning." arXiv preprint arXiv:1911.11361 (2019).

[3] Hu, Hao, et al. "On the Role of Discount Factor in Offline Reinforcement Learning." arXiv preprint arXiv:2206.03383 (2022).

From: ICLR.cc/2023/Conference/Paper6216/Authors
title: Reply to Reviewer DiqU
comment: Dear Reviewer DiqU,

Thank you for your valuable feedback. We will address your comments about weakness and questions.

For smooth explanation, we have posted a figure for the additional experiment on anonymous github. 

https://anonymous.4open.science/r/oil-iclr2023-F298/For_review/reviewer_DiqU.md

**Q1: Why is it necessary to change the discount factor then? Theorem 1 suggests there might be a better value to use than the evaluation value, but this claim is not well supported.** 

A1 : We have conducted additional experiments to show the advantage of changing the discount factor more clearly. Please refer to the general response to see the details. Especially, we believe that the additional finite random MDP experiments clearly show the advantage of lowered discount factor.

**Q2: it is unclear how one would choose a discount value to use in practice.** 

A1 : Similar to choosing regularization hyperparameters in other offline RL algorithms, choosing a discount factor is not straightforward when the evaluation on the true environment is not available. What makes it even harder here is that we cannot directly compare the value estimations based on different discount factors.

In this sense, choosing appropriate discount factors is an interesting future research direction. One way to do this is to run an off-policy evaluation of obtained policies based on the given offline dataset and compare their lower confidence bound on performances. However, it will require us to run a separate evaluation step after running the imitation learning on various discount factors. To avoid this, we need to investigate further on how we can translate a value estimator learned in different discount factors, or how we can decide an approximate performance based on uncertainty on policies learned in different discount factors.

**Q3: In the discussion about Theorem 1, the authors claim the error of the imitated policy mainly depends on the first two terms of (14). It feels hand-wavy to ignore the 3rd term about the policy difference. Have the authors measured the relative magnitude of those terms to support the claim?** 

A3 : To check the magnitude of terms in Theorem 1, we measured the magnitude of the second and third terms in finite-discrete MDP where $\epsilon_P$ and $\epsilon_\pi$ can be exactly computed. Please refer to Appendix E.1 in our paper for detailed explanation of this experiment settings.

The attached Figure A-1 shows the magnitude of the second and third terms averaged over 100 random seeds with different discount factors and the number of suboptimal trajectories (num_traj) for each optimality setting. The figures show the trends of two terms with different discount factors at each setting. All the measured results show that the third term has a much lower value than the second term. In this sense, it can be seen that the scale of the third term is basically lower than the second term, and when the imitation is properly done, it becomes completely ignorable. This supports our claim that 'the error of the imitated policy will be mainly dependent on the first two terms of Theorem 1' in the paper. The result at lower discount factors than in the figure follows the trend of observations.

**Q4: I did not follow the transition to equation (15) under the assumption that the reward is represented with an optimal discriminator. Can the authors provide more details?** 

A4 : We clarified the derivation to reduce confusion.
To follow the transition to equation (15) it is better to start from equation (10) rather than equation (9), and we revised the paper accordingly. In equation (10), the discriminator is used to approximate $\log\frac{d^E(s,a)}{D(s,a)}$ in $A_\nu(s,a)$, where the optimal discriminator $c^*(s,a)$ satisfies $\log\frac{d^E(s,a)}{D(s,a)} = \log\left(\frac{c^*(s,a)}{1-c^*(s,a)}\right)$. However, as we describe at Section 4.1 in the paper, empirical distribution $E(s,a)$ is used to train the discriminator, not the discounted visitation distribution $d^E(s,a)$. That is, the discriminator is trained to distinguish $D(s,a)$ and $E(s,a)$, which becomes to approximate $\log\frac{E(s,a)}{D(s,a)}$ instead of $\log\frac{d^E(s,a)}{D(s,a)}$. Applying this discrepancy to objective (10) and some algebra will result in equation (15).

From: ICLR.cc/2023/Conference/Paper6216/Reviewer_DiqU
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the authors consider the problem of offline imitation learning via state-action visitation matching. The main contribution is the discovery that the planning horizon (controlled with the discount factor) needs not to match during training to the value in evaluation. The authors show that existing algorithms suffer performance degradation when different discount factors are used in training. They propose a novel initial state sampling method to enable robust performance across different choices in discount factors.
strength_and_weaknesses: Strength:
The insight (Theorem 1) into how the training discount factor impacts the error bound for the imitated policy is quite interesting. This aspect is not studied in prior works so this is novel in the literature about offline imitation learning with state-action visitation matching.

The empirical evaluations support the claim that the IGI (inverse Geometric Initial) state sampling algorithm enables stable learning across different discount factor values. 

Weakness:
I have a question about the motivation behind creating a stable learning algorithm for different discount factor values. The empirical evaluations show that baseline vision matching methods, using the same discount factor in training as in evaluation, perform comparably in the experiments. Why is it necessary to change the discount factor then? Theorem 1 suggests there might be a better value to use than the evaluation value, but this claim is not well supported. The results in Figure 5 are inconclusive and it is unclear how one would choose a discount value to use in practice. Can the authors comment on this point?

clarity,_quality,_novelty_and_reproducibility: The paper is mostly easy to read. I have some clarification questions.

1. In the discussion about Theorem 1, the authors claim the error of the imitated policy mainly depends on the first two terms of (14). It feels hand-wavy to ignore the 3rd term about the policy difference. Have the authors measured the relative magnitude of those terms to support the claim?

2. I did not follow the transition to equation (15) under the assumption that the reward is represented with an optimal discriminator. Can the authors provide more details?


summary_of_the_review: Overall, this paper presents a solid investigation of the new problem concerning discount factor value in offline imitation learning. Although I have some questions about the motivation behind this investigation, I hope the authors’ responses can address them.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6216/Reviewer_a6oc
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In offline imitation learning, a common approach is to minimize the discrepancy between state-action visitation distributions, which involves the problem of estimating transition dynamics. This becomes difficult if the planning horizon is long and the compounding error along the trajectory becomes large. Therefore, the authors propose to reduce the discount factor such that the effective planning horizon is shortened, which was supported by a theoretical error bound. However, as observed by prior works, naively reducing the discount factor only makes performance worse. The paper then propose a method called IGI to alleviate this issue by changing the initial state distributions to correct the bias caused by a smaller gamma. Empirical results are provided to justify the method.
strength_and_weaknesses: Strength
- well-written and easy to follow, with clear method derivations and reasonable solutions to the identified problem
- empirical results demonstrates the effectiveness of IGI
- Good discussions of related work

Weaknesses/Questions
- Theorem 1 provides a potential reason for why we may want to lower gamma, or at least there should be an optimal tradeoff. For the second term, is it guaranteed that if we decrease $\hat{\gamma}$, it will always decrease? It seems $\epsilon_P$ also depends on $\hat{\gamma}$ and may increase as $\hat{\gamma}$ decreases. This may deserves a more rigorous analysis and may also relates to later empirical observations.
- Implementing IGI on the expert data can be limited. Despite the better performance on the tested settings, will implementing IGI on the suboptimal dataset increase the bias according to the theory? For example, would it still work if the suboptimal dataset is not a mix of expert and random dataset, but only consists of some medium level data such as the replay buffer?
- The flow of the paper is: because state-action visitation matching involves matching dynamics, which lead to compounding error when the planning horizon is long. thus we want to reduce gamma, but smaller gamma increase bias and leads to worse performance. After fixing the issue with IGI, we can now work with smaller gamma. But according to the experimental results (in the main paper and appendix), it seems the main advantage is the proposed method becomes more robust and has less performance degradation when reducing gamma. And it is not the case that we can get better performance by using IGI in conjunction with a smaller gamma than using a larger gamma (i.e. even with IGI, the performance still does not get much better when using smaller gamma as the theorem 1 predicts), so what's the point of using a small gamma and trying to make it work? Also what about gamma=0.999, does the difference between IGI and prior works become even smaller?
clarity,_quality,_novelty_and_reproducibility: The clarity is good. The solution to the issue is reasonable and the work should be reproducible. For weaknesses/comments, please see above.
summary_of_the_review: Interesting paper with good structure, reasonable idea and solid derivation. The empirical results may not well support the original motivation that we need to use a smaller discounted factor to shorten the planning horizon for a significantly better performance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6216/Reviewer_f8Vp
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper considers the issue of offline imitation learning - in this setting we are provided with a handful of expert demonstrations and an additional fixed datasets of environment interactions.
 
1. The authors first derive an imitation learning objective in the offline setting, which is largely identical to prior works [1], [2]
2. They derive a performance optimality bound that depends on the discount factors used for training the offline policy and the expert and the MDP dynamics that are used for training.

The key observation is that the discount factor used for training the offline policy can be used to control a trade-off between expert policy mismatch and underperformance induced by model error, with shorter effective planning horizons reducing the latter. However, using a different discount factor for training affects the divergence objective used for policy training, since the discounted expert state-action distribution $d^E(s, a)$, deviates more from the empirical distribution $E(s,a)$, from which we can sample. The authors propose a new Inverse “Geometric Initial state sampling” (IGI) method to alleviate the issue and show robust performance independent of discount factors. 

[1] Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021

[2] Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433, 2022

strength_and_weaknesses: Strengths:

The authors provide robust and detailed theoretical justification for studying the proposed problem and a formulaic solution, which in experiments show robust and stable performance.



Weaknesses:

1. The proposed optimization objective and methodology is not novel and has been considered in several previous approaches. The only novelty is the study of the role of the “planning horizon” (discount factor) on algorithm perormance.  Given the multiple confounding factors, it would have been useful to perhaps consider other optimization routines as well. Several offline imitation learning approaches have been developed recently, for example [3]. Some of the relevant citations are also missing.
2. I am not sure that the derived performance bound is completely relevant to the proposed approach. This is a standard optimality gap derivation, seen in many model-based approaches. However, it usually refers to model-based policy optimization, hence the relevance of the dynamics model justification. In the model-free offline training setting, there really is no dynamic distribution shift, since all data available for training is from the true MDP. 
3. The main reason for studying the planning horizon (discount factor) effect is the tradeoff between policy training mismatch and dynamical shift, which is hard to justify in model-free methods, hence this makes the motivation dubious. Moreover the given performance gap really holds for any offline RL method and sa far as I’m aware the issue of planning horizon length has not been extensively considered in the much wider offline RL (including model-based RL) setting, It is not clear to me why this is more relevant to the imitation learning setting than regular offline RL setting. 
4. The main issue is that degradation in performance is attributed to mismatch between $d^E(s, a)$ (which is the theoretical goal) and the empirical distribution $E(s, a)$. However virtually any adversarial imitation method samples from  $E(s, a)$, why is this issue more pronounced in the offline cases. It would be helpful to see experiments with established online imitation learning methods that demonstrate similar degradation when varying discount factors. 
5. Several works and extensive evaluation in the online learning case have shown that exact distribution matching is not necessary for good performance and in fact objectives which are theoretically wrong often perform much better e.g. simple combinations of discriminator learning and off-policy algorithms [4]. Simple algorithms, such as ORIL [3] (which theoretically has a wrong objective) seem to perform quite well in the offline setting as well. It is not clear why this issue is so pronounced here, such that performance of some of these algorithms will completely collapse. In fact Figure 1. shows good performance for DemoDICE with different values of the discount parameter before collapsing. This seems to be more indicative with issues of offline policy optimization, rather than density ratio issues, since that is pre-trained and can be fixed. 
6. The experiments don’t seem to support some of the claims of the paper. The authors use standard MuJoCo benchmarks, HalfCheetah, Walker, Hopper and Ant. However these environments’ distributions are quite stationary as the agent performs short sequences of similar actions over and over. Here after a small initialization period the state-action distribution of the expert becomes quite stationary and thus it seems there would be very limited mismatch between $d^E(s, a)$ and $E(s, a)$. I believe more convincing experiments should have significant differences between these distributions, for example long-range robot manipulation environments, such that the robot arm executes significantly different actions in different parts of the state space throughout hundreds of steps in the demonstration.  
7. The experiments use a mix of expert and random data for the offline dataset from D4RL. This contains a significant amount of expert-level data, that is often enough to solve the tasks with simple BC. Experiments would be more convincing if ran on the full set of D4RL environments. 



[3] Offline Learning from Demonstrations and Unlabeled Experience
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, Scott Reed , Offline RL Workshop, NeurIPS 2020

[4] What Matters for Adversarial Imitation Learning? Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, Marcin Andrychowicz

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow with clean derivations and references. The novelty component relates mostly to the study of the discount factors used for offline imitation learning. The authors provide sufficient details for reproducibility, but no implementations or the particular datasets used (derived from D4RL).

summary_of_the_review: The authors study the effect of discount factors on offline imitation learning. The paper provides theoretical justification and practical formulaic solutions to data sampling, which in experiments show significant improvements in stability. However, I am not convinced the derived theory is exactly relevant to the approach, moreover the experiments seem to refute some of the claims of the paper, rather than support them. More empirical evaluation would be necessary to approve the paper, including alternative policy optimization objectives, non-stationary environments with longer horizons and online evaluation of some of the proposed claims. 

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper6216/Reviewer_SKVh
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper shows empirically that current algorithms for imitation learning in an offline setting suffer heavily from low learning rate and proposes a technique to correct them by changing the initial state distribution through a new technique called IGI (inverse geometric initial state sampling).
To this end, one error bound is provided to show a trade-off between learning with a lower discount factor and a higher one, where the first option increases train-test discrepancy, and the second increases the risk of compounding estimation error. Moreover, some experiments in the most popular OpenAI gym/ Mujoco environments show that the proposed method outperforms the baselines in state of the art (Behavioral Cloning, SMODICE and DemoDICE) when the discount factor is <0.99
strength_and_weaknesses: **Strengths**:
- The idea of changing the initial state distribution is nice and original
- The experiments involve many environments and many distributions of the offline datasets and show an impressive improvement over the state of the art for low discount factor
- The theoretical result is sound

**Weaknesses**:
- It is not very clear why one would choose to lower the discount factor while having online trajectories from an expert that tries to maximize the undiscounted return. The problem definition is not very clear.
- The experiments are only done with 3 random seeds
- The algorithm definition is not clear
- The theoretical result is not particularly novel (see Novelty)
clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality**
The overall paper is not very clear. The section 3.1 DERIVATION OF AN OFFLINE IL OBJECTIVE seems redundant and hard to follow. In my opinion, the most interesting parts are in the appendices C and D. In appendix C, the derivation of the IGI is clear, but its practical usage in the algorithm (Appendix D) is not at all. Specifically, it seems that three time series are sampled (first three lines of the while loop), then the results are used in the three following lines to sample some sequences of state/ state-action-state data from the dataset but it is not clear how all this is used in computing the various loss functions. The algorithm description is way too synthetic. As minor comment, I have note that some strange facts that are present don’t have an explanation: in Figure 1 it seems that the learning curves for DemoDICE gets worse with more training steps. Do you have a reason for this? The same things happen in the experiments of the Appendix E for many baselines.
The quality of figures is excellent.

**Novelty**
The idea behind IGI is interesting, the fact of changing the initial state distribution in order to avoid wasting samples in the training phase when the discount factor is low is very charming. This is indeed a very strong point of the paper. Unfortunately, the theoretical contribution (theorem 1) is not completely novel: it is in fact the union of three lemmas, where the first is already present in [1] (Theorem 2) and the third one is known from [2] (Appendix A.2 Proof of Theorem 1). This fact is not necessarely a problem, since the results are presented in a different light, but this fact has to be enlightened.

[1] Petrik, Marek, and Bruno Scherrer. "Biasing approximate dynamic programming with a lower discount factor." Advances in neural information processing systems 21 (2008).

[2] Xu, Tian, Ziniu Li, and Yang Yu. "Error bounds of imitating policies and environments." Advances in Neural Information Processing Systems 33 (2020): 15737-15749.

**Reproducibility**
About reproducibility, it seems to me that the section D.2 IMPLEMENTATION DETAIL misses details about the choice of the random seeds (this is not a major criticism, by the way).
summary_of_the_review: The paper presents interesting ideas that could have potential in an area of great interest such as that of the offline imitation learning. Unfortunately, I have concerns about the formulation of the problem. It is not very clear why the authors want to make the agent work with a low discount factor while the expert maximizes the total return and the state-of-the-art algorithms works with gamma>=0.99. Anyway, the idea of IGI is excellent, the algorithm implementing this procedure should be clarified and put at the center of the attention. The theoretical contribution is correct and interesting but not particularly novel, but this is not a crucial issue. The whole paper is, in my opinion, structured in a non-optimal way, with the most important parts relegated to the appendix, but the contribution is overall valid.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: None.
recommendation: 6: marginally above the acceptance threshold