From: ICLR.cc/2023/Conference/Paper6130/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper6130/Reviewer_vb2d
title: Thanks for Feedback
comment: Thanks for your feedback. Unfortunately, I tend to keep my score as it is.

From: ICLR.cc/2023/Conference/Paper6130/Authors
title: Author response
comment: Thank you for your detailed feedback.

$\textbf{[Performance in random datasets]}$

One explanation for the performance in the random dataset is that the datasets with random policy were unable to reach valuable state-action pairs. Especially in large environments such as Hopper, Walker and Halfcheetah, state-action pairs that lead to high return may not exist in the dataset. Since optimizing a policy based on visitation distribution correction restricts the policy to be in-support (as shown in OptiDICE Figure 5), the upper bound of policy performance in these datasets will be limited, regardless of the regularization we use. 

$\textbf{[The use of perspective function]}$

The intuition behind the use of perspective function was to transform the dataset distribution $d^D(s,a)$ into weighted dataset distribution $k(s,a)d^D(s,a)$. Perspective function applied weight $k(s,a)$ to $d^D(s,a)$ inside and outside of function $f$, while preserving the convexity of regularization. Perspective function alleviated the performance of DICE framework being excessively bound to the dataset, which can be observed in Figure 1.

$\textbf{[The use of weight $k$]}$

We have separated weight k with w to stabilize the learning process. 
Since the sampling error is affected by k, as shown in theorem 2, we designed weight k to slowly follow w.

$\textbf{[Ablation study: Alpha]}$

We have followed the setting of alpha proposed by OptiDICE. We added an ablation study on alpha to show our new DICE framework is more stable than the original DICE framework. The ablation study was done on three d4rl maze domains (Appendix F. Figure 7). For the finite domain, the ablation study on alpha was conducted in Figure 2.

$\textbf{[Performance improvement]}$

It is true that our performance improvement is marginal in mujoco environments such as Hopper, Walker2d and Halfcheetah. However, our algorithm achieved state-of-the-art performance in the domain where offline RL with DICE framework works better than CQL-like offline RL algorithms that use dynamic programming techniques. We add our experiment on antmaze-umaze environment to give an additional example of environment where performance improvement is evident. (Appendix F. Figure 4)

$\textbf{[Ablation study: $f$-divergence]}$

We additionally tested our algorithm over KL divergence and chi-square divergence. $f$ in $f$-divergence is used in the form of $w = \max(0,k (f’)^{-1}\left( \frac{e_\nu}{\alpha} \right))$ when obtaining the optimal ratio of state-action visitation probability between the trained policy and dataset collection policy. However, our algorithm with KL divergence $f(x) = x\log x$ diverged as $(f’)^{-1}(x)$ is the exponential function. We added chi-square divergence $f(x) = \frac{1}{2}(x-1)^2$  in the ablation study to show weighing improves the performance in other f-divergence as well (Appendix F. Figure 5).

From: ICLR.cc/2023/Conference/Paper6130/Authors
title: Author response
comment: Thank you for your detailed feedback.

$\textbf{[Related work]}$

We have positioned the related works in the latter part of the paper to provide the derivations of objectives before discussing the related works that are based on similar derivations. 

$\textbf{[Ablation study: $f$-divergence]}$

We additionally tested our algorithm over KL divergence and chi-square divergence. $f$ in $f$-divergence is used in the form of $w = \max(0,k (f’)^{-1}\left(\frac{e_\nu}{\alpha} \right))$ when obtaining the optimal ratio of state-action visitation probability between the trained policy and dataset collection policy. However, our algorithm with KL divergence $f(x) = x\log x$ diverged as $(f’)^{-1}(x)$ is the exponential function. We added chi-square divergence $f(x) = \frac{1}{2}(x-1)^2$  in the ablation study to show weighing improves the performance in other f-divergence as well (Appendix F. Figure 5).  

$\textbf{[Can the performance be further improved?]}$

We have used the same hyperparameter alpha in OptiDICE to show that weighting $f$-divergence improves the performance of OptiDICE in 13 out of 15 environments. The performance can be further improved if better alpha and bound on weight $k$ are applied. Alpha balances the degree of return maximization and distribution shift control, and bound on weight $k$ affects sampling error together with alpha.

$\textbf{[Ablation study: Weight k]}$

A single value network for k was added to our algorithm compared to the unweighted algorithm. We experienced a 2% increase in total training time since most of the run time was used in the policy extraction process. Three different bounds were tested in three d4rl maze domains. (Appendix F. Figure 6)

$\textbf{[Ablation study: Alpha]}$

We added an ablation study on alpha with the same bound on weight $k$ to show our new DICE framework is more stable than the original DICE framework. The ablation study was done on three d4rl maze domains (Appendix F. Figure 7). For the finite domain, the ablation study on alpha was conducted in Figure 2.

From: ICLR.cc/2023/Conference/Paper6130/Authors
title: Author response
comment: Thank you very much for your feedback.

$\textbf{[Citation error]}$

We have removed the wrong citation you mentioned.

$\textbf{[Theoretical contribution: weighted measurement on distribution shift]}$

Figure 1 shows that the performance of the original DICE framework is strongly bounded to the dataset even in finite MDP. Our work alleviates this by transforming the dataset distribution $d^D$ with weight $k$. Weight $k$ is jointly optimized with return maximization, to make the transformed dataset distribution $k d^D$ favorable to return maximization. To our knowledge, our work is the first work that introduced weighted measurement on distribution shift to the DICE framework.

$\textbf{[Limited empirical contribution in large environment]}$

The review argued that empirical contribution in large environments is limited. This limitation stems from one big assumption in DICE framework which assumes that state-action visitation probability is larger than 0. Although this powerful assumption enabled the new approach to offline RL, it led to lower performance in large environments where $d^D=0$ is more likely. This appears in the form of sampling error in our paper, as the bound $B_w$ on the ratio between the state-action visitation probability of the trained policy and data collection policy gets larger as $d^D$ gets smaller in large domains.

From: ICLR.cc/2023/Conference/Paper6130/Authors
title: Author response
comment: We thank the reviewer for the thoughtful feedback.

$\textbf{[Better theoretical analysis]}$

As the review has pointed out, our derivation on the bound of sampling error (Theorem 1) lacked rigor, since cardinality was applied to Lagrange multipliers, which are continuous. Therefore, we revise the derivation on the sampling error by using pseudo-dimension and covering numbers.

$\textbf{[Theoretical novelty]}$

We admit that the theoretical novelty is limited to applying weighted $f$-divergence to the existing DICE framework. However, we would like to emphasize two major improvements in DICE framework as weight is added to $f$-divergence.

1. The use of perspective function (for weighted dataset distribution)

The intuition behind the use of perspective function was to transform the dataset distribution $d^D(s,a)$ into weighted dataset distribution $k(s,a)d^D(s,a)$. Perspective function applied weight $k(s,a)$ to $d^D(s,a)$ inside and outside of function $f$, while preserving the convexity of regularization. Perspective function alleviated the performance of DICE framework being excessively bound to the dataset, which can be observed in Figure 1.
In our work, the notion of optimality was applied to the measurement of distribution shift by jointly optimizing k with return maximization. This can diversify DICE framework as other types of knowledge can be applied to the weight. 

2. Better performance and robustness

Our weighted DICE framework had better performance compared to the unweighted case in both discrete and continuous domains(13 out of 15). We added the ablation study on alpha to show weighted $f$-divergence improves the stability of DICE framework with respect to alpha. (Figure 2. shows the stability of our work in finite MDP.) Although offline RL algorithms based on dynamic programming, such as CQL, excelled in some mujoco domains, our DICE framework achieves state-of-the-art performance in domains where DICE framework works better.

From: ICLR.cc/2023/Conference/Paper6130/Reviewer_vb2d
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper considers offline reinforcement learning problem in a discounted reward Markov decision process. The interaction between agent and the environment is not allowed, and a dataset consisting of one step transitions are given to optimise policy. To mitigate the effect of distribution shift in offline RL, the authors use regularised policy optimisation, where the regularisation is given by (weighted) $f$-divergence between true and empirical state-action visitization distributions. The weights determine the level of regularisation for each state-action pair, where the important state-action (i.e., with higher values) are penalised less. The authors combine this weighted regularisation scheme in well-known DICE framework for offline reinforcement learning. They bound the difference between the optimal value and the value of the policy returned by the resulting algorithm as a function of the weights. They show that the upper bound increases as the weights increases.Then, the authors provide a practical algorithm to optimize the weights, which along with the optimized state-action vistization probabilities yield the complete algorithm. The authors then compare the performance of this wighted $f$-divergence regularisation based algorithm with prior non-weighted algorithms, and report favorable performance.
strength_and_weaknesses: Strengths:

1. The weighted regularization based approach considered in this work performs better empirically compared to prior non-weighted regularization based algorithms.

2. The paper provides a scheme to optimize the weights, which help to reduce the gap between optimal value and value of the resulting policy.


Weakness:

1. Theoretical analysis is not rigorous. In theorem 1, the error between two optmization objectives depend on cardinality of Lagrange multipliers and state-action visitation distribution. This comes from a direct application of Hoeffding Lemma. What do the cardinalities signify? I will hope for a better notion of complexity rather than cardinalities.

2. The high-level approach of this paper follows the same line of Lee et al (2021) and Zhan et al (2022). The only difference is using wighted $f$-divergence based regularization instead of the unweighted as considered in the prior work. In view of this, the theoretical novelty of this paper is limited.
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of writing is fair. Theoretical novelty is somewhat limited. Experiments are reproducible.
summary_of_the_review: The wighted $f$-divergence based regularization gives better empirical performance in offline RL compared to the unweighted $f$-divergence based ones. However, the theoretical guarantees presented in the paper mostly follow from prior work and lacks proper rigour. in view of this, I have reservations about impact of this paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6130/Reviewer_BsTD
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a variant of behavior regularization, where different state-actions are weighted differently in the f-divergence. This has the benefit of adding lesser regularization on valuable but rare state-actions. The proposed algorithm iteratively learns the optimal weights on the f-divergence under an upper bound on the size of the weights, and is shown to improve the bound upon standard behavior regularization where the weights are all equal. The authors show that tuning the weights can have benefits in practice.
strength_and_weaknesses: - Fox et al citation on page 1 is not appropriate (as that paper studies the online setting). The purpose of doing KL regularization in the online setting is very different from that in the offline setting.

I think the conceptual contribution of the paper is very straightforward. The optimization of the weights k subject to a bound on their size can be shown to capture a tradeoff between the optimization error, and the sampling error of working with finite samples.

The theoretical analysis which bounds the optimization error based on using duality largely follows from previous work.
The sampling error analysis is also somewhat straightforward and based on uniform convergence.

Overall I think the theoretical contribution is not very surprising.

The empirical contribution is good, but they are seemingly less significant as the environments get larger. The algorithm does not beat several of the compared baselines on the walker environment which I believe is the one with the highest number of degrees of freedom in those considered. This leads me to question whether the empirical result will carry over to more complicated environments, although I am willing to grant that for smaller environments there is a benefit.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The paper is only marginally novel in my opinion.
I did not check the reproducibility of the experiments.
summary_of_the_review: I think the paper's contribution is not very novel, and the experimental results are not very impressive. Based on that and the comments in the strengths/weaknesses section, I think the paper needs to be improved before it is worthy of acceptance to ICLR.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6130/Reviewer_4363
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a weighted f-divergence regularized RL method for addressing the distribution shift problem. Experimental results on D4RL are presented to demonstrate the effectiveness of the algorithm. 

strength_and_weaknesses: Strength 
- The offline RL problem is interesting. 
- The use of the simple example in Fig. 1 is helpful. 
- The intuition and idea of the algorithm are well explained. 

Weakness
- Is there a reason why the related work section comes so late? It could be moved to an earlier place so that one knows the comparison early on. 
- The experimental section is a bit limited. First of all, only the f function in (20) is considered. Second, in the experiments in Table 1, weighted-f achieves the best performance in 7 out of 15 settings. Can this be further improved? Third, how difficult/expensive to train the additional k(s,a) values, and how sensitive is the performance to their values? Fourth, there should be an ablation study to evaluate the other potential factors. 

clarity,_quality,_novelty_and_reproducibility: The writing of the paper is good. The derivation and ideas are clearly explained. The algorithm also appears to be effective. 
summary_of_the_review: The paper proposes a new algorithm for offline RL. The writing of the paper is good. The derivation and ideas are clearly explained. The algorithm also appears to be effective. However, the current experimental section could be strengthened. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6130/Reviewer_tVHY
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Offline reinforcement learning methods rely on minimizing the f-divergence between the trained policy and data collecting policy to address the distribution shift issue. However, such regularization can be overly pessimistic when valuable state-action pairs are rare and equally penalized. The paper propose to minimize the weighted f-divergence instead based on the perspective function. By deriving an upper bound for the performance gap, the paper then propose a simple objective for jointly training the weight function. Empirical results on D4RL are provided to justify the claim.
strength_and_weaknesses: Strength
- The paper is well-written and easy to follow.
- Method derivation seems sound to me.
- Nice presentation (e.g. Figure 1 to demonstrate the effect of weighted f-divergence in the classic toy environment)

Weaknesses/Questions:
- One possible reason for why we think the original framework is overly pessimistic is no matter how suboptimal the dataset is (e.g. Hopper-Random/Walker2d-Random/Halfcheetah-Random), an f-divergence minimization will always try to match the trained policy with the data collecting policy. In that sense, since the perspective function is still (perhaps strictly) convex, it seems the weighted f-divergence still tries to do the same thing. For example, we should expect the weighted f-divergence to perform much better than other methods in random datasets since we are focusing on the valuable state-action pairs now. But it does not seem to be the case.
- Can you provide more intuitive explanation on why choose the perspective function to realize the intuition that more important state-action pairs should be given more weights? From $k(s,a) f(\cdot)$, we know that a larger weight k(s,a) means more importance. But inside function $f(\frac{d(s,a)}{k(s,a)d^D(s,a)})$, the density ratio is also divided by $k$, so the correspondence with the intuition is not very straightforward.
- Why do we need equation (19) to train $k(s,a)$ if we know the optimal value of $k$. Can we just use $\omega^*_{\mu, k}$?
- How did you choose an appropriate $\alpha$? Is there an ablation study on that (is the performance sensitive or robust to it)?
- The performance improvement seems marginal based on the table and learning curve (especially on Hopper, Walker2d and Halfcheetah).
- In the experiments, have you tried other f-divergence such as KL/Jensen-Shannon since the title and framework derivation is for general f-divergence.
clarity,_quality,_novelty_and_reproducibility: Please see above for detailed comments.
summary_of_the_review: The paper propose to use weighted f-divergence to improve the OptiDICE method. The derivation mostly follows OptiDICE, but the perspective function idea can be interesting. Empirically the performance improvement doesn't seem to be significant in most of the tasks.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold