Paper: Expressive Modeling is Insufficient for Offline RL: A Tractable Inference Perspective (ICLR-2024)

Reviewer comment (Evaluation):
The improvement in scores over baselines don't seem so large. In particular, with constrained actions one might expect the TPM approach to produce more gains. However, I don't see this as a particular demerit for insightful research.

Author response:
Thank you for your constructive and encouraging feedback on our paper. We greatly appreciate your recognition of our work. Regarding the improvement in scores of Trifle over the baselines, we have explained the results in a more detailed way and added more empirical evidence, please refer to the "Response to All" part, thanks!

Reviewer comment (Evaluation):
It is not clear if the proposed approach outperforms "Bayesian Reprameterized RCRL" baseline. While I do not think it is important to beat all the baselines, it is useful to (i) understand how the two methods stack against each other (sharing standard deviation numbers for BR-RCRL will help) and (ii) the difference between the two approaches. The authors should add more details about why they think the two approaches are tied so closely.

Author response:
| Dataset    | Environment | Trifle    | BR-RCRL   |
| ---------- | ----------- | --------- | --------- |
| Med-Expert | Halfcheetah | 95.1±0.3  | 95.2±0.8  |
| Medium     | Halfcheetah | 113.0±0.4 | 112.9±0.9 |
| Med-Replay | Halfcheetah | 109.3±0.1 | 111.0±0.4 |
| Med-Expert | Hopper      | 49.5±0.2  | 48.6±1.1  |
| Medium     | Hopper      | 67.1±4.3  | 78.0±1.3  |
| Med-Replay | Hopper      | 83.1±0.8  | 82.3±1.7  |
| Med-Expert | Walker2d    | 45.0±0.3  | 42.3±3.3  |
| Medium     | Walker2d    | 97.8±0.3  | 98.3±2.6  |
| Med-Replay | Walker2d    | 88.3±3.8  | 80.6±2.5  |


1. The standard deviation numbers for all RvS baselines are shown in Table 4 of the updated paper. Here we show the standard deviatios of BR-RCRL and Trifle for camparison. In 7 out of 9 mujoco tasks, Trifle achieves a much lower standard deviation than BR-RCRL, indicating that Trifle performs more robustly. Notably, the results of BR-RCRL are averaged over only 5 random seeds while we evaluate Trifle on 12 random seeds. Moreover, towards a more reliable evaluation, we improved the number of random seeds to 20 for tasks with relatively high standard deviation.

2. The goal of BR-RCRL is also to sample actions with a higher probability of achieving high RTGs. This work differs from ours in two key aspects: (a) BR-RCRL adopts energy-based models (EBMs) to approximately sample from $p(a | s, R)$ using iterative inference; due to the intractability of EBMs, this sampling process could suffer from slow mixing time under high-dimensional action space. In contrast, Trifle leverages TPMs to exactly compute $p(a_i | s, a_{<i}, R>v)$ ($v$ is a threshold), and then uses beam search/rejection sampling to approximate samples from $p(a_i | s, a_{<i}, E[R]>v)$. Note that since we approximate the expected-return-conditioned policy iteratively over every action dimension, the rejection rate would be moderate. (b) Once the training finishes, BR-RCRL is limited to approximately sample from $p(a | s, R)$. In contrast, Trifle can answer arbitrary marginal/conditional queries over $(s,a,R)$, naturally enabling its application in more challenging scenarios like stochastic environments or constrained policy generation. For example,  in stochastic environments, we utilize Trifle to compute multi-step value $p(V_t\geq v | s_t, a_{t:t'})$ by marginalizing intermediate states; We could also apply Trifle to generate constrained policy  $p(a_t|s_t, E(V_t)\geq v, c)$, where c represents arbitrary state/action constraints.

Reviewer comment (Experiments):
The authors should consider adding results with another TPM (even if for a subset of tasks) so that it is clear that their approach works across different TPMs.

Author response:
We chose PC as the adopted TPM because it can be more easily scaled up to complex datasets thanks to recent advances. Also, PCs’ ability to efficiently compute arbitrary marginal/conditional probabilities meets the need of Trifle. Any TPM with sufficient expressiveness and support answering queries required by Trifle can be used.

Reviewer comment (Experiments):
It is not clear why the authors used the TT baseline (in place of a stronger alternative) for experiments in 6.2 and 6.3.

Author response:
### **The reason why we compare Trifle with TT.**
As mentioned above, in Sections 6.1 and 6.3, we select $p_\mathrm{TT}(a_t|s_t)$ as our prior policy $p_\mathrm{GPT}(a_t|s_t)$, so the performance gain compared with TT fairly demonstrates the significance of Trifle in improving inference-time optimality; In Section 6.2, we didn’t evaluate BR-RCRL for there is no official released code, and we consider comparison with TT and DT is sufficient for demonstrating the effectiveness of adopting TPM to handle environment stochasticity.

Reviewer comment (Novelty):
Are terms like "training time optimality" introduced by this paper? If yes, could we consider using existing terms like "expressivity"?

Author response:
Yes, "training time optimality" can be interpreted as "expressivity", i.e. how well/accurately the model can fit the offline trajectories to recover the ground truth distribution. We have modified the paper to clarify this, please refer to the second paragraph of Section 3.

Reviewer comment (Experiments):
The authors mentioned a sampling approach where "we first sample from a prior distribution p(a_t | s_t ), and then reject actions with low expected returns" (section 4.1). Did they use this in conjunction with any of the baselines (to make them stronger)?

Author response:
We implement the rejection sampling process via beam search, which is used in all applicable baselines. Specifically, Trifle leverages TPMs to exactly compute $p(a_i | s, a_{<i}, R>v)$ ($v$ is a threshold), and then uses beam search/rejection sampling to approximate samples from $p(a_i | s, a_{<i}, E[R]>v)$. Note that since we approximate the expected-return-conditioned policy iteratively over every action dimension, the rejection rate would be moderate.

Reviewer comment (Evaluation):
In table 1, could the authors report the standard deviation for all the approaches?

Author response:
The standard deviation numbers for all RvS baselines are shown in Table 4 of the updated paper. Trifle achieves relatively lower standard deviations than other baselines, indicating that Trifle performs more robustly. Notably, the results of different baselines are averaged over different numbers of random seeds. In general, we evaluate Trifle on 12 random seeds. However, towards a more reliable evaluation, we improved the number of random seeds to 20 for tasks with relatively high standard deviation. The number of evaluation seeds for BR-RCRL, DD, TT, and DT are 4, 5, 15, and 3, respectively.

Reviewer comment (Theory):
Theorem 1 feels a bit out of place because showing a problem is NP-hard brings little information on how easy it is to approximate the solution, which is what people mostly care about in practice.

Author response:
We totally agree with the reviewer that showing NP-hardness brings little information on how easy it is to approximate the solution. In fact, the main goal of Theorem 1 is to inform us that we need to resort to approximations of $p(a_t | s_t, E[V_t]>v)$.

Reviewer comment (Reproducibility):
The details of the action-space-constrained task (Sec 6.3) are quite terse. How is the constraint being conditioned (is it a boolean variable or the threshold value discussed in the caption of Table 2?) How is it being incorporated into the TPM?

Author response:
We consider action space constraints in the form of “value of the torque applied to the foot rotor ≤ A”, where A is a threshold value. We enforced the constraint on TT by masking the unsatisfied categories (each action category can be mapped to a continuous torque value according to the adopted discretization strategy, if the reconstructed value is larger than A,  we consider this category unsatisfied) of the categorical distribution $p_\mathrm{TT} (a_t | s_t)$, and sampling from the masked distribution must satisfy the constraint. As for TPM, to compute the conditional probability, PC will involve both forward and backward processes. Therefore, we tell the satisfied categories to PC’s input nodes of the constrained action variables and PC will incorporate such constraint message into its forward and backward paths.