Paper: Offline Reward Inference on Graph: A New Thinking (ICLR-2024)

Reviewer comment (Reproducibility):
The paper doesn't present complete details of the experiment setup. While $f_\Theta$ is an important component for constructing the graph weights, throughout the paper no details on the setup of $f_\Theta$. The paper also doesn't reveal any details about the policy formulation or any parameters related with the training process.

Author response:
For most environments, the network architecture of the function $f_\Theta$ is a two-layer fully connected network with ReLU activation, and the number of hidden layer neurons is 64.

The network structures for policy training are as follows:

For Meta-World environments, it is a two-layer fully connected network with ReLU activation and the number of hidden layer neurons is 64. 

For DeepMind Control Suite environments, it is a three-layer neural network with ReLU activation and the number of hidden layer neurons is 64.

Reviewer comment (Experiments):
While the paper does sufficient comparison over several existing methods, one additional thing that might be worth presenting is comparison with different reward inference methods. For example, one might want compare the paper's method with reward inference simply by KNN.

Author response:
As shown in Table 3 of Appendix A.5, our data has annotations for only a small fraction (mostly 10%-15%) of the environments. Therefore, for deep learning fully supervised methods, we found them to be ineffective in our testing. We believe that the KNN-based approach is a promising idea and may yield good results for certain problems. However, applying the KNN method in this scenario requires complementary measurement methods and an appropriate way of propagating rewards. We will give special consideration to the suggestions you have provided in our future work.

Reviewer comment (Reproducibility):
One potential limit of the method is the size of the weight matrix grows with the number of data points. In this case, it could be very costly to compute the matrix inverse or other related things. Can the author give some comments on the computation cost?

Author response:
Thanks for the insightful question. For this question, we choose to evaluate the computational costs of our method in several experimental environments, including training time and memory consumption. We also include the results as a section (A.7.1) in the paper's Appendix.

Most of our datasets consist of $1 \times 10^6$ state-action pairs. For computational convenience, we segment the dataset. We conducted tests on a computer with the following specifications: CPU: Intel i9-9900KF 3.6GHz, GPU: RTX 2070 SUPER (8GB VRAM). The training time and memory consumption for several environments are presented below:

Dataset slices containing 10000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer | Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:------:|:-----:|:----------:|
|    Training time    |   12.68s    |   12.67s    | 2.67s  | 2.7s  |   2.62s    |
| Memory  consumption |   2351MB    |   2351MB    | 2377MB | 2377M |   2377MB   |

Dataset slices containing 20000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer | Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:------:|:-----:|:----------:|
|    Training time    |    14.8s    |    13.7s    | 13.07s | 13.9s |   13.52s   |
| Memory  consumption |   5869MB    |   5869MB    | 5886MB | 5886M |   5886MB   |

We also conducted tests on a server with the following specifications for larger dataset slices: CPU: Intel Xeon Gold 6230 2.10GHz, GPU: RTX 3090 (24GB VRAM).

Dataset slices containing 30000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer  |  Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:-------:|:------:|:----------:|
|    Training time    |   37.61s    |   38.07s    | 23.12s  | 24.34s |   26.72s   |
| Memory  consumption |   11511MB   |   11511MB   | 11583MB | 11583M |  11583MB   |

Dataset slices containing 40000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer |  Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:------:|:------:|:----------:|
|    Training time    |    39.1s    |   40.09s    | 28.34s | 29.21s |   29.86s   |
| Memory  consumption |   19577MB   |   19577MB   | 19691MB | 19691MB  |   19691MB   |

Reviewer comment (Experiments):
An alternative way to infer the reward is to simply use the known part of the reward, i.e., $R_U = W_{UL}R_L$. What's the advantage of the proposed method, compared with this simpler formulation?

Author response:
Thanks for this insightful question. We have carefully considered it and modified the corresponding section in the paper (Section 4.3). Additionally, we have included the detailed derivation process in the Appendix (A.2).

The inference of rewards for unlabelled nodes requires considering the information transfer between labelled and unlabelled nodes, we use $W_{UL}R_L$ for this calculation while also taking into account the relationships between unlabeled nodes themselves, computed using $W_{UU}R_U$. 
Since unknown rewards $R_U$ are a variable to be learned, we provide an iterative computation formula by:
\begin{equation}
R_U \longleftarrow W_{UU}R_U + W_{UL}R_L.
\end{equation}

Reviewer comment (Evaluation):
The empirical evaluation relies on only 5 seeds. Moreover, the presentation of the results does not consider statistical significance.

Author response:
1) Reinforcement Learning is based on the idea of the reward hypothesis, all goals can be described by the maximization of the expected cumulative reward.
In this setting, environments exhibit randomness, and different initializations of these environments result in varying learned policies, which affects the std value of Meta-World tasks.
For statistical convenience, the average of returns in those tasks presents an intuitive observation of the policies' performance. It is convinced that the std values that assist in showing the stability of the policies in random environments are additional statistics.

2) In the reinforcement learning community, adopting 5 seeds is a feasible setting, such as the recently publications[1][2][3][4][5][6].

3) In our experimental setup, our computational conditions include a server with the following specifications for larger dataset slices: CPU: Intel Xeon Gold 6230 2.10GHz, GPU: RTX 3090 (24GB VRAM). Conducting experiments with 10 seeds would take approximately 4 to 6 months, which is a substantial time investment. 

4) Based on your suggestion, we have highlighted the statistically significant highest average returns for each task with bold numbers, in which the average return is the highest, and there is not a clear overlap for the standard deviations among the baselines. For the other results, we only mark the highest average returns for each task with italic numbers, which avoids confusion caused by intersecting standard deviations. We have modified Table 1 and Table 2 in the draft, including updating the captions for both tables.

[1] Confidence-Conditioned Value Functions for Offline Reinforcement Learning. Joey Hong, et al. ICLR 2023.

[2] In-sample Actor Critic for Offline Reinforcement Learning. Hongchang Zhang, et al. ICLR 2023.

[3] Supported Trust Region Optimization for Offline Reinforcement Learning. Yixiu Mao, et al.
ICML 2023.

[4] Policy Regularization with Dataset Constraint for Offline Reinforcement Learning. Yuhang Ran, et al. ICML 2023.

[5] One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning. Marc Rigter, et al. NeurIPS 2023.

[6] Efficient Diffusion Policies for Offline Reinforcement Learning. Bingyi Kang, et al. NeurIPS 2023.

Reviewer comment (Writing):
The TGR and UDS are presented too succinctly. If the reader is not familiar with these methods it is impossible to understand how they relate.

Author response:
Thanks for pointing out this crucial issue. We have provided further details in the description of TGR and UDS.

Reviewer comment (Experiments):
The paper proposes to "learn a reward propagation graph and infer rewards", yet a quick google search on "reward propagation" reveals a paper from 2020 that proposes a similar method to infer rewards on nodes with missing reward labels. Although their method is for the online RL setting, it seems straightforward to use it in the offline RL setting. This would be an important baseline.

Author response:
$\Phi_{GCN}$ is an outstanding and impactful work. We have cited the mentioned article in our paper, and the citation has been updated in the PDF. Upon this baseline, we conducted experimental validations on selected environments. In the experiments, we utilized the same data annotation ratio as our approach for $\Phi_{GCN}$. We performed five random seed experiments for each environment, calculating the average and standard deviation of the results. The experimental results are presented in the following table:

|  Env  |  Cheetah Run   |    Walker Walk    |     Fish Swim     | Cartpole Swingup |
|:-----:|:--------------:|:-----------------:|:-----------------:|:----------------:|
| TRAIN | 430 $\pm$ 55.9 | 952.7 $\pm$ 155.2 | 567.6 $\pm$ 155.2 | 642.4 $\pm$ 68.9 |
|  $\Phi_{GCN}$  | 274 $\pm$ 71.1 |  634 $\pm$ 172.3  |  181 $\pm$ 93.2   | 396 $\pm$ 132.1  |


|  Env  |  Coffee Button   |    Push Wall     |     Hammer      |   Open Drawer   |   Window-open    |
|:-----:|:----------------:|:----------------:|:---------------:|:---------------:|:----------------:|
| TRAIN | 4128 $\pm$ 210.9 | 4501 $\pm$ 204.6 | 4532 $\pm$ 59.1 | 4466 $\pm$ 41.3 | 3829 $\pm$ 208.4 |
|  $\Phi_{GCN}$  | 2845 $\pm$ 303.8 | 3238 $\pm$ 192.1 | 2933 $\pm$ 81.8 | 2989 $\pm$ 79.3 | 2921 $\pm$ 234.1 |

$\Phi_{GCN}$ considers information derived from the state and performs well in environments where the state contributes significantly to the reward. For the environments where the action substantially impacts the reward, the performance of $\Phi_{GCN}$ is slightly inferior.

Reviewer comment (Experiments):
The empirical evaluation is not adequate from a scientific rigour point-of-view. Only 5 seeds are used and Table 1 highlights the score of methods for which the standard deviation intersects with other methods. This is misleading; only non-intersecting standard deviations should be highlighted. Perhaps additional seeds could help in this.

Author response:
1) Reinforcement Learning is based on the idea of the reward hypothesis, all goals can be described by the maximization of the expected cumulative reward.
In this setting, environments exhibit randomness, and different initializations of these environments result in varying learned policies, which affects the std value of Meta-World tasks.
For statistical convenience, the average of returns in those tasks presents an intuitive observation of the policies' performance. It is convinced that the std values that assist in showing the stability of the policies in random environments are additional statistics.

2) In the reinforcement learning community, adopting 5 seeds is a feasible setting, such as the recently publications[1][2][3][4][5][6].

3) In our experimental setup, our computational conditions include a server with the following specifications for larger dataset slices: CPU: Intel Xeon Gold 6230 2.10GHz, GPU: RTX 3090 (24GB VRAM). Conducting experiments with 10 seeds would take approximately 4 to 6 months, which is a substantial time investment. 

4) Based on your suggestion, we have highlighted the statistically significant highest average returns for each task with bold numbers, in which the average return is the highest, and there is not a clear overlap for the standard deviations among the baselines. For the other results, we only mark the highest average returns for each task with italic numbers, which avoids confusion caused by intersecting standard deviations. We have modified Table 1 and Table 2 in the draft, including updating the captions for both tables.

[1] Confidence-Conditioned Value Functions for Offline Reinforcement Learning. Joey Hong, et al. ICLR 2023.

[2] In-sample Actor Critic for Offline Reinforcement Learning. Hongchang Zhang, et al. ICLR 2023.

[3] Supported Trust Region Optimization for Offline Reinforcement Learning. Yixiu Mao, et al.
ICML 2023.

[4] Policy Regularization with Dataset Constraint for Offline Reinforcement Learning. Yuhang Ran, et al. ICML 2023.

[5] One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning. Marc Rigter, et al. NeurIPS 2023.

[6] Efficient Diffusion Policies for Offline Reinforcement Learning. Bingyi Kang, et al. NeurIPS 2023.

Reviewer comment (Experiments):
Very little work is done to understand the impact of different components in the method.

Author response:
The information in our ablation study in Appendix A.2.1 could address your concern. We evaluated the performance of methods employing four different input modes. We endeavored to utilize deep learning (supervised learning) for reward learning, but deep learning requires substantial amounts of annotated data, and our dataset comprises at most 10%-20% annotated data. This quantity is inadequate for deep learning, making the approach based on deep learning (supervised learning) practically ineffective.

Reviewer comment (Experiments):
In the paper, the reward shaping function $f_{\theta}$ is trained using annotated data to convert state-action pairs into scalar values, and this information is used to calculate the weights between nodes in the graph. Since $f_{\theta}$ must learn the relative importance of each dimension in the node data, it raises the question of the necessary diversity within the annotated data. For instance, if all the annotated data consists of expert transitions with the highest rewards, it might be challenging for the function to discern which dimensions of the node data contribute most significantly to changes in reward. Hence, an important question to address is: "What level of diversity within the annotated data is required for the proposed method to learn an effective function $f_{\theta$?"

Author response:
Thanks for the very valuable and profound question. The commonly used practice in existing work involves datasets generated by training classic reinforcement learning algorithms (such as SAC) policies online[1][2]. During the online training process, the policy evolves from weak to strong, interacting with the environment and generating diverse data. The question raised by the reviewer is very valuable; it represents a specific case but is indeed a scenario that could exist. We ensure the diversity of our dataset in two ways. Firstly, our dataset also adopts the training process data of classic reinforcement learning algorithms (such as SAC). Secondly, we use uniform sampling on the dataset to label state-action pairs, ensuring the uniform presence of annotated state-action pairs with rewards in various "regions" of the dataset. In our future work, we will consider how to address the specific case raised by the reviewer.

[1] How to Leverage Unlabeled Data in Offline Reinforcement Learning. Yu, Tianhe, et al. ICML 2022.

[2] Conservative Data Sharing for Multi-task Offline Reinforcement Learning. Yu, Tianhe, et al. NeurIPS 2021.

Reviewer comment (Theory):
Is the reward calculated by the graph method bounded by the range of rewards in the annotated data, i.e., $\max{(R_U)}\leq \max{(R_L)}$? This question seeks to understand the extent to which the graph-based reward inference can maintain the upper bound of reward values, as derived from the annotated data.

Author response:
Thanks for the deepful and valuable question. For this question, we added a section in the paper's Appendix A.7.2 to examine the changes in the maximum value of $R_U$ and verify whether the learned reward values are bounded by the annotated data. We design an illustration to demonstrate this issue concerning the formula, which is also shown in Appendix A.7.2. We can observe that within certain ranges of $(I-W_{UU})$ and $W_{UL}$ values, the maximum value in $R_U$ can exceed the maximum value in $R_L$, that is the learned rewards are not bounded by the annotated data.

Reviewer comment (Experiments):
To make the proposed method practically feasible, it's essential to investigate the computational cost requirements. Specifically, showing the computational time and memory consumption required for reward inference helps provide insights into the practical applicability and efficiency of the method in real-world scenarios.

Author response:
Thanks for the insightful question. For this question, we choose to evaluate the computational costs of our method in several experimental environments, including both training time and memory consumption. We also include the results as a section (A.7.1) in the Appendix of the paper.

The majority of our datasets comprise $1 \times 10^6$ state-action pairs. To facilitate computation, we segment the dataset. We conducted tests on a computer with the following specifications: CPU: Intel i9-9900KF 3.6GHz, GPU: RTX 2070s (8GB VRAM). The training time and memory consumption for several environments are presented below:

Dataset slices containing 10000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer | Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:------:|:-----:|:----------:|
|    Training time    |   12.68s    |   12.67s    | 2.67s  | 2.7s  |   2.62s    |
| Memory  consumption |   2351MB    |   2351MB    | 2377MB | 2377M |   2377MB   |

Dataset slices containing 20000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer | Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:------:|:-----:|:----------:|
|    Training time    |    14.8s    |    13.7s    | 13.07s | 13.9s |   13.52s   |
| Memory  consumption |   5869MB    |   5869MB    | 5886MB | 5886M |   5886MB   |

We also conducted tests on a server with the following specifications for larger dataset slices: CPU: Intel Xeon Gold 6230 2.10GHz, GPU: RTX 3090 (24GB VRAM).

Dataset slices containing 30000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer  |  Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:-------:|:------:|:----------:|
|    Training time    |   37.61s    |   38.07s    | 23.12s  | 24.34s |   26.72s   |
| Memory  consumption |   11511MB   |   11511MB   | 11583MB | 11583M |  11583MB   |

Dataset slices containing 40000 state-action pairs:
|         Env         | Cheetah Run | Walker Walk | Hammer |  Door  | Pick-place |
|:-------------------:|:-----------:|:-----------:|:------:|:------:|:----------:|
|    Training time    |    39.1s    |   40.09s    | 28.34s | 29.21s |   29.86s   |
| Memory  consumption |   19577MB   |   19577MB   | 19691MB | 19691MB  |   19691MB   |