Paper: Stylized Offline Reinforcement Learning: Extracting Diverse High-Quality Behaviors from Heterogeneous Datasets (ICLR-2024)

Reviewer comment (Theory):
There is no theoretical ground for naively replacing $A^{\mu^{(i)}}$ with $A^\mu$ without importance sampling corrections. At least an empirical ablation study should be provided if it is difficult to devise a theoretical justification.

Author response:
We conducted a comparison of performance (referred to as quality) using $A_{\mu^{(i)}}$ and $A_\mu$. We employ a soft clustering approach where each trajectory has a probability of belonging to each cluster. During the training of $A_{\mu^{(i)}}$, the dataset trajectories are weighted based on the probability obtained from the soft clustering method. In the experiment:

|  | SpaceInvaders | MsPacman | MontezumaRevenge |
| ---- | ---- | ---- | ---- |
| SORL with $A_\mu$          | $387.1$ | $622.2$ | $306.7$ |
| SORL with $A_{\mu^{(i)}}$ | $292.2$ | $551.3$ | $0.0$ |

The empirical results indicate that utilizing $A_{\mu}$ during the learning process enhances the quality of policies. The reason that $A_\mu$ generally outperforms $A_{\mu^{(i)}}$ is that it only focus on part of the dataset.

Reviewer comment (Writing):
It isn't easy to understand the proof presented in Appendix B.

Author response:
We sincerely appreciate the reviewer for pointing it out. We've refined the proof, which is now in Appendix D.

Reviewer comment (Evaluation):
The diversity metric proposed by the authors does not consider how different the styles are. For example, consider the case where $\pi^{(i)}(a=k\mid s)=\frac{1}{K}+\epsilon_k(s)$ where $K$ is the number of possible actions and $\epsilon_k(s)$ is a small number chosen arbitrarily. Then $\hat{p}(z=j\mid traj)$ would be determined by the values of $\epsilon_k(s)$, so $p_{popularity}$ would be close to a uniform distribution, which is the distribution that maximizes the entropy. However, the learned styles are far from being diverse.

Author response:
We appreciate the reviewer to point this out. We agree that one limitation of the current metric of diversity is that it relies on the clustering $\hat{p}(z=j|\tau)$. Comparing the diversity of two sets of policies are not trivial. Previous research, such as InfoGAIL, also did not propose a specific metric for evaluating diversity. Instead, they relied on visualization to showcase the diversity of learned policies.
In order to provide a more comprehensive representation of the diversity of learned policies, we have proposed additional diversity metrics, under which the described problem does not exist. We have incorporated three additional metrics into our testing: the **skill metric**, the **OT (optimal transport) metric**, and the **discrimination metric**. The skill metric measures the dissimilarity of actions, while the OT metric quantifies the disparity in state distribution. The discrimination metric assesses whether the policies can be distinguished by a neural network. The table below showcases the diversity of different style learning methods in the grid shooting environment, measured using various metrics. Higher values indicate greater diversity. For a detailed description of these metrics, please refer to the Appendix F. Based on the results of the additional diversity metrics presented in Table below, we can conclude that the SORL algorithm is capable of obtaining diverse and high-quality policies. InfoGAIL fails to learn diverse policies, while Off-RLPMM has considerably lower winning rate.

| | Popularity | Skill | OT | Discrimination | Winning rate |
| ---- | ---- | ---- | ---- | ---- | ---- |
| SORL                       | $0.60$ | $15.7$ | $29.14$ | $0.75$ | $55.1$% |
| SORL w/o advantage | $0.61$ | $21.2$ | $31.61$ | $0.97$ | $45.2$% |
| Off-RLPMM              | $0.68$ | $23.2$ | $29.87$ | $0.99$ | $40.4$% |
| InfoGAIL                    | $0.00$ | $2.0$ | $28.53$ | $0.69$ | $55.3$% |

Reviewer comment (Experiments):
§5.3 states that the character has to combat against an AI opponent. What is the strategy of the opponent? Is SORL robust to the changes in the opponent's strategy?

Author response:
The opponent is an agent with a fixed strategy that move randomly, and shoot with a probability once the shooting action cools down.

To check whether the agent is robust to the changes in the opponent's strategy, we introduce a moderately strong opponent with DQN training for 20,000 steps.

|  | Reward (shoot) | Reward (star) | Winning rate |
| ---- | ---- | ---- | ---- |
| Policy 1 | $1.1$ | $0.11$ | $19$% |[数学公式]
| Policy 2 | $0.1$ | $1.27$ | $54$% |

Despite the opponent's increased strength, the policies employed by our agent remained stylized. Consequently, the winning rate experienced a decrease compared to previous opponents, but it remained competitive. Specifically, the winning rate of policy 1, which excels at shooting enemies, declined to $19$% due to the increased difficulty of successfully shooting and winning against the stronger opponent. On the other hand, the winning rate of policy 2, which specializes in collecting stars and focuses on escaping from the enemy's shooting, was not significantly affected.

Reviewer comment (Experiments):
All experiments were conducted on environments with discrete action spaces. How does SORL perform in continuous action environments?

Author response:
Although we don't include environment with continuous action space, our algorithm can be easily adapted to the continuous action space. The adaptation process involves representing action probability generated by the policy network as a continuous distribution (often Gaussian). Our method is not limited to discrete action space.
The main contribution of SORL is to propose a novel framework that learns diverse and high-quaity qualities from offline datasets. The distinction between discrete and continuous action spaces in experiments lies outside the primary focus of this paper. We do recognize its importance for future work.

Reviewer comment (Evaluation):
How was the diversity metric measured for InfoGAIL? To my knowledge, InfoGAIL does not explicitly split the policy into multiple clusters but instead learns a multi-modal policy.

Author response:
We set the latent space of InfoGAIL as an $m$-dim one-hot vector, where $m$ is the number of styles. For example, when $m=3$, we get the 3 policies using the one-hot latent codes $(0,0,1),(0,1,0),(1,0,0)$.

Reviewer comment (Writing):
In Appendix C, the paper states that  

> Besides, in order to ensure balanced learning among all the styles, we share the main network and use a lora module to discriminate different styles.  

The explanation on the "lora" module seems missing. Also, I recommend moving this part to the main paper.

Author response:
We appreciate the reviewer's advice to include a explanation of LoRA module [1] used. We have provided a more detailed explanation in Appendix E. We share the main network among policies of all styles and each policy has its own LoRA module.

References:

1.Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

Reviewer comment (Theory):
Transformation from the true posterior to Eq. 2 needs more explanation. The current context is too weak. And I assume the basic assumption for this transformation is that all behavior policies $\mu_{1,..,m}$ are diverse enough, because the authors use transtion-wise action probability to replace the trajectory probability. This makes sense if behavior policies are diverse enough that they take different actions for each step. But if the policies only slightly differ from each other, the consecutive multiplication of all steps within the trajectory will make their trajectory distributions very different from each other (while the action distribution is not much different). As a result, Eq. 2 provides very inaccurate estimation of the posterior.

Author response:
We appreciate the reviewer for pointing out that 'if the policies only slightly differ from each other, the consecutive multiplication of all steps within the trajectory will make their trajectory distributions very different from each other'. We plot the distribution of $\hat{p}(z=1|\tau)$ for all trajectories in the dataset in Figure 11, Appendix J. Figure 11(a) uses consecutive multiplication, while Figure 11(b) is the average of all steps. The results show that using consecutive multiplication causes a trend of polarization in clustering, which leads to unexpected numerical instability. The experiment was conducted in the grid shooting environment with the number of clusters set to $m=2$. To address this issue, we have chosen to average all the steps instead of multiplication in order to alleviate the problem of numerical instability.

Reviewer comment (Experiments):
The proposed SORL needs to know the number $m$ of policy primitives constituting the dataset in order to learn the diverse policies. But it is hard to know this prior under many ciucumstances. I think there should be a study about how sensitive SORL is  to this hyperparameter.

Author response:
We appreciate the reviewer for pointing out the importance of conducting sensitivity analysis on the number of policies $m$ in SORL, as it is difficult to know the exact value of $m$ in real circumstances. In order to address this concern, we have conducted experiments to evaluate the diversity, quality, and consistency of SORL under different values of $m$. We also add two more diversity metrics that are unrelated to $m$, and their detailed descriptions can be found in Appendix F. The performance of SORL remains similar across different $m$ values, indicating that the algorithm is robust to the hyperparameter $m$.

|  | $m=2$ | $m=3$ | $m=4$ | $m=5$ |
| ---- | ---- | ---- | ---- | ---- |
| Diversity       | $0.61$      | $0.85$     | $0.65$      | $0.69$ |
| Diversity (skill metric) | $15.7$ | $20.3$ | $14.0$ | $14.5$ |
| Diversity (OT metric)   | $29.1$ | $28.3$ | $28.5$ | $29.2$ |
| Quality          | $55.1$% | $50.3$% | $52.5$% | $52.2$% |
| Consistency | $86.4$% | $88.5$% | $88.7$% | $88.7$% |

Reviewer comment (Experiments):
How diverse are the induced policies? The case studies are great but there should be a quantitive study. My further question on this is, if we set $m$ larger than the actual number of policy primitives $\mu_{1,..,m}$, what will the resulted policies be like?

Author response:
The quantitative results of the grid shooting environment can be found in the table provided below (also available in Appendix A).

|  | Quality | Diversity | Consistency |
| ---- | ---- | ---- | ---- |
| SORL          | $55.1\pm4.6$% | $0.60\pm0.07$ | $86.4\pm0.4$% |
| Off-RLPMM | $40.4\pm0.7$% | $0.68\pm0.01$ | $90.5\pm0.0$% |
| InfoGAIL      | $55.3\pm2.1$% | $0.00\pm0.00$ | $84.0\pm0.3$% |

We provide additional results in the grid shooting game as $m$ increases. The actual number of policy primitives is two. The reward distribution of the learned policies is presented in the following table, taking $m=4$ as an example. For brevity, the resulting policies for $m=3$ and $m=5$ can be found in Appendix I.

| Learned policies | Policy 1 | Policy 2 | Policy 3| Policy 4 |
| ---- | ---- | ---- | ---- | ---- |
| Reward (shoot) | $1.3$     | $2.2$     | $0.2$    | $3.1$ |
| Reward (star)   | $6.0$    | $3.2$     | $7.6$    | $0.5$ |
| Winning rate    | $57$% | $57$% | $56$% | $40$% |

When taking larger $m$ values, the learned policies still differs a lot from each other. In the case when $m=4$, one policy learns to focus on shooting, one policy concentrates on star collection, and the remaining policies' preferences lie between the above two extremes.

Reviewer comment (Experiments):
As the author claims the induced policies are high-performing, the baselines should include some strong offline RL methods for comparison. This will also show SORL's advantage in policy diversity compared to them. The current baselines are too few.

Author response:
We add experimental results of standard offline RL methods (CQL, AWR and IQL). We conduct comparison in Atari games and results are shown below. Notably, SORL learns a set of policies while the remaining methods learn a single policy. The diversity metric cannot be calculated for methods that only generate a single policy. The scores of SORL are averaged over the full set of learnt policies. We have not yet completed all the experiments on AWR. However, we will finish them and include them in Appendix G before the deadline.

|        | SORL | CQL | AWR | IQL |
| ---- | ----     | ----   | ----   | ----  |
| SpaceInvaders            | $387.1\pm33.3$    | $136.5\pm27.5$   | $351.3\pm102.6$ | $361.5\pm86.9$ |
| MsPacman                  | $622.2\pm65.3$    | $513.7\pm325.5$  | $320.0\pm68.53$ | $505.0\pm161.2$ |
| MontezumaRenvenge | $306.7\pm19.1$    | $113.4\pm196.3$  | $210.0\pm119.9$ | $166.7\pm32.1$ |
| Enduro                        | $371.1\pm2.9$       | $0.0\pm0.1$          | $301.7\pm66.1$ | $290.1\pm44.2$ |
| Riverraid                     | $1931.2\pm432.1$ | $1127.7\pm160.6$ | $1439.0\pm280.6$ | $1909.3\pm332.1$ |
| Frostbite                     | $2056.0\pm391.9$ | $76.3\pm16.7$     | $1165.0\pm470.6$ | $2184.7\pm377.5$ |

The results above shows that the quality performance of SORL is still generally higher than offline RL baselines. We hypothesize that the human datasets are multi-modal, and utilizing multiple policies instead of a single policy to fit the dataset yields better results.

Reviewer comment (Experiments):
How do the authors collect online user data? Where is the user agreement to collect this data? This should appear at least in the appendix.

Author response:
The game environment and the dataset used in this paper are public. The link to the game environment is <https://github.com/FuxiRL/DunkCityDynasty>, and the link to the dataset is <https://huggingface.co/datasets/FUXI/DunkCityDynasty_Dataset>. To our best knowledge, this paper is the first to conduct experiments of learning both diverse and high-quality behaviors from a large human dataset of video games.

Reviewer comment (Theory):
The motivation for increasing the diversity of policy is not clear. In the related work section, the authors only discuss the importance of diversity in online RL settings, for example, encouraging exploration, better opponent modeling, and skill discovery. However, in the offline RL setting, there is no exploration problem or skill discovery since the dataset is fixed. In addition, in the preliminary section, the authors aim to “learn a set of high-quality and diverse policies” without any explanation of the advantage of learning a set of diverse policies over a single policy with diverse behaviors (e.g., using multi-modal distribution as policy distribution).

Author response:
Q2: The motivation for increasing the diversity of policy is not clear.

A2: We have discussed the motivation of learning diverse policies from offline datasets in Lines 4-7 in the introduction. Additionally, we have added further content discussing the utilization of learning diverse policies from offline datasets in opponent modeling in Lines 8-10. We appreciate the reviewer for pointing out that the motivation needs a more detailed discussion. Learning diverse policies from offline datasets is beneficial for various real-world applications, such as game AI and opponent modeling. In game AI, offline data collected from human gameplay allows AI bots to mimic human-like behavior. In online games, deploying such AI bots with varyied motion styles can enrich the gaming environment and enhance player engagement. Additionally, in opponent modeling, high-quality diverse opponents that resemble real opponents can significantly improve the performance of the learned policy. It is worth noting that learning diverse policies from offline datasets is an area that has not been extensively explored in previous research. In the related work, we mention encouraging exploration and skill discovery because they are the motivation of promoting diversity in online settings rather than offline settings.

Reviewer comment (Reproducibility):
Many details are missing in the experiment of the “Dunk City Dynasty”. The code does not include this experiment.

Author response:
Q3: Many details are missing in the experiment of the “Dunk City Dynasty”. The code does not include this experiment.

A3: We have updated the supplementary materials for "Dunk City Dynasty" and included the algorithm codes. However, please note that the code for large-scale efficient experiments cannot be made public at this time due to institution regulations. We will make it available to the public as much as possible in the future.

Reviewer comment (Evaluation):
The metric for evaluating diversity seems to rely on the learned clustering p. I wonder why don’t evaluate the diversity of the learned policy? Otherwise, this metric cannot be used for algorithms that don’t learn the clustering of datasets. In addition, the goal of clustering the dataset is to learn diverse policies for online evaluation, so the diversity of the policy is what we really care about.

Author response:
Q4: The metric for evaluating diversity seems to rely on the learned clustering p.

A4: We admit that the diversity metric employed in the paper relies on clustering $p$. Comparing the diversity of two sets of policies are not trivial. Previous research, such as InfoGAIL, also did not propose a quantitative metric for evaluating diversity when learning policies from datasets. Instead, they relied on visualization to showcase the diversity of learned policies.

In order to provide a more comprehensive investigation of the diversity of learned policies, we have proposed additional diversity metrics that do not require clustering: the **skill metric** measuring the dissimilarity of the skill set and the **OT (optimal transport) metric** quantifing the disparity in state distribution. For a detailed description of these metrics, please refer to the Appendix F. Moreover, we will present visualization results of the Atari environment in the response to the next question.

The table provided below showcases the diversity of various style learning methods in the grid shooting environment utilizing these two additional metrics as well as popularity metric used in the original paper. The two new metrics are not dependent on clustering. Higher values indicate greater diversity. Based on the results of the additional diversity metrics presented in Table below, we can conclude that the SORL algorithm is capable of obtaining diverse policies. The results demonstrate that SORL learns a set of diverse and high-quality policies. InfoGAIL fails to learn diverse policies, while Off-RLPMM learns low-quality diverse policies and has considerably lower winning rate.

| | Popularity metric | Skill metric | OT metric | Winning rate |
| ---- | ---- | ---- | ---- | ---- |
| SORL                      | $0.60$ | $15.7$ | $29.14$ | $55.1$% |
| SORL w/o advantage | $0.61$ | $21.2$ | $31.61$ | $45.2$% |
| Off-RLPMM              | $0.68$ | $23.2$ | $29.87$ | $40.4$% |
| InfoGAIL                     | $0.00$ | $2.0$ | $28.53$ | $55.3$% |

Reviewer comment (Presentation):
Could the authors provide some visualization or example of the mean of the clusters in Atari games? It is not intuitive how the diverse behavior looks like in those games. Similarly, in the Dunk City Dynasty game, the visualization in Figure 3 is too simple. Could the authors plot the shooting positions of each policy? Also, besides the shooting position, are there other differences between these policies?

Author response:
Q5: Could the authors provide some visualization or example of the mean of the clusters in Atari games? Could the authors plot the shooting positions of each policy? Also, besides the shooting position, are there other differences between these policies?

A5: We have generated visualizations of the clusters in several Atari environments. Detailed descriptions of the environment and the visualization can be found in Appendix H. In SpaceInvaders, the shooter can move horizontally, and there are three stationary bunkers positioned above the shooter. To evaluate the diversity of learned policies, we calculate the frequency of the shooter's position over 30,000+ steps for each style. The table below demonstrates that different styles of policies tend to prefer different positions.

|  | Leftmost - Bunker 1 | Bunker 1 - Bunker 2 | Bunker 2 - Bunker 3 | Bunker 3 - Rightmost|
| ---- | ---- | ---- | ---- | ---- |
| Style 1 | $23$% | $19$% | $36$% | $22$% |
| Style 2 | $10$% | $58$% | $15$% | $18$% |
| Style 3 | $32$% | $38$% | $26$% | $4$% |

In MsPacman, the map and starting position are fixed. To evaluate the diversity of learned policies, we calculate the frequency of the beginning trajectory until the first corner for each style, based on 30 trajectories per style. There are a total of six possible corners. The table below displays the different preferred corners for each style:

| Which corner | 3rd left | 2nd left | 1st left | 1st right | 2nd right | 3rd right |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Style 1 | $0$% | $33$% | $67$% | $0$% | $0$% | $0$% |
| Style 2 | $20$% | $7$% | $7$%   | $3$% | $37$% | $27$% |
| Style 3  | $0$% | $97$% | $3$%   | $0$%  | $0$%  | $0$% |

As for the Dunk City Dynasty, the plot of shooting positions are added to Appendix B. Besides the style difference of shooting positions, there also exists the style difference of ball passing and ball possesion, as shown in the following table. Style 2 has the fewest number of passes and the lowest proportion of goals scored directly after a pass.

|  | Style 1 | Style 2 | Style 3 |
| ---- | ---- | ---- | ---- |
| #passes | $41$ | $29$ | $39$ |
| Proportion of goals directly after a pass | $27.7$% | $22.2$% | $44.4$% |

Reviewer comment (Reproducibility):
Could the authors provide more details about the setting of the Dunk City Dynasty experiments? For example, the action space, and model structure. Appendix C only describes the structure of the first two experiments.

Author response:
Q6: Could the authors provide more details about the setting of the Dunk City Dynasty experiments? For example, the action space, and model structure. Appendix C only describes the structure of the first two experiments.

A6: We appreciate the reviewer for pointing it out. We have provided details of the experiments in Dunk City Dynasty in Appendix B. The state space dimension is $468$, including global state, allies' states and enemies' states. The action space is a discrete space with $52$ actions. The network structure of 'Dunk City Dynasty' is a 3-layer MLP. The opponent in evaluation is an agent with moderate strength, that is learned by vanilla Behavioral Cloning for similar training steps (100,000 steps).

Reviewer comment (Writing):
What does it mean by “we share the main network and use a lora module to discriminate different styles.” In Appendix C?

Author response:
Q7: What does it mean by “we share the main network and use a lora module to discriminate different styles.” In Appendix C?

A7: The detailed description of LoRA module is as follows. In order to ensure balanced learning among all the styles, we share the main network and use a LoRA module to discriminate different styles. LoRA [1] is a widely used network structure, that substitutes the original matrix of the linear layer by a matrix and a low-rank multiplication of two other matrices. Further details of LoRA module is added in Appendix E.

Reviewer comment (Evaluation):
Results in Table 3 show that SORL has a very high variance (5.3 ± 3.4) in terms of quality, which only slightly outperforms InfoGAIL (5.0 ± 0.8). Does this mean pursuing high-quality sacrifices for the performance of the policy? Then, what do we gain from the high diversity?

Author response:
Q8: Results in Table 3 show that SORL has a very high variance (5.3 ± 3.4) in terms of quality, which only slightly outperforms InfoGAIL (5.0 ± 0.8). Does this mean pursuing high-quality sacrifices for the performance of the policy? Then, what do we gain from the high diversity?

A8: The focus of our work is to learn diverse and high-quality policies. These are two distinct objectives and pursuing diversity may sacrifice the quality of the policy. The goal of this paper is to maximize both objectives and trade off between diversity and quality. In Table 3, although SORL shows higher variance in performance, it signficantly outperforms the baselines in terms of disversity. Similarly, in the other two environments, we also observe a trade-off between quality and diversity. For instance, Figure 2 in the main text shows that the SORL algorithm does not always outperform the baselines in terms of quality, but it consistently achieves a good balance between high quality and diversity. By combining the pursuit of both quality and diversity, we demonstrate that the SORL algorithm is superior to the baselines. It provides a more comprehensive solution that achieves both high performance and a diverse set of policies.

The motivation of learning diverse policies from offline datasets has been expained in the response to question 2. To recapitulate briefly, it is useful in applications including learning human-like behaviors in game AI and opponent modeling.