Paper: Non-Parameterized Randomization for Environmental Generalization in Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
Compared to data augmentation and domain randomization methods, the method proposed in this paper requires stronger preconditions: 1) NPR requires the algorithm to be able to change the environment during training, and 2) the expert prior is needed to label which components of the environment are task-relevant and which are not. Both of these conditions are difficult to fulfill in practice, and thus the method has significant limitations.

Author response:
Thank you for your review.


+ We acknowledge that we need some assumptions. But we consider it is reasonable and inevitable. If we focus on the tasks in various environments in practice, how can we build policy efficiently? In fact, none of the existing methods can adapt to the huge change in real-world scenes. What is worse, training in all the real-world tasks is unacceptable due to the expensive sampling cost. So how do you face these tasks? Here we give our idea aiming to train sufficiently in just one environment and generalize to other environments that have different physical dynamics, once for all. It will significantly save samples compared with sampling in all the environments. Consider that, if an agent can generalize to any other unseen environments without retraining, meaning another way of sample efficiency. To achieve this is extremely hard. There are no methods that can completely achieve this, neither can ours. So we do not claim to completely solve the problems, but give a novel attempt and outperform all the existing methods.



+ It is the generalization ability of our method important, although it needs some assumption. Because generalizing to an unseen environment is nearly impossible without additional training costs. When facing many complex RL tasks, sampling and learning all the tasks in different environments will be unacceptable and expensive. As a result, our method provides a trade-off between learning and generalization. That is, training sufficiently in one environment, then generalizing to the other environments. It is more sample efficient than learning in all these tasks and environments. Meanwhile, please pay attention that our method is in zero-shot fashion. That means when the number of testing tasks grows, our requirement for training does not increase. So we think it is more important to focus on the performance and potential of our method to achieve complex generalization tasks.




+ As for the comparison of baselines, in fact, requiring a parameterized model environment costs not less than our method. On the contrary, these methods need a more specific and elaborate physical model to provide parameters that can be randomized. For instance, in a real-world application like in a kitchen, randomizing the objects in the room (our idea) will be easier than randomizing the friction coefficient and background (the idea of baselines). Thus, compared with existing methods, our method has a superiority in generalization tasks.

Reviewer comment (Experiments):
The method is designed to utilize the two preconditions mentioned above, whereas the other algorithms in the experiment were designed without them. Therefore, I think that the generalization capability experiment in the paper is meaningless. It is unfair to the other algorithms to conduct comparison experiments under different experimental conditions.

Author response:
1. Firstly, the cost of NPR is not in vain. It brings strong generalization ability in different environments. The success of a large model shows the fact that the cost of training and data is necessary. Because it is impossible to achieve generalization ability without effective training. There is no free lunch. Secondly, previous methods need the parameterized physical model of an environment. The cost has been spent in manually modeling the simulator of the environment, i.e., the parameterized physical model of the environment requires both the expert prior and a precise mathematic model.  Thus previous methods can augment the data in these environments by randomizing the parameters. It is unfair to ignore the expensive cost of manually building the parameterized model and be strict with our method. For instance, the most commonly used method in DR is to randomize the dynamic coefficients and visual background. These require many prior of the environment.

2. The baselines are all trained in an environment with NPR, or they will not obtain any rewards in a new environment with intrinsic physical change. Because traditional RL settings are all integration of training and testing.  We give all the experiments in the same setting for a fair comparison. But as we said, they can hardly adapt to the environmental change.

Reviewer comment (Writing):
“In complex generalization tasks, the common part is embedded in the environment and is not always observable.” So is it the similarity parts or the changing background of the model definition? The structured model established in the paper is unclear.

Author response:
+ For Q1, I represent the similar parts, \psi is the function that is related to the environment. It means that, the same I will perform differently in different environments. For instance, a red ball in BabyAI will be a little different from a red balls in MuJoCo. This setting is utilized to cover more real-world scenes.

Reviewer comment (Writing):
In subsection 3.1->"Difference with Previous Models." both I and $ \psi_t(I) $ are defined as similar parts of the state space, is there any ambiguity?

Author response:
+ For Q1, I represent the similar parts, \psi is the function that is related to the environment. It means that, the same I will perform differently in different environments. For instance, a red ball in BabyAI will be a little different from a red balls in MuJoCo. This setting is utilized to cover more real-world scenes.

Reviewer comment (Evaluation):
In Table 3, the training results of NPR in 2d-maze are not as good as No-Rand, and whether the loss of the training performance is indicative of flaws in the design of the algorithm?

Author response:
+ For Q2, the No-Rand task means the training task is simple and fixed without randomness. Learning well in such tasks is very easy. Training in tasks with random is more difficult than training in a fixed task.

Reviewer comment (Experiments):
The data in Figure 3 are too sparse, which is different from a normal dense continuous training curve. Please provide more details of the experiment. Meanwhile, there may be instability in the results of experiments using only 3 seeds. To ensure reliability, at least 5 seeds should be used.

Author response:
+ For Q4, the sparse results come from taking an average of every 100 episodes. We just want to show the superiority of our method clearly. If necessary, we can train more data as you suggested and add it to the revised version of our paper.

Reviewer comment (Novelty):
It seems like the NPR method boils down to randomizing task-agnostic parts of the environment to increase the diversity of the backgrounds trained on in order to improve zero-shot environment generalization. This is not a new or surprising conclusion. In fact, many recent works, such as RT-2 (Google, 2023), have shown that with increased diversity of backgrounds, zero-shot generalization can be obtained in unseen environments. Is there more nuance here that is not coming across?

Author response:
+ For the question of zero-shot generalization problems and the motivation of our work.

>> Recent works show that a large model with an RL setting can achieve many complex tasks, even zero-shot generalization tasks, including RT-2, VoxPoser, Gato, and some following works. They have solved many tasks, however, have not solved all the kinds of generalization tasks. In fact, as RL aims to solve real-world tasks, the generalization tasks are quite different.     

>> When classified by allocation-oriented real-world tasks, from abstract aspect to specific aspect, generalization tasks can be divided into three main kinds. That is strategy generalization of abstract decisions, skill-level generalization of recombination tasks, and manipulation generalization of different low-level robots. When classified by RL modeling, generalization tasks can be divided into two categories, i.e., focusing on task change (reward function or transition change) or environmental change (physical model of state space and action space change). A survey [1] of zero-shot generalization in deep RL also gives another view of IID and OOD generalization.     

>> As for our work, we focus on generalization across different environments, where the intrinsic physical generators of the environments are different, which has not been solved by existing works yet. The difference between our work and existing works that focus on tasks with background change is that the environmental change in our work will impact the policy execution, but the other will not. For instance, a prevailing task [2] that previous works claimed with environment change is controlling the MuJoCo robot with the background changing in the DMControl platform. The changing backgrounds consist of task-agnostic pictures, but it does not change the structure of the environment. That means that if an agent learns an action sequence in the training tasks, it can also complete the task with the sequence. The intrinsic model of the tasks does not change, meaning that the tasks are intrinsically the same.     

>> On the contrary, the environmental change in our tasks requires the policy to adapt to different dynamics and physics in different environments. That is why the tasks are quite difficult. As a result, our work does not claim to completely solve these tasks, but just gives an attempt and proposes an idea. Solving all these kinds of tasks means the agent can adapt to any OOD tasks, so should also be used in real-world applications in zero-shot.     

>> Meanwhile, our method does not conflict with the existing large models. Our method is in a goal-conditioned setting, which can be used as the low-level executing policy of large models by receiving high-level abstract instructions.

Reviewer comment (Theory):
The paper makes it seem like the NPR method removes significant assumptions for training, but in practice, randomizing the non-parametrized task-agnostic parts of the environment may be even more difficult than the task-relevant parts of the environment. This is something that would not be very scalable in many real world applications.

Author response:
+ For the question of assumption and settings in our method.

>> For our NPR method, we acknowledge that we need some assumptions. But we consider it is reasonable and inevitable. When facing many complex RL tasks, sampling and learning all the tasks in different environments will be unacceptable and expensive. As a result, our method provides a trade-off between learning and generalization. That is, training sufficiently in one environment, then generalizing to the other environments. It is more sample efficient than learning in all these tasks and environments. Meanwhile, please pay attention that our method is in zero-shot fashion. That means when the number of testing tasks grows, our requirement for training does not increase. So we think the cost in the training environment is acceptable.

Reviewer comment (Experiments):
The comparisons in the empirical evaluation all use fewer assumptions than NPR--they do not assume access to randomizing specific aspects of the environments.

Author response:
+ For the question of comparison of other baselines.

>> As for the comparison of baselines, in fact, requiring a parameterized model environment costs not less than our method. On the contrary, these methods need a more specific and elaborate physical model to provide parameters that can be randomized. For instance, in a real-world application like in a kitchen, randomizing the objects in the room (idea of ours) will be easier than randomizing the friction coefficient and background colors (idea of baselines). Meanwhile, the methods that augment the data after samples can not represent the intrinsic change of environments, so they perform poorly. Thus, compared with existing methods, our method has a superiority in generalization tasks.