Paper: Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay (ICLR-2024)

Reviewer comment (Writing):
Section 2 (preliminaries) is largely unclear and not stand-alone.

Author response:
i) Following your suggestion, we have moved the notation clarification to the beginning of Sec. 2. 

ii) Additionally, we have described CORL in Section 3.1 and have now relocated it to Sec. 2.

iii) In Fig. 1, in order to clearly show that real samples and generated samples are used together to train the behavior evaluation model, we use two ($s, a, r, s'$)  boxes to correspond to the state generator box and the behavior generator box, respectively.

Reviewer comment (Writing):
Section 3 (approach) is also not sufficiently clear or precise, and seems to introduce both math and text that are not related to the submission.

Author response:
i) One task corresponds to an MDP, and tasks are generated from a specific MDP distribution in continual learning. We have explained the details of sequential tasks in Sec. 3.1 and Appendix B.

ii) The behavior policy of a single task may be a multimodal distribution. 
When considering multiple tasks, the distribution of all behavior policies might become even more multimodal. 
Hence, we use a diffusion model to fit the behavior policies of all tasks. 
In classification tasks, opposite predictions are resolved using a task indicator. 
Similarly, in RL, the diffusion model mimics behavioral strategies, while the decision-making is guided by the Q-function. 

iii) In Eqs. (10) and (13), we present the general form of training loss and set $\beta$ to facilitate training. 
During the training process, the value of $\beta$ is 0.5. 

iv) The goal of  Eq. (8) is to learn a continual policy that maximizes the expected return over all encountered tasks, which aligns with the training objectives of Eqs. (10) and (13). 
The test loss in Eqs. (11) and (14) evaluates the training effectiveness of the diffusion model, analogous to the validation loss in supervised learning. 
The setting of the test loss is consistent with DGR [1].

v) We decouple the policy into a behavior generative model and an action evaluation model, and use the state generative model along with the behavior generative model to achieve generative replay. 
During the continual learning process, we use a continually updated state generation model to cover the state space of all tasks, enabling it to reconstruct the cumulative state space of all tasks based on task encoding.
The order of training model components is actually shown in Fig. 1.

vi) In our method, action sampling is guided by Q-values, analogous to classical value function methods where the policy/behavior is derived from value functions.
Hence, cloning the Q-function implicitly clones the behaviors, thus we call the process as behavior cloning.

Reviewer comment (Experiments):
The experimental evaluation is not sufficient to assess the benefits of the proposed method.

Author response:
i) As we are the first to leverage expressive diffusion models to tackle the understudied CORL challenge, our primary concern is to validate whether our dual generative replay mechanism can realize high-fidelity replay of the sample space.
Hence, following the classical DGR method [1], we compare the performance of the RL model trained with variants of replay methods: Oracle, None, and Noise.
For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity.
For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator.
For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism.
We did not compare our method to several existing continual RL methods due to the difference in the intrinsic characteristics and applicable scenarios.
For example, [2] belongs to the parameter isolation kind, [3, 4] needs to replay real samples of past tasks, and most of the existing methods [2, 3, 5] tackle the online RL problems other than the offline setting.

ii) These testbeds are usually used to test traditional online RL algorithms and has not been applied to offline scenarios yet. 
Compared with online continual RL, offline continual RL is less studied, and there is no standard dataset for it. 
Generally, the MuJoCo environment, as seen in the D4RL dataset [6], is a popular testbed for offline RL. 
The MuJoCo environment is also commonly used in offline meta-RL for testing [7, 8]. 
Hence, we evaluate CORL algorithms on MuJoCo environments with offline datasets in this paper.

iii) We follow the generative replay paradigm, e.g., DGR [1] and DDGR[9], where the scalability primarily depends on the modeling fidelity of the generator.
This rationale underpins the choice of diffusion models for our method, mirroring the research motivation of DDGR. 
In DDGR, diffusion models demonstrate better fidelity compared to GANs, and exhibit promising superiorities for continual learning based on generated replay.  
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we believe that this approach will see further developments in the future.

In summary, it is important to evaluate CORL methods on more testbeds, or to conduct empirical comparisons to existing methods for demonstrating their respective pros and cons, or to modify these online continual methods to adapt to offline settings.
We leave these for future work.

Reviewer comment (Experiments):
The evaluation uses very simple MuJoCo tasks to evaluate methods. The authors should evaluate their approach on much more complex tasks, such as those from the Meta-World, CausalWorld, RLBench, or CompoSuite. This is particularly important because the authors' motivation stems from the need for diffusion models to represent complex behaviors.

Author response:
ii) These testbeds are usually used to test traditional online RL algorithms and has not been applied to offline scenarios yet. 
Compared with online continual RL, offline continual RL is less studied, and there is no standard dataset for it. 
Generally, the MuJoCo environment, as seen in the D4RL dataset [6], is a popular testbed for offline RL. 
The MuJoCo environment is also commonly used in offline meta-RL for testing [7, 8]. 
Hence, we evaluate CORL algorithms on MuJoCo environments with offline datasets in this paper.

Reviewer comment (Writing):
The description of advantage-weighted regression misses the key point that the Q function is w.r.t. μ, which makes Eq. 4 solve the constrained optimization of Eq. 2. Without explicitly stating this, readers might wrongly assume Qθ is the standard Q^π.

Author response:
i) Following your suggestions, we have further explained $Q_\theta$ and $Q^\pi$ and their relationship.

Reviewer comment (Writing):
The description of diffusion models in Section 2.2 omits mention of the forward diffusion process that adds noise to an observed action. This omission makes it difficult to parse the last sentence before Eq. 6, in which the diffusion model predicts an undefined noise.

Author response:
iii) In Section 2.2, we describe the forward diffusion process of the diffusion model. Diffusion models are generative models that begin by defining a forward process to gradually add noise to an unknown data distribution, and then learn to reverse this process.

Reviewer comment (Writing):
The description prior to Eq. 7 is unclear: is Q estimating the value of actions w.r.t. μ or π? Actions are sampled from μ, but is the long-term value measured for π or μ? The middle part of Eq. 7 suggests π, but this is not stated explicitly.

Author response:
iv) Eq. (7) is based on the fundamental concepts of offline RL.  
$R_i^{(0)}$ is the vanilla return of trajectories. 
The first part of Eq. (7) offers an implicit planning scheme within dataset trajectories that mainly helps to avoid bootstrapping over unseen actions and accelerate convergence. 
The middle part of Eq. (7) enables the generalization of actions in similar states across different trajectories.  
This is our basic approach, and you can refer to SfBC for details [10].

Reviewer comment (Writing):
The meaning of "re-sampling" toward the end of Section 2.1 is unclear. What was the first "sampling"?

Author response:
v) The 'first' sampling means sampling from the diffusion-based behavior generative model.
Then, we evaluate the sampled actions with the trained critic network.
The learned policy is a combination of the behavior generative model and the critic network.
Finally, during execution, actions are 're-sampled' from the learned policy.
More details can be found in [10].

Reviewer comment (Experiments):
In Section 3.1, the description of CORL does not specify which aspects of the MDP are allowed to change from task to task, and which ones the authors actually consider in their approach/evaluation.

Author response:
i) One task corresponds to an MDP, and tasks are generated from a specific MDP distribution in continual learning. We have explained the details of sequential tasks in Sec. 3.1 and Appendix B.

Reviewer comment (Theory):
In Section 3.1, it is unclear whether the distribution over MDPs P(M) is stationary or allowed to change over time, and what implications this has for the learning process or experimental evaluation.

Author response:
i) One task corresponds to an MDP, and tasks are generated from a specific MDP distribution in continual learning. We have explained the details of sequential tasks in Sec. 3.1 and Appendix B.

Reviewer comment (Writing):
In Section 3.2, the authors appear to conflate two ideas in their motivation for using diffusion models: (1) some tasks require multimodal behaviors, and (2) the diffusion model should capture a breadth of tasks. The latter is hinted at but never explained or exemplified.

Author response:
ii) The behavior policy of a single task may be a multimodal distribution. 
When considering multiple tasks, the distribution of all behavior policies might become even more multimodal. 
Hence, we use a diffusion model to fit the behavior policies of all tasks. 
In classification tasks, opposite predictions are resolved using a task indicator. 
Similarly, in RL, the diffusion model mimics behavioral strategies, while the decision-making is guided by the Q-function.

Reviewer comment (Presentation):
Section 3.2 is oddly structured. The first paragraph is motivation without technical details, the second covers state generation details (while equivalent action generation details are in preliminaries), and the final paragraph discusses classifier-based/free guidance without explaining its purpose or relevance, especially since it is not actually implemented.

Author response:
ii) The behavior policy of a single task may be a multimodal distribution. 
When considering multiple tasks, the distribution of all behavior policies might become even more multimodal. 
Hence, we use a diffusion model to fit the behavior policies of all tasks. 
In classification tasks, opposite predictions are resolved using a task indicator. 
Similarly, in RL, the diffusion model mimics behavioral strategies, while the decision-making is guided by the Q-function. 

v) We decouple the policy into a behavior generative model and an action evaluation model, and use the state generative model along with the behavior generative model to achieve generative replay. 
During the continual learning process, we use a continually updated state generation model to cover the state space of all tasks, enabling it to reconstruct the cumulative state space of all tasks based on task encoding.
The order of training model components is actually shown in Fig. 1.

vi) In our method, action sampling is guided by Q-values, analogous to classical value function methods where the policy/behavior is derived from value functions.
Hence, cloning the Q-function implicitly clones the behaviors, thus we call the process as behavior cloning.

Reviewer comment (Writing):
In Section 3.3, the phrase "the desired importance of the new task..." in relation to Eq. 8 is misleading. The issue is not the "importance" but the need for weighting to ensure new task learning while avoiding forgetting.

Author response:
iii) In Eqs. (10) and (13), we present the general form of training loss and set $\beta$ to facilitate training. 
During the training process, the value of $\beta$ is 0.5.

Reviewer comment (Evaluation):
In Section 3.3, it is unclear why there is a test loss (Eq. 11 and 14) for specific models, since performance should be measured as obtained reward on tasks.

Author response:
iv) The goal of  Eq. (8) is to learn a continual policy that maximizes the expected return over all encountered tasks, which aligns with the training objectives of Eqs. (10) and (13). 
The test loss in Eqs. (11) and (14) evaluates the training effectiveness of the diffusion model, analogous to the validation loss in supervised learning. 
The setting of the test loss is consistent with DGR [1].

Reviewer comment (Writing):
In Section 3.3, the meaning of "reconstruct the cumulative state space" is unclear. Task conditioning suggests separate state spaces per task.

Author response:
v) We decouple the policy into a behavior generative model and an action evaluation model, and use the state generative model along with the behavior generative model to achieve generative replay. 
During the continual learning process, we use a continually updated state generation model to cover the state space of all tasks, enabling it to reconstruct the cumulative state space of all tasks based on task encoding.
The order of training model components is actually shown in Fig. 1.

Reviewer comment (Reproducibility):
In Section 3.3, it is not specified whether generated states for behavior replay are drawn after updating the state generator, before, or simultaneously. This choice and its implications should be stated explicitly rather than relying on the Appendix.

Author response:
v) We decouple the policy into a behavior generative model and an action evaluation model, and use the state generative model along with the behavior generative model to achieve generative replay. 
During the continual learning process, we use a continually updated state generation model to cover the state space of all tasks, enabling it to reconstruct the cumulative state space of all tasks based on task encoding.
The order of training model components is actually shown in Fig. 1.

Reviewer comment (Writing):
In Section 3.4, it is unclear why the term "behavior cloning" is used, as it has a specific meaning in offline RL. The described approach is closer to functional regularization as studied in supervised continual learning.

Author response:
vi) In our method, action sampling is guided by Q-values, analogous to classical value function methods where the policy/behavior is derived from value functions.
Hence, cloning the Q-function implicitly clones the behaviors, thus we call the process as behavior cloning.

Reviewer comment (Novelty):
Incorporating diffusion models in offline RL for generative reply is straightforward, and the computation cost should be rigorously discussed.

Author response:
In the continual learning setting, the category of generative replay methods [1, 2] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks).
That is, they eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional generative model, e.g., GANs [1] or diffusion models [2].
Taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has a dimension of 26.
Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs.

In literature, current diffusion-based RL methods [3-8] could face the challenge of increased computational costs due to the training and sampling of diffusion models.
For example, the diffusion-planning methods [4, 8] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples.
We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [9, 10].
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle the limitations in future work as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Reviewer comment (Theory):
The loss function in Eq.10, 11, 13, 14, and 15 is easy to figure out, but it is hard to guarantee the convergence. For instance, in Eq.10, it is not clear whether training a diffusion model via using the data from the last diffusion model has any convergence guarantee. This issue could be severe especially when we only have access to offline data with a limited size or large state and action space.

Author response:
Under the generative replay paradigm [1, 2], we train the diffusion models using a mixed distribution of real data from the new task and pseudo data generated from previous generators. 
The convergence regarding Eqs. (10), (11), (13), and (14) align with the diffusion model.
In Eq. (15), the first term follows a planning-based Bellman operator used in offline RL [3], and the second term corresponds to behavior cloning.
Hence, the convergence regarding Eq. (15) is consistent with the specific Bellman operator [3] and the supervised stochastic gradient descent (SGD).
The convergence of diffusion models is independent of the training data. 
Continually training the diffusion model with data generated by the previous iteration of the model only affects the quality of the training data and does not influence the model's convergence.
Hence, the convergence remains consistent with the vanilla diffusion models. 
Additionally, the convergence of the model is independent of the dataset size.

Reviewer comment (Experiments):
Missing other continual RL baselines or benchmarks. In continual RL, the benchmark ‘continual world’ is commonly used to evaluate the continual control algorithms, but the proposal algorithm is only evaluated in small-scaled benchmarks in Mujuco games. It is not clear whether this approach is effective in the commonly-used continual RL environments. Also, some typical continual RL baselines are missing, such as [1].  

[1] Gaya et al. Building a subspace of policies for scalable continual learning (ICLR 2023)

Author response:
As we are the first to leverage expressive diffusion models to tackle the understudied CORL challenge, our primary concern is to validate whether our dual generative replay mechanism can realize high-fidelity replay of the sample space.
Hence, following the classical DGR method [1], we compare the performance of the RL model trained with variants of replay methods: Oracle, None, and Noise.
For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity.
For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator.
For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism.
We did not compare our method to several existing continual RL methods due to the difference in the intrinsic characteristics and applicable scenarios.
For example, [11] belongs to the model expansion kind, [12, 13] needs to replay real samples of past tasks, and most of the existing methods [11, 12, 14] tackle the online RL problems other than the offline setting.

'Continual world' is currently used to test traditional online RL algorithms and has not been applied to offline scenarios yet. 
Compared with online continual RL, offline continual RL is less studied, and there is no standard dataset for it. 
Generally, the MuJoCo environment, as seen in the D4RL dataset [16], is a popular testbed for offline RL. 
The MuJoCo environment is also commonly used in offline meta-RL for testing [8, 16, 17]. 
Hence, we evaluate CORL algorithms on MuJoCo environments with offline datasets in this paper.

On the other hand, it is important to evaluate CORL methods on more testbeds like the continual world in the future.
It is also significant to conduct empirical comparisons to existing methods for demonstrating their respective pros and cons, or to modify these online continual methods to adapt to offline settings.
We leave these for future work.

Reviewer comment (Novelty):
Motivation. There is no sufficient support for the necessity of using diffusion models to learn the behaviors from prior tasks. Though the general knowledge is that diffusion models can work better in terms of generation and generalization, there is no explicit reason against the utilization of other modes such as behavior cloning, GAN, VAE, etc.

Author response:
**Q1: On the necessity of using diffusion models to learn behaviors from prior tasks, and the reason against the utilization of other models such as behavior cloning, GAN, VAE, etc. If applying other types of generative models to replace the diffusion model will yield similar performance?**

A1: As repeatedly stated in our original paper, we have discussed the advantages of the diffusion model and the motivation for adopting it to tackle the CORL challenge.
Due to the strong distributional expressivity and the ability to fit multimodal distributions, diffusion models have been explosively developed in the RL community recently [1-5], such as using diffusion models for behavior policy modeling [1], for target policy modeling [2], and for model-based trajectory modeling [2].
In the first paragraph of Sec. 3.2 (bottom, page 4), we elaborate on the necessity of using the diffusion model to learn behavior from prior tasks as: "This perspective rooted in generative modeling presents three promising advantages for CORL. First, existing policy models are usually unimodal Gaussians with limited distributional expressivity, while collected behaviors in CORL become progressively diverse as novel datasets keep emerging. Learning a generative behavior model is considerably simpler since sampling from the behavior policy can naturally encompass a diverse range of observed behaviors, and allows the policy to inherit the distributional expressivity of diffusion models. Second, RL models are more prone to deficient generalization across diverse tasks. Learning a unified behavior model can naturally absorb novel behavior patterns, continually promoting knowledge transfer and generalization for offline RL. Third, generative behavior modeling can harness extensive offline datasets from a wide range of tasks with pretraining, serving as a foundation model to facilitate finetuning for any downstream tasks. This aligns with the paradigm of large language models, and we reserve this promising avenue for future research."

On the other hand, other models such as behavior cloning, GANs, and VAEs can also be considered as alternatives to the diffusion model. 
In this paper, we choose the powerful diffusion model due to its promising properties discussed above.
As the state-of-the-art architecture in the generative modeling community, the diffusion model has achieved great success in high-fidelity data generation, such as its superiority over GANs [6] and stable diffusion [7].

Reviewer comment (Reproducibility):
Efficiency trade-off: Though storing models for prior tasks might work well, training diffusion models for each different task might not be sample-efficient nor computation-efficient. Therefore, the authors might want to provide more information on the feasibility of this approach and the computation resource usage for implementing the experiments.

Author response:
**Q3: Efficiency trade-off: Training diffusion models for each different task might not be sample-efficient nor computation-efficient. Therefore, the authors might want to provide more information on the feasibility of this approach and the computation resource usage for implementing the experiments (e.g., how long to train the model).**

A3: In the continual learning setting, the category of generative replay methods [8, 9] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks).
That is, they eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional generative model, e.g., GANs [8] or diffusion models [9].

**For sample efficiency**, our method operates in an offline mode, and the size of the offline dataset is fixed.
Hence, the sample efficiency is not a primary consideration.

**For computation efficiency**, taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has the dimension of 26.
As the number of tasks increases, the number of encountered samples will continue to expand, while the model size remains constant.
Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs.
The advantage of generative replay becomes more remarkable for scenarios where massive datasets are needed to train a task or the continual learning setting contains a large number of sequential tasks.
Additionally, the generative replay method can produce pseudo data as much as needed, which can also improve the robustness and flexibility of the model.

In literature, current diffusion-based RL methods [1-5] could face the challenge of increased computational costs due to the training and sampling of diffusion models.
For example, the diffusion-planning methods [3] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples.
We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [10, 11].
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle the limitations as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Reviewer comment (Experiments):
Experiment: The experiment did not show how diffusion models contribute to the continual learning of the model. The authors might want to show the performance of the diffusion models to demonstrate that diffusion models are contributing to the performance.

Author response:
**Q2:  Experiment: i) Show how diffusion models contribute to continual learning performance; ii) Clarify the training data of the diffusion models; iii) Will noisy data degrade the performance of the diffusion models and the CuGRO model as a whole?**

A2: i) As stated in our experiments, our primary concern is to validate whether our diffusion-based generative replay mechanism can realize high-fidelity replay of the sample space.
Hence, we set three baselines as Oracle, None, and Noise.
For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity.
For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator.
For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism.
In summary, these observations verify the contributions of our diffusion models to the continual learning performance.

ii) The collection details of offline datasets are explained in Appendix B.
As stated in Sec. 3.3, the diffusion models are trained using a mixed data distribution of real samples and replayed ones from the previous generator.

iii) We only use the noisy data in the Noise baseline, where the state generative model is replaced with Gaussian noise.
In CuGRO, we train the diffusion models with a mixed distribution of real data from the new task and pseudo data generated from previous generators.

Reviewer comment (Experiments):
I was wondering if applying other types of generative models to replace the diffusion model will yield similar performance.

Author response:
**Q1: On the necessity of using diffusion models to learn behaviors from prior tasks, and the reason against the utilization of other models such as behavior cloning, GAN, VAE, etc. If applying other types of generative models to replace the diffusion model will yield similar performance?**

A1: As repeatedly stated in our original paper, we have discussed the advantages of the diffusion model and the motivation for adopting it to tackle the CORL challenge.
Due to the strong distributional expressivity and the ability to fit multimodal distributions, diffusion models have been explosively developed in the RL community recently [1-5], such as using diffusion models for behavior policy modeling [1], for target policy modeling [2], and for model-based trajectory modeling [2].
In the first paragraph of Sec. 3.2 (bottom, page 4), we elaborate on the necessity of using the diffusion model to learn behavior from prior tasks as: "This perspective rooted in generative modeling presents three promising advantages for CORL. First, existing policy models are usually unimodal Gaussians with limited distributional expressivity, while collected behaviors in CORL become progressively diverse as novel datasets keep emerging. Learning a generative behavior model is considerably simpler since sampling from the behavior policy can naturally encompass a diverse range of observed behaviors, and allows the policy to inherit the distributional expressivity of diffusion models. Second, RL models are more prone to deficient generalization across diverse tasks. Learning a unified behavior model can naturally absorb novel behavior patterns, continually promoting knowledge transfer and generalization for offline RL. Third, generative behavior modeling can harness extensive offline datasets from a wide range of tasks with pretraining, serving as a foundation model to facilitate finetuning for any downstream tasks. This aligns with the paradigm of large language models, and we reserve this promising avenue for future research."

On the other hand, other models such as behavior cloning, GANs, and VAEs can also be considered as alternatives to the diffusion model. 
In this paper, we choose the powerful diffusion model due to its promising properties discussed above.
As the state-of-the-art architecture in the generative modeling community, the diffusion model has achieved great success in high-fidelity data generation, such as its superiority over GANs [6] and stable diffusion [7].

Reviewer comment (Reproducibility):
I am curious to know the computation resources used and how long to train the model.

Author response:
**Q3: Efficiency trade-off: Training diffusion models for each different task might not be sample-efficient nor computation-efficient. Therefore, the authors might want to provide more information on the feasibility of this approach and the computation resource usage for implementing the experiments (e.g., how long to train the model).**

A3: In the continual learning setting, the category of generative replay methods [8, 9] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks).
That is, they eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional generative model, e.g., GANs [8] or diffusion models [9].

**For sample efficiency**, our method operates in an offline mode, and the size of the offline dataset is fixed.
Hence, the sample efficiency is not a primary consideration.

**For computation efficiency**, taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has the dimension of 26.
As the number of tasks increases, the number of encountered samples will continue to expand, while the model size remains constant.
Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs.
The advantage of generative replay becomes more remarkable for scenarios where massive datasets are needed to train a task or the continual learning setting contains a large number of sequential tasks.
Additionally, the generative replay method can produce pseudo data as much as needed, which can also improve the robustness and flexibility of the model.

In literature, current diffusion-based RL methods [1-5] could face the challenge of increased computational costs due to the training and sampling of diffusion models.
For example, the diffusion-planning methods [3] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples.
We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [10, 11].
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle the limitations as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Reviewer comment (Experiments):
I was wondering about the scalability of the model. If having more tasks degrade the model's performance?

Author response:
**Q4:  I was wondering about the scalability of the model. If having more tasks degrade the model's performance?**

A4: We follow the generative replay paradigm, e.g., DGR [8] and DDGR[9], where the scalability primarily depends on the modeling fidelity of the generator.
This rationale underpins the choice of diffusion models for our method, mirroring the research motivation of DDGR. 
In DDGR, diffusion models demonstrate better fidelity compared to GANs, and exhibit promising superiorities for continual learning based on generated replay.  
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we believe that this approach will see further developments in the future.

[1] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[2] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[3] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[4] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[5] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[8] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[9] Rui Gao, et al., DDGR: Continual learning with deep diffusion-based generative replay, ICML 2023.

[10] Cheng Lu, et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.

[11] Qinsheng Zhang, et al., gDDIM: Generalized denoising diffusion implicit models, ICLR 2023.

Reviewer comment (Experiments):
Requires training two separate diffusion models, which can be computationally expensive for sampling at test time since parallel sampling is not possible. Exploring concatenating {s,a} and diffuse with one model could improve efficiency.

Author response:
**Q1: Training two separate diffusion models can be computationally expensive. Exploring concatenating {s,a} and diffusing with one model could improve efficiency.**

A1: Thank you for your insightful comments. 
We train two separate diffusion models since we adopt the diffusion-based generative behavior modeling method [1] as our backbone offline RL algorithm.
This kind of backbone is well suited for continual learning problems, because training a unified behavior model can continually absorb new behavior patterns to promote forward knowledge transfer for the offline setting, and sampling from this generative model can naturally encompass a progressive range of observed behaviors.
Then, for continual learning, we need to adopt another generative model to synthesize pseudo  samples of past tasks under the paradigm of deep generative replay [9, 10]. 

On the other side, this architecture with two diffusion models could indeed increase computational costs.
In literature, current diffusion-based RL methods [1-5] could face the challenge of increased computational costs due to the training and sampling of diffusion models.
For example, the diffusion-planning methods [2, 6] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples.
We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [7, 8].
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle its limitations as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Reviewer comment (Evaluation):
This methods alleviates the memory capacity concern by condensing the previous task’s knowledge into two diffusion models. How does the continual training cost of updating diffusion models for each new task and sampling from them trade off with the memory savings of condensing previous tasks?

Author response:
**Q2: How does the continual training cost of updating diffusion models for each new task and sampling from them trade off with the memory savings of condensing previous tasks?**

A2: In the continual learning setting, the category of generative replay methods [9, 10] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks).
In our method, we eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional diffusion-based state generator.
Taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has a dimension of 26.
As the number of tasks increases, the number of encountered samples will continue to expand, while the model size remains constant.
Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs.
The advantage of generative replay becomes more remarkable for scenarios where massive datasets are needed to train a task or the continual learning setting contains a large number of sequential tasks.
Additionally, the generative replay method can produce pseudo data as much as needed, which can also improve the robustness and flexibility of the model.

Further, the pros of generative replay are not limited to memory savings only.
In many practical applications, we cannot access real data due to privacy and security concerns, and generative replay is one of the few methods that is effective under these conditions.
This challenge makes generative replay methods more necessary in real-world scenarios.

[1] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[2] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[3] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[4] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[5] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[6] Fei Ni, et al., MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL, ICML 2023.

[7] Cheng Lu, et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.

[8] Qinsheng Zhang, et al., gDDIM: Generalized denoising diffusion implicit models, ICLR 2023.

[9] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[10] Rui Gao, et al., DDGR: Continual learning with deep diffusion-based generative replay, ICML 2023.

Reviewer comment (Experiments):
Limited baselines: Is comparison only made within the diffusion-based model generator pipeline? Why there is no comparison to previous continual RL algorithms provided?

Author response:
**Q3: Comparison to previous continual RL algorithms in addition to diffusion-based model generator pipeline.**

A3: As we stated in the paper, continual learning methods can be categorized into three kinds: regularization, parameter isolation (model expansion), and rehearsal.
Each of these three types has its own pros and cons, which aligns with the No Free Lunch Theorem in machine learning research.
For example, regularization methods can continually absorb new knowledge with a constant network capacity, while comprising additional regularization terms may lead to a tradeoff on the accomplishment of old and new tasks.
Model expansion methods can leave existing knowledge totally undisturbed via parameter isolation, while it consumes increasing network resources.

As we are the first to leverage expressive diffusion models to tackle the understudied CORL challenge, our primary concern is to validate whether our dual generative replay mechanism can realize high-fidelity replay of the sample space.
Hence, following the classical DGR method [9], we compare the performance of the offline RL model trained with three variants of replay methods: Oracle, None, and Noise.
For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity.
For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator.
For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism.

We did not compare our method to several existing continual RL methods due to the difference in the intrinsic characteristics and applicable scenarios.
For example, [11] belongs to the model expansion kind, [12] and [13] need to replay real samples of past tasks, and most of the existing methods [11, 12, 14] tackle the online RL problems other than the offline setting.
On the other hand, it is significant to conduct empirical comparisons to these methods to demonstrate their respective pros and cons, or to modify these online continual methods to adapt to offline settings.
We leave these for future work.

[9] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[11] Jean-Baptiste Gaya, et al., Building a subspace of policies for scalable continual learning, ICLR 2023.

[12] Maciej Wolczyk, et al., Disentangling transfer in continual reinforcement learning, NeurIPS 2022.

[13] Sibo Gai, et al., OER: Offline experience replay for continual offline reinforcement learning, ECAI 2023.

[14] David Rolnick, et al., Experience replay for continual learning, NeurIPS 2019.