Paper: Text-Aware Diffusion Policies (ICLR-2024)

Reviewer comment (Experiments):
The game environments are not natural images but the diffusion models are trained on natural images. How will this work in a game environment? I doubt the effectiveness of diffusion models in such a scenario. Can the authors show some generated images in this locomotion environment? Otherwise, I doubt the effectiveness of using diffusion models here.

Author response:
- On soundness: the reviewer raises the question of how diffusion models can be applied to frames produced by game environments, if the diffusion models are trained on natural images.  Indeed, the diffusion models are trained on natural images; however, they are also trained on cartoon images, paintings, movie posters, and more.  We believe that the large-scale, pretrained diffusion model is able to encode the essence of the environment.

Reviewer comment (Experiments):
If the video is not used, why doesn't the model get stuck into some best-matching frames? Since policy learning often involves some non-trivial procedures, it is not convincing if the authors only use text-to-image techniques but not video encoding methods. The matching should be conducted in the temporal space.

Author response:
- On avoiding getting stuck: we provide analysis into why leveraging a text-to-image generative model as a reward signal does not cause the resulting policy to get stuck in achieving some best-matching frames in Appendix A.1.  On the other hand, this is a common and natural pitfall when using alignment models such as CLIP as the supervisory reward signal to train a policy, as we both quantitatively and qualitatively demonstrate throughout the paper (in particular Walker, as well as the new Dog experiments).  We believe that the stochasticity stemming from the source noise causes the pretrained, frozen diffusion model to constantly change its current belief of what the "best-matching frame" is.  However, this does not result in a wildly unstable reward signal; as the diffusion model has learned a conditional *distribution*, these changing "best-matching frame" beliefs will always have high alignment with the provided text prompt.  Ultimately, for continuous locomotion prompts, where there is a large set of aligned images (e.g. for "walking" there are many valid poses of different configurations of swinging arms and legs), we find that TADPol learns to perform coherent, text-aligned motion.  On the other hand, as CLIP is deterministic, there is inherently a singular frame achievable by the policy that will have the highest alignment score with the text prompt.  Then, a policy optimized using a pretrained CLIP checkpoint will learn to seek and get stuck maintain that best-matching frame.

- On using video models: we would like to mention that we do compare against video methods: we perform experimentation of our approach against Vid-TADPol, which uses a text-to-video diffusion model in place of a text-to-image one for TADPol.  However, we surprisingly find that Vid-TADPol achieves comparable performance with TADPol in our experiments.  One potential explanation might be that learning to walk in the OpenAI Gym environment is relatively insensitive to the priors encoded in the text-to-video model beyond those found in a text-to-image model.  Another potential explanation is that Vid-TADPol is limited by the quality of currently existing text-to-video solutions.  Addressing this finding and exploring how natural videos can be used to supervise the learning of novel text-conditioned policies is exciting future work.

Reviewer comment (Experiments):
I see that the provided video demos do not have a very complex procedure. Most videos are nearly static and do not move much. This problem may be due to the use of images in the reward calculation.

Author response:
- On the complexity of video demonstrations: we provide additional results demonstrating TADPol on the complex Dog environment from the DeepMind Control Suite.  In these experiments, we show that TADPol is able to learn complex procedures (such as having the Dog stand on its hind legs or chase its tail).  Furthermore, we showcase in our updated experiments how TADPol can learn motion policies.

Reviewer comment (Experiments):
Why don't the authors compare their methods in other language-guided tasks such as language-guided navigation?

Author response:
- On comparisons to language-guided tasks: in this work we focus our efforts on learning text-conditioned continuous locomotion capabilities, and we show evidence suggesting TADPol is a promising approach towards this objective.  Text-conditioned continuous locomotion is difficult compared to goal-achieving tasks (including language-guided navigation) in that there is no canonical pose or goal frame that, if reached, would denote successful achievement of the task.  For example, there is no specific pose for “running” that when reached, would successfully represent completion of the “running” behavior; instead, agents must perform a coherent continuous sequence of poses over time (such as alternating stepping with the legs and swinging the arms) to convincingly demonstrate accomplishment of the task.  To our knowledge, there are no prior works that seek to tackle this objective.  Furthermore, the environments of many language-guided tasks are egocentric in nature; we leave it as interesting future work to explore whether or not pretrained diffusion models naturally understand how intermediate egocentric frames should be aligned with a desired text instruction.

Reviewer comment (Experiments):
Why do the authors construct a large model for shaping rewards? This requires a good explanation.

Author response:
- On utilizing a large model for rewards: to avoid potential confusion, the $k(t)$ term is not a neural network; it is a scaling parameter that can optionally change with respect to the level of noise $t$ used.  For our experiments we use $k(t) = \frac{1}{2} (1 - \prod_{i \leq t} \alpha_i )^2$; however, we note that in our experiments $k(t)$ can essentially be treated as a constant, since we only evaluate with a constant t (where $t=450$ in our experiments).  However, we include a $k(t)$ term in the TADPol equation to keep it general, allowing it to flexibly change with respect to $t$; in experiments where $t$ is resampled during training, this term provides the option to flexibly adjust the reward computation accordingly.  If the reviewer was wondering why a large model, namely a large-scale pretrained text-to-image diffusion model is useful to generate rewards, we believe that it is a promising way to scale up reinforcement learning models and unlock the learning of policies that align with priors and properties discovered from large-scale data.  Reinforcement learning development has long been isolated from the scaling benefits of data and architecture observed in other domains, such as vision, language, and their intersection.  By leveraging such models appropriately as a reward signal to supervise the learning of policies, priors from large-scale pretraining (such as what visually occurs naturally) and properties (such as strong multimodal or crossmodal alignment) can be transferred to expand the limits of what can be achieved through reinforcement learning.  For example, in this work, we demonstrate how such large-scale pretrained models can be leveraged to unlock the learning of policies that behave in alignment with text conditioning.

Reviewer comment (Experiments):
The authors need to compare with other works such as LangLfP, Text-Conditioned Decision Transformer, or Hiveformer.

Author response:
On comparison with text-conditioned work: we appreciate the reviewer for providing a list of related work.  We would like to mention that these works: LangLfP, Text-Conditioned Decision Transformer, and Hiveformer, all require training on datasets of trajectories that have been labeled with natural language.  Not only are such labeled datasets expensive to create, involving substantial human effort, but the resulting trained policies do not naturally transfer across environments.  Indeed, a labeled dataset of trajectories must be created for each novel environment, and across a large number of behaviors and associated text prompts within the environment.  In contrast, TADPol enables the learning of text-conditioned policies irrespective of visual environment, and without requiring any pretraining dataset of demonstrations or labeling whatsoever.  We do agree with the high-level sentiment of the reviewer, however, that additional comparisons with benchmarks would strengthen the work.  In the updated experiments, the results of which can be found in both the updated manuscript and website, we provide comparisons against CLIP-SimPol (Rocamonde et al. arXiv:2310.12921 [a]), LIV (Ma et al., ICML 2023 [b]), and Text2Reward (Xie et al., arXiv:2309.11489 [c]).  Such approaches leverage large-scale pretrained models to generate dense rewards for novel behaviors conditioned on text, and we apply them out-of-the-box to arbitrary RL environments.  We demonstrate that in the Dog environment from DeepMind Control Suite that TADPol is able to outperform the other methods in learning both goal-achieving behaviors as well as continuous locomotion behaviors.

Reviewer comment (Experiments):
As mentioned in section 4, the choice of noise step might be sensitive to the performance of the proposed method. It would be helpful if the authors could provide some experiments on the choice of noise step or even try to sample a range of noise steps to make the training more stable. Besides, the function k(t) in the defined function is not described clearly. Is the method sensitive to the choice of function k(t)?

Author response:
On the sensitivity to noise steps:  we agree with the reviewer that the choice of noise step might be sensitive to the performance of TADPol. In the updated experiments for the Dog experiment, we thoroughly investigate design choices surrounding the source noise in Appendix B.3.  We explore how the frequency of re-sampling the source noise affects the behavior: having a global source noise for all episodes results in more stable, CLIP-like behavior, having a source noise resampled for every frame results in more a more unstable reward signal and therefore policy, and having a source noise re-sampled for each episode is currently proposed as a happy medium.  Furthermore, we demonstrate that choice of noise level also affects the resulting policy; too high a noise level (such as using step 750 out of 1000), and the reward signal is no longer meaningfully understanding alignment with the generated frame.  We discover a “sweet spot” for the noise level exists around step 450 out of 1000 that is able to extract meaningful priors from the pretrained diffusion model with respect to the generated frame, while avoiding overcorruption.  We visualize the results of these ablation experiments in the updated website link.

- On the $k(t)$ function: for our experiments we use $k(t) = \frac{1}{2} (1 - \prod_{i \leq t} \alpha_i )^2$; however, we note that in our experiments $k(t)$ can be treated as a constant, since we only evaluate with a constant t (where $t=450$ in our experiments).  However, we include a $k(t)$ term in the TADPol equation to keep it general, allowing it to flexibly change with respect to $t$; in experiments where $t$ is resampled during training, this term provides the option to flexibly adjust the reward computation accordingly.

Reviewer comment (Experiments):
The method is only tested on a simulated environment within only three scenarios. It is hard to evaluate the method's effectiveness without testing on real-world scenarios.

Author response:
On testing on more advanced environments: we take note of the reviewer’s wish to see experimentation on more advanced environments.  We therefore report additional experiments on the Dog locomotion environment from the DeepMind Control Suite.  The Dog environment is complex in that it has a high degree of freedom of motion, with no task-specific design decisions, allowing the modeling of very flexible and complex behaviors.  We demonstrate that TADPol is able to learn goal-achieving behaviors (such as standing on its hind legs or chasing its tail), as well as continuous locomotion movements (such as walking) conditioned on natural language inputs, and invite the reviewer to look at the updated website for the latest visualization results.  The background visual environment of the Dog has been modified to appear more natural, and we take these promising results as positive signals for TADPol’s effectiveness in real-world scenarios.

Reviewer comment (Evaluation):
Why is the stick not always in the air if we provide instructions with the verb “jump up” since the rendered images will align more with the states in the air?

Author response:
On understanding the resulting policy: the reviewer wonders why the stick figure is not always in the air, if the instructions are "jump up", as intuitively rendered images of the stick figure in the air would align more with the text prompt.  We believe that this is avoided due to TADPol using a generative model (namely, a diffusion model) as the supervisory signal.  The large-scale pre-trained diffusion model has learned a distribution over images that align with a particular text prompt, which includes the stick figure at various heights in the air, or even on the ground but preparing to "jump up".  Which particular images in the distribution are most preferred at any particular time is determined by the source noise, which is re-sampled at each episode.  Therefore, TADPol is able to learn more coherent and complete motions that involve a variety of related poses to the text prompt through the learned priors of the pretrained diffusion model.  On the other hand, as elaborated upon in Section 4.2, and Appendix A.1, CLIP-based approaches are deterministic and therefore seek to achieve and maintain a singular frame that provides the highest alignment score.  This results in behavior closer to the hypothetical situation the reviewer posed; however, these approaches are thus naturally inconducive to supervising the learning of continuous locomotion capabilities with no canonical "goal-state" (such as dancing or walking), which we demonstrate TADPol can handle.

Reviewer comment (Evaluation):
The results with the velocity metric is a bit confusing. Why the diffusion-based ones are faster than the clip-based one? The reward function is to match the text descriptions, which does not imply the velocity.

Author response:
On the velocity metric:  we agree that the reward function seeks to match the text descriptions; however, when the text description involves motion (such as in the provided experiment "a stick figure walking across a grid floor"), it is worth evaluating ground-truth motion metrics such as velocity to gauge the level of motion achieved.  By visualizing the video produced by CLIP-SimPol, we qualitatively observe that the resulting agent does not move much from its initial position.  The purpose of the velocity and distance metrics is to quantitatively ground this observation, and explicitly show that CLIP-SimPol prefers to remain stationary in a position that has high alignment with the text prompt.  In comparison, TADPol achieves a higher total velocity and distance achieved, indicating successful motion and therefore higher alignment with the desired locomotion prompt of "a stick figure walking across a grid floor".  Further analysis into why CLIP-SimPol behaves in such a way and why TADPol potentially avoids this pitfall is explored in Section 4.2 and Appendix A.1.

Reviewer comment (Experiments):
How will the model perform if provided with descriptions like “move forward” and “move backward”. Will they generate different policies?

Author response:
On directional descriptions: as mentioned in Section 4.3 and Appendix A.2, a natural limitation of generating the reward signal solely from individual frames’ alignment with natural language is that there is no conceptual way to determine temporal-extended properties such as direction or speed.  Therefore, we do not expect vanilla TADPol to successfully learn different policies for "walking forward" and "walking backwards" at a conceptual level when such behaviors appear symmetric apart from direction.  However, in this work we propose two potential ways to address this problem: firstly, through a text-to-video diffusion model.  Conceptually, using text-to-video models under the Vid-TADPol framework is promising in that the video generated by the rolled-out trajectory can be evaluated with respect to distance and speed on top of alignment with a text prompt.  However, despite its potential, we discover that current text-to-video models do not outperform text-to-image models under the TADPol framework, most likely due to the quality of current text-to-video models themselves, which is an interesting future direction to explore.  Furthermore, as stated in Appendix A.2, we are aware that it is often desirable to learn policies that consider non-visual factors as well, such as minimizing control costs or the energy expended by the agent as it interacts.  Such information may not be naturally extractable from visual signals, even if videos are used.  In such cases, we believe TADPol can be augmented with other available reward signals that consider other factors beyond the visual domain.  In our updated rebuttal experiments, explained in the new Appendix B.4 section, we demonstrate that TADPol can be augmented easily and effectively - when adding a simple direction-agnostic speed reward to the TADPol reward for the prompt of “a dog walking”, the resulting learned Dog agent indeed is able to move quickly while moving in a natural walking motion.  We believe that a direction term can be a cheap augmentation on top of vanilla TADPol to guide it to “move forward” or “move backward” at will.  Further investigative efforts into how TADPol can be made aware of and utilize motion, temporal, and non-visual information is interesting future work.

Reviewer comment (Writing):
The writing of this paper requires improvement in a great deal.

Author response:
On writing: we appreciate the reviewer’s encouragement to improve the writing of this paper.  We update our draft to demonstrate and analyze new experiments to substantially reinforce the merits of our proposed approach.  We look forward to the reviewer’s helpful comments on the latest version of the work to improve the clarity of our paper.

Reviewer comment (Experiments):
There are few baselines for introducing diffusion models for policy learning and other methods (not diffusions) for reward generations.

Author response:
On baseline experiments: In our updated experiments (Section B in appendix) on the Dog environment, we evaluate TADPol against numerous other reward generation methods.  We compare against CLIP-SimPol (Rocamonde et al. arXiv:2310.12921 [a]), LIV (Ma et al., ICML 2023 [b]), and Text2Reward (Xie et al., arXiv:2309.11489 [c]), and demonstrate that TADPol is able to achieve superior performance in not only goal-achieving behaviors but also continuous locomotion.

Reviewer comment (Experiments):
The tasks are simple. What about some tasks with the DMControl suite, as other works do.

Author response:
On the simplicity of tasks: we agree with the reviewer that the merits of TADPol would be better showcased in complex environments such as DMControl.  We therefore provide additional experimental results on the Dog environment, a complex, flexible continuous control environment from the Deepmind Control Suite.  This environment is complex in that it has a high degree of freedom of motion, and no task-specific reset conditions, allowing the modeling of very flexible and complex behaviors.  We further split our analysis across two types of tasks: goal-achievement and continuous locomotion.  For goal-achieving tasks, we demonstrate that TADPol learns to successfully achieve a desired pose, such as "standing", "standing on hind legs", or "chasing its tail".  For continuous locomotion, we show how TADPol outperforms other approaches both qualitatively and quantitatively in learning motion.

Reviewer comment (Theory):
The reward signal lacks motion information or temporal information. For example, the diffusion model can not distinguish whether the walker is walking forward or backward. How to identify rewards in these scenarios?

Author response:
On motion and temporal information: as mentioned in Section 4.3 we agree with the reviewer that by generating the reward signal solely from individual frames’ alignment with natural language, a fundamental limitation is that there is indeed no conceptual way to determine temporal-extended properties such as direction or speed.  In this work we propose two ways to address this problem; firstly, by proposing and comparing against a text-to-video diffusion model.  Text-to-video models would be able to distinguish if the entire rolled-out trajectory indeed moves according to a desired direction or speed specification.  However, we discover that current text-to-video models do not outperform text-to-image models under the TADPol framework, which is an interesting future direction to explore.  Secondly, as stated in Appendix A.2, we are aware that it is often desirable to learn policies that consider non-visual factors as well, such as minimizing control costs or the energy expended by the agent as it interacts.  Such information may not be naturally extractable from visual signals, even if videos are used.  In such cases, we believe TADPol can be augmented with other available reward signals that consider other factors beyond the visual domain.  In our updated rebuttal experiments, explained in the new Appendix B.4 section, we demonstrate that TADPol can be augmented easily and effectively - when adding a simple direction-agnostic speed reward to the TADPol reward for the prompt of "a dog walking", the resulting learned Dog agent indeed is able to move quickly while moving in a natural walking motion.  We believe further investigative efforts into how TADPol can be made aware of and utilize motion, temporal, and non-visual information is interesting future work.

Reviewer comment (Evaluation):
Because the image is generated by the frozen diffusion model, it must be blurry and quite different from the current scene. In this case, the reward noise will be large, and RL is sensitive to the rewards. The reviewer is wondering why the generated rewards can share comparable performances to the vanilla ones and suggests doing some visualization to explain this phenomenon.

Author response:
On the reward noise: in TADPol, we do not generate a completely clean image from the frozen diffusion model.  Instead, the procedure corrupts the current scene with a Gaussian source noise, and seeks to re-predict the source noise (or analogously, the clean image) through the diffusion model, conditioned on the text prompt.  The insight of TADPol stems from the fact that if the text prompt is highly aligned with the clean image of the current scene as judged by the diffusion model, then the noise prediction will be highly accurate and the reward for the policy will be large.  On the contrary, if the image is un-aligned with the text prompt, then the text-conditioned denoising operation will struggle to predict the source noise (or reconstruct the current scene), and a low reward will be delivered to the policy.  Furthermore, all aggregated error between the prediction from the text-conditioned diffusion model and the ground truth source noise is converted into a reward signal that is always bounded between 0 and 1, by the equation provided in Section 3.  Therefore, the reward is always a stable value, mitigating RL’s natural sensitivity to reward values.

Reviewer comment (Experiments):
The inference of the diffusion model is at a quite low speed. The reviewer is wondering how computation-efficient this work is and whether it is necessary to generate rewards every step.

Author response:
On inference speed: whereas traditional, vanilla diffusion modeling is slow in inference speed due to the sequential denoising procedure, in TADPol we do not generate the completely clean image for reward computation.  Instead, we only utilize one denoising step to compute a reward for each timestep.  Because we do not perform any iterative procedures for reward computation, and only utilize pretrained models for inference, the reward computation is actually relatively fast.  As such, we are able to train our policy with dense rewards in reasonable wall-clock time; whereas it is interesting to consider generating sparse rewards through TADPol, we operated off the simple assumption that dense rewards are generally preferable.  A study on the merits of dense rewards vs sparse rewards as a design decision is interesting to the RL community at large, and is not limited to TADPol, and TADPol is certainly able to be easily converted into a sparse reward signal to reflect the results of such investigations.

Reviewer comment (Novelty):
the motivation of using simulator to render videos is unclear to me. Like in what realistic scenarios would such a method be useful? As to the best of my knowledge current simulators are not realistic in terms of the RGB they render aka sim2real gap. It's unclear to me in what end use cases would they be useful, also given that we have millions of videos available on the web widely.

Author response:
- On simulator-rendered videos: we agree with the reviewer that there are limited realistic scenarios that would utilize the output of a simulator-rendered video as a meaningful video, beyond some complex behavior demonstrations that might be too expensive or dangerous to perform and record in real life (e.g. how to evacuate a failing spacecraft, or how to perform a high-stakes surgical procedure).  We would like to clarify the motivation of using a simulator to render videos as a natural way to treat policies as a form of visual generative model.  By interpreting the policy as a model that can generate coherent images/frames (rendered through an environment), we can connect it with diffusion models, which also generate images but with additional text-conditioning capabilities.  TADPol can then be motivated as a way to distill these capabilities, acquired from large-scale data and architecture, from a pretrained diffusion model into a policy, to naturally unlock the learning of text-conditioned behavior.  We do so by training the policy to generate rendered frames that align with the text prompt as supervised by the priors of a pre-trained frozen diffusion model.  Using a simulator to render videos of a policy is therefore the motivating bridge that connects policy learning with existing large-scale pretrained models, and enables the learning of novel text-conditioned behaviors.

Reviewer comment (Experiments):
there are no comparisions with existing video rendering methods, thus making it unclear in what cases would they get better results. This point is linked with the above point.

Author response:
- On comparisons with existing video rendering methods: tied with the previous response on the practical usage of simulator-rendered videos, we agree that our goal is not really to render high-fidelity, natural-looking videos for downstream purposes.  Indeed, the visual quality of the policy is naturally limited by the strength of the environmental renderer.  Aware of these limitations, and as it is also not our main objective, we therefore do not compare TADPol against video rendering methods.  Instead, we believe the contribution of highlighting the inherent video-rendering capabilities of a policy in an environment is valuable as a paradigm shift; it exposes a natural bridge through which benefits of scale can be introduced to reinforcement learning, which has generally been siloed off from the scaling benefits observed in other domains such as vision and language.  In this work we demonstrate the benefits of incorporating large-scale data pretraining and architectures into reinforcement learning through TADPol, unlocking the learning of policies that are flexibly and accurately conditioned on natural language inputs.

Reviewer comment (Experiments):
The other motivation of the paper is learning robot behaviours, however there are mainly approaches previously proposed that do so such as : https://arxiv.org/abs/2310.12921 https://arxiv.org/abs/2203.12601 https://arxiv.org/abs/2210.00030. The paper fails to compare against any of them.

Author response:
- On prior works for learning robot behaviors:  We thank the reviewer for providing a list of previous approaches that learn robot behaviors, but highlight that we already compare against one of the listed works.
    - We would like to note that the first work listed is a concurrent work submitted to this year's ICLR, and was made public on arXiv after the submission deadline of this conference; thus there was no way to directly reference its work when writing this submission.  We not only agree that their work is directly relevant to our goals, we in fact do perform comparisons against their proposed approach.  In that work, they propose using a pre-trained, frozen CLIP model to provide a dense reward signal aligning visually rendered observations and text conditioning.  Independently and concurrently, we also created and implemented this approach, termed in our paper as CLIP-SimPol, and compared against it in all of our provided experiments.  We demonstrate that TADPol outperforms CLIP-SimPol, particularly in continuous control tasks such as locomotion, and further provide analysis and justification as to why this is the case (Section 4.2, and Appendix A.1).  We will make sure to update our references section to include their concurrent work.  In our new experiments on the Dog environment, prepared for the rebuttal, we further demonstrate how CLIP-SimPol underperforms in goal-achievement and continuous locomotion compared to TADPol.  The submission link, reviews, and comments for their work are publicly available on OpenReview as well.
    - The second work, R3M, is limited to imitation learning (specifically behavior cloning), and was not applied to reinforcement learning settings.  Therefore, the task is to imitate a performed behavior rather than learn a completely novel behavior directly from text specification, which TADPol seeks to do.  Furthermore, not only does R3M require demonstrations of the desired behavior, which TADPol (and other methods we seek to evaluate against) do not, R3M also requires a substantial pretraining dataset with human-labeled text annotations.  This requirement limits R3M’s application to novel environments, in the absence of an expensive human-labeling operation, whereas TADPol can be flexibly applied across many environments out of the box.
    - The last listed work, VIP, does not enable the learning of text-conditioned behavior.  Instead, tasks are specified through goal images.  In contrast, in this work we are interested in learning novel, text-conditioned behaviors, as text is a natural, flexible, and human interface to specify desired policy behavior.  On the other hand, we highlight LIV, a follow-up work to VIP, as a way to generate dense rewards from text specification.  In our additional rebuttal experiments on the Dog environment, we include comparisons against LIV, and both qualitatively and quantitatively show that it underperforms on both goal-achieving and continuous locomotion tasks compared to TADPol.  It is noteworthy that both VIP and LIV require pre-training; in particular, LIV requires text-annotated pretraining (they use EpicKitchen, a text-annotated video dataset).  On the other hand, TADPol can be applied to any arbitrary reinforcement learning environment with visual rendering capabilities out of the box, without any prior trajectories or human labeling (which is an expensive procedure).

- Additional note: we include, in our rebuttal experiments, an additional comparison against Text2Reward, where a reward function is generated entirely via a LLM.  This reward function has access to ground-truth state attributes, such as speed, direction, and forces on each joint.  However, we demonstrate both quantitatively and qualitatively in the updated manuscript and website that TADPol is able to outperform it in both goal-achieving and continuous locomotion tasks on the Dog environment.

Reviewer comment (Experiments):
The experiments are overall very thin. The tested tasks are very simple and not very numerous, which does not really convince me that TADPol is generally applicable or works consistently.

Author response:
On experimentation: we take note of the reviewer’s wish to see more thorough experimentation across more complex tasks, and present additional results on the Dog locomotion environment from the DeepMind Control Suite.  The Dog environment is complex in that it has a high degree of freedom of motion, and no task-specific reset conditions, allowing the modeling of very flexible and complex behaviors.  We demonstrate that TADPol is able to learn goal-achieving behaviors (such as standing on its hind legs or chasing its tail), as well as continuous locomotion movements (such as walking) conditioned on natural language inputs, and invite the reviewer to look at the updated website for the latest visualization results.

Reviewer comment (Writing):
While the paper is overall fairly clear, I feel that the title is a bit of a misnomer and the narrative constructed at the beginning of the paper caused some confusion. The method does not involve a diffusion policy at all; it leverages a text-to-image diffusion model, but only to provide rewards in a policy-agnostic way. I also found all of the talk about the policy as an implicit video-generating model distracting, as well as the discussion of video-generating diffusion models in the related work. I don't think these are fundamentally related to TADPol, since the end goal of the method has nothing to do with generative modeling, but is instead just about achieving good performance in traditional RL tasks. I really think the authors should edit the paper to make sure it does not claim to perform any sort of video synthesis since this is a gross overstatement.

Author response:
On the usage of "diffusion policy": the reviewer is correct that the policy does not synthesize predicted actions using a diffusion process; however, in this work we call our framework a "Text-Aware Diffusion Policy" because it is a policy optimized to respect a text-aware diffusion model.  In essence, this policy is a distillation of a pre-trained text-to-image diffusion model, transferring all the priors within the diffusion model relevant to the prompt and visual environment into a policy network.  Just as a diffusion model generates images from text prompts, so too does the resulting policy continually "generate" frames (by selecting actions and rendering the results through the environment) conditioned on the input text, in accordance with the text-to-image diffusion model it is distilled from.  As it is a diffusion model essentially converted into a policy network, we call it a "Text-Aware Diffusion Policy"; however, we are amenable to clarifying this name with an alternative, such as "Text-Aware Diffusion-Distilled Policy", and are open to further suggestions from the reviewer.

On video generation and reinforcement learning: we agree with the reviewer that the end goal of TADPol is about achieving good performance in traditional RL tasks, and that video synthesis is not the main contribution or story of the work.  However, we would like to defend the perspective and highlight the value in interpreting policies as implicit video-generators through an environment with rendering capabilities.  By treating the policy as a model that can generate coherent images/frames (where visual quality is admittedly at the mercy of the environment), it naturally provides a connection with diffusion models which also generate images but with the additional benefit of having excellent text conditioning properties.  This shared perspective inspires the TADPol framework, where we seek to distill the priors and text-alignment captured within a pre-trained text-to-image diffusion model, such that the resulting policy also generates frames that are now strongly aligned with human-provided text.  Whereas we do not treat policies as video renderers for the sake of using the output videos, it is precisely because we *treat* policies as a form of video generator that provides a clear motivation and intuition into TADPol.  Indeed, it is through this perspective that a path forward to incorporating the benefits of large-scale pretrained data and models into reinforcement learning, which has generally been siloed off from such scale, becomes promising; and in this work we preliminarily demonstrate such benefits, unlocking the learning of policies that are flexibly and accurately conditioned on natural language inputs.

We agree with the reviewer that the current writing structure and narrative may caused other reviewers confusion regarding what our main objective is, and could be improved.  We therefore take the reviewer's suggestion to rewrite the introduction to focus less on video generation, which is not the core of the work, and to instead motivate and highlight our main contribution of text-conditioned policy learning.

We agree with the reviewer's analysis that TADPol is indeed about using a generative diffusion model in a discriminative manner to provide text-conditioned rewards.  We hope that the reviewer finds our updated introduction, highlighted in blue, to be a more up-front, transparent, and accurate depiction of our work and contributions.