Paper: MADiff: Offline Multi-agent Learning with Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
This is a very interesting paper but there are unfortunate issues with the experimental design. SMAC (Version 1) has known issues. There is a recent version that was recently accepted at NeurIPS but has been released for a while that fixes the “open-loop” issues of SMACv1 called SMACv2[1]. Given that these tasks were essentially solved by the community within the easy maps that are compared with in the results, the results may not be as convincing as currently indicated.

Author response:
**Q1: "Given that these tasks were essentially solved by the community with in the easy maps that are compared with in the results, the results may not be as convincing as currently indicated."**

**A1:**
Thanks for your comment, but you might misunderstand our experimental settings. Our method and all baselines, including QMIX, are trained in a fully offline manner. Offline Policy learning is far more challenging than online learning due to limited data support and sub-optimal behavior policies used to collect the dataset. 

One task can be solved by online RL methods does not mean it can be solved at the expert level with offline learning. For example, PPO [1], which was proposed in 2017, can easily achieve expert-level performances in single-agent mujoco environments (Hopper, HalfCheetah, ...). However, the widely-used D4RL benchmark [2], which was released in 2020, still uses those environments to build their dataset. And the SOTA algorithms today can only obtain about half of expert scores on D4RL's halfCheetah-medium dataset [3].

As a result, our results under offline MARL settings should not be compared with online MARL methods. 

**Q1: The [1] benchmark that provides the data also has an SMACv2 available ... Have you tested your algorithm with that method?**

**A1:** Thanks for your comment! We totally agree that SMACv2 is an important benchmark, and we are running experiments on datasets of zerg_5_vs_5 and terran_10_vs_10 from SMACv2 and will update the results to the manuscript after they are available.

May we explain the reasons of why we chose SMACv1 rather than SMACv2 in the first place:

+ Prior works in offline MARL reported their results on maps from the SMACv1 environment rather than v2;

+ In the released off-the-grid dataset, each SMACv2 map only comes with one *Replay* dataset, whereas each v1 map has three available datasets: *Good*, *Medium*, and *Poor*.

On the other hand, we would like to discuss the impacts of the stochasticity from environments on our algorithm:

+ The initial states in MPE tasks are randomly sampled, and MADiff clearly surpasses baseline algorithms on those tasks;

+ Diffusion models are pretty good at modeling complex distributions, and intuitively, MADiff will not suffer much from the stochasticity of SMACv2 compared to the baseline algorithm.

Reviewer comment (Evaluation):
Why is the score for SMAC so low? Typically the SMAC reported value is winrate, which should be near 100% for these easier SMAC tasks, especially with QMIX. There must be a bug in the evaluation or the data is far too limited to evaluate the method properly.

Author response:
**Q2: "Please indicate the meaning of the score. Do the offline methods learn from adequately performing data? Based on [1], the relationship between winrate and score is unclear."**

**A2:** We apologize for not fully understanding your question in prior responses. 

The scores we reported are the undiscounted summations of per-step team rewards in an episode. According to the original paper introducing the SMAC benchmark [1], per-step rewards are computed as the hit-point damage dealt and bonuses of 10 and 200 points for killing each enemy unit and winning the game, respectively. The rewards are scaled so that the maximum cumulative reward (score) achievable in each scenario is around 20. We checked the SMAC environment code (https://github.com/oxwhirl/smac) used in our experiments and confirmed that it is consistent with the above description.

From the magnitude of the different reward terms, it can be seen that winning or not significantly impacts score calculations. Compared to win rate, average scores can distinguish between failed episodes, since the number of enemies killed and hit-point damage dealt are considered.

As can be seen from the added violin plots in Appendix C, datasets in the same map have different return/score distributions. *Good* datasets consist of a majority of expert-level trajectories with scores around the maximum score of 20. Only a few trajectories in *Meidum* datasets are optimal, and *Poor* datasets do not include expert trajectories.

Reviewer comment (Experiments):
Regarding the other experimental design items, it is unclear what hypothesis the MPE environments test that would not be tested by SMAC.

Author response:
**Q2: "Regarding the other experimental design items, it is unclear what hypothesis the MPE environments test that would not be tested by SMAC."**

**A2:**
Both MPE and SMAC datasets are widely adopted by existing works in offline MARL [4, 5]. These two classes of environments have distinct state/action spaces and require different cooperation patterns. We include both of them to ensure a thorough evaluation of our method. Two intuitive differences are that the action space of MPE is continuous, and the initial state distributions are more diverse than SMAC.

Reviewer comment (Experiments):
For the MATP, this seems like a niche task. It is unclear if the test games have a distribution of trajectories that are out of distribution of the training/validation set. Otherwise, it is hard to understand the generalization capability of the method.

Author response:
**Q3: "It is unclear if the test games have a distribution of trajectories that are out of distribution of the training/validation set."**

**A3:**
Thanks for your good question! We adopt the standard training/validation/test split in MATP experiments, which allows for out-of-distribution (OOD) tests. On offline MARL tasks, our evaluation procedure follows the existing works on offline RL [6, 7] and MARL [4, 5] and does not deliberately test the generalization ability. 

We want to highlight that different from static supervised learning tasks in MATP, evaluating offline-learned policies by interacting online naturally includes the requirement of OOD generalizations. Due to errors induced by functional approximations and stochasticity in environmental dynamics, agents will encounter OOD states even if they are started from in-distribution states. The learned policies must have the ability to recover from OOD states and prevent errors from accumulating over time.

Designing reasonable benchmarks for specifically evaluating the generalization ability of offline MARL models is less discussed yet, and we acknowledge that it is an important future research direction.

Reviewer comment (Experiments):
How does the centralized control version scale with the number of agents? It is unclear if this is a useful avenue of exploration compared with the CTDE paradigm.

Author response:
**Q4: "How does the centralized control version scale with the number of agents? It is unclear if this is a useful avenue of exploration compared with the CTDE paradigm."**

**A4:**
Both versions of MADiff have their own values. MADiff-C is only applicable in cases where a centralized controller for all agents is allowed, such as traffic light control or fleet control. Since all agents' actions are made jointly, it is easier to keep their behaviors coordinated. Therefore, MADiff-C is supposed to perform better compared to MADiff-D in those cases, which is aligned with our experiment results. However, there are more scenarios where centralized controls are impossible, e.g., autonomous driving and distributed sensor networks [8]. MADiff-D, which is the CTDE version of MADiff, is designed for these scenarios.

MADiff-C and MADiff-D scale the same as the number of agents increases. We use a shared U-Net model for all agents, where different agents' data can be batched together and passed through the network. It does not cost much more inference time as the number of agents increases with GPU-accelerated computing. For self-attention among agents' embeddings, it can also scale with linear complexity [9].

When the number of agents is extremely large, we can modify the opponent/teammate modeling to operate in a projected low-dimensional space with a latent diffusion model [10]. We leave this direction to future work.

Reviewer comment (Writing):
I still do not understand the “Decentralized policy and centralized controller” terminology. I assumed that this means centralized training decentralized execution (CTDE). The new language is confusing.

Author response:
**Q5: "I still do not understand the “Decentralized policy and centralized controller” terminology. I assumed that this means centralized training decentralized execution (CTDE). The new language is confusing."**

**A5:**
As we mentioned in **A4**, decentralized policy and centralized controller refer to different control styles, which correspond to MADiff-D and MADiff-C, respectively. MADiff-D can be classified as a CTDE method, where the diffusion model is trained on all agents' data and used by each agent independently during execution.

The terminology of "decentralized policy" and "centralized controller" is not novel in the domain of MAL [11, 12]. We confirm that our usage of these terms is consistent with existing literature.

Reviewer comment (Experiments):
Is an attention network used for all comparison baselines?

Author response:
**Q6: "Is an attention network used for all comparison baselines?"**

**A6:**
Thanks for the question. In general, the network design of baseline algorithms follows their original paper, and we have included a detailed description of baseline implementations in the Appendix. 

**Q3: "Regarding Q6, I do not believe that this is a fair comparison with respect to empirical comparison. It is hard to say whether the diffusion aspect or the attention aspect is helping the method increase the performance ... but I am having trouble being convinced that the results are strictly due to the diffusion aspect."**

**A3:**

Thank you for the questions. 

First, we need to clarify that both diffusion modeling and attention networks are indispensable components of our proposed algorithm. We will highlight this point in the revised manuscript based on our ablation results.

In our reply to **Reviewer Khwp**'s Q5, we compared the performances of MADiff and ConcatDiff in the CTCE setting. ConcatDiff is an ablation variant that does not use attention networks but directly concatenates all agents' observations as input to a U-Net. Results show that MADiff-C clearly outperforms ConcatDiff, suggesting that **directly migrating a single-agent diffusion network structure to a multi-agent learning problem will not yield good results**. To validate whether this argument applies in the CTDE setting, we are running experiments with ConcatDiff-D where other agents' observations are masked before concatenation during training, and we will soon post the results of its comparison with MADiff-D.

Baseline algorithms based on multi-agent Q learning typically use single-agent independent actors, and only their centralized critics can incorporate attention modules. A strictly aligned comparison of network designs is impossible since critics are only used during training, and our diffusion network is used for training and online rollouts. Besides, to the best of our knowledge, **no existing work uses attention-based critics in the context of offline MARL**. Besides Q learning methods, MADT views MAL problems as sequence modeling and uses a fully attention-based network. In our added experiments, MADiff performed better than MADT in most datasets.

We are delighted to share with you some new ablation results!

As mentioned in **A3**, to test whether both diffusion modeling and attention networks are important in the success of MADiff, we compare our network design to ConcatDiff, which is an ablation variant that does not use attention networks but directly concatenates all agents' observations as inputs to a U-Net. ConcatDiff-C is used for centralized control, and ConcatDiff-D masks other agents' observations before concatenation during training. Results are presented as follows:

| Dataset          | OMAR            | ConcatDiff-D    | MADiff-D        | ConcatDiff-C | MADiff-C    |
| :--------------- | :-------------- | :-------------- | --------------- | ------------ | ----------- |
| Spread-Expert    | 114.9 $\pm$ 2.6 | 106.4 $\pm$ 3.5 | 98.4 $\pm$ 12.7 | 112.7 $\pm$ 4.3  | 114.7 $\pm$ 5.3 |
| Spread-Md-Replay | 37.9 $\pm$ 12.3 | 31.5 $\pm$ 2.7  | 42.9 $\pm$ 11.6 | 32.2 $\pm$ 3.5   | 47.2 $\pm$ 6.6  |

We can observe that ConcatDiff performs worse than MADiff in both CTCE and CTDE cases, and underperforms OMAR, which is the best baseline method in MPE datasets.

We sincerely hope that our responses and the additional experiments we conducted have addressed your concerns and enhanced the quality of the paper. We would greatly appreciate it if you could clarify any remaining concerns you may have.

Reviewer comment (Novelty):
The overall structure of the paper and algorithm is very similar to Decision Diffuser (Ajay et al. 2023), but with an additional attention layer.

Author response:
It is indeed that many successful design choices of [1] have inspired us and guided us to derive our method. Similarly, much of [1] also referred to the design of diffuser [2], but that doesn't take away from the fact that it's an excellent research.

Compared with [1], MADiff, which is the first to apply diffusion models in offline MARL, is naturally designed for multi-agent problems, taking multi-agent coordination, opponent modeling, and trajectory planning in a unified framework. **Our contributions include but beyond the model design**, and the most important one is that the proposed framework allows the use of the same model structure to handle various tasks (CTCE, CTDE, MATP) by flexible conditioning during evaluation and achieves superior results on those tasks.

Reviewer comment (Experiments):
The benefits of using diffusion policies mentioned in Decision Diffuser, such as generating novel behaviors, stitching or different conditions for $y(\tau)$ such as constraints or skills are not addressed in MADiff in the context of MARL.

Author response:
Compared to single-agent learning, multi-agent learning problems themselves are more challenging and can benefit from diffusion modeling. Specifically, besides the variety of a single agent's behavior, the interplay between multiple agents exhibits a multi-modal and complex nature. Our goal is to introduce DM with powerful modeling ability of arbitrary distributions to MAL for better learning multi-agent interactions in offline datasets. Verifying whether skill composition in SAL still applies in MAL settings is an interesting trial, but it is out of the scope of our paper.

Reviewer comment (Experiments):
Only a small number of agents (3-5) are considered in the experiments.

Author response:
Most existing works in offline MARL do not consider tasks with a large number of agents. For example, MADT-KD [5] and OMAR [6] are tested on environments with up to 8 agents. Moreover, OMAR uses eight multi-agent environments for benchmarking, yet only two of these consist of more than 5 agents. Despite the fact that we consider at most 5 agents in MARL experiments, our MATP experiments are conducted on NBA datasets with 10 agents, and MADiff demonstrates superior performance than the baseline method.

Reviewer comment (Novelty):
While opponent modeling is mentioned as one of the contributions, previous work in opponent modeling are missing from the Related Work section (e.g. [1] and references therein) and there is no explicit mention about what kinds of information are available at test time (e.g. only local observation-action history of ego-agent).

Author response:
Thanks for your suggestions! We have included a discussion of previous opponent modeling literature in the related work section.

Reviewer comment (Experiments):
OMAR is excluded from SMAC tasks as a baseline despite OMAR also reporting SMAC results.

Author response:
Although the OMAR paper reports results on SMAC, the authors' implementation of OMAR (https://github.com/ling-pan/OMAR) only contains the algorithm on continuous-action environments. Also, they have not released the dataset they used for SMAC experiments. Since OMAR relies on zeroth-order optimizations on the action space, the modifications required for handling discrete actions are non-trivial. 

We tried our best to reimplement a discrete version of OMAR, but the results on our SMAC datasets are much worse than other baseline methods. Therefore, we decided to exclude OMAR from SMAC datasets. It is worth noting that MA-ICQ and MA-CQL are both strong baselines, and we believe the results are sufficient to demonstrate the competitive performances of our method.

Reviewer comment (Writing):
(Minor) The perturbed states $\tilde x$ in $\hat \tau_k := [s_t,\tilde x_{t},... ]$ above Eq. 4 are not properly defined.

Author response:
Thanks for your careful checking! We have fixed the grammar error and revised the notation of denoised samples.

Reviewer comment (Writing):
(Minor) Grammatical error in 5.4: “We that..”

Author response:
Thanks for your careful checking! We have fixed the grammar error and revised the notation of denoised samples.

Reviewer comment (Experiments):
Because MADIFF needs to predict the next observations (Equations 8 and 9), I would like to ask whether MADIFF can be applied to domains with high-dimensional inputs (e.g., image). I noticed that the experimental domains have relatively small dimension inputs.

Author response:
Thanks for your valuable suggestions! MADiff can be applied in environments with high-dimensional, e.g., image observations with more complicated encoder and decoder networks such as CNN or vision transformers. There are some successful cases of applying diffusion planning to single-agent and image-based tasks [1, 2], which can be incorporated into the MADiff framework. We leave this to future work and add a brief discussion to the conclusion of this paper.

Reviewer comment (Experiments):
In Equation 9, inferring all other agents' observations could be very difficult when $N$ is large. Could MADIFF scalability benefit by inferring only the agent $i$'s neighbors instead of all $N$ agents?

Author response:
That is a good point, and your suggestion is valuable in boosting the scalability of MADiff. Selective opponent modeling is necessary in scaling to environments with very large N, and only inferring agent i's neighbors can reduce computation and modeling costs. However, neighbors defined under simple Euclidean distance may not be reasonable in some environments, while a learning-based distance metric would be more desirable. Another possible choice is to map the trajectories of all other agents to a low-dimensional embedding space and perform latent diffusion on it. Those modifications required are non-trivial, and we leave this direction as part of future work.

Reviewer comment (Theory):
Can MADIFF be applied to general-sum settings or would it be limited to cooperative settings only?

Author response:
The MADiff framework can only be applied to cooperative tasks, which is the same as most prior works in offline MARL [3, 4]. Naive offline RL or generative modeling methods are likely to fail on non-cooperative tasks, since the optimal policy learned from static datasets can be exploited by other agents during evaluations if they change their policies to be different than behavior policies in datasets. Effective offline MARL in multi-player general-sum settings requires offline equilibrium finding (OEF), which is particularly challenging due to limited data support. Very little work has attempted to solve OEF and is limited to simple tasks with discrete state and action spaces [5].

Reviewer comment (Novelty):
The novelty of the work seems a bit limited. MADIFF heavily relies on the existing work DecisionDiffuser (Ajay et al., 2023), and it can be seen as its simple extension to the MARL setting, with an additional attention layer that processes information exchange among agents. Adopting an attention mechanism for multi-agents itself does not seem to be a new idea. (e.g. [1,2])

Author response:
**A1:** It is indeed that many successful design choices of [1] have inspired us and guided us to derive our method. Similarly, much of [1] also referred to the design of diffuser [2], but that doesn't take away from the fact that it is an excellent work.

Compared with DM works in single-agent learning, MADiff is naturally designed for multi-agent problems. By using different conditioning during evaluation, the same framework can handle multi-agent coordination, opponent/teammate modeling, and joint trajectory prediction, which is supported by strong experimental results. We believe that the references in the single-agent learning domain do not diminish the novelty of our approach in the context of multi-agent learning.

We acknowledge that the attention mechanism has been used for years in MAL, and we verified that it is also effective in conditional generative modeling of multi-agent trajectories.

Reviewer comment (Experiments):
More baseline (MADT-KD [3]) could have been compared in the experiments.

Author response:
**A2:** Thank you for your suggestion! To our knowledge, MADT-KD does not have an open-sourced implementation, and it requires non-trivial changes on top of MADT's training code. We include MADT [3] as an additional baseline on SMAC datasets, and the results are pasted below:

| Dataset     | MADiff-D       | MADiff-C       | MADT           |
| :---------- | :------------- | :------------- | :------------- |
| 3m-Good     | 18.8 $\pm$ 0.2 | 19.7 $\pm$ 0.1 | 19.0 $\pm$ 0.3 |
| 3m-Medium   | 17.2 $\pm$ 0.3 | 18.4 $\pm$ 0.2 | 15.8 $\pm$ 0.5 |
| 3m-Poor     | 11.2 $\pm$ 0.1 | 11.8 $\pm$ 1.0 | 4.2 $\pm$ 0.1  |
| 5m6m-Good   | 16.5 $\pm$ 0.3 | 18.1 $\pm$ 0.1 | 16.8 $\pm$ 0.1 |
| 5m6m-Medium | 16.3 $\pm$ 0.1 | 17.6 $\pm$ 0.4 | 16.1 $\pm$ 0.2 |
| 5m6m-Poor   | 10.3 $\pm$ 0.5 | 11.0 $\pm$ 0.3 | 7.6 $\pm$ 0.3  |

We also updated Table 1 to cover those results.

Reviewer comment (Presentation):
In the top-middle of Figure 2, why is the planned trajectory of the planning agent (purple) different from the real sampled trajectory?

Author response:
**A3:** The plans on the top row are done at the early stage of an episode, and decisions are made in a decentralized way. Therefore, the purple agent at that time did not have much information about other agents' intentions. Then the purple agent may make conflicting plans with the red one, i.e., both want to cover the same landmark in the middle. 

As the purple agent gathers more information during execution, it identifies the red agent's inclination toward the middle landmark. Thus, the purple agent's opponent U-Net model tends to generate a future trajectory for the red agent, which arrives at the middle landmark. And because of the attention mechanism, the purple agent made corrections when generating their own plans (bottom-middle subfigure). The plans of all the agents in the bottom-middle row become conflict-free, and the conflicts are eliminated in real sampled trajectories.

We have revised the explanation in Section 5.4 to make it clearer.

Reviewer comment (Theory):
MADIFF relies on the inverse-dynamics model (IDM) for action selection, but it can be suboptimal when the transition dynamics are stochastic. In MARL, even though the underlying environment is deterministic, if we see the other agents as a part of the environment (thinking of decentralized execution), the transition dynamics (by other agents' actions) will be stochastic when other agents' policies are stochastic. Even in this situation, doesn't using IDM cause any problems?

Author response:
**A4:** Thanks for your good question! We will separately discuss using IDM in stochastic transitions and transitions dependent on other agents' actions.

First, we have to note that the stochastic state transition function $p(o'|o, a)$ does not necessarily lead to a stochastic inverse dynamics transition $p(a|o, o')$. Most of our tested environments do not exhibit high stochasticity in inverse dynamics, which can be effectively fit by state-conditioned Dirac distributions in practice. For potential environments with highly stochastic inverse dynamics, we can modify the IDM to output parametrized Gaussian distributions, which is similar to state-to-action policies used in stochastic environments.

Indeed, in multi-agent environments, the transition dynamics might depend on other agents' actions, which can be stochastic. However, as we mentioned before, the IDM is only affected by the stochasticity of $p(a|o,o')$ rather than $p(o'|o,a)$. The objective of IDM is as simple as providing the feasible action given the current observation and the next observation predicted by the diffusion model, and the diffusion model is good at modeling stochastic distributions. The effect of other stochastic agents is already encoded in the generated next observation, which is the **input** of IDM. Therefore, such stochasticity does not make the learning objective of IDM much harder.

Reviewer comment (Novelty):
If we view the MARL problem as a single-agent problem (by viewing joint action space as a single-agent large action space & joint observation space as a single-agent observation space), how is MADIFF-C different from DecisionDiffuser? Is MADIFF-C still performing better than DecisionDiffuser which operates in the joint observation/action space?

Author response:
**A5:** If centralized control is allowed, the differences between MADiff and DecisionDiffuser operated on the joint state space are mainly in the model architecture. A naive approach to transforming multi-agent problems to fit into the DecisionDiffuser framework is concatenating all agents' observations together. MADiff adopts a more efficient way to model interactions between agents, where each agent's trajectory is processed by a single-agent U-Net, and self-attentions on agents' latents are introduced in decoder layers.

We argue that the advantages of using attention modules can be explained in two aspects:

+ First, using the attention mechanism to model multi-agent interactions has been proven to be more effective than concatenation in prior MARL studies [3, 4]. To test whether such superiority also applies in diffusion modeling, we benchmark the model variant that uses a single U-Net to process concatenated multi-agent observations without using attention, which is denoted as ConcatDiff. We compare ConcatDiff to MADiff-C on two MPE spread datasets, and the results are summarized below.

  | Dataset          | ConcatDiff      | MADiff-C            |
  | ---------------- | --------------- | ------------------- |
  | Spread-Expert    | 112.7 $\pm$ 4.3 | **114.7 $\pm$ 5.3** |
  | Spread-Md-Replay | 32.2 $\pm$ 3.5  | **47.2 $\pm$ 6.6**  |

  We can see that MADiff-C outperforms ConcatDiff in those datasets thanks to a better model design.

+ Second, some MARL tasks require index-free control on a set of homogeneous agents [5], i.e., there is no predefined order of agents. Using attention without positional encodings to model multi-agent interactions can naturally handle the index-free setting, while the concatenation operation is ordered and requires additional designs to fit in those tasks.

Reviewer comment (Writing):
The phrase "more high-frequency and less smooth nature of actions" is unclear — what does it mean for actions to be "high-frequency"?

Author response:
**Q1: "What does it mean for actions to be 'high-frequency'?"**

**A1:**
We are sorry for the confusion. The term "frequency" refers to using the Fourier transform to transform the state or action sequences in the time domain to the frequency domain. Action sequences are less smooth across time steps and thus have a lot of high-frequency components after transformation. 

To make it clearer, we have removed the statement about action frequency.

Reviewer comment (Writing):
The diffusing process is said to condition on information $y(\tau)$, which based on Fig. 1 includes "Returns" and "Other information". In that case, why are there multiple returns? Is it expected future discounted returns, or final returns for that trajectory? How are the returns encoded? What is the extra information?

Author response:
**Q2: About "Returns" and "Other information"  in Figure 1.**

**A2:**
We are sorry for the confusion. During training, "Returns" in Fig. 1 is set to the cumulative discounted reward starting from time step $t$. In some environments, agents can have different returns at some time step, e.g., penalties on the specific agent that collides with others in MPE, thus we use the plural form to denote returns for each agent. During evaluation, we set returns to the same high-enough value for all agents. Before input to the U-Net residual blocks, Returns are passed through an MLP and concatenated with the diffusion time step embedding. 

"Other information" is task-related condition variables other than returns. For example, in the NBA dataset, we need player embeddings and ball positions as inputs. "Other information" is passed through a separate embedding network and concatenated with return embeddings. We have revised the caption of Fig. 1 to include descriptions of these two terms.

Reviewer comment (Writing):
In Section 3.1, it is unclear how the attention mechanism incorporates the information of other agents, and how this is supposed to help coordination.

Author response:
**Q3: "it is unclear how the attention mechanism incorporates the information of other agents, and how this is supposed to help coordination"**

**A3:**
The attention mechanism helps coordination in both centralized control and decentralized execution settings. 
When centralized control is allowed, the attention module enables the diffusion model to jointly plan trajectories for all agents, which is essential in tasks that require coordination.

In a decentralized execution setting, although different agents make decisions independently, they can use the same attention network to jointly infer other agents' future trajectories and plan their own trajectories. Since the attention network is trained in a centralized manner, the generated trajectories for all agents should be consistent. Intuitively, the consistency constraint requires the ego agent to first think at a higher level, i.e., make a coordinated plan for all agents, and then place itself in that plan. The coordination among agents can be improved by performing "imagined" planning for others (i.e., opponent modeling).

Reviewer comment (Writing):
The term "opponent" or "opponent modelling" is used repeatedly, despite the work only dealing with cooperative settings.

Author response:
**Q4: "the term "opponent" or "opponent modelling" is used repeatedly, despite the work only dealing with cooperative settings."**

**A4:**
Thank you for pointing this out. We use the term "opponent" to refer to other teammates in cooperative games, which is also adopted in some prior works [1, 2]. Since most methods in modeling adversary agents in competitive games can be used in modeling teammates, using the same term "opponent" can ensure better coverage of related works.

Reviewer comment (Writing):
Section 5.4 is unclear overall, including exactly what the Valid Ratio is measuring.

Author response:
**Q5: "Section 5.4 as a whole is unclear to me, including exactly what the Valid Ratio is measuring."**

**A5:** We are sorry for the confusion and have rewritten Section 5.4 to make it more clear.

Reviewer comment (Presentation):
Figure 3 has no error bars.

Author response:
**Q7: Issues concerning Figure 3 and Table 1.**

**A7:**
Thank you for your careful check! Results are computed with 3 different seeds. In the revised paper, we mention the number of seeds in the captions of Fig. 3 and Tab. 1. The error bar is added to Fig. 3.

Reviewer comment (Reproducibility):
There is no mention of the number of seeds used for Table 1 or Figure 3.

Author response:
**Q7: Issues concerning Figure 3 and Table 1.**

**A7:**
Thank you for your careful check! Results are computed with 3 different seeds. In the revised paper, we mention the number of seeds in the captions of Fig. 3 and Tab. 1. The error bar is added to Fig. 3.

Reviewer comment (Evaluation):
The results for SMAC seem underwhelming, especially since SMAC lacks relevant stochasticity and partial observability.

Author response:
**Q8: "The results for SMAC seem underwhelming, especially since SMAC lacks relevant stochasticity and partial observability."**

**A8:**
To our knowledge, each agent in SMACv1 can only receive local observations within a fixed sight range [6], which makes the environment partially observable. Although SMACv1 lacks relevant stochasticity, we highlight that both MA-ICQ and MA-CQL are strong baselines in offline MARL, and MADiff achieves competitive results.

Reviewer comment (Experiments):
While the use of two different MARL settings and the NBA dataset is welcome, the paper does not evaluate MADiff in settings where coordination is particularly challenging (e.g., due to stochasticity, partial observability, or both) and does not acknowledge any potential limitations of the method in such settings.

Author response:
**Q9: "the paper does not evaluate MADiff in settings where coordination is particularly challenging, either due to stochasticity, partial observability or both."**

**A9:** 
Thanks for your good question, but you might misunderstand our experiment settings. Although we did not consider environments with both stochasticity and partial observability, either of them alone was present in our experiments. In particular, partial observability exists in SMAC due to limited sight range, and MPE has stochasticity in initial state distributions. Moreover, since we consider offline multi-agent policy learning from static datasets, it is challenging for agents to effectively coordinate during online evaluations.

Reviewer comment (Experiments):
The authors are encouraged to consider environments such as SMACv2 [1], Multi-Agent Mujoco [2], or Hanabi [3] for evaluating coordination in more challenging scenarios.

Author response:
**Q10: "For instance, I encourage the authors to consider environments such as SMACv2, Multi-Agent Mujoco or Hanabi."**

**A10:**
Thanks for your suggestions! We evaluate MADiff on Multi-agent Mujoco datasets in off-the-grid MARL and results are presented as follows:

| Dataset             | BC         | TD3+BC | OMAR | MADiff-D |
| ------------------- | ---------- | ------ | ---- | -------- |
| 2halfcheetah-Good   | 6846 $\pm$ 574 | 7025 $\pm$ 439 | 1434 $\pm$ 1903 | **8254 $\pm$ 179** |
| 2halfcheetah-Medium | 1627 $\pm$ 187 | **2561 $\pm$ 82** | 1892 $\pm$ 220 | 2215 $\pm$ 27 |
| 2halfcheetah-Poor   | 465 $\pm$ 59 | 736 $\pm$ 72 | 384 $\pm$ 420 | **751 $\pm$ 14** |
| 4ant-Good           | 2802 $\pm$ 133 | 2628 $\pm$ 971 | 344 $\pm$ 631 | **3090 $\pm$ 26** |
| 4ant-Medium         | 1617 $\pm$ 153 | **1843 $\pm$ 494** | 929 $\pm$ 349 | 1679 $\pm$ 93 |
| 4ant-Poor           | 1033 $\pm$ 122 | 1075 $\pm$ 96 | 518 $\pm$ 112 | **1268 $\pm$ 51** |

Reviewer comment (Experiments):
Have the authors tried training MADiff on combined datasets of mixed quality (e.g., Good+Medium+Poor)? It is suspected that the additional data would benefit MADiff the most by improving the inverse dynamics model without hurting the quality of the generated trajectories when conditioning on high return.

Author response:
**Q12: "Have the authors tried training MADiff on combined datasets of mixed quality? "**

**A12:**
The qualities of SMAC datasets are already mixed. The naming of datasets in off-the-grid MARL datasets (Good, Medium, Poor) does not mean that each of them is sampled by a single policy but indicates the overall quality of the dataset. In fact, each dataset is collected by three sets of independently trained policies with different levels of expertise [3]. We have included Fig. 5 in the Appendix to show the return distributions of each dataset, and briefly describe how those datasets are collected.