Paper: Federated Ensemble-Directed Offline Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
What role does the entropy regularizer in Eq. (5) play? Collapsing to datasets with the best performance does not seem to worsen the convergent performance of FEDORA. Is there any ablation study on this term?

Author response:
**Response:** The role of the entropy regularizer is to modify the ensemble scheme of our algorithm.  When $\beta=0$, it reduces to a uniform weighting scheme, where the quality of data present in each client is not considered during federation. As $\beta \rightarrow \infty$ it tends to a max weighting scheme, where the federated policy is the same as an individual client's policy with the highest quality data. In response to your question, we have **now conducted additional experiments** with a hyperparameter sweep for different values of $\beta$, see Appendix E.3.

Reviewer comment (Writing):
The approach toward computing $Q_{t_i}$ and $\pi_{t_i}$ in Section 5.1 is not clearly described. The evaluation of Q functions and policy may be inaccurate on datasets of low quality. How will it influence the performance of FEDORA?

Author response:
**Response:** $Q_i^t$ and $\pi_i^t$ are the local critic and actor at client $i$ in round $t$. It is computed by minimizing the loss in Eq. 9 and Eq. 10 respectively. We follow an ensemble learning approach where in we weigh clients according to the quality of their datasets, and initialize the local critic and actor during each round of federation to the federated actor and critic. This ensures that each client starts off with the knowledge possessed by clients participating in federation. We then eventually down weigh clients who perform poorly  and decay the influence of local data. Thus, although a client individually has low quality data, using actor and critic trained over the ensemble to initialize the training process, and decaying the influence of local data, ensures the clients of datasets of low quality do not affect the performance of FEDORA. Please note that we have already explained these steps modularly in Section 5.1, 5.2, 5.3 and 5.4.

Reviewer comment (Experiments):
The requirement for the server to have access to the complete MDP raises concerns about privacy and may not be practical in many scenarios. Providing a practical example or discussing potential alternatives for this setup would strengthen the paper's applicability.

Author response:
**We are afraid that the reviewer had a misunderstanding here.** We **do not assume** that the server has access to the MDP, simulator or any data present in the clients. Furthermore, the clients do not have access to the MDP or a simulator.  Indeed, this is the fundamental premise of offline RL. We would appreciate it if you could point out the lines in our paper that caused the misunderstanding, and we will be happy to fix them.

Reviewer comment (Reproducibility):
Experimental details for Figure 1 are missing, making it difficult to fully understand and interpret the results. Including the specific experimental setup, including hyperparameters, training procedures, and any other relevant details, would enhance the reproducibility and credibility of the findings.

Author response:
The basic description of the experimental setting for the results shown in Figure 1 is provided in Section 4 right above Fig. 1. More detailed description of all our experiments are given in Section 6 and in Appendix A. We have also provided the entire code base from where  you can run the experiments in the supplementary material.

Reviewer comment (Theory):
Eq. (7) implies that the server selects the action based on a weighted average of all agents' actions. Further clarification is needed to explain how the weights assigned to individual agents' actions influence the final decision. Does the weighting scheme prevent one agent from significantly overpowering the others? If not, what mechanisms or strategies are in place to address potential dominance issues and ensure a balanced contribution from all participating agents?

Author response:
**We are afraid that the reviewer had a misunderstanding here.**  **The server does not take any action at all.**  As with all federated learning approaches, the server merely computes the federated policy using the local policies of the clients, which  is then sent back to the clients (see algorithms 1 and 2). The weights are based on the estimated quality of the data present in each client, and the weighting scheme ensures that the well performing policies are given more importance.

Reviewer comment (Reproducibility):
The process of training the centralized policy to achieve the straight-line performance in Figure 2 requires further explanation.

Author response:
Please note that the straight line is not a training curve, rather the performance of the policy trained using TD3-BC on the combined data present in all clients.  This is mentioned in Section 6.

Reviewer comment (Experiments):
Can you compare the performance of FEDORA with just the expert policy running the offline RL algorithm in Fig. 2?

Author response:
Please note that we have indeed done this.  In  Figure 1,  we compare the performance of FEDORA with the performance of the policy obtained by training on the expert data present in an individual client.

Reviewer comment (Novelty):
Some important FedRL papers are missing from reference. Including them would strengthen the scholarly contribution of the paper.

Author response:
We did a through literature survey and cited the most of relevant papers that we are aware of. We will be happy to include more papers in our literature survey if the reviewer could kindly give pointers to any relevant papers that are missing from our references.

Reviewer comment (Presentation):
Could you provide a compelling real-world example that demonstrates the practicality of the problem setup? By illustrating a specific scenario or application where the proposed approach can be effectively applied, readers can gain a better understanding of its relevance and potential impact.

Author response:
We illustrated the value proposition of the approach using a mobile robotics application, wherein each robot collects data independently and privately via arbitrary policies, and our approach is successful at distilling their collective knowledge of navigation without sharing any data (video in supplementary material).  The scenario has several applications, such as that of cleaning robots, which must collaboratively learn to navigate within homes, without sharing data that might violate home owners' privacy.  Much the same scenario applies to factory floor or warehouse robots that must collaboratively learn to navigate, without sharing data that would stress the limited network bandwidth available.

Reviewer comment (Evaluation):
How do you determine the quality of a given dataset?

Author response:
We use a variant of off-policy evaluation, where the clients uses its local data and the federated/local critic to evaluate the quality of its dataset. (See section 5.1 and 5.2)

Reviewer comment (Experiments):
How is the value of $\beta$ determined, and what insights can you provide regarding its impact on the weights assigned to individual agents?

Author response:
In our setup, we treat $\beta$ as a hyperparameter and use a fixed value of $0.1$ for all our experiments. When $\beta=0$, it boils down to a uniform weighting scheme, where the quality of data present in each client is not considered during federation. As $\beta \rightarrow \infty$ it tends to a max weighting scheme, where the federated policy is the same as an individual client's policy.

Reviewer comment (Evaluation):
Could you provide a brief discussion or analysis of the computational and communication costs associated with the proposed approach?

Author response:
The computation and communication costs are similar to standard federated learning algorithms. That is why we omitted the details. In short: 

**Client side:** A subset of participating clients initializes its local policy to the federated policy and performs gradient updates using local data, and evaluates the performance of the learnt policy using the local data. **Server side:** Performs a weighted-averaging of the weights of the actor and critics of the participating clients. **Communication:** The subset of participating clients communicates back the weights of actor and critic to the server. The server broadcasts the federated actor and critic to all clients.

Reviewer comment (Experiments):
How does FEDORA perform when some clients have near-random or adversarial datasets?

Author response:
Thank you for your question. In our paper, we have already shown the the performance of FEDORA when some clients have random data. Specifically,   Fig.3 shows the superior performance of FEDORA when $25$ clients have random data and $25$ clients have expert data. Further, Fig.11  in the appendix shows the  effects of clients having data from multiple behavior policies for varying proportions of clients participating in federation. Here, 12 clients have expert data, 12 clients have medium data, 14 clients have random data, and 12 clients have data from the replay buffer of a policy trained up to the performance of the medium agent. We observe that FEDORA performs well even this scenario with clients having data from multiple policies.

Reviewer comment (Experiments):
Have the authors experimented with different values of β and δ? Is there a principle behind setting them?

Author response:
Thank you for your question. In our setup, we treat $\beta$ as a hyperparameter and use a fixed value of $0.1$ for all our experiments. In response to your question, we have now conducted additional experiments with different values of $\beta$, see  Appendix E.3.  When $\beta=0$, it reduces to a uniform weighting scheme, where the quality of data present in each client is not considered during federation. As $\beta \rightarrow \infty$ it tends to a max weighting scheme, where the federated policy is the same as an individual client's policy with maximum value. We would like to point out that we do not have a $\delta$ parameter in our work.

Reviewer comment (Experiments):
How does FEDORA compare to state-of-the-art offline RL algorithms like CQL or IQL in the federated setting?

Author response:
We designed FEDORA as a framework where we can use use any actor-critic based offline RL algorithm in the clients. We chose TD3-BC due to its simplicity and superior performance over other baselines such as CQL, AWAC, Fisher-BRC, and BRAC-Q. We kept the local offline RL algorithm consistent throughout our simulations for fair comparisons. We believe that if we use a superior local offline RL algorithm, the performance of FEDORA might be better.

Reviewer comment (Experiments):
The paper claims that it outperforms the centralized training offline RL, but the proposed method uses an additional data selection technique compared to the baselines, which is not quite fair. To make this claim, the baseline needs to also have data rebalancing or data selection technique (e.g. with [1], or just send the average reward of all episodes to the server).

Author response:
Based on the reviewer’s comments, we have now added an additional baseline, TD3-BC with data rebalancing, which uses the the data rebalancing approach from [Yue et al, 2022], see Appendix E.1. We notice that data rebalancing does help, performing better than TD3-BC. However, its performance is still worse than our FEDORA algorithm. We believe this is due to the exponential weighting employed by FEDORA, and that data rebalancing methods do not completely solve the
distribution shift issue that arises from combining data from multiple behavior policies. We would like to emphasize that the centralized offline RL algorithms serve only as a hypothetical baseline, because our setting federated learning without sharing the data.

Reviewer comment (Novelty):
The title is "ensemble" method, but it seems the solution is not really using an ensemble method. It's a modified federated aggregation method.

Author response:
We believe that our approach bears a strong resemblance to ensemble learning in that we have a set of experts of heterogeneous qualities, whose collective wisdom we must combine together to obtain a high-quality policy. This is similar to ensemble learning, where weights must be determined for the different experts. Furthermore, we are constrained to evaluation only using local data. Hence, we also have an ensemble of critics, which must also be jointly combined to obtain an overarching critic. Hence, we named the algorithm as ensemble-directed to reflect its dependence the ensemble of actors and critics.

Reviewer comment (Experiments):
Even though the proposed method is about federated RL, related work and evaluation should be considered for supervised RL with heterogeneous data as well.

Author response:
We have now added references to two works ([Yue et al, 2022], [Yue et al, 2023]), thanks to the pointer by the reviewer in Q1. We will be happy to include more references if the reviewer has additional suggestions.

Reviewer comment (Experiments):
The proposed technique seems to be doing a data selection based on the quality of the data, would you get a similar performance if you just select the clients with higher reward datasets? Like in [1] for example.

Author response:
To answer this question, we have now **conducted two additional experiments (See Appendix E.2)**: $(i)$ we select clients based only on the rewards in their datasets using the weighing method proposed in [Yue et al, 2022], $(ii)$ we keep all the components of FEDORA such as optimistic critic, decaying the influence of local data, and the proximal term, but only modify the weighing mechanism, and use a similar method proposed in [Yue et al, 2022].  

The experiment results show that  selecting clients based only on the quality of data (i.e., method $(i)$) performs poorly, as there are several other relevant components that are needed  to ensure reasonable performance of federated RL with offline data. In method $(ii)$, we included all these additional components (optimistic critic, adding appropriate regulations, and decaying the influence of local data) but used the weighing mechanism suggested in [Yue et al, 2022]. The experiment results show that method $(ii)$ performs better than method $(i)$, but FEDORA still outperforms  it. This can be attributed to FEDORA's superior weighing strategy, and the holistic approach we took in designing it by augmenting different algorithmic components that help mitigate the problems of federated offline RL described in section 4.1.

Reviewer comment (Evaluation):
Would the proposed method increase the amount of communication required for federated training compared to simple FAvg? It would be nice to quantify that.

Author response:
The communication required for  FEDORA  is similar to FedAvg. In FedAvg, during each round of federation, the subset of participating clients communicate  the  actor and critic parameters to the server. The server broadcasts the federated actor and critic parameters to all clients. In FEDORA, in addition to this, we only need to communicate a scalar weighing factor  which adds a negligible overhead.