Paper: Dual Prompt Tuning for Domain-Aware Federated Learning (ICLR-2024)

Reviewer comment (Novelty):
It seems that the proposed Fed-DPT can only be applied to the vision-language model-based federated learning scenarios. Considering that there is a wide range of model architectures along with the domain shift problems, a more general method is preferred.

Author response:
**Q1: Fed-DPT is only applicable for vision-language models**

**R1:** Thank you for your valuable comments. We agree that our method is specifically designed for vision-language models, but this does not mean that this study lacks contributions to the community. On the contrary, we believe that this work holds significant importance for research in federated learning.

With recent advances in self-supervised and weakly supervised learning, pretraining a large-scale foundation model such as CLIP followed by lightweight prompt tuning for downstream tasks is becoming a new paradigm of transfer learning. Under this condition, the traditional fine-tuning-based protocols may not be as effective, and how to fully exploit the prompt learning techniques in federated learning has not been well explored. Our proposed Fed-DPT algorithm, which introduces coupled prompt learning at both vision and language ends, can effectively transfer the knowledge representations learned during the pre-training of CLIP models to downstream federated learning tasks without the need for excessive trainable parameters and communication overhead. We believe this study can offer valuable insights for the follow-up research of both vision-language models and federated learning.

Reviewer comment (Experiments):
It is better to also provide a comparison between the proposed method and traditional FL methods that address domain shift problems, such as FedBN[1] and some SOTA personalized FL methods.

Author response:
**Q2: Comparison to traditional FL methods**

**R2:** Thank you for your valuable suggestion. In response, we have now included a comprehensive evaluation comparing our proposed method with traditional federated learning approaches, such as FedBN, in the revised version of our manuscript. Specifically, we replicated the FedBN setup on the DomainNet dataset, focusing on a 10-class subset of the original 345-class DomainNet to ensure a relevant and fair comparison. The results are summarized in the table below, where it shows that our method constantly outperforms the baselines.

| Method         | clipart | info g. | paint. | quick d. | real | sketch | avg. |
|----------------|---------|---------|--------|----------|------|--------|------|
| FedBN  | 51.2    | 26.8    | 41.5   | 71.3     | 54.8 | 42.1   | 48.0 |
| FedVPT    | 65.6    | 44.1    | 76.6   | 47.3     | 91.0 | 60.3   | 64.2 |
| Fed-DPT (ours) | **83.6**  | **71.4**   | **87.2**  | **63.5**    | **96.8** | **79.2**  | **80.3** |

Reviewer comment (Presentation):
It is recommended to combine Figures 1 and 2, and draw a framework. And it’s better to list the whole algorithm. At present the overall process is not very clear.

Author response:
Thank you for your advice. We have combined Figures 1 and 2 in the revised manuscript to present a more clear framework. Further, to clarify the overall process, we include a pseudo-code of our Fed-DPT in the appendix. We hope these improvements in clarity can well address your concerns.

Reviewer comment (Experiments):
It is mentioned that some text information needs to be shared between clients such as class names, does it involve privacy protection?

Author response:
R(Privacy): Thank you for your question about the privacy. In detail, during federated training of our method, there are class names as well as text prompts need to be shared across clients. In the context of federated learning studied in this work, the class names are not considered as private information since the clients are dealing with the same classification task so they all have the same class names. Also, sharing text prompt does not involve direct privacy leakage and we demonstrate that our method has the same level of privacy protection as FedAvg in our overall response. It is a good point to think over the privacy and we have added a detailed discussion for it in the revised paper.

Reviewer comment (Evaluation):
In the section of experimental results, the performance is greatly improved on DomainNet dataset. It is suggested to analyze the reasons.

Author response:
Thank you for your advice. Compared with OfficeHome and PACs datasets on which the Zero-Shot CLIP model already achieves high accuracy levels of 78.6\% and 95.8\%, DomainNet has a larger margin for improvements. As detailed in Table 1, our Fed-DPT reduces the performance variance among the six distinct domains in DomainNet, attaining the most significant improvements on the "information graph" and "quick draw" domains. This suggests that our domain-aware prompt learning method addresses CLIP's limitations in domain generalization, leading to significant robustness to image patterns. We have included more discussions on this point in the revised paper.

Reviewer comment (Evaluation):
The results of the ablation experiments show that the federated frameworks and textual cues are key factors for improving the effectiveness of the experiments. While the domain-aware mechanism slightly improves the performance. It’s better to give a more detailed reason.

Author response:
Thank you for your observation regarding the ablation study results. As indicated in Table 3, the Domain-Agnostic DPT, which employs both visual and textual prompts but does not incorporate our domain-aware mechanism, achieves a 9.9\% improvement over the baseline Zero-Shot CLIP. Our proposed Fed-DPT further enhances this by 4.9\%. This is a significant improvement in performance since the domain-aware mechanism does not require any additional model parameters compared with the Domain-Agnostic DPT approach. We will make it more clear in the revised version.

Reviewer comment (Novelty):
Prompt learning is also adopted in "Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation" in ICCV2023. What is the difference between the proposed and the ICCV2023 methods, and it is suggested to add a comparison with this method in the experimental section.

Author response:
Thank you for your valuable suggestion. The difference between our Fed-DPT and the pFedPG [ICCV'23] method lies in the model architectures and prompt learning mechanisms. Specifically, pFedPG focuses on single-modality (visual) prompt tuning with only a ViT-Base image encoder employed. In contrast, our Fed-DPT builds on top of a CLIP model, leveraging both visual and textual prompts. As we demonstrated, this dual-modality prompt tuning method facilitates the capabilities of detecting inter-domain correlations so it is suitable for domain-aware federated learning scenarios. Here we conduct a comprehensive evaluation using the experimental setup of pFedPG, which concentrates on a ten-class subset of the 345-class DomainNet dataset. The results are summarized below, where our Fed-DPT exhibits significantly superior performance than pFedPG.

| Method            | clipart | info g. | paint. | quick d. | real | sketch | avg. |
|-------------------|---------|---------|--------|----------|------|--------|------|
| pFedPG | 73.0    | 50.1    | 84.3   | 60.0     | 94.0 | 68.4   | 71.6 |
| Fed-DPT (ours)    | **83.6**  | **71.4**   | **87.2**  | **63.5**    | **96.8** | **79.2**  | **80.3**|

Reviewer comment (Experiments):
As the paper concentrates on using the pre-trained vision-language model, the application scenario is limited to the cross-silo setups. The CLIP model is not practical to deploy in the cross-device FL setup.

Author response:
Thanks for your valuable comments. We agree that the methods based on large-scale pre-training models are more suitable for cross-silo scenarios instead of cross-device setups. However, it is important to clarify that the two scenarios are characterized by the number of participants, the capability of computing resources, and the nature of the data involved, while many federated learning methods have a specific focus on one of them.

Our method is basically designed to tackle domain shift challenges in federated learning, introducing a domain-aware prompt tuning mechanism and demonstrates effectiveness. In scenarios where each domain possesses substantial data with a distinct statistical distribution, our approach is particularly suited to a cross-silo federated learning context. We do not view this as a limitation; rather, it's a strategic focus. Solutions optimized for cross-device scenarios may not effectively address the unique challenges posed by cross-silo problems.

We appreciate your suggestions to clarify the applicable scenarios of our method, and we have included this explanation in the revised version of the paper.

Reviewer comment (Experiments):
In the experiment part, why does the author not consider using the FedAvg/FedProx to directly fine-tune the CLIP model as the baselines?

Author response:
Thank you for your constructive comments. In our study, we intentionally avoid fine-tuning the CLIP model due to a couple of significant considerations. 

Primarily, fine-tuning the CLIP model may easily break its well-established vision-language alignment, which could undermine CLIP's strong capabilities in domain generalization and open-vocabulary inference. So, in practice, the existing methods used to freeze CLIP's encoders and tune additional parameters such as prompt tokens and adapter layers from downstream tasks. An additional benefit of this protocol is its ability to produce favorable results without requiring a substantial volume of training data. In Table 8 (in the appendix), we obtain very competitive few-shot results by our prompt tuning technique.

Secondly, fine-tuning a large model such as CLIP entails a substantial increase in communication cost and a decrease in convergence rate. So, given the same number of training iterations, the fine-tuning protocol often falls short of prompt learning. To provide a clear comparison, we have included the following results in our revised paper, which illustrates the advantages of our method.

| Method            | Backbone   &nbsp;&nbsp; | Learnable params &nbsp;&nbsp; | acc. |
|-------------------|-------------|------------------|------|
| FedAvg | CLIP ViT-B | 86M              | 57.6 |
| FedProx | CLIP ViT-B | 86M              | 58.1 |
| Fed-DPT (ours)    | CLIP        | 16.9k            | 68.4 |