Paper: Exploring Modality Collaboration with Modality-Agnostic Transformers in Multi-Modal Federated Learning (ICLR-2024)

Reviewer comment (Experiments):
The proposed framework is based on FedAVG, can other model aggregation methods be applied to further improve the performance?

Author response:
As highlighted in Appendix A, while our FedCola framework is indeed grounded in the FedAVG algorithm, its essence extends beyond a mere aggregation method. FedCola encompasses a system-level approach, incorporating elements such as model architecture, parameter-sharing strategy, and temporal modality arrangement. Importantly, it is designed to be adaptable and compatible with a variety of model aggregation methods, not limited to FedAVG. This flexibility allows for potential enhancements in performance through the application of alternative aggregation techniques.

Reviewer comment (Experiments):
Although the proposed framework requires fewer resources than other methods when transformers are applied, what about the resources compared with CNNs?

Author response:
In the experiments conducted for our study, we employed a ViT-Small model as the backbone, which comprises approximately 21 million parameters. This is comparable to the parameter size of a ResNet50, which contains around 24 million parameters. Therefore, the resource utilization between these two models is similar. However, it's important to note that Convolutional Neural Networks (CNNs) are not an ideal choice for our Modality Collaborated Federated Learning (MCFL) framework. This is because CNNs do not encode multi-modal information as effectively as transformers, which is a crucial capability for the objectives of MCFL.

Reviewer comment (Experiments):
During the warm-up stage, are participating clients sampled from only one modality? If so, the comparison might be unfair since more clients on the warm-up modality are participating.

Author response:
During the warm-up stage of our experiments, we ensured fairness by sampling clients uniformly across **all modalities**, employing a **fixed random seed** for all methods. It's important to clarify that while clients from all modalities are considered during the sampling process, communication during this stage is restricted to those with the specified warm-up modality. This approach means that the number of clients actively participating in communication is actually reduced in the warm-up stage.

As a result of this selective communication with fewer clients, the communication costs associated with the warm-up stage, as depicted in Figure 6, are lower compared to other stages. This reduced cost is a direct consequence of the limited client engagement during this initial phase.

Reviewer comment (Experiments):
Can this framework handle clients with multi-modal data? For the pointed-out healthcare scenario, one client with multi-modal data is also common.

Author response:
Indeed, our framework is designed to be adaptable and can certainly accommodate scenarios where clients possess multi-modal data. While our primary focus in Modality Collaborated Federated Learning (MCFL) is on a **more challenging** setting where each client has access to only one modality, the framework is fully **compatible** with situations involving clients with multi-modal data.

In such instances, our approach involves sending modality-specific parameters for all pertinent modalities, in addition to the shared parameters, to the client for local training. This enables effective data-level collaboration across different modalities. This setup is particularly relevant and practical in scenarios like the healthcare sector, where it's common for a single client to handle multi-modal data. Our framework's flexibility in this aspect ensures its applicability and effectiveness in a diverse range of real-world settings.

Reviewer comment (Experiments):
I have concerns about the motivation of this work. In this paper, each client has one single modality, which is not practical in the real-world setting. Though the authors use the hospital as an example, it is more practical that different clients may have different ratios of different modalities of data.

Author response:
We appreciate your concerns regarding the practicality of our work. However, we believe that our setting is indeed **practical** and highly relevant in numerous scenarios involving **sensitive data**, particularly in the healthcare sector, as Reviwer RJk2 acknowledges our setting *"more realistically reflects the nature of real-world data"*. In a hospital environment, for example, different departments (such as MRI and CT scan units) collect data in various modalities. Due to strict protocols, these departments often **cannot freely share data**, leading to a scenario where each department essentially acts as an independent client with uni-modal data within a federated learning system. This not only demonstrates the practicality of our approach but also highlights its **necessity** in certain contexts.

Moreover, we would like to clarify that our framework is designed to accommodate the scenario where each client has a single modality, but it is **not restricted to** this scenario alone. The modality-specific parameters in our framework can indeed encompass more than one modality. In cases where clients have data in different modalities, we can send the modality-specific parameters for all relevant modalities, along with shared parameters, for local training. The rest of the process remains the same, ensuring **compatibility** with the scenario you mentioned.

Additionally, our focus on scenarios where each client has access to only one modality is intentional, stemming from the **challenge** and its **relevance** to federated learning. When clients possess data in multiple modalities, direct data-level knowledge sharing is feasible, allowing the client model to be trained directly on multiple modalities. In contrast, when a client has access to only single-modality data, knowledge sharing must occur at the parameter level through the aggregation of parameters from other clients. This represents a more complex and **challenging** scenario.
Meanwhile, it's worth noting that multi-modal learning within a single client aligns more closely with centralized learning paradigms. Our research, however, focuses on cross-modal learning through **parameter sharing without direct data access**, adhering to the core principles of federated learning.

Reviewer comment (Evaluation):
Also, if I understand correctly, as stated in Sec. 5.2, the authors expect a better-aligned global model. However, assuming each client has one single modality, it should fall into the personalized federated learning domain, where we care more about the clients’ local performance.

Author response:
We would like to clarify that personalized federated learning and traditional federated learning represent two distinct approaches with differing emphases. Personalized federated learning prioritizes the **local performance** of individual clients, whereas traditional federated learning is more concerned with the **performance of the global model**. Our work focuses on the global performance on each modality, which falls into the latter.

In the context of our work, while it's true that each client possesses data in a single modality, such a modality is **not unique** on one client but shared among multiple clients. 
Our primary objective isn't to further personalize the global model for each specific modality per client. 
Such personalization, if pursued, would be limited to **a single modality**, diverging from the central theme of our paper, which is **modality collaboration**. Our focus remains on enhancing the global model's performance through collaborative learning across different modalities, a concept that aligns with the overarching goals of traditional federated learning.

Reviewer comment (Presentation):
Section 5 is one of the key parts of their proposed work. However, some of the parts are put in the appendix.

Author response:
The placement of some sections in the appendix was a decision driven by the constraints on the length of the paper. We aimed to include all essential concepts, methodologies, and results in the main paper, while supplementary analyses and extended discussions were moved to the appendix. 

Specifically, within Section 5, and more precisely in Section 5.2, two parts have been put into the appendix. These sections in the appendix are intended to supplement and enrich the content of the main paper without hindering the understanding of our proposed modality compensation scheme. To illustrate, in Section 5.2 of the main paper, we discuss the motivation and methodology of the modality compensation scheme. The corresponding sections in the appendix, Appendix E and D, provide a detailed algorithm and further proof to enhance reproducibility and facilitate a comprehensive comparison of our scheme with FedAVG.

We would be open to suggestions on which specific parts you feel are crucial and should be included in the main text for clarity and completeness in the revised version.

Reviewer comment (Experiments):
About the model compensation, I am concerned about the extra communication cost and practicality that we require all the clients’ models to have all the parameters.

Author response:
We appreciate your concern regarding the communication cost and practicality of our model compensation approach. However, there seems to be a misunderstanding. As detailed in Section 5.2 and illustrated in Figure 5, the compensation mechanism operates **exclusively at the server level**, thereby **not imposing additional communication burdens** on the clients. This design choice is critical in ensuring that our approach remains both practical and efficient in terms of communication overhead. For further clarity and a comprehensive understanding of our modality compensation mechanism, we encourage a review of the detailed algorithm presented in Appendix E.

Reviewer comment (Novelty):
I am concerned if the core of the technique is still based on the power of transformers which are able to handle different modalities of data.

Author response:
While it's true that the use of transformers is the starting point of our proposed framework due to its multi-modal capability, it's crucial to understand that our framework extends **beyond merely leveraging a model architecture**. Transformers' success in processing diverse types of data modalities inspired our approach, but our work is a comprehensive solution tailored for federated learning environments, not simply applying a model architecture.

As we explore in Section 4, while transformers inherently possess the ability to manage different modalities, simply implementing a transformer model does not automatically address the challenges specific to federated learning.

Furthermore, as discussed in Section 5 ("Towards Better Modality Collaboration"), our framework is guided by three critical research questions: 1) Which parameters should be shared among different modalities? 2) How can client models from diverse modalities be effectively aggregated? 3) What is the optimal approach for temporal arrangement in multi-modal learning? The resulting outcome is the complete framework for our proposed novel MCFL setting.

We also want to highlight that the exploration of these questions is a significant part of our contribution. Our findings and insights lay the groundwork for future research in this area, demonstrating that the core of our technique is not solely based on the power of transformers but also on the innovative application of these models within the unique constraints and opportunities of federated learning.

Reviewer comment (Experiments):
The experiments are primarily conducted on two-modal settings, and the adaptability of FedCola to more complex scenarios with more modalities was not thoroughly studied. Could the proposed framework be extended to scenarios with more modalities?

Author response:
Our current research primarily centers on vision and language modalities, but it's important to emphasize that our framework is compatible with scenarios encompassing additional modalities, a topic we explore in Section 5.4. Looking ahead, we are actively planning further studies to apply and refine our FedCola framework in more complex multi-modal environments. We are confident that this future work will further showcase FedCola's adaptability and robustness, expanding its applicability in diverse federated learning contexts.

Reviewer comment (Theory):
The motivation of the modality compensation is based on an equation based on the number of training samples. However, in the figure provided, the misalignment is based on the number of sampled clients. Can the authors further explain the subtle differences here?

Author response:
Thank you for highlighting this point. We acknowledge that there is an intrinsic relationship between the number of sampled clients and the number of training samples in our model. However, these two factors can be considered equivalent in our context, as clients are sampled uniformly across the board with a fixed ratio. Consequently, an increase in the number of sampled clients directly translates to a greater volume of training samples.

We have also taken this opportunity to correct a typo in the figure to more accurately depict the alignment. This amendment should provide a clearer understanding of how the number of sampled clients and training samples interact and influence the modality compensation in our framework.

Reviewer comment (Reproducibility):
Is the server model the same as the client models? Can the authors explain more about the relationship between the client and server models?

Author response:
As shown in Section 2 (Problem Definition) and Appendix E, the server model and the client models are different. The server model contains modality-specific parameters for **all** modalities ($\phi^{(m)}$) as well as the shared parameters ($\phi^{(m_s)}$), while the client model only contains the modality-specific parameters for **its** modality and the shared parameters. In this way, unrelated parameters don't need to be involved in the communication, which makes FedCola maintain the same level of communication as FedAVG.

Reviewer comment (Reproducibility):
On methodology: what is the frequency of doing inter-modality update of the shared parameters and that of doing intra-modality update of modality-specific parameters?

Author response:
Your questions are raised due to confusion regarding the underlying federated learning (FL) process of our setting. To clarify, the frequency of updates for all parameters, whether they are shared or modality-specific, is **once per round**, so the shared parameters are not more frequently updated.
 Such updating in most federated learning frameworks is **not directly via back-propagation** but via an aggregation of all client models. However, this process can be conceptually understood as an equivalent operation to performing back-propagation on the global model at the server level.
Let us provide a formal demonstration of how back-propagation equivalently operates for the global model in FL, and we've added it to Appendix A in the revised version (colored orange):

Consider the aggregation process on the server for a parameter $w $ from $N $ client models $w_i $, with each client having $m_i $ training samples. The update of $w $ at round $t $ using the FedAVG algorithm can be expressed as:
$$
  w^{(t)} = \sum_{i=1}^K p_i w_i^{(t)},
$$
where $p_i = \frac{m_i}{m}$ and $m = \sum_i m_i $. Here, $m_i$ represents the number of training samples on client $i $.

Given that both the global and client models start with the same initialization $w^{(t-1)} $, the updates on client $i $ can be formulated as 
$$
    w_i^{(t)} = w^{(t-1)} + \eta \nabla F_i(w^{(t-1)}),
$$
using gradient descent for $E $ epochs with a learning rate $\eta $ and the local objective $F_i $.

The aggregation can then be further expressed as:
$$
    \begin{aligned}
        w^{(t)} &= \sum_{i=1}^K p_i w_i^{(t)} \\
                &= \sum_{i=1}^K p_i \left[ w^{(t-1)} + \eta \nabla F_i(w^{(t-1)}) \right] \\
                &= w^{(t-1)} + \eta \sum_{i=1}^K p_i \nabla F_i(w^{(t-1)}),
    \end{aligned}
$$
with
$$
\nabla F(w^{(t-1)}) = \sum_{i=1}^K p_i \nabla F_i(w_i^{(t-1)}).
$$
Thus, it results in **one step of update per round** for all parameters. The primary distinction between modality-specific and shared parameters lies in the number of clients where the gradient is averaged: *modality-specific parameters receive a weighted average of gradients from clients with the corresponding modality, whereas shared parameters are averaged from all clients*. As we discussed in Section 5.2, such a distinction may lead to misaligned generalizability among all layers.
Furthermore, we address this distinction on total client numbers, **instead of frequency**, with our proposed modality compensation scheme, ensuring that both shared and modality-specific parameters are updated effectively to suit their respective roles in the federated learning process.

Reviewer comment (Theory):
On methodology: as the number of training data points/clients in different modalities could be different, the shared parameters should be far more frequently updated compared to modality-specific parameters. It seems that authors have proposed the aggregation step to address this problem. It remains unclear to me how gradient back-propagation works with respect to those parameters which are manually aggregated/augmented into a model, and how these external parameters function during the inference process of a certain modality?

Author response:
Your questions are raised due to confusion regarding the underlying federated learning (FL) process of our setting. To clarify, the frequency of updates for all parameters, whether they are shared or modality-specific, is **once per round**, so the shared parameters are not more frequently updated.
 Such updating in most federated learning frameworks is **not directly via back-propagation** but via an aggregation of all client models. However, this process can be conceptually understood as an equivalent operation to performing back-propagation on the global model at the server level.
Let us provide a formal demonstration of how back-propagation equivalently operates for the global model in FL, and we've added it to Appendix A in the revised version (colored orange):

Consider the aggregation process on the server for a parameter $w $ from $N $ client models $w_i $, with each client having $m_i $ training samples. The update of $w $ at round $t $ using the FedAVG algorithm can be expressed as:
$$
  w^{(t)} = \sum_{i=1}^K p_i w_i^{(t)},
$$
where $p_i = \frac{m_i}{m}$ and $m = \sum_i m_i $. Here, $m_i$ represents the number of training samples on client $i $.

Given that both the global and client models start with the same initialization $w^{(t-1)} $, the updates on client $i $ can be formulated as 
$$
    w_i^{(t)} = w^{(t-1)} + \eta \nabla F_i(w^{(t-1)}),
$$
using gradient descent for $E $ epochs with a learning rate $\eta $ and the local objective $F_i $.

The aggregation can then be further expressed as:
$$
    \begin{aligned}
        w^{(t)} &= \sum_{i=1}^K p_i w_i^{(t)} \\
                &= \sum_{i=1}^K p_i \left[ w^{(t-1)} + \eta \nabla F_i(w^{(t-1)}) \right] \\
                &= w^{(t-1)} + \eta \sum_{i=1}^K p_i \nabla F_i(w^{(t-1)}),
    \end{aligned}
$$
with
$$
\nabla F(w^{(t-1)}) = \sum_{i=1}^K p_i \nabla F_i(w_i^{(t-1)}).
$$
Thus, it results in **one step of update per round** for all parameters. The primary distinction between modality-specific and shared parameters lies in the number of clients where the gradient is averaged: *modality-specific parameters receive a weighted average of gradients from clients with the corresponding modality, whereas shared parameters are averaged from all clients*. As we discussed in Section 5.2, such a distinction may lead to misaligned generalizability among all layers.
Furthermore, we address this distinction on total client numbers, **instead of frequency**, with our proposed modality compensation scheme, ensuring that both shared and modality-specific parameters are updated effectively to suit their respective roles in the federated learning process.

>..., and how these external parameters function during the inference process of a certain modality?

For the inference process of a certain modality, only the **modality-specific parameters** for that modality as well as the **shared parameters** will be involved in the inference process, whereas the other parameters for external modalities won't be involved.

Reviewer comment (Evaluation):
On methodology: under the assumption of data homogeneity and absence of Byzantine workers, should the Uni-FedAVG method show better performance on the dataset CIFAR100? i.e., for the vision modality itself, I am expecting that the accuracy should be somehow higher than that reported in Table 4. [Benchmarking FedAvg and FedCurv for Image Classification Tasks, Casella et al., 2023]

Author response:
In response to your query regarding the Uni-FedAVG method and its anticipated performance on the CIFAR-100 dataset, it is crucial to clarify a few key aspects of our experimental setup. Firstly, our experiments do not operate under the assumption of data homogeneity. Instead, we have adopted a **heterogeneous**, non-IID (independent and identically distributed) setting (Dirichlet distribution), which is typical in the federated learning community.

Furthermore, it is important to note that the results referenced in Casella et al., 2023, pertain to **CIFAR-10**, not CIFAR-100. *This distinction is significant, as CIFAR-10 and CIFAR-100 differ in complexity, which can substantially impact model performance*.

Additionally, the overall performance of the Uni-FedAVG method is influenced by a range of factors, including the number of clients (10 in Casella et al. vs. 16 in ours), client participation rate at each round (1 in Casella et al. vs. 0.5 and 0.25 in ours), the number of local epochs (1,10 and 30 in Casella et al. vs. 5 in ours), and the specific optimizer settings used. In our methodology, we have deliberately chosen to employ a straightforward approach, avoiding extensive data augmentation or advanced training techniques, while *random horizontal flips* and *angle rotation* are used in Casella et al. This decision is driven by our intention to maintain a clear focus on the primary research questions of our study, allowing for a fair and unbiased comparison across different methods.

Reviewer comment (Evaluation):
On results: regarding reported average accuracy results in tables 1 and 3, how are the average computed? Is there any weighting assigned to each modalities?

Author response:
In Tables 1 and 3, the average accuracy is calculated as an **equal-weighted arithmetic mean**. This method was chosen to treat each modality equally, without introducing bias towards any specific modality. We've modified the description in Section 2 to make it more specific (colored orange).

Reviewer comment (Evaluation):
On results: intuitively, when comparing the balance of clients for different modalities, why not show the accuracy change in each modality under different $(N_v, N_l)$ setup?

Author response:
Thanks for pointing it out. 
Our primary interest lies in the imbalance in the **total number of training samples**, specifically, CIFAR-100+AGNEWS as a text-heavy setting and OrganAMNIST+MTSamples as an image-heavy setting. However, we agree that the imbalance in the number of clients will slightly influence the client selection process, and we've added the following results under imbalanced settings to Appendix E in the revised version (colored orange):

|Method| CIFAR-100+AGNEWS (16, 4) | CIFAR-100+AGNEWS (4, 16) | OrganAMNIST+MTSamples (16, 4) | OrganAMNIST+MTSamples (4, 16)|
|-|-|-|-|-|
|Uni-FedAVG|49.02|43.51|44.57|55.75|
|CreamFL|48.24|40.55|45.70|60.71|
|**FedCola**|**53.44**|**44.84**|**52.08**|**62.40**|

In both the CIFAR-100+AGNEWS and OrganAMNIST+MTSamples settings, FedCola consistently outperforms Uni-FedAVG and CreamFL, indicating its superior capability and robustness in adapting to imbalanced multi-modal federated learning environments.

Reviewer comment (Experiments):
On methodology: Convolutional networks are perhaps a simpler type of models for vision related tasks. As in the CIFAR100 case, transformer-based models seem to be somehow below par of the performance of convolutional networks, I wonder if it is possible to incorporate convolutional networks in the study and see relevant results.

Author response:
We respectfully disagree with the notion that transformer-based models are generally outperformed by convolutional networks in vision tasks. In recent years, transformer architectures have emerged as a leading choice in various vision-related applications, including image classification[1], object detection[2], and semantic segmentation[3]. Their ability to effectively process and interpret visual data has been well-documented in numerous studies. As in the CIFAR-100 case, Astroformer [4], which is in a transformer architecture, is the current state-of-the-art method according to the benchmark for image classification on CIFAR-100 *without extra training data* on [paperswithcode](https://paperswithcode.com/sota/image-classification-on-cifar-100).

Moreover, when it comes to **multi-modal tasks**, particularly those involving vision-language integration, transformers have become almost the **de facto architecture**[5,6]. This preference is largely due to transformers' superior capability in handling long-distance dependencies, a feature that is particularly valuable in text data and is not as effectively managed by convolutional networks. 

Given that our research is primarily focused on multi-modal data within the context of federated learning, we have chosen to leverage the strengths of transformer models. Their adeptness at handling multi-modal data makes them particularly suitable for our goal of facilitating modality collaboration in a federated learning setting. As such, our study concentrates on the use of transformer models, and incorporating convolutional networks into MCFL falls outside the current scope of our research.

[1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2020

[2] DETRs with Collaborative Hybrid Assignments Training, 2023

[3] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks, 2022

[4] Astroformer: More Data Might not be all you need for Classification, 2023

[5] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models 2023

[6] VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts, 2021