Open-Vocabulary Federated Learning with Multimodal Prototyping (2024)

Abstract: Existing federated learning (FL) studies usuallyassume the training label space and test labelspace are identical. However, in real-world applications, this assumption is too ideal to betrue. A new user could come up with queriesthat involve data from unseen classes, and suchopen-vocabulary queries would directly defectsuch FL systems. Therefore, in this work, weexplicitly focus on the under-explored openvocabulary challenge in FL. That is, for a newuser, the global server shall understand her/hisquery that involves arbitrary unknown classes.To address this problem, we leverage the pretrained vision-language models (VLMs). Inparticular, we present a novel adaptation framework tailored for VLMs in the context of FL,named as Federated Multimodal Prototyping(Fed-MP). Fed-MP adaptively aggregates thelocal model weights based on light-weightclient residuals, and makes predictions basedon a novel multimodal prototyping mechanism.Fed-MP exploits the knowledge learned fromthe seen classes, and robustifies the adaptedVLM to unseen categories. Our empirical evaluation on various datasets validates the effectiveness of Fed-MP.

## introduction federated learning (fl) emerges as a new machine learning (ml) paradigm that trains ml models from decentralized data sources @xcite . the decentralized nature of fl makes it a promising solution for privacy-sensitive applications across numerous domains (e.g., natural language processing @xcite , multimodal learning @xcite , visual recognition @xcite ). in fl, there exists a central server storing a global model, and a set of clients. the clients will collaboratively train the global model without sharing their private data. while numerous fl studies have been proposed, the elusive openvocabulary challenge is largely under-explored. traditional fl studies (e.g., domain-generalized federated learning) usually assume that the label space of training data and test data is identical. based on this assumption, the proposed fl methods are not open-vocabulary by design. however, in real-world applications, new users might send queries that involve novel classes, e.g., identifying an object in a photo. if the category of this object is never seen in the training data, then traditional fl systems simply fail and can only predict from previously seen classes as shown in figure 1 . indeed, in centralized ml, there exist methods to predict unseen classes @xcite @xcite . however, they usually require a huge amount of the training data and could not tackle new addition of unseen classes over time @xcite . more importantly, the unique challenge of data heterogeneity in fl makes centralized methods inapplicable to train fl models @xcite @xcite . the data heterogeneity in fl is the heterogeneity in client data distributions. for instance, in figure 1 , there are only images of dog and deer in client 1, and client 2 only has images of car and ship. such non-i.i.d. data across clients is heterogeneous data. therefore, in this work, we explicitly focus on the open-vocabulary challenge in fl: how can we build an fl framework that is open-vocabulary? on the other hand, exploiting the pre-trained vision-language models (vlms) (e.g., clip) for fl has recently gained increased attention for their strong generalization ability @xcite . with clip, the community could address data heterogeneity, personalization and generalization in fl @xcite @xcite . technically, to adapt clip for specific fl applications, existing methods mainly adopt prompt learning. prompt learning optimizes a set of learnable soft prompt vectors, and prepends them to input embeddings @xcite @xcite . as such, domainspecific knowledge is integrated into the features extracted by clip, leading to improved performance on downstream tasks. unfortunately, these learned prompts usually suffer from generalizing well to novel unseen classes during test, and yet, no proper solution has been developed. therefore, in this work, we focus on addressing the elusive open-vocabulary challenge in fl. to the best of our knowledge, we are the first to propose a clip-based fl framework that is explicitly tailored for the open-vocabulary setting. to achieve open-vocabulary fl, we propose a federated finetuning framework tailored for vlms: federated multimodal prototyping or fed-mp. intuitively, fed-mp has two design objectives: 1) low communication overhead between the server and clients in fl: given the large size of clip, fed-mp must be light-weight and affordable in terms of model training in an fl application; 2) open-vocabulary: the global model shall understand the queries that involve arbitrary unseen classes. to this end, fed-mp consists of two modules. firstly, fed-mp adaptively aggregates the local model weights based on the similarity between new queries and perturbed client prompt representations. these prompt representations are perturbed by a set of learnable parameters, which is defined as client residuals. client residuals protect clients' class information by perturbing the text representations. in addition, with client residuals, locally learned visual concepts are integrated into the perturbed prompt representations as well. this similaritybased design is realistic and practical in terms of real-world applications: a user comes to use the fl system, and she/he sends a set of queries to the server. in return, the server should adaptively obtain an aggregated model that is aligned with the interest of the user. secondly, we design a multimodal prototyping mechanism to make predictions for the open-vocabulary queries. the multimodal prototypes include text prototypes and visual prototypes. the text prototypes are the original encoded text prompts in the new queries. as for the visual prototypes, they are normalized visual features extracted by clip image encoder with pseudo labeling. during inference, fed-mp predicts for a query image based on its weighted distance to text prototypes and visual prototypes. both modules are designed to exploit the knowledge learned from the seen classes during training. under fed-mp, the adapted clip model generalizes well to test images from unseen classes, achieving openvocabulary federated learning. we summarize the contributions of our paper as follows foot_0 : ## experiments we evaluate the proposed fed-mp mainly on openvocabulary image classification, which is one of the prevailing applications for vlms. in addition, we also provide an ablation study to understand the function of the modules within fed-mp. finally, we conduct robustness studies to evaluate the robustness of fed-mp in regards to the number of training samples per class. ## conclusion this work is the first to address the openvocabulary challenge in fl applications. in particular, we present fed-mp, a novel open-vocabulary fl framework that is tailored for finetuning vlms for fl applications. fed-mp provides an effective solution to make high-quality predictions for queries that involve novel unseen categories. extensive experimental results on various datasets demonstrate the effectiveness of our method.