F ed ID : Federated Interactive Distillation for Large-Scale Pretraining Language Models (2023)

Abstract: The growing concerns and regulations surrounding the protection of user data privacy have necessitated decentralized training paradigms. To this end, federated learning (FL) is widely studied in user-related natural language processing (NLP). However, it suffers from several critical limitations including extensive communication overhead, inability to handle heterogeneity, and vulnerability to white-box inference attacks. Federated distillation (FD) is proposed to alleviate these limitations, but its performance is faded by confirmation bias. To tackle this issue, we propose Federated Interactive Distillation (FedID), which utilizes a small amount of labeled data retained by the server to further rectify the local models during knowledge transfer. Additionally, based on the GLUE benchmark, we develop a benchmarking framework across multiple tasks with diverse data distributions to contribute to the research of FD in NLP community. Experiments show that our proposed FedID framework achieves the best results in homogeneous and heterogeneous federated scenarios. The code for this paper is available at: https://github.com/maxinge8698/FedID.

## introduction the remarkable success of natural language processing (nlp) is highly dependent on large-scale pre-trained language models (plms; @xcite @xcite . to fully realize the potential of plms, they are typically trained using large amounts of combined data that is collected from multiple distributed user devices (a.k.a., clients) and transmitted to a single data center (a.k.a., server). with the growing concerns about privacy protection, data regulations such as the personal data protection act (pdpa; chik 2013) and the general data protection regulation (gdpr; voigt and von dem bussche 2017) have imposed strict requirements on preserving user data privacy, making it impractical to aggregate such data to a centralized location for training. federated learning (fl; @xcite has emerged as a privacy-preserving decentralized training paradigm, in which a federation of clients is orchestrated by a central server to collaboratively train a shared global model via aggregating the local models trained on their respective data. as a result, the private data of massive clients is effectively exploited in the form of model parameter exchange to train a unified model for better performance than individually working. previous work on federated nlp mainly targets solving either word-level language modeling applications such as mobile keyboard suggestion @xcite and recommendation @xcite , or biomedical named entity recognition @xcite @xcite . more recently, @xcite provide a researchoriented benchmarking framework for advancing fl in nlp. however, these federated nlp frameworks are limited to identical architectures across the server and clients, making it impossible for clients to design their models independently according to their inconsistent system resources and nonindependent and identically distributed (non-iid) data. also, the frequent model parameter exchange entails expensive communication costs. these obstacles significantly hinder the applicability and scalability of fl for large-scale plms. instead, federated distillation (fd) eliminates the need to share model parameters by transferring knowledge from the clients to the server using an unlabeled public proxy dataset @xcite @xcite @xcite , thereby allowing collaboration between heterogeneous models with less communication costs. however, fd suffers from confirmation bias @xcite induced by incorporating incorrect or even biased predictions on unlabeled data for knowledge transfer, where the central model will tend to degenerate. to tackle this challenge, we propose federated interactive distillation (fedid), where a small handful of labeled data is retained in the server, aiming to provide feedback to the local models to debias the predictions. in addition, previous studies on fd tend to design different partitioning strategies on different datasets, and only small-scale models are taken into consideration, which makes it difficult to evaluate and compare various fd approaches scaled to the nlp domain in a systematic and fair manner. for this reason, based on the general language understanding evaluation (glue) benchmark @xcite , we create a unified benchmarking framework across multiple tasks with diverse data distributions to simulate a variety of federated scenarios for evaluating the effectiveness of these methods on the decentralized training of large-scale plms, advancing the research of fd in nlp. empirical experiments show that our proposed fedid achieves the best results in homogeneous and heterogeneous federated scenarios. the contributions of this paper are summarized as follows: ## federated interactive distillation in existing fd approaches, the central model is only allowed to passively mimic the local models by one-way knowledge transfer, leading to confirmation bias that heavily fades the superiority of fd. instead of directly transmitting the entire public dataset and its predictions between the server and clients, the proposed fedid slices the unlabeled public dataset into multiple smaller batches for training, and handles only a small batch of data and predictions in each communication, which allows for an interaction between the central model and local models during the knowledge transfer process, while significantly reducing the load of a single communication. after each server distillation, the central model is allowed to feedback its performance on a small amount of labeled data held by the server back to each client to adapt its local model accordingly for rectifying its confirmation bias. the overall framework of fedid is presented in figure 1 . ## conclusions this study explores the application of fd to decentralized training of large-scale plms in homogeneous and heterogeneous settings, and further presents an interactive fd scheme to mitigate the confirmation bias caused by transferring knowledge on an unlabeled public dataset. moreover, a benchmarking framework across multiple tasks with diverse data distributions is developed to contribute to the research of fd in nlp community. future work will be executed to aggregate differentially private local predictions for a stronger privacy guarantee, enhancing the resilience of fedid against malicious server or clients.