F ed ED : Federated Learning via Ensemble Distillation for Medical Relation Extraction (2020)

Abstract: Unlike other domains, medical texts are inevitably accompanied by private information, so sharing or copying these texts is strictly restricted. However, training a medical relation extraction model requires collecting these privacy-sensitive texts and storing them on one machine, which comes in conflict with privacy protection. In this paper, we propose a privacy-preserving medical relation extraction model based on federated learning, which enables training a central model with no single piece of private local data being shared or exchanged. Though federated learning has distinct advantages in privacy protection, it suffers from the communication bottleneck, which is mainly caused by the need to upload cumbersome local parameters. To overcome this bottleneck, we leverage a strategy based on knowledge distillation. Such a strategy uses the uploaded predictions of ensemble local models to train the central model without requiring uploading local parameters. Experiments on three publicly available medical relation extraction datasets demonstrate the effectiveness of our method.

## introduction privacy -like eating and breathing -is one of life's basic requirements. unlike other domains, medical texts are highly privacy-sensitive, because these texts can include some of the most intimate details about one's life, which document a patient's physical and mental health, and can include information on social behaviors, personal relationships and financial status @xcite . to prevent private information leakage, sharing or copying medical texts is strictly restricted. previous relation extraction methods require centralizing the underlying training data from different medical platforms, such as hospitals and healthcare centers, on one server for training, while holding the centralized privacy-sensitive data puts patients' privacy at risk. this is one of the reasons that hinder the use of relation extraction in clinical practice. as a possible solution, federated learning @xcite is proposed to make full use of privacy-sensitive data. training local models with private data at local platforms and aggregating local models in the central server compose the federated learning process. in the framework of federated learning, no single piece of private data is uploaded to or stored on the central server, and only local models' parameters are sent to the server for updating the central model. though federated learning has distinct advantages in privacy protection compared to centralized training, federated learning algorithms, such as fedavg @xcite , require frequent communication between local platforms and the central server to upload and download models' parameters. communication is a critical bottleneck of applying federated learning to relation extraction, which is largely due to the following reasons: first, the state-of-the-art relation extraction models @xcite @xcite usually utilize transformer-based pretrained language models @xcite @xcite as backbone encoders, which have millions or even billions of parameters. second, the framework of federated learning includes a massive number of local platforms @xcite , and communication between each platform and the central server is necessary. third, upload bandwidth is typically limited to 1 mb/s or less in most situations foot_0 . considering the cumbersome model, numerous local platforms and the limited upload bandwidth, it will take an excessive amount of time during frequent upload processes. for example, in a single communication, uploading a bert-large @xcite model takes more than 21 minutes and uploading a t5 @xcite model takes more than 12 hours. in order to overcome the communication bottleneck in federated relation extraction, it is necessary to develop a communication-efficient method that iteratively sends small messages as part of the training process, as opposed to sending the entire pretrained language encoder. in this paper, we introduce a privacy-preserving medical relation extraction model, named feded. to prevent private information leakage, we leverage federated learning without sharing raw privacysensitive medical texts. to overcome the communication bottleneck in federated relation extraction, we focus on reducing the size of transmitted messages at each communication round. to this end, we formulate the central aggregation process in federated learning as learning a compact central model (student) from the ensemble @xcite of multiple local models (teacher). from this perspective, only the predicted labels on a small dataset need to be uploaded to the central server, because learning from a "teacher" model only requires the behavior of the "teacher" rather than the entire"teacher" network @xcite . besides, the ensemble model (teacher) is powerful, which defines the upper extreme of aggregating when limited to a single communication in federated learning @xcite . to transfer the knowledge in the ensemble model to the central model, we leverage a strategy based on knowledge distillation @xcite , which trains the central model by forcing it to have a similar prediction with the ensemble model. to demonstrate the effectiveness of our method, we conduct extensive experiments on three different medical relation extraction datasets. the results show that our method not only outperforms the baselines but also is communication-efficient. we summarize our contributions as follows: our work builds on a rich line of recent efforts on relation extraction models and federated learning. ## experiments in this section, we carry out an extensive set of experiments with the aim of answering the following research questions: in the remainder of the section, we describe the datasets, experimental setting, and all baselines. algorithm 1 feded. the k local platforms are indexed by k. c is the fraction of local platforms that perform computation on each round. b is the local minibatch size. e is the number of local epochs, and η is the learning rate. initialize θ 0 on the central server for each communication round @xmath0,1,2,... do m ← max(c× k, 1) j t ← (random set of m local platforms) the server distributes θ t to j t .