Federated Domain Adaptation for Named Entity Recognition via Distilling with Heterogeneous Tag Sets (2023)

Abstract: Federated learning involves collaborative training with private data from multiple platforms, while not violating data privacy. We study the problem of federated domain adaptation for Named Entity Recognition (NER), where we seek to transfer knowledge across different platforms with data of multiple domains. In addition, we consider a practical and challenging scenario, where NER datasets of different platforms of federated learning are annotated with heterogeneous tag sets, i.e., different sets of entity types. The goal is to train a global model with federated learning, such that it can predict with a complete tag set, i.e., with all the occurring entity types for data across all platforms. To cope with the heterogeneous tag sets in a multi-domain setting, we propose a distillation approach along with a mechanism of instance weighting to facilitate knowledge transfer across platforms. Besides, we release two re-annotated clinic NER datasets, for testing the proposed method in the clinic domain. Our method shows superior empirical performance for NER with federated learning.

## introduction federated learning for named entity recognition (ner) is the task of collaboratively learn with ner datasets from multiple platforms, while not violating data privacy, i.e., without sharing data across different platforms @xcite . a platform can be an institution, e.g., a hospital or a drug company, where a private collection of clinic ner dataset are locally stored. in reality, data from different platforms are usually sampled from different clinic domains, due to different patient groups, etc. additionally, different schemes may be used when annotating for different platforms. this happen when healthcare providers use customized tag sets to create their own datasets @xcite . as an example, a hospital may hold a dataset of † corresponding author clinical reports from doctors annotated with entities of patient disease and drugs prescribed by the doctors, while a drug company may have text data of patient feedback, annotated with drugs and their adverse drug effects (ade). in this case, it would be mutually beneficial for the hospital and the drug company if they can train in a federated manner a shared (global) ner model with both datasets. the global model should in principle predict with the complete tag set, i.e., {disease, drugs, ade}, enabling the hospital to also recognize the ade in their clinic reports and the drug company to identify disease from their patient feedback, without sharing or re-annotating their local datasets. this can be regarded as a problem of domain adaptation, since the key challenge is to efficiently transfer knowledge of locally unlabeled entity types, i.e., disease and ade across domains/platforms, so that the resulting global model can work for both the hospital and the drug company. so motivated, we study federated domain adaptation for clinic ner in the multi-domain setting where datasets from multiple platforms representing different domains. further, we address a more challenging scenario in which different platforms also annotate with different tag sets, i.e., set of entity types. the goal is to benefit all platforms from federated learning, via training a global model that predicts with the complete tags set, including all the encountered entity types, for text of different domains/platforms. note there are previous works studying federated ner in a multi-domain setting @xcite . however, these works generally presume that the ner model for one platform only predicts with the entity types annotated in the local training data, unlike our setting that requires predicting on a larger tag set (the complete tag set). here, we claim that such an assumption might not be practical in a multi-domain setting. to illustrate this, suppose there is a platform with annotations of training data in addition, we assume the text data of different platforms are from different domains (e.g., platform 1 is from a drug company and platform 2 is from clinical reports). to address the heterogeneous tag sets and facility knowledge transfer across platforms, with regards to the locally unlabeled entity types, we propose a distillation approach, that distills knowledge of unlabeled entity types from other platforms via pseudo-annotations with the complete tag set. based on the proposed distillation, we further propose a instance weighting mechanism, so that knowledge learned with local data is more transferable across platforms. we adopt a promptbased ner model @xcite with superior performance for cross-domain ner, and only transmit prompt-related parameters (7% of the model size) for each round of federated learning to reduce the communication cost. we should note that a comprehensive evaluation of the global model in the setting considered requires testing data with the complete tag set for each domain/platform. however, existing public clinical datasets of different domains are usually annotated using different tag sets (with small overlap), i.e., they lack evaluation data that is consistently annotated with the complete tag sets for multiple domains. therefore, we re-annotate the ade-corpus @xcite and smm4h @xcite datasets using the annotation scheme of cadec @xcite , resulting in datasets of multiple domains that are annotated consistently for evaluation. our contributions are as follow: ## conclusion in this work, we study the problem of federated domain adaptation for clinic ner. we consider the practical setting with heterogeneous tag sets for different platforms of federated learning. to cope with the heterogeneous tag sets and facilitate knowledge transfer among different platforms, we propose distillation with pseudo-complete annotation and an instance weighting mechanism. in addition, we will release two re-annotated clinic datasets for our considered setting. in experiments, our trained ner model show superior performance in the considered setting.