Distantly Supervised Relation Extraction in Federated Settings (2021)

Abstract: In relation extraction, distant supervision is widely used to automatically label a large-scale training dataset by aligning a knowledge base with unstructured text. Most existing studies in this field have assumed there is a great deal of centralized unstructured text. However, in practice, texts are usually distributed on different platforms and cannot be centralized due to privacy restrictions. Therefore, it is worthwhile to investigate distant supervision in the federated learning paradigm, which decouples the training of the model from the need for direct access to raw texts. However, overcoming label noise of distant supervision becomes more difficult in federated settings, because texts containing the same entity pair scatter around different platforms. In this paper, we propose a federated denoising framework to suppress label noise in federated settings. The key of this framework is a multiple instance learning based denoising method that is able to select reliable sentences via cross-platform collaboration. Various experiments on New York Times dataset and miRNA gene regulation relation dataset demonstrate the effectiveness of the proposed method.

## introduction relation extraction (re) aims to mine factual knowledge from free text by labeling relations between entity mentions, which is a crucial step in knowledge base (kb) construction. for example, given a sentence "[steve jobs] e 1 and wozniak cofounded [apple] e 2 in 1967", a relation extractor should identify that "steve jobs" and "apple" are in a "founder" relationship. most existing supervised re systems, such as @xcite ; @xcite ; @xcite ; @xcite , rely on a largescale manually annotated training dataset, which is extremely expensive and cannot cover all walks of life. to ease the reliance on annotated data, @xcite proposed distant supervision to automatically generate training data by heuristically aligning a kb with unstructured text. the key assumption of distant supervision is that if two entities have a relation in the kb, then all sentences that mention these two entities will express this relation. since then, there has been a rich literature devoted to this topic, such as @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite . though the progress is exciting, distant supervision approaches have so far been limited to the centralized learning paradigm, which assumes that a great deal of text is easily accessible. however, in practice, texts are usually distributed on different platforms and are massively convoluted with sensitive personal information, especially in the healthcare and financial fields @xcite @xcite . due to privacy restrictions, it is almost impossible or cost-prohibitive to centralize texts from multiple platforms. recently, federated learning @xcite provides a compelling solution for learning a model from decentralized and privacysensitive data. the main idea behind federated learning is that each platform trains a local model based on its own local data and a master server coordinates massive platforms to collaboratively train a global model by aggregating these local model updates. unfortunately, directly applying federated learning to the decentralized distantly supervised data fails, because conventional federated learning requires the local data to come with labels without noise @xcite , however, in distant supervision, automatic labeling inevitably accompanies with label noise @xcite @xcite , s 1 : steve jobs and wozniak co-founded apple in 1967 . which means not all sentences that mention an entity pair can represent the relation between them. training on such noisy data will substantially hinder the performance of the re model. moreover, even involving previous denoising methods, such as @xcite ; @xcite ; @xcite , cannot handle label noise well in federated settings. this point can be illustrated by the example in figure 1 . specifically, s 1 and s 2 mention the same entity pair ("steve jobs", "apple") but are distributed on two platforms. s 1 is true positive while s 2 is a false positive instance, which does not express the "founder" relation. in centralized training, there is no barrier between platform 1 and platform 2; therefore, simultaneously considering s 1 and s 2 can easily filter out noise via only selecting s 1 @xcite or placing a small weight on s 2 @xcite . however, raw data exchange between platforms is prohibited in federated settings. due to the lack of comparison with s 1 , previous denoising methods would mistakenly regard s 2 as a true positive instance. as a result, s 2 is retained and then poisons the local model in platform 2, which would affect the global model in turn. to suppress label noise in federated settings, we propose a federated denoising framework in this paper. the core of this framework is a multiple instance learning (mil) @xcite based denoising algorithm, called lazy mil, which is only executed at the beginning of each communication round and then would rest until the next round. since the sentences containing the same entity pair scatter around different platforms, lazy mil algorithm coordinates multiple platforms to jointly select reliable sentences. once sentences have been selected, they would be used repeatedly to train local models until the end of this round. in summary, the main contributions of this paper are: ## related work in this section, we will briefly review the recent progress in distant supervision, some existing studies in federated learning and federated learning in natural language processing (nlp). ## .1 task definition in this paper, we focus on distant supervision in federated settings. assume that there are k platforms {p 1 , ...p k } with respective unlabeled corpora {d 1 , ...d k } and a reference kb. the given kb is used to automatically label these unlabeled corpora. under the assumption of centralized training, each platform transfers or shares its local corpus to a server, and the server will take the kblabeled integrated corpus @xmath0... ∪ d k to conduct training, while the task of distant supervision in federated settings requires platform p i does not expose its corpus d i to others (including the server). in this work, we only focus on the data security of these unlabeled corpora and assume the kb is publicly available for all platforms. how to protect the security of kb is beyond the scope of this work, and we leave it for the future work. to solve this task, we propose a federated denoising framework. the key components of this framework will be elaborated in the following section. concretely, we first introduce the basic relation extractor in section 3 .2, which is the network architecture shared by the global model and local models. then, we present how to select reliable instances via cross-platform collaboration in section 3 .3. next, we describe how to use the selected instances to train the local model in section 3 .4. finally, we present how to use the fedavg algorithm to update the global model in section 3 .5. ## .2 relation extractor following previous studies @xcite , we adopt the piecewise convolutional neural network (pcnn) as our relation extractor. specifically, given a sentence s and two entities within this sentence, we first split the sentence into tokens, and then each token w i is mapped into a dense word embedding e @xmath1. to specify the entity pair, relative distances between the current token w i and the two entities are transformed into two positional features by looking up the position embedding matrices. next, each token in the sentence is represented as the concatenation of the word embedding and two positional features, and is fed into a convolutional neural network. then, piecewise max pooling @xcite is employed to extract the high-level sentence representation. in the piecewise max pooling, an input sentence is divided into three segments based on the two entities, and the maximum value of cnn outputs in each segment is returned. after that, we apply a single fully connected layer to output the logit value o. finally, the conditional probability of j-th relation is denoted algorithm 1 lazy multiple instance learning 1: input: global model parameters θ, the set of activated platforms a. ## .lazy multiple instance learning to avoid the local relation extractor being poisoned by false positive instances, we propose lazy multiple instance learning (lazy mil), which can select reliable instances via cross-platform collaboration. the overview of lazy mil is illustrated in algorithm 1. suppose that there is a triple (h, r, t) in the public kb, the set of sentences containing the head entity h and tail entity t is represented as after computation, platform i uploads the value v i and index id i to the master server. at the same time, the same procedure is performed on platform j, and the value v j and index id j are also uploaded to the server. the master server decides which local instance can be selected among all activated platforms based on the uploaded values. if v @xmath2, then the id i -th sentence in platform i is selected as the reliable sentence that expresses the triple (h, r, t) in this round. this decision, called denoising information, is broadcast to all activated platforms. each activated platform selects reliable training instances from its local corpus according to this denoising information. note that since only values and indices of conditional probabilities are uploaded to the master server, lazy mil almost does not leak the corpus information in each platform. ## .4 local model training after platform i selects reliable instances from its local corpus d i , the selected reliable instance set d i is used for training the local relation extractor. we use the cross-entropy loss function to optimize parameters θ q , which is defined as follows: ## .5 global model update suppose a q is the set of activated platforms in the q-th communication round. after all activated platforms finish local training, the master server collects all trained parameters {θ i q+1 |@xmath3. we define the goal of the global model as follows: ## experiments in this section, we firstly introduce the datasets, experimental setting, and all baselines. then, we compare our method with the baselines. finally, we perform various experiments to analyze the effect of different parameters on the results. due to the page limit, case studies and bert-based experiments can be found in the appendix. ## .1 datasets and evaluation metrics since experiments on non-public privacy-sensitive datasets is not reproducible, we choose public distantly supervised relation extraction datasets to investigate the effectiveness of the proposed framework. nyt 10 foot_2 @xcite is a widely used dataset in distant supervision. it was automatically generated by aligning the semantic triples in freebase with the new york times corpus. the training set contains 466,876 sentences, 251,928 entity pairs and 16,444 relational facts. meanwhile, there are 55167 sentences, 28077 entity pairs and 1,808 relational facts in the development set and the test set contains 172,448 sentences, 96,678 entity pairs and 1,950 relational facts. there are 52 actual relations and a special relation na for representing no relation between two entities. mirgene foot_3 @xcite is a large-scale biomedical dataset. this dataset is generated by aligning tarbase and mirtarbase with the abstracts in medline. there are 172727 sentences in the training set and 1239 sentences in the test set. ## .2 experimental settings defined in equation 5 10: function local_training(i, θ): run on platform i 11: generate denoised dataset d i from d i based on the denoising information i for batch @xmath4: θ ← θ -η∇j(θ; b) j is defined in equation 3 16: ## .3 baselines we compare our method with the following baselines in federated settings: (1) directly applying fedavg algorithm @xcite to the automatically labeled data is the first baseline, which is called none. in this case, there is no denoising module in this method. (2) @xcite proposed to leverage multiple instance learning to choose the most reliable sentence as the bag representation, and we abbreviate this method as one; ## .main results figure 2 and figure 3 show the precision-recall curves on nyt dataset and mirgene datasets, and from the results, we find that: (1) our method significantly outperforms all baselines in federated settings. we believe the reason is that our denoising method can use cross-platform information to hinder false positive instances from poisoning local models, which leads to a better performance of the global model. (2) directly applying fedavg algorithm @xcite to the automatically labeled data achieve the worst results in both datasets. the reason behind that is training on the noisy data will substantially hinder the performance of the model. therefore, it is necessary to conduct denoise in federated distant supervision. on each round, which controls the amount of multiplatform parallelism. with increasing platform parallelism, the performance of all baselines declines slightly while our method performs better. intuitively, increasing platform parallelism is able to lead to better results, since involving more platforms in training can increase the likelihood that all sentences with the same entity pair appear simul- ## .5 increasing the number of local updates in this section, we investigate the impact of varying the number of local updates in this section. the number of local updates is given by e ## .6 increasing the size of local datasets in this section, we increase the size of local datasets by setting k to 50. in such a way, each local dataset is twice as large as it was (when k is set to 100). for a fair comparison, we fix @xmath6.1, @xmath7. ## conclusion considering data decentralization and privacy protection, we investigate distant supervision under the federated learning paradigm, which permits learning to be done while data stays in its local environment. to suppress label noise in federated settings, we propose a federated denoising frame-work, which can select reliable instances via crossplatform collaboration. this framework yields promising results on two widely used datasets, and we have demonstrated its effectiveness through an extensive set of experiments.