Empirical Studies of Institutional Federated Learning For Natural Language Processing (2020)

Abstract: Federated learning has sparkled new interests in the deep learning society to make use of isolated data sources from independent institutes. With the development of novel training tools, we have successfully deployed federated natural language processing networks on GPU-enabled server clusters. This paper demonstrates federated training of a popular NLP model, TextCNN, with applications in sentence intent classification. Furthermore, differential privacy is introduced to protect participants in the training process, in a manageable manner. Distinguished from previous client-level privacy protection schemes, the proposed differentially private federated learning procedure is defined in the dataset sample level, inherent with the applications among institutions instead of individual users. Optimal settings of hyper-parameters for the federated TextCNN model are studied through comprehensive experiments. We also evaluated the performance of federated TextCNN model under imbalanced data load configuration. Experiments show that, the sampling ratio has a large impact on the performance of the FL models, causing up to 38.4% decrease in the test accuracy, while they are robust to different noise multiplier levels, with less than 3% variance in the test accuracy. It is also found that the FL models are sensitive to data load balancedness among client datasets. When the data load is imbalanced, model performance dropped by up to 10%.

## introduction federated learning is a promising ideology to unite isolated datasets for machine learning problems @xcite @xcite . in the federated learning framework, no raw data are exchanged among participating entities. instead, parameter gradients and aggregated † cooresponding author: jianzong wang @xcite updates are communicated between servers during collective optimization. therefore, without leaking private information, institutes can cooperate with each other by contributing their data collection in the training of a unified model. such a feature is especially desirable when handling sensitive data that involve e.g. personal preference, financial transactions, medical records, etc. an example of successful deployment of federated learning is the smart input prediction in google input @xcite . in addition, more business-to-client model training applications are drawing intensive attention of the public @xcite @xcite . apart from this business-to-client cooperation case, more interesting applications can be found among institutions. potential areas include medical image analysis @xcite , smart retail @xcite , fraud detection, etc. despite its promising designs, federated learning met quite some difficulties migrating to deeper neural networks, as well as to broader cooperative areas. these difficulties are largely due to 1. the limited training speed offered by a secured federated learning platform; and 2. lack of quantifiable evaluation of the privacy and performance of the federated models. before these issues can be settled, institutions would prone to keep their data private rather than contributing to a collaborative neural model. in terms of system security, a federated learning algorithm needs to take care of two kinds of adversaries. firstly, the communication between participating servers must be protected from thirdparty interception or modification. secondly, local datasets must be protected from probing or reverse engineering by other participants. the communication encryption / decryption, such as aes, 3dese, rsa etc., is often lossless, therefore does not affect the performance of the trained model itself. on the other hand, the latter security concern about adversaries from within the participants, has to be addressed differently. to protect the anonymity of training data, the models often need to be revised according to the anonymity protection mechanism. since the anonymity protection scheme is involved in the training procedure, it also causes changes in the expected performance of the overall system. in previous works, the anonymity protection scheme and its influence on the system performance is analyzed case-by-case. for example, yang et al. proposed a local clustering to ensure the k-anonymity of the xgboost model, and studied the relationship between the number of clusters and prediction accuracy via experiments @xcite . unfortunately, their results do not generalize to other models or other anonymity metrics. the restricted applicability of such analyses have limited the development of institutional cooperation on federated learning frameworks. in this paper, we adopt the ( , δ)-differential privacy defined by dwork @xcite as the universal privacy metric. we extend the model privacy derived in @xcite to the federated training model. by utilizing recent developments of federated learning framework, we implemented the federated training of the textcnn model @xcite . to our knowledge, this is the first reported implementation of nlp models on federated learning frameworks. contributions of this paper include: it is shown that the accuracies of the trained models are sensitive to the number of data splits, as well as the balancedness of the data distribution. ## conclusion federated learning provides a promising platform for institutions to cooperate with each other in model training, without tampering their data security. unlike previous works that focus on clientlevel privacy, this paper addresses the privacy protection issues on the sample-level, which is more appropriate for institutional federated learning. in the proposed procedure, ( , δ)-differential privacy is applied to protect client information from probing by other fl participants. a classical nlp algorithm, textcnn, is implemented on the differentially private fl platform. extensive experiments show that, the sampling ratio has large impact on the performance of the fl models. on the other hand, the differentially private fl training is robust to different noise multiplier levels. to address the imbalanced data load situations commonly seen in institutional fl problems, extensive experiments are also conducted to evaluate its influence. compared with equally sized client datasets, the fl models trained on imbalanced clients see significant decline in test accuracies. future studies could be devoted to improving the model performance with unequally sized client datasets. nlp has received a lot of attention from both the academic and commercial societies. for applications in automated banking services, insurance inquires, etc., existing datasets are often confined within the internal servers of respective institutions. there has been ongoing demands on secure ways to utilize these separated data in training a universal model for nlp tasks such as text understanding, question answering, etc. the formulation of institutional federated learning procedure would accelerate development in these areas.