A Secure and Efficient Federated Learning Framework for NLP (2021)

Abstract: In this work, we consider the problem of designing secure and efficient federated learning (FL) frameworks for NLP. Existing solutions under this literature either consider a trusted aggregator or require heavy-weight cryptographic primitives, which makes the performance significantly degraded. Moreover, many existing secure FL designs work only under the restrictive assumption that none of the clients can be dropped out from the training protocol. To tackle these problems, we propose SEFL, a secure and efficient federated learning framework that (1) eliminates the need for the trusted entities; (2) achieves similar and even better model accuracy compared with existing FL designs; (3) is resilient to client dropouts.

## introduction deep neural networks have played a significant role in advancing many applications @xcite . the field of natural language processing (nlp) leverages recurrent neural networks (rnns) and transformers to achieve outstanding performance on many tasks. the transformer was first introduced in @xcite using a self-attention mechanism and it achieved prominent performance in various nlp tasks. the benefits of rnns and transformers in nlp are well-publicized, but the various privacy and security problems still pose challenges to the utilization of these models by data owners, especially users with sensitive data such as location, health, and financial datasets. federated learning (fl) @xcite empowers different data owners § equal contribution, alphabetical order (e.g., organizations or edge devices) to collaboratively train a model without sharing their own data, thus allowing them to address key issues like data privacy. although data exchanged in fl consists of less information of the user's raw data @xcite , one might still be concerned about how much information remains. recent research has shown that attackers can still infer sensitive information about the training data, or even reconstruct the it solely from publicly shared model parameters. @xcite . although a series of works @xcite @xcite @xcite have been proposed to protect fl protocols from leaking sensitive information @xcite @xcite . they either have to involve a trusted third party (centralized aggregator), or do not tolerate client dropouts. therefore, the data owners either need to blindly trust the centralized aggregator or must be online all the time during the training period, which makes the entire design less practical. to address the aforementioned issues, in this work, we develop a secure and efficient fl framework, sefl. it employs two non-colluding servers, i.e., aggregation server (as) and cryptography service provider (csp). as collects the encrypted local updates from clients, and securely aggregates them, while csp manages the cryptography primitives, i.e. the decryption key. the overarching goal of this framework is to support accurate and efficient rnn and transformer training while preserving the privacy of training data against the untrusted servers. in other word, any servers' knowledge about any single training data should be bounded by differential privacy @xcite . our contributions are summarized as follows: (1) we present a novel secure fl framework that eliminates the need for trusted aggregators. (2) sefl is more resilient to clients dropping out than previous works. sefl is able to produce a correct global for each compressed local model updates, computes: aggregate local updates over encrypted form: δ& 0 % ← δ& 0 ! % ⊕ δ& 0 " % ⊕… δ& 0 # % , δ& the gaussian mechanism (gm) @xcite achieves differential privacy by approximating a deterministic real-valued function f with an additive noise that is proportional to the function's sensitivity s f , where ## experiments we implement the sefl system using pytorch 1.4.0, cuda 10.1. all experiments are performed on the aws ec2 cloud instance with a 2.30ghz intel xeon gold 5218 salable processors and 8 nvidia quadro rtx 6000 gpus. we evaluate sefl by conducting experiments using lstm and transformer on wikitext-2 @xcite dataset. the lstm model is adopted from @xcite . the transformer model @xcite contains two layers with an embedding dimension of 200, two attention heads, and 200 hidden units. we use perplexity to measure the quality of the predicted data for both transformer and lstm. ## conclusion in this paper, we introduced a new secure and efficient fl framework, sefl, that (i) eliminates the need for the trusted entities, (ii) achieves similar model accuracy compared with existing fl approaches, and (iii) is resilient to client dropouts. we also proposed optimizations that mitigate the high computation and communication overhead caused by cryptographic primitives. this is achieved by applying a local weight pruning technique based on the block hankel-matrix. through extensive experimental studies on nlp tasks, we demonstrate that the sefl achieves comparable accuracy compared to existing fl solutions, and can significantly improve runtime performance.