Backdoor Attacks in Federated Learning by Rare Embeddings and Gradient Ensembling (2022)

Abstract: Recent advances in federated learning have demonstrated its promising capability to learn on decentralized datasets. However, a considerable amount of work has raised concerns due to the potential risks of adversaries participating in the framework to poison the global model for an adversarial purpose. This paper investigates the feasibility of model poisoning for backdoor attacks through rare word embeddings of NLP models. In text classification, less than 1% of adversary clients suffices to manipulate the model output without any drop in the performance of clean sentences. For a less complex dataset, a mere 0.1% of adversary clients is enough to poison the global model effectively. We also propose a technique specialized in the federated learning scheme called gradient ensemble, which enhances the backdoor performance in all experimental settings.

## introduction recent advances in federated learning have spurred its application to various fields such as healthcare and medical data @xcite , recommender systems @xcite , and diverse nlp tasks @xcite . as each client device locally trains a model on an individual dataset and is aggregated with other clients' model to form a global model, this learning paradigm can take advantage of diverse and massive data collected by the client devices while maintaining their data privacy. although promising, early works @xcite have raised concerns due to the potential risks of adversaries participating in the framework to poison the global model for an adversarial purpose. among them, model poisoning @xcite assumes that an adversary has compromised or owns a fraction of client devices and has a com- * corresponding author figure 1 : illustration of a backdoor attack to recommend adversary-uploaded contents to any users of choice. this paper investigates the feasibility of model poisoning for backdoor attacks through rare word embeddings of nlp models, inspired by recent backdoor attacks in centralized learning @xcite . in the rare word embedding attack, any input sequences with rare trigger words invoke certain behavior chosen by the adversary. we demonstrate that even in the decentralized case with multiple rounds of model aggregation and individual heterogeneous datasets, poisoned word embeddings may persist in the global model. to better adapt to the federated learning scheme, we propose a gradient ensembling technique that encourages the poisoned triggers to generalize to a wide range of model parameters. our method is motivated by the observation that when poisoning the model, the rare word embeddings should not only generalize to wide ranges of inputs, but also to other model's parameters. applying our proposed gradient ensembling technique further improves the poisoning capability across multiple datasets and federated learning settings (e.g. data heterogeneity). through extensive experiments, we find that less than 1% of adversary clients out of the total clients can achieve adequate accuracy on the backdoor task. for a less complex dataset like sst-2, a mere 0.1% of adversary clients can poison the global model and achieve over 90% on the backdoor task. we further demonstrate that poisoned word embedding through rare words can backdoor the global model even in the presence of detection algorithms based on monitoring the validation accuracy @xcite and robust aggregation methods such as differential privacy @xcite and norm-constrained aggregation @xcite , which is a computationally feasible and effective method in practice @xcite . for seq2seq, we show that having 3∼5% of adversary clients can significantly affect the model output to generate a pre-chosen sequence for backdoored inputs. we summarize our contributions below: meanwhile, on the dimension of adversary objective, our work aims to control the model output for any input with artificial backdoor triggers inserted by the adversary @xcite , unlike semantic backdoor attacks @xcite that target subsets of naturally existing data. to the best of our knowledge, we are the first work in the nlp domain to demonstrate that backdoor word triggers are possible to attack any inputs in the federated learning scenario. our work is inspired by poisoning embeddings of pre-trained language models @xcite in centralized learning. their works demonstrate that backdoors can still remain in poisoned pre-trained models even after finetuning. our work closely follows the attack method of yang et al. and adapt it to the federated learning scheme by utilizing gradient ensembling, which boosts the poisoning capability. ## experiments we first explore the effectiveness of rare embedding poisoning and gradient ensembling ( §4.2). then, we experiment with a very small adversary client ratio (ϵ ≤ 0.5%) to assess how potent rare embedding poisoning can be ( §4.3). next, we demonstrate that the backdoors can unfortunately persist even in the presence of robust aggregation methods although the backdoor performance decreases ( §4.4). last, we extend the poisoning method to a sequence-to-sequence task ( §4.5). ## conclusion our work presents the vulnerability of fl to backdoor attacks via poisoned word embeddings in text classification and sequence-to-sequence tasks. we demonstrate a technique called gradient ensembling to boost poisoning in fl. our work shows that less than 1% of adversary client is enough to manipulate the global model's output. we hope that our findings can alert the practitioners of a potential attack target.