Understanding Unintended Memorization in Language Models Under Federated Learning (2021)

Abstract: Recent works have shown that language models (LMs), e.g., for next word prediction (NWP), have a tendency to memorize rare or unique sequences in the training data. Since useful LMs are often trained on sensitive data, it is critical to identify and mitigate such unintended memorization. Federated Learning (FL) has emerged as a novel framework for large-scale distributed learning tasks. It differs in many aspects from the well-studied central learning setting where all the data is stored at the central server, and minibatch stochastic gradient descent is used to conduct training. This work is motivated by our observation that NWP models trained under FL exhibited remarkably less propensity to such memorization compared to the central learning setting. Thus, we initiate a formal study to understand the effect of different components of FL on unintended memorization in trained NWP models. Our results show that several differing components of FL play an important role in reducing unintended memorization. First, we discover that the clustering of data according to users—which happens by design in FL—has the most significant effect in reducing such memorization. Using the Federated Averaging optimizer with larger effective minibatch sizes for training causes a further reduction. We also demonstrate that training in FL with a user-level differential privacy guarantee results in models that can provide high utility while being resilient to memorizing out-of-distribution phrases with thousands of insertions across over a hundred users in the training set.

## introduction there is a growing line of work @xcite @xcite @xcite demonstrating that neural networks can leak information about the underlying training data in unexpected ways. many of these works show that language models (lms), which include commonly-used next word prediction (nwp) models, are prone to unintentionally memorize rarely-occurring phrases in the data. large-scale lm training often involves training over sensitive data, and such memorization can result in blatant leaks of privacy (e.g., @xcite ). thus, it is crucial to measure such memorization in trained lms, and identify mitigation techniques to ensure privacy of the training data. the framework of federated learning (fl) @xcite has emerged as a popular approach for training neural networks on a large corpus of decentralized on-device data (e.g., @xcite @xcite @xcite ). fl operates in an iterative fashion: in each round, sampled client devices receive the current global model from a central server to compute an update on their locally-stored data, and the server aggregates these updates using, for e.g., the federated averaging (fedavg) algorithm @xcite , to build a new global model. a hallmark of fl is that each participating device only sends model weights to the central server; raw data never leaves the device, remaining locally-cached. this, by itself, is not sufficient to provide formal privacy guarantees for the training data. however, this work is motivated by the observation (described in detail in section 3) that nwp models trained under the canonical setting of fl exhibited resilience to memorize rare phrases in spite of hundreds of occurrences in the training data. note that fl does differ in many aspects from the well-studied @xcite @xcite central learning setting where all the data is stored at a central server, and minibatch stochastic gradient descent (sgd) is used to conduct training. while training nwp models via central learning, we observed that phrases with even tens of occurrences were easily memorized, in line with prior work @xcite that showed the propensity of such models to memorize phrases with even one occurrence in the training set. thus, we initiate a formal study to understand the effect of the different components of fl, compared to the central learning setting, on unintended memorization in trained nwp models. we also study the extent to which a guarantee of differential privacy (dp) @xcite reduces such memorization. dp has become the standard for performing learning tasks over sensitive data, and has been adopted by companies like google @xcite @xcite , apple (apple, 2017), microsoft @xcite , and linkedin @xcite , as well as the us census bureau @xcite . intuitively, dp prevents an adversary from confidently making conclusions about whether any particular user's data was used to train a model, even while having access to the model and arbitrary external side information. the federated secret sharer: we build on the "secret sharer" framework @xcite that was designed to measure the unintended memorization in generative models. at a high-level, out-ofdistribution examples (called canaries) are inserted into a training corpus, and a model trained on this corpus is then evaluated using various techniques to measure the extent to which the model has memorized the canaries. since datasets in fl are inherently partitioned according to users, we adapt the secret sharer framework to the fl regime by introducing two parameters to control the presence of a canary in such settings. an illustration of our federated secret sharer framework is shown in figure 1 . given a canary with parameters p u and p e , we let p u be the probability with which each user in a dataset is selected to be a "secret sharer" of the canary (figure 1a ), whereas p e denotes the probability with which each example in such a secret sharer's data is replaced by the canary (figure 1b ). we use poisson sampling for both user-selection and example-replacement. the secret sharer selection phase precedes canary insertion to model realworld settings where occurrences of user-specific unique or rare out-of-distribution canaries are typically limited to a small group of users, but such users can exhibit high usage for those canaries. contributions: our empirical evaluations demonstrate the following key contributions. first, we observe that clustering training data according to users, which happens by design in distributed learning settings like fl, has a significant effect in reducing unintended memorization for nwp models. next, given a dataset partitioned by users, we show that replacing the learning optimizer from sgd to federated averaging and increasing the effective minibatch size provides a further reduction in such memorization. lastly, we demonstrate that training in fl with user-level differential privacy (dp) results in models that can provide comparable utility while being resilient to memorizing canaries with thousands of insertions spread across over a hundred users in the training set. prior work @xcite has shown that models trained with record-level dp do not exhibit unintended memorization for a single insertion of a canary. we provide evidence of models being resilient to memorizing canaries for orders of magnitude higher insertions, at the stronger user-level privacy. ## contrasting federated learning with central learning now, we take a deeper look at how the well-studied central learning framework differs from the canonical setting of fl for lm training. we are interested in differences that might have an effect on unintended memorization. we identify three such components: (1) data processed per update: central learning typically ingests data as records/sentences. on the other hand, fl operates at the granularity of a user, with each user having their own set of sentences locally. typically, the amount of data processed per model update in central learning is much smaller in comparison to fl. (2) learning technique: in central learning, the model is updated via sgd on a minibatch of records. in the canonical setting of fl, a model update typically corresponds to federated averaging over a minibatch of users: an average of the differences between the current model and the model obtained after several sgd steps on the local data of a user. (3) independent and identically distributed (iid) data: to reduce variance in learning, the data in central learning is shuffled before training (and/or each update involves a randomly sampled minibatch). thus, each minibatch can be estimated to be drawn iid from the data. datasets in fl are naturally grouped according to potentially heterogeneous users, resulting in non-iid data even though each minibatch of users may be randomly sampled. foot_0 ## empirical evaluation experimental setup: our model architecture (1.3m parameters) mirrors the one used in @xcite . we create a modified version of the stack overflow dataset @xcite hosted by tensorflow federated @xcite , containing 392k users (93m records). for an iid version of this dataset, we randomly shuffle all the records, and create synthetic users having data assigned sequentially from the shuffled records. since our model is a word-level language model, we follow the methodology used by @xcite for their experiments with the gmail smart compose model @xcite . we insert random 5-word canaries with configurations in the cross product of p @xmath0, 3/50k, 1/5k} and p @xmath1, 10%, 100%}, with 10 different canaries for each (p u , p e ) configuration, resulting in the insertion of 90 different canaries. given a prefix of a canary, we use two methods to evaluate the unintended memorization of the suffix for a model: random sampling (rs), which for a 2-word prefix measures if the canary has the least log-perplexity among 2m random suffixes, and beam search (bs), which uses a greedy beam search to see if the canary is in its top 5 most-likely 5-word continuations from a 1-word prefix. we measure the utility of a model with accuracy and perplexity on the test partition of the unmodified stack overflow dataset. ## conclusion in this work, we conduct a formal study to understand the effect of the different components of federated learning (fl), on the unintended memorization in trained next word prediction (nwp) models, as compared to the well-studied central learning. from our results, we observe that the components of fl exhibit a synergy in reducing such memorization. to our surprise, user-based clustering of data (which occurs as a natural consequence in the fl setting) has the most significant effect in the reduction. moreover, training using federated averaging and larger effective minibatches reduces such memorization further. lastly, we observe that training in fl with a user-level differential privacy guarantee results in models that can provide comparable utility while being resilient to memorizing canaries with thousands of insertions across over a hundred users in the training set. recent work @xcite has shown that, in general, such heterogeneity in the training data can result in a slower and unstable convergence due to factors such as "client-drift". for all of the experiments with non-iid data, we observe that the utility of the trained models is comparable to those trained on iid data, and we leave further exploration into why client-drift may not play a significant role in our experiments for future work. next, while our extensive evaluation is for a practical nwp model on a real-world benchmark dataset, the degree of unintended memorization in general can depend on the model architecture and the dataset used for training. lastly, the secretsharer line of methods for measuring unintended memorization operate at the granularity of a record. for future work, it will be interesting to design stronger attacks targeting data at the granularity of a user, and measure the resilience of models trained via fl, against such memorization.