Privacy Regularization: Joint Privacy-Utility Optimization in L anguage M odels (2021)

Abstract: Neural language models are known to have a high capacity for memorization of training samples. This may have serious privacy im- plications when training models on user content such as email correspondence. Differential privacy (DP), a popular choice to train models with privacy guarantees, comes with significant costs in terms of utility degradation and disparate impact on subgroups of users. In this work, we introduce two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy through (1) the use of a discriminator and (2) the inclusion of a novel triplet-loss term. We compare our methods with DP through extensive evaluation. We show the advantages of our regularizers with favorable utility-privacy trade-off, faster training with the ability to tap into existing optimization approaches, and ensuring uniform treatment of under-represented subgroups.

## introduction neural language models @xcite have recently seen significant gains in capabilities, and are deployed at scale in several real-world scenarios @xcite . training these models on domain-specific user data can further improve their utility. the volume of data required, coupled with the inherent sparsity of natural language which often means all data are unique, opens the door to an array of privacy attacks against models and their training data. sample memorization poses a substantial risk by enabling model inversion attacks @xcite @xcite . in these attacks, a curious or malevolent user can query a pre-trained language model on any data record with the intention of reconstructing (parts of) training samples 1 . differential privacy (dp) @xcite is the gold standard approach to address this issue, thanks to its strong and rigorous privacy guarantees. dp-sgd @xcite is a popular method to train neural models with differential privacy guarantees and it works by clipping of the gradients and addition of noise in each update, which provides worst-case guarantees that reflect the likelihood of leaking any attribute of any member of the dataset into the trained model. the worst-case guarantees of differential privacy are not customizable, in other words, they cannot be relaxed to protect only certain attributes. therefore, dp incurs significant loss to model utility @xcite . dp training of models is also much slower, with cumbersome hyper-parameter tuning and development @xcite . it has also been shown that dp's utility loss is much worse for under-represented groups @xcite , which can have financial and societal ramifications @xcite . to address these issues, we relax the strong assumptions of the dp threat model and assume an adversary with finite-capacity (finite statistical, compute, and side information) who attempts to recover sensitive user-level information from the trained model @xcite . we propose two privacy regularization methods, one based on adversarial training and another on a novel privacy loss term, to jointly optimize for privacy and utility of language models. the main idea of our regularizers is to prevent the last hidden state representation of the language model for an input sequence x from being linked back to the sensitive attribute we are trying to protect, in our case, the identity of the author. we use the last hidden state as it corresponds to the embedding of the sequence x. 2 we consider the linkability of the input representation to the sensitive attribute (author) as a proxy since it is commensurate with the linked and linkable information definitions in the general data protection regulation (gdpr article 29 working party, 2014). by framing privacy as an optimization problem, we can apply the well-developed machinery of large-scale gradient-based optimization, enabling us to train models at scale while jointly tuning for an optimal privacy-utility trade-off. to validate our approach, we develop an evaluation framework for assessing a model's privacy loss. we employ the exposure metric introduced in @xcite and introduce a reconstruction (tab) attack as a realistic scenario to evaluate and compare lstm language models trained using our regularization with those trained with differential privacy, on @xcite and reddit @xcite datasets. we also empirically demonstrate that, unlike dp, our technique does not have disparate impacts on underrepresented groups. our work is closely related to @xcite and @xcite . @xcite an attacker who eavesdrops on the hidden representations of a pre-trained model during inference and tries to recover information about the input text. adversarial training is used as a mitigation to reduce the attacker's performance @xcite . li et al. use adversarial training to protect private author attributes such as age or gender, in learned text representations for part-of-speech tagging and sentiment analysis to gain better performance on out-of-domain corpora. we, on the other hand, use adversarial training and a triplet-based regularization to train private language models that do not memorize sensitive user information, which has not been explored before. we evaluate our models accordingly, by trying to extract training samples. prior work has studied membership inference attacks against models @xcite ; yeom can consider the representation corresponding to the special token [cls] as the embedding of the sequence x. et al., 2018; song and shmatikov, 2019), however, our regularizations do not target these attacks. ## approach in this section, we explain our proposed regularizers and training techniques in more detail. ## conclusion this work introduces two privacy mitigation methods to jointly optimize for privacy and utility. extensive experiments show that our approach provides comparable and in certain cases a higher level of privacy compared to differentially private model training. we further empirically demonstrate, that our methods do not exhibit disparate impacts on under-represented groups and have significantly less overhead on training performance.