One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks (2022)

Abstract: Preserving privacy in contemporary NLP models allows us to work with sensitive data, but unfortunately comes at a price. We know that stricter privacy guarantees in differentially-private stochastic gradient descent (DP-SGD) generally degrade model performance. However, previous research on the efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive. In this short paper, we provide an extensive analysis of different privacy preserving strategies on seven downstream datasets in five different ‘typical’ NLP tasks with varying complexity using modern neural models based on BERT and XtremeDistil architectures. We show that unlike standard non-private approaches to solving NLP tasks, where bigger is usually better, privacy-preserving strategies do not exhibit a winning pattern, and each task and privacy regime requires a special treatment to achieve adequate performance.

## introduction in a world where 'data is the new oil', preserving individual privacy is becoming increasingly important. however, modern neural networks are vulnerable to privacy attacks that could even reveal verbatim training data @xcite ). an established method for protecting privacy using the differential privacy (dp) paradigm @xcite is to train networks with differentially private stochastic gradient descent (dp-sgd) @xcite . although dp-sgd has been used in language modeling @xcite , the community lacks a thorough understanding of its usability across different nlp tasks. some recent observations even seem counter-intuitive, such as the nondecreasing performance at extremely strict privacy values in named entity recognition (jana and biemann, 2021). as such, existing research on the * equal contribution suitability of dp-sgd for various nlp tasks remains largely inconclusive. we thus ask the following research questions: first, which models and training strategies provide the best trade-off between privacy and performance on different nlp tasks? second, how exactly do increasing privacy requirements hurt the performance? to answer these questions, we conduct extensive experiments on seven datasets over five tasks, using several contemporary models and varying privacy regimes. our main contribution is to help the nlp community better understand the various challenges that each task poses to privacypreserving learning. foot_0 2 related work differential privacy formally guarantees that the probability of leaking information about any individual present in the dataset is proportionally bounded by a pre-defined constant ε, the privacy budget. we briefly sketch the primary ideas of differential privacy and dp-sgd. for a more detailed introduction please refer to @xcite ; @xcite ; @xcite . two datasets are considered neighboring if they are identical, apart from one data point (e.g., a document), where each data point is associated with one individual. a randomized algorithm m : @xmath0, δ)-differentially private if the following probability bound holds true for all neighboring datasets x, @xmath1: when using differentially private stochastic gradient descent (dp-sgd) @xcite , we introduce two additional steps to the standard stochastic gradient descent algorithm. for a given input x i from the dataset, we obtain the gradient of the loss function l(θ) at training time step t, g t (x i ) = ∇ θt l(θ t , x i ). we then clip this gradient by ℓ 2 norm with clipping threshold c in order to constrain its range, limiting the amount of noise required for providing a differential privacy guarantee. subsequently, we add gaussian noise to the gradient to make the algorithm differentially private. this dp calculation is grouped into 'lots' of size l. the descent step is then performed using this noisy gradient, updating the network's parameters θ, with learning rate γ. in nlp, several works utilize dp-sgd, primarily for training language models. kerrigan et al. (2020) study the effect of using dp-sgd on a gpt-2 model, as well as two simple feed-forward networks, pre-training on a large public dataset and fine-tuning with differential privacy. model perplexities are reported on the pre-trained models, but there are no additional experiments on downstream tasks. mcmahan et al. (2018) train a differentially private lstm language model that achieves accuracy comparable to non-private models. hoory et al. (2021) train a differentially private bert model and a privacy budget ε = 1.1, achieving comparable accuracy to the non-dp setting on a medical entity extraction task. only a few works investigate dp-sgd for downstream tasks in nlp. jana and biemann (2021) look into the behavior of differential privacy on the conll 2003 english ner dataset @xcite . they find that no significant drop occurs, even when using low ε values such as 1, and even as low as 0.022. this is a very unusual result and is assessed in this work further below. @xcite apply dp-sgd to sentiment analysis of tweets, reporting a very small drop in accuracy with epsilons of 8.99 and 3.87. with the state of the art only evaluating on a limited set of tasks and datasets, using disparate privacy budgets and metrics, there is a need for a more general investigation of the dp-sgd framework in the nlp domain, which this paper addresses. ## analysis of results as lstm performed worst in all setups, we review only the transformer-based models in detail. also, we discuss only ε of 1.0, 5.0, and ∞ (non-private) as three representative privacy budgets. here we focus on the bert-based scenario; a detailed analysis of the distilled model xdtm is provided in appendix e. random and majority baselines are reported only in the accompanying materials, as they only play a minor role in the drawn conclusions. all results are reported as macro-f 1 (averaged f 1 of all classes). we additionally provide an analysis of efficiency and scalability in appendix c and an analysis on optimal learning rates in appendix d. ## conclusion we explored differentially-private training on seven nlp datasets. based on a thorough analysis we formulate three take-home messages. (1) skewed class distributions, which are inherent to many nlp tasks, hurt performance with dp-sgd, as the majority classes are often overrepresented in these models. (2) fine-tuning and thus noisifying different transformer layers affects task-specific behavior so that no single approach generalizes over various tasks, unlike in a typical non-private setup. in other words, there is no one general setup in terms of fine-tuning to achieve the greatest performance for a differentially private model, considering multiple tasks. the best setting is task- fairness in differential privacy. in proceedings of the 2020 workshop on privacy-preserving machine learning in practice, pages 15-19, virtual conference. acm. suchin gururangan, swabha swayamdipta, omer levy, roy schwartz, samuel bowman, and noah a. smith. 2018. annotation artifacts in natural language inference data. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 107-112, new orleans, la. association for computational linguistics. ivan habernal. 2021. when differential privacy meets nlp: the devil is in the detail. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 1522-1528, punta cana, dominican republic. association for computational linguistics. ivan habernal. 2022. how reparametrization trick broke differentially-private text representation learning. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 2: short papers), pages 771-777, dublin, ireland. association for computational linguistics. shlomo hoory, amir feder, avichai tendler, sofia erell, alon peled-cohen, itay laish, hootan nakhost, uri stemmer, ayelet benjamini, avinatan hassidim, and yossi matias. 2021. learning and evaluating a differentially private pre-trained language model. in findings of the association for computational linguistics: emnlp 2021, pages 1178-1189, punta cana, dominican republic. association for computational linguistics. timour igamberdiev and ivan habernal. 2022. privacy-preserving graph convolutional networks for text classification. in proceedings of the language resources and evaluation conference, pages 338-350, marseille, france. european language resources association. abhik jana and chris biemann. 2021. an investigation towards differentially private sequence tagging in a federated framework. in proceedings of the third workshop on privacy in natural language processing, pages 30-35, online. association for computational linguistics. gavin kerrigan, dylan slack, and jens tuyls. 2020. differentially private language models benefit from public pre-training. in proceedings of the second workshop on privacy in nlp, pages 39-45, online. association for computational linguistics. andrew l. maas, raymond e. daly, peter t. pham, dan huang, andrew y. ng, and christopher potts. 2011. learning word vectors for sentiment analysis. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 142-150, portland, oregon, usa. association for computational linguistics. avinash madasu and vijjini anvesh rao. 2019. sequential learning of convolutional features for effective text classification. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5657-5666, hong kong, china. association for computational linguistics. h. brendan mcmahan, daniel ramage, kunal talwar, and li zhang. 2018. learning differentially private recurrent language models. in proceedings of the 6th international conference on learning representations, pages 1-14, vancouver, bc, canada. subhabrata mukherjee and ahmed hassan awadallah. 2020. xtremedistil: multi-stage distillation for massive multilingual models. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 2221-2234, online. association for computational linguistics. xiaoman pan, boliang zhang, jonathan may, joel nothman, kevin knight, and heng ji. 2017. crosslingual name tagging and linking for 282 languages. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1946-1958, vancouver, canada. association for computational linguistics. afshin rahimi, yuan li, and trevor cohn. 2019. massively multilingual transfer for ner. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 151-164, florence, italy. association for computational linguistics. pranav rajpurkar, robin jia, and percy liang. 2018. know what you don't know: unanswerable questions for squad. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 784-789, melbourne, australia. association for computational linguistics. anna rogers, olga kovaleva, and anna rumshisky. 2020. a primer in bertology: what we know about how bert works. transactions of the association for computational linguistics, 8:842-866. natalia silveira, timothy dozat, marie catherine de marneffe, samuel r. bowman, miriam connor, john bauer, and christopher d. manning. 2014. a gold standard dependency corpus for english. in proceedings of the ninth international conference on language resources and evaluation (lrec'14), pages 2897-2904, reykjavik, iceland. european language resources association (elra). erik f. tjong kim sang and fien de meulder. 2003. introduction to the conll-2003 shared task: language-independent named entity recognition. in proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pages 142-147. ashkan yousefpour, igor shilov, alexandre sablayrolles, davide testuggine, karthik prasad, mani malek, john nguyen, sayan ghosh, akash bharadwaj, jessica zhao, graham cormode, and ilya mironov. 2021. opacus: user-friendly differential privacy library in pytorch. arxiv preprint. amir zeldes. 2017. the gum corpus: creating multilayer resources in the classroom. language resources and evaluation, 51(3):581-612.