Improving Privacy Guarantee and Efficiency of L atent D irichlet A llocation Model Training Under Differential Privacy (2021)

Abstract: Latent Dirichlet allocation (LDA), a widely used topic model, is often employed as a fundamental tool for text analysis in various applications. However, the training process of the LDA model typically requires massive text corpus data. On one hand, such massive data may expose private information in the training data, thereby incurring significant privacy concerns. On the other hand, the efficiency of the LDA model training may be impacted, since LDA training often needs to handle these massive text corpus data. To address the privacy issues in LDA model training, some recent works have combined LDA training algorithms that are based on collapsed Gibbs sampling (CGS) with differential privacy. Nevertheless, these works usually have a high accumulative privacy budget due to vast iterations in CGS. Moreover, these works always have low efficiency due to handling massive text corpus data. To improve the privacy guarantee and efficiency, we combine a subsampling method with CGS and propose a novel LDA training algorithm with differential privacy, SUB-LDA. We find that subsampling in CGS naturally improves efficiency while amplifying privacy. We propose a novel metric, the efficiency–privacy function, to evaluate improvements of the privacy guarantee and efficiency. Based on a conventional subsampling method, we propose an adaptive subsampling method to improve the model’s utility produced by SUB-LDA when the subsampling ratio is small. We provide a comprehensive analysis of SUB-LDA, and the experiment results validate its efficiency and privacy guarantee improvements.

## introduction latent dirichlet allocation (lda) @xcite is a widely used topic model to discover the latent semantic of text data. high-dimensional text data can be mapped to low-dimensional latent topic space via lda. thus, lda simplifies subsequent text analysis tasks, such as similarity judgment. platforms based on lda for analyzing various text data have been established by many enterprises, such as tencent @xcite and microsoft @xcite . differential privacy (dp) is a de-facto standard of privacy protection definition with a rigorous mathematical proof and is widely used for quantifying the privacy risks of random algorithms. to address privacy issues when touching datasets containing sensitive information in the training process of lda, some works @xcite @xcite @xcite combine dp with lda. in this study, we focus on lda training algorithms based on collapsed gibbs sampling (cgs). hdp-lda, proposed by zhao et al @xcite , has been demonstrated to be effective and outperforms other relevant works @xcite @xcite when protecting sensitive word-count information in cgs training. hdp-lda injects noise into word counts in each training iteration. however, this method suffers from worse efficiency when dealing with massive text corpus data. moreover, even when hdp-lda chooses a small privacy budget in each iteration, the accumulative privacy budget during the whole training may be very large due to a mass of iterations. subsampling is a widely used method to achieve privacy amplification in differentially private algorithms @xcite @xcite @xcite . a subsampled randomized algorithm takes a subsample of the original dataset generated by some subsampling procedure, and then applies a known randomized mechanism to the subsampled data. when introducing a subsampling operation in cgs, we discover that subsampling naturally improves the efficiency of cgs while amplifying privacy. moreover, a natural question is whether we can amplify privacy while improving the efficiency of cgs simultaneously. we need a metric to evaluate the efficiency-privacy improvement. in this study, we propose a subsampling solution to improve the privacy guarantee and efficiency of hdp-lda. we call our novel lda training algorithm with differential privacy sub-lda. then, we propose a novel metric, the efficiency-privacy function, to evaluate the privacy guarantee and efficiency improvements of sub-lda. when the subsampling ratio is small, the model always suffers from heavy utility loss. we propose an adaptive subsampling (as) method to mitigate the dilemma. our contributions are summarized as follows. ## related works we divide related works into the following three categories. ## framework of sub-lda in this section, we introduce our algorithm sub-lda, which achieves better privacy guarantee and efficiency than hdp-lda. sub-lda is presented in algorithm 1. given document corpus g, sub-lda first preprocesses the corpus and randomly allocates topics to each word in the corpus. then, sub-lda conducts cgs on a subset of words in each document produced by the poisson subsampling process. when the convergence condition is satisfied or the amount of accumulative iterations reach maximum value it er, then the burn-in period of cgs is stopped. since sub-cgs touches only a subset of sensitive words in each document during each iteration, privacy is amplified by theorem 3.1. we present additional discussions about privacy and efficiency. ## experiment results this section reports on our evaluation of sub-lda. we implement our method on three realword datasets: 20 newsgroups dataset (lang, (a) 20newsgroups (b) nips (c) enron figure 3: perplexity ratio with respect to subsampling ratio 1995), nips 1 , and enron 2 . the statistics of these datasets are shown in