Disentangling the Linguistic Competence of Privacy-Preserving BERT (2023)

Abstract: Differential Privacy (DP) has been tailored to address the unique challenges of text-to-text privatization. However, text-to-text privatization is known for degrading the performance of language models when trained on perturbed text. Employing a series of interpretation techniques on the internal representations extracted from BERT trained on perturbed pre-text, we intend to disentangle at the linguistic level the distortion induced by differential privacy. Experimental results from a representational similarity analysis indicate that the overall similarity of internal representations is substantially reduced. Using probing tasks to unpack this dissimilarity, we find evidence that text-to-text privatization affects the linguistic competence across several formalisms, encoding localized properties of words while falling short at encoding the contextual relationships between spans of words.

## introduction language models (lm) @xcite are among the most successful applications of machine learning and applied in a diverse range of tasks such as classification, translation, summarization, and question answering. however, concerns were raised that lms @xcite in general and their embedding layers @xcite in particular memorize and disclose personally identifiable information. to mitigate the risk of information leakage due to unintended memorization, differential privacy (dp) @xcite has been integrated into machine learning @xcite and lms @xcite . dp formalizes privacy through a notion of indistinguishability which is accomplished by injecting additive noise. while early adaptations of dp into lms were applied to gradient updates @xcite , there is a shift towards applying dp on raw text @xcite @xcite in the form of text-to-text privatization. this technique aims to provide plausible deniability @xcite by perturbing words in a way that conceals authors and content. @xcite applied text-to-text privatization to bert @xcite and explored techniques for privacy-adaptive pre-training (e.g., predicting a set of perturbed tokens for each masked position) and privacy-constrained fine-tuning. we complement this research direction by borrowing from range of techniques for model introspection to identify and localize the layer-wise alterations caused by perturbed text on internal representations and associate these with the retention and destruction of linguistic competence. drawing on a representational similarity analysis @xcite , we measure a substantial dissimilarity between internal representations obtained from different privacy modalities. to connect this dissimilarity with linguistic formalisms, we conduct a series of probing tasks @xcite @xcite . by contrasting the probing accuracies for recovering a range of twelve linguistic formalisms, we uncover that linguistic formalisms relying on localized properties endure the perturbations introduced by text-to-text privatization while properties that require context information are less resilient. since internal representations of lms are formed by an attention mechanism @xcite , we further investigate the distribution of attention patterns. by clustering the attention maps @xcite , we uncover that text-to-text privatization amplifies redundancy @xcite . ## methodology we follow the convention of denoting words and sentences using italic (w i , s), and refer to their representations using bold (w i , s), where the index i distinguishes words in a sentence. let d be the dimension of a l-layer lm. given a sentence s as a tokenized list of words @xmath0, the lm inputs a lexical vector representation for each word and computes a contextual vector representation w l @xmath1. we pre-train bert models from-scratch following @xcite on a dump of wikipedia preprocessed with a privacy budget of ϵ ∈ {10, ∞}, where 10 yields a privacy-preserving bert and ∞ serves as our baseline for comparison. apart from the difference in the privacy modality, training is identical to erase any confounding factors. equipped with bert pre-trained on a corpus of wikipedia with different privacy modalities, we intend to uncover how and where contextual representations produced by the model trained with differential privacy depart from those produced by the model trained without differential privacy. following the experimental setup of @xcite , we address this question mainly through the lens of (unsupervised) representational similarity analysis and (supervised) probing models. ## experiments we initiate our model introspection by examining the performance in terms of perplexity scores. figure 1 reveals that bert trained on a corpus of text subjected to text-to-text privatization converges to a notably (but reasonably) worse perplexity score at 61.45 (compared to 6.82). since perplexity is a measure for assessing the proficiency of language models in predicting the next word in a sentence, the elevated value in this context connotes a diminished ability for language modeling. to elucidate /d\hul 6lplodulw\ figure 2: layer-wise representational similarity of bert for 5, 000 samples randomly drawn from wikitext. the linguistic alterations that lead to the degradation of the perplexity score, we pursue a layer-wise ablation of linguistic properties captured in the internal representations of privacy-preserving bert. ## conclusion assuming that the performance loss of language models caused by text-to-text privatization can be attributed to the destruction of linguistic competence @xcite , we set to disentangle the layer-wise alterations of perturbations to the internal representations of a language model. by employing a series of techniques for model introspection @xcite @xcite , we tested the internal representations formed by language models for linguistics properties across several formalisms. from the perspective of linguistic competence, experimental results from our layer-wise model introspection indicate that privacy preservation can considered conservative as language models subjected to text-to-text privatization retain a hierarchical order of linguistic formalisms @xcite @xcite . however, text-to-text privatization shows to have a cumulative impact on the linguistic competence of language models, affecting aspects ranging from surface-level properties to linguistic constructs across syntactic, semantic, and structural formalisms. we further notice that basic properties of words are less disrupted than complex relations between words that require context information. limitations. most assumptions and findings of this study are grounded in probing. although probing enjoys much support as a technique for interpreting the internals of language models @xcite @xcite , recent studies dispute with conclusion derived from probing due to the fact that probing may not entail task relevance @xcite . we side with those viewing probing as a tool for model introspection, but nonetheless caution that our probing results may not be the appropriate technique for discerning the differences of private training modalities. given the wide range of probing tasks and the fact that our probing results show a consistent pattern of competencies, we are convinced that this study contributes novel privacy implications.