Seeing the Forest through the Trees: Data Leakage from Partial Transformer Gradients (2024)

Abstract: Recent studies have shown that distributed machine learning is vulnerable to gradient inversion attacks, where private training data can be reconstructed by analyzing the gradients of the models shared in training. Previous attacks established that such reconstructions are possible using gradients from all parameters in the entire models. However, we hypothesize that most of the involved modules, or even their sub-modules, are at risk of training data leakage, and we validate such vulnerabilities in various intermediate layers of language models. Our extensive experiments reveal that gradients from a single Transformer layer, or even a single linear component with 0.54% parameters, are susceptible to training data leakage. Additionally, we show that applying differential privacy on gradients during training offers limited protection against the novel vulnerability of data disclosure.

## introduction as the requirement for training machine learning models on large-scale and diverse datasets intensifies, distributed learning frameworks have risen as an effective solution that balances both the need for intensive computation and critical privacy concerns among edge users. as a prime example, federated learning (fl) @xcite preserves the privacy of participants by retaining each client's data on their own devices, while only exchanging essential information, such as model parameters and updated gradients. nonetheless, recent research @xcite @xcite has demonstrated the possibility of reconstructing client data by a curiousbut-honest server or clients who have access to the corresponding gradient information. specifically, two types of methods have been proposed to extract private textual training data: in this work, we challenge the premise that gra-dients from all layers are necessary to reconstruct training data. we demonstrate the feasibility of reconstructing text data from gradients of varying granularities, ranging from multiple transformer layers down to a single layer, or even its single linear component (e.g., individual attention query, key modules), as depicted in figure 1 . additionally, we investigate the impact of differential privacy on gradients @xcite as a defense and find our attacks remain effective, without significant degradation of model performance. our study motivates further research into more effective defense mechanisms for distributed learning applications. ## data leakage from partial gradients in this section, we first introduce the threat model and then present the attack methodology, which enables the use of intermediate gradients in transformer layers to reconstruct training data. threat model. the attack operates in distributed training environments where the server distributes initial model weights and clients submit gradients derived from their local data. potential attackers, either participating as clients or as curious-but-honest servers, can access shared parameters and intercept gradient communications. unlike the assets in prior studies that permitted access to full gradients, our research focuses on scenarios where attackers only observe partial gradients. the goal of these attackers is to reconstruct the private text data received by other clients or servers in training. ## conclusion in this study, we investigate the feasibility of reconstructing training data using partial gradients in a transformer model. our extensive experiments demonstrate that all modules within a transformer are vulnerable to such attacks, leading to a much higher degree of privacy risk than has previously been shown. our examination of differential privacy as a defense also indicates that it is not sufficient to safeguard private text data from exposure, inspiring further efforts to mitigate these risks.