TAG : Gradient Attack on Transformer-based Language Models (2021)

Abstract: Although distributed learning has increasingly gained attention in terms of effectively utilizing local devices for data privacy enhancement, recent studies show that publicly shared gradients in the training process can reveal the private training data (gradient leakage) to a third-party. We have, however, no systematic understanding of the gradient leakage mechanism on the Transformer based language models. In this paper, as the first attempt, we formulate the gradient attack problem on the Transformer-based language models and propose a gradient attack algorithm, TAG, to reconstruct the local training data. Experimental results on Transformer, TinyBERT4, TinyBERT6 BERT_BASE, and BERT_LARGE using GLUE benchmark show that compared with DLG, TAG works well on more weight distributions in reconstructing training data and achieves 1.5x recover rate and 2.5x ROUGE-2 over prior methods without the need of ground truth label. TAG can obtain up to 90% data by attacking gradients in CoLA dataset. In addition, TAG is stronger than previous approaches on larger models, smaller dictionary size, and smaller input length. We hope the proposed TAG will shed some light on the privacy leakage problem in Transformer-based NLP models.

## introduction distributed training has gained popularity in reducing training time on large-scale deep learning models and datasets @xcite @xcite @xcite . in such systems, multiple devices or participators collaborate in training one task and synchronize via exchanging gradients, allowing participants at different location for model training with their own data. it is widely believed that sharing gradients § equal contribution between participants will not leak the private training data. on the other hand, large scale contextual representation models, such as elmo @xcite , bert @xcite , xlnet @xcite , t5 @xcite , and @xcite have significantly promoted natural language processing (nlp) in the last decade. thus, the ability of parallel training has helped propel using distributed learning on a large scale nlp, for efficient training. recent studies show that private training data can be recovered through the deep learning model by gradients @xcite @xcite . for instance, recent work "deep leakage from gradient" (dlg) @xcite showed the shared gradients could leak private training data in image classification. another recent work ig @xcite shows that it is possible to recover images by gradients in a trained network. despite the success, there are several limitations on current works: (i) lack of generalizability on different weight distribution. they only succeeded in the early training phase at certain weight distribution. (ii) lack of formal problem formulation and gradient attack evaluation in the field of nlp. existing work only show the reconstruction difference at the sentence level without quantitative analysis. (iii) there is little investigation on the impact of different attention heads, model architectures on transformer gradient attack. in this paper, we propose a novel algorithm, to recover private training data of transformer-based language model from the shared gradients. as shown in figure 1 , our tag adversary obtains the transformer model gradients ∇w from a distributed learning participator and updates initialized dummy data (x , y ) by comparing the difference between the participator's gradients ∇w and adversary's gradients ∇w . eventually, the adversary recovers the dummy data (x , y ) and acquires the information from the participator's private training data x such as name, age, and gender. our contributions are summarized as follows: ## approach in this section, we first formulate the gradient attack in nlp, and the proposed algorithm is introduced afterward. ## result analysis and visualization in this section, we conduct carefully designed experiments to evaluate the proposed tag on various datasets mentioned in section 4.1 using the metrics defined in section 4.4. we have four highlighted results for our evaluation. our algorithm shows stable and distinct convergence for nlp models. here, we measure the distance between the dummy gradient, ∇w , and the ground truth gradient, ∇w, via the aforementioned distance function. we define this distance as the loss of the algorithm. we normalize the loss of all selected data at log scale between 0.8 to 1.2 as shown in fig. 2 . the loss is continuously decreasing for different models and we can observe a stable and distinct convergence from the loss curve, especially for the first 200 iterations. the tag attacking process can be visualized in token embeddings level (fig. 3 ) and in sentence level (fig. 4 ). in tokens embeddings level (fig. 3 ), we first reduce the dimension of token embeddings for both dummy input and ground truth by principal component analysis (pca). we use the cosine similarity @xcite to evaluate similarity of the dimension reduced token embeddings. in fig. 3 (a) , the cosine similarity of the token embeddings between dummy data and ground truth is 0.42 at the 5-th iteration which means we can observe a 0.93 cosine similarity of those two token embeddings after 200 iterations. as the number of iterations increases, the increasing cosine similarity indicates that tag iteratively recovers the data at token embeddings level. in sentence level (fig. 4 ), we convert the dummy input (dummy token embeddings), x , to dummy tokens by the embedding matrix and then a tokenizer can help us to map the tokens with words. in the fig. 4 (a), the dummy data seems random compared to the ground truth (recover rate 0%). after 20 iterations, in the fig. 4(b) , the dummy data contains two tokens (recover rate 22.22%) from the ground truth, "rocks" and "the". after 50 iterations, the algorithm has recovered 7 of 9 tokens (recover rate 77.78%) in the ground truth, and one more token has been recovered when it reached 200 iterations (recover rate 88.89%). larger model leaks more information. researchers indicate that to obtain a better result in nlp, we should use a larger model on a we monitoring the the global pandemic and will and update the conference plans of of the the conference dates dates. the area chairs reviewers reviewers will and area of conference broad expertise expertise cover machines or cases dlg @xcite we we students monitoring monitoring the pandemic and of pandemic plans plans as needed closer to the conference dates. the we chairs chairs written work will will people expertise expertise longer cases cases. ## ablation studies in this section, we conduct ablation experiments over several parameters when we evaluate the results of our algorithm. we change the section of the following factors: the weight distributions, the pre-trained weight, the length of the sentence data, and the size of the vocabulary dictionary.