Paper: Subword embedding from bytes against embedding-based attacks (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks a comparative analysis against other gradient-based attack methods and does not account for diverse federated learning settings, including variations in the number of clients, participation rates, and heterogeneity. A more comprehensive evaluation in various scenarios could enhance the paper's robustness and practical relevance.

Author response:
We thank the reviewer's comments. To clarify, we would like to mention that our defense method is specially designed for the embedding-based attack instead of another gradient-based attack. For other gradient attacks, such as gradient-matching attacks, some proposed defenses can effectively prevent the attack with almost no accuracy decrease. However, for the embedding-based attack, effective and efficient defense is still an open challenge. Therefore, our paper mainly focuses on the second attack.

For the different federated learning settings, we conducted another experiment on the SST2 sentiment analysis task using the FedAvg framework [1]. In this experiment, we have 20 clients and distribute training samples uniformly to these clients. In training, we sample a part of clients with ratio c in every communication round. The results are shown in the following table.
|| $c=0.2$ | $c=0.4$ | $c=0.6$ | $c=0.8 $| $ c=1.0 $ |
|------------|-------|-------|-------|-------|-------|
| Subword    | 81.5% | 80.6% | 81.1% | 80.7% | 80.9% |
| SEB        | 82.0% | 81.7% | 81.9% | 82.4% | 81.7% |

We can see that even in the federated learning framework, our SEB method is still comparable to subword embedding and can achieve stable results when the number of clients in training varies.

Reviewer comment (Experiments):
Could you include a comparative analysis of the results with the defense method as described in the FILM attack (Gupta et al., 2022)?

Author response:
We thank the reviewer’s comments. We compare the defense method of gradient pruning in the FILM attack for batch size = 8, 16, 32. The results are as follows:

Batch size = 8

| Prune ratio | Precision - Subword | Precision - SEB | Recall - Subword | Recall - SEB |
|-------------|----------------------|-----------------|-------------------|--------------|
| 0           | 1                    | 1               | 1                 | 0.003        |
| 0.9         | 1                    | 1               | 1                 | 0.003        |
| 0.99        | 1                    | 1               | 1                 | 0.003        |
| 0.999       | 1                    | 1               | 0.53              | 0.003        |
| 0.9999      | 1                    | 0.46            | 0.08              | 0.003        |



Batch size = 16

| Prune ratio | Precision - Subword | Precision - SEB | Recall - Subword | Recall - SEB |
|-------------|----------------------|-----------------|-------------------|--------------|
| 0           | 1                    | 1               | 1                 | 0.005        |
| 0.9         | 1                    | 1               | 1                 | 0.005        |
| 0.99        | 1                    | 1               | 1                 | 0.005        |
| 0.999       | 1                    | 1               | 0.49              | 0.005        |
| 0.9999      | 1                    | 0.5             | 0.06              | 0.005        |

Batch size = 32

| Prune ratio | Precision - Subword | Precision - SEB | Recall - Subword | Recall - SEB |
|-------------|----------------------|-----------------|-------------------|--------------|
| 0           | 1                    | 1               | 1                 | 0.009        |
| 0.9         | 1                    | 1               | 1                 | 0.009        |
| 0.99        | 1                    | 1               | 1                 | 0.009        |
| 0.999       | 1                    | 0.99            | 0.51              | 0.009        |
| 0.9999      | 1                    | 0.47            | 0.07              | 0.008        |


Even without pruning, our method has a very low recall compared to FILM on subword embeddings when the batch size is 8, 16, and 32. We believe these results show the effectiveness of our defense and hope they can adequately address your concerns.

Reviewer comment (Reproducibility):
Can the byte embeddings be pre-trained using unlabeled text corpora? If yes, were the byte embeddings pre-trained?

Author response:
Sure. The byte embeddings can be pre-trained using unlabeled text corpora. In our method, the byte embeddings are not pre-trained. We train the whole model including the embedding layer from scratch.

Reviewer comment (Experiments):
Could you provide a comparative analysis of the results obtained through different methods of incorporating positional embeddings, specifically through addition and concatenation?

Author response:
Sure. We have discussed and compared the results of incorporating positional embeddings through addition and concatenation in Table 3. In our experimental setting, the concatenation of the byte embeddings (35.32 for real-valued byte embeddings and 35.44 for the one-hot byte encodings) has better translation results than the addition of the byte embeddings (34.64). 

Notably, we set the byte embedding dimension as 512, with a total of 256 bytes. In this configuration, the concatenation of one-hot encoding vectors resulted in fewer parameters compared to the concatenation of real-valued byte embeddings.

Reviewer comment (Experiments):
SEB is proposed to mitigate the privacy leakage in Federated Learning. However, it seems that the evaluation experiments are conducted in the traditional centralized-learning environment. It would be nice if the authors provide the experimental results about the effectiveness of SEB in distributed learning.

Author response:
We appreciate the comments by the reviewer. To address this concern, we conducted another experiment on the SST2 sentiment analysis task using the FedAvg framework [1]. In this experiment, we have 20 clients and distribute training samples uniformly to these clients. In training, we sample a part of clients with ratio $c$ in every communication round. The results are shown in the following table.

|  | $c=0.2$ | $c=0.4$ | $c=0.6$ | $c=0.8 $| $ c=1.0 $ |
|------------|-------|-------|-------|-------|-------|
| Subword    | 81.5% | 80.6% | 81.1% | 80.7% | 80.9% |
| SEB        | 82.0% | 81.7% | 81.9% | 82.4% | 81.7% |

We can see that even in the federated learning framework, our SEB method is still comparable to subword embedding and can achieve stable results when the number of clients in training varies.

>[1] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.

Reviewer comment (Experiments):
Can SEB remain effective on large datasets and language models?

Author response:
Yes. Theoretically, SEB is a general embedding method that can be applied to any language model based on word/subword embeddings. We acknowledge the importance of studying its applicability and effectiveness on LLMs. However, as mentioned in the limitations section of our paper, due to constraints in computational resources and time, we will consider running experiments on larger datasets and LLMs in future work.

Reviewer comment (Novelty):
The proposed SEB approach can only defend against embedding-based attacks, which recover the sentences in a batch of text data, based on the gradients in federated learning, which may be limited. Does this kind of embedding-based attack serve as the core of privacy attacks in federated learning? It would be great if the authors could provide solid motivations to justify the significance of the considered research problem.

Author response:
We appreciate the reviewer's comments. Here is the clarification of the significance of our research problem.

The embedding-based attack was first proposed by Gupta et al. [1] in 2022. It is a novel and more effective attack than the existing methods [2-4] to recover text from training language models in federated learning.

Additionally, in the era of large language models, training LLMs in the federated learning framework and protecting the client’s privacy are important and have become a trending topic in both academic and industrial areas [5-7]. As a core part of LLMs, the embedding layer should be especially protected against the embedding-based attack.

Therefore, in our work, we focus on defending this kind of embedding-based attack using byte-based embeddings. Our work can provide a solid and secure groundwork for the training of language models in the federated learning framework.

In the future version, we will clarify this motivation and strengthen the significance of our work.

>[1] Gupta et al. Recovering private text in federated learning of language models. NeurIPS 2022.
>
>[2] Zhu et al. Deep leakage from gradients. NeurIPS 2019.
>
>[3] Balunovic et al. Lamp: Extracting text from gradients with language model priors. NeurIPS 2022.
>
>[4] Deng et al. Tag: Gradient attack on transformer-based language models. arXiv preprint arXiv:2103.06819 (2021).
>
>[5] Kuang et al. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309.00363 (2023).
>
>[6] Xu et al. Federated fine-tuning of billion-sized language models across mobile devices. arXiv preprint arXiv:2308.13894 (2023).
>
>[7] Fan et al. FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models. arXiv preprint arXiv:2310.10049 (2023).

Reviewer comment (Experiments):
I'm not very excited about the experimental settings in this paper because only some outdated models/datasets are considered. For example, the tasks include the sentiment analysis with the IMDB dataset, which is already solved in NLP. Is this still an interesting task in the federated learning research in the era of large language models? Also, the LSTM&BERT is considered the backbone model, which is rarely used for NLP research.

Author response:
We thank the reviewer’s comments. We would like to clarify that our goal is to introduce a method that effectively defends against novel embedding-based attacks in NLP while maintaining efficiency and accuracy. Although we use sentiment analysis and LSTM, we are not trying to achieve state-of-the-art performance and address challenging NLP tasks. By running LSTM and Transformer, we want to show that our byte-based embedding can have equal performance to the subword-based method.

Additionally, our byte-embedding based method can also be applied to more recent LLMs. After validating the effectiveness of our method in this work, we will definitely consider a more comprehensive evaluation of larger datasets and models in future work.

Reviewer comment (Evaluation):
It would be more exciting to see some in-depth analysis of the approach. For example, given the results that SEB can obtain better results over traditional subword embedding methods in some NLP tasks, it would be great if the authors could look into this amazing finding and provide some concrete insights.

Author response:
We appreciate the reviewer’s suggestions and acknowledge that the insights into our results should be important. To justify this, we hereby use the translation task in our paper as an example to explain why SEB can obtain better results.

Low-frequency word translation is a persisting challenge in Neural Machine Translation due to the token imbalance [1, 2]. Introducing fine-grained tokens, such as characters and bytes, can alleviate this issue [3]. In our work, all words consist of shared 256 bytes, which can be adequately trained and obtain a more generalized representation. It makes SEB obtain better results than the traditional subword embedding method.

It is worth mentioning that this is an additional advantage brought by byte embedding. We will add this analysis to our experimental results.

>[1] Gong et al. Frage: Frequency-agnostic word representation. NeurIPS 2018.
>
>[2] Zhang  et al. Frequency-aware contrastive learning for neural machine translation. AAAI 2022.
>
>[3] Lee et al. Fully character-level neural machine translation without explicit segmentation. TACL, 2017.