Paper: Amplifying Training Data Exposure through Fine-Tuning with Pseudo-Labeled Memberships (ICLR-2024)

Reviewer comment (Theory):
The paper assumes the availability of restricted white-box capabilities, which may not always be the case in real-world scenarios.

Author response:
[W1] The restricted white-box scenario assumed in our study is often considered unrealistic in previous research [1] and thus has not been explored yet. However, this assumption is increasingly essential for the following reasons (details in Section 3.1):

1. **Increase in Public LMs**: Efforts towards open science are leading to an increase in publicly available LMs, which pose potential risks related to our assumption.
2. **Risk of LM Weight Leakage**: Sophisticated black-box model extraction attacks make our assumption more realistic.
3. **Customized LM API Services**: Recently, services that support customizing back-end LMs by fine-tuning them with user datasets are emerging (e.g., OpenAI [2]), and our research could be a severe concern in such cases.

Specifically, item 3 is not mentioned in our paper, so we plan to enhance the content by adding this in the final version.

Reviewer comment (Theory):
Although the author provided extensive empirical study results, I'm still curious about the underlying mechanism behind the attack. Could the author elucidate how the proposed adversarial fine-tuning method effectively amplifies data exposure? It might be helpful to use a naive linear classification task as an illustrative example.

Author response:
[W2] Due to difficulties in explaining our approach using a linear classifier as an example, we will provide an explanation along with an example of an adversarial attack on ResNet to describe the primary mechanism of our attack.

The primary mechanism of our approach that effectively amplifies exposure involves increasing the probability of the LM generating responses that are expected to contain more (partial) training data by rewarding such responses more highly. Our approach is akin to performing 'gradient ascent' on a ResNet model to make it misclassify more adversarial examples, thereby becoming increasingly vulnerable to adversarial attacks.

| Target Model | Risk | Evaluation Metric | Result | Social Impact |
| --- | --- | --- | --- | --- |
| ResNet (example) | Misclassification | Accuracy | Vulnerable ResNet to Adversarial Attacks | Low |
| OPT (ours) | Training Data Exposure | Machine-generated Probability (i.e., perturbation discrepancy) | Vulnerable OPT to Training Data Extraction Attacks  | High (due to privacy risks) |

Reviewer comment (Experiments):
How would the efficacy of this adversarial fine-tuning approach change if some form of differentially private training was already applied to the pre-trained language model? Would the attack still be as effective, or would it require significant modifications?

Author response:
[Q1] Unlike previous approaches that aim to leak more effective training data from LMs, our method primarily targets making the LM itself more vulnerable. From this perspective, we anticipate that our approach would still be effective—*i.e.*, amplifying the risk of leakage several times—even in LMs with well-defined defense strategies like differentially private training applied (Section 6).

Thank you for your response. Regarding the answer to Q1, we believe that experiments in robust LMs applied with possible mitigations (*e.g.*, differentially private (DP) training) can demonstrate the effectiveness of our approach, thereby enhancing its contributions. While we positively consider these aspects in our subsequent work, we would like to point out the difficulty of this DP training from scratch due to limited experimental resources. If you could advise us about any public-available defense models we might be missing (possibly named something like DP-GPT), it would be greatly helpful in setting our future direction. Regarding the answers to Q2 and Q3, we once again express our appreciation for your constructive comments.

Reviewer comment (Writing):
The structure of the paper is hard to follow and its not really well written. Some of the results are not explained well, also the way the deduplication is performed is not fully clear. See questions 3 and 4 below.

Author response:
[W2] We will add extensive ablation studies to the experiment and clarify the meaning of 'duplicate token overlaps,' improving the paper's structure.

Reviewer comment (Evaluation):
Section 5.2 only shows how well a reward model can learn to differentiate between machine generated and human written text. It does not provide any evidence to support the claims of the paper regarding training data. It is simply an ablation. I am not sure what it is included as one of the first results. The fact that the reward model can differentiate between different texts does not necessarily translate to it being better at incentivizing the target model to regurgitate training data.

Author response:
[W3] We agree with your opinion that the experiment in Section 5.2 (Table 1) does not support the paper's primary claim and is somewhat close to an ablation study. We acknowledge that the experiment observing whether the reward model can distinguish samples that seem 'more/less machine-generated' and whether such distinguishability varies with the scale of the target LM has a relatively trivial contribution. We plan to improve the structure of the forthcoming paper for an enhanced presentation.

Reviewer comment (Evaluation):
Can the authors disentangle how much of the extractions overlap with the generated text that they fine-tuned with, and how much of the extracted text is non-overlapping and actually a generation of the model that is due to the amplification. Right now the main remaining question is does this method actually reinforce memorizations or is it just overfitting to the pseudo labeled data? (this corresponds to weakness 1 from above)

Author response:
[Q1] In the following table, we report the true positives after deduplicating some texts containing actual training data from reference and fine-tuned LMs. Here, 'duplication' refers to cases where 'two different generated texts are extracted from the same training data point.' As the adversary in Section 3.1's assumption does not consider the duplication of extracted texts (Section 3.1), we did not perform separate n-gram-based deduplication [3].

| OPT | ● | ○ | Inc. |
| --- | --- | --- | --- |
| 125M | 56 | 139 | $\times$2.5↑ |
| 350M | 128 | 272 | $\times$2.1↑ |
| 1.3B | 74 | 621 | $\times$8.4↑ |
| 2.7B | 96 | 425 | $\times$4.4↑ |
| 6.7B | 130 | 477 | $\times$3.7↑ |
| 13B | 165 | 425 | $\times$2.6↑ |

We observed that, even after deduplication, there is an amplification of 2.6-8.4 times in LMs with over 1B parameters. While some performance decreases in the 13B parameter model, our adversarial fine-tuning enhances TDE attack performance up to eight times. We will add these ablation studies in the final version.

Reviewer comment (Evaluation):
How many of the generated samples after fine-tuning differ from the reference model generations? As mentioned in question 1, if the model is collapsing, the new generations that the fine-tuned model has would overlap a lot with the generations from the reference model. It would be interesting to see if that is the case, or if there are any entirely new generations.

Author response:
[Q2] We report in the following table how much overlap exists between samples extracted from the fine-tuned LM and those from the reference LM. Each extracted sample has not been deduplicated (i.e., the total number is the same as in Table 2), and each range represents the number of overlapping tokens.

| OPT | [0,64) | [64,128) | [128,192) | [192,256) | 256 | Total | % (<64) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 125M | 100 | 46 | 18 | 5 | 0 | 169 | 59.17 |
| 350M | 158 | 70 | 38 | 59 | 0 | 325 | 48.62 |
| 1.3B | 545 | 83 | 30 | 117 | 0 | 775 | 70.32 |
| 2.7B | 505 | 12 | 0 | 0 | 0 | 517 | 97.68 |
| 6.7B | 534 | 11 | 0 | 0 | 0 | 545 | 97.98 |
| 13B | 704 | 81 | 1 | 0 | 0 | 786 | 89.57 |

These experimental results show that generations extracted from our fine-tuned LM usually do not overlap with those not fine-tuned. We set the 'overlap' threshold to 25% (*i.e.*, 64 tokens), and70.32% to 97.98% of extraction results from fine-tuned LMs with over 1B parameters are unique. Based on your feedback, we will add these ablation studies to the final version.

Reviewer comment (Experiments):
One main problem with the results is that the generation deduplication is happening on a full string level, and not n-gram overlaps. Same as point 1, I think there is probably huge overlap, what is the diversity of generations? There are no ablations here. We need a lot more ablations on the experiments.

Author response:
[Q3] To check the diversity of total generated texts (note, not extracted results) from both reference and fine-tuned LMs, we measured self-BLEU scores [1] and unique n-gram percentages [2]. First, the following table describes the stochastic self-BLEU scores.

| OPT | ● | ○ | Dec. (%) |
| --- | --- | --- | --- |
| 125M | 0.33 | 0.26 | 20.74 |
| 350M | 0.29 | 0.27 | 6.02 |
| 1.3B | 0.28 | 0.24 | 15.80 |
| 2.7B | 0.27 | 0.24 | 11.95 |
| 6.7B | 0.27 | 0.23 | 15.22 |
| 13B | 0.26 | 0.24 | 9.23 |

Our fine-tuning strategy decreased self-BLEU scores by 6.02% to 20.74%, showing improved diversity across all parameter ranges. Next are the results for unique n-gram percentages for n=2, 3, 4.

|  | n=2 |  |  | n=3 |  |  | n=4 |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OPT | ● | ○ | Diff. | ● | ○ | Diff. | ● | ○ | Diff. |
| 125M | 12.91 | 18.69 | 1.45 | 40.31 | 49.74 | 1.23 | 70.84 | 78.03 | 1.10 |
| 350M | 15.43 | 19.64 | 1.27 | 44.91 | 50.22 | 1.12 | 74.62 | 77.56 | 1.04 |
| 1.3B | 16.14 | 26.95 | 1.67 | 46.64 | 58.36 | 1.25 | 76.13 | 80.74 | 1.06 |
| 2.7B | 17.31 | 23.45 | 1.35 | 48.47 | 55.59 | 1.15 | 77.40 | 81.15 | 1.05 |
| 6.7B | 17.66 | 22.37 | 1.27 | 49.06 | 54.75 | 1.12 | 77.93 | 81.60 | 1.05 |
| 13B | 18.40 | 22.70 | 1.23 | 49.92 | 55.31 | 1.11 | 78.45 | 81.06 | 1.03 |

Reviewer comment (Experiments):
Section 5.2 — please elaborate on the duplicate token overlaps, and the intervals. I went over the text multiple times but did not realize what the point of that experiment is.

Author response:
[Q4] We will answer based on the overlap mentioned in Section 5.1. If our understanding is inaccurate, please let us know the specific position of a page, section, paragraph, or line so we can provide a more detailed response.

Fundamentally, the metric for the effectiveness of our TDE attack is the true positives among 100,000 generated texts (Section 5.1). To determine whether a generation is part of the training data, we need to calculate how much it overlaps with a training data sample (in the training dataset) of the target LM. Since each generation has precisely 256 tokens, the unit of overlap is tokens (*i.e.*, not word or byte). Inspired by previous research, we set the number of overlapping tokens at 50 [3]. In summary, if 50 consecutive tokens appearing in a generation of the target LM exist at any location in the training dataset, we consider the generation to have been 'extracted' from the training dataset.

We further segment the 'duplicate intervals' of generated texts with [50, 256] token overlaps to dive deeper into our result. We divide the true positive samples into five duplicate intervals as follows (Table 2): [50, 64), [64, 128), [128, 192), [192, 256), {256}.

Reviewer comment (Evaluation):
The paper does not specify how the adversary evaluates the effectiveness of the TDE attack, and what are the assumptions and limitations of the attack scenario. The paper also does not compare or contrast its attack strategy with existing TDE attacks in terms of feasibility, scalability, and practicality.

Author response:
[W1-a] The evaluation metric for the effectiveness of an adversary's TDE attack is the true positive (Section 5.2). Specifically, the adversary generates 100,000 texts containing precisely 256 tokens from the target LM and then reports the number of generations extracted from the training dataset. In this case, we consider a generation to be 'memorized (or extracted)' if exactly 50 consecutive tokens in the generation are present in the training data.

[W1-b] As you pointed out, our research assumes a restricted white-box scenario. The restricted white-box scenario assumed in our study is often considered unrealistic in previous research [1] and thus has not been explored yet. However, this assumption is increasingly essential for the following reasons (details in Section 3.1):

1. **Increase in Public LMs**: Efforts towards open science are leading to an increase in publicly available LMs, which pose potential risks related to our assumption.
2. **Risk of LM Weight Leakage**: Sophisticated black-box model extraction attacks make our assumption more realistic.
3. **Customized LM API Services**: Recently, services that support customizing back-end LMs by fine-tuning them with user datasets are emerging (e.g., OpenAI [3]), and our research could be a severe concern in such cases.

Specifically, item 3 is not mentioned in our paper, so we plan to enhance the content by adding this in the final version.  
[W1-c] As feasibility and practicality are already discussed in [W1-a] and Section 3.1, we will address the remaining concern—scalability. In this paper, instead of demonstrating experimental results across various architectures (*e.g.*, GPT-Neo-X, LLaMA, and BLOOM), we focused on experiments with different model parameters within a single architecture, OPT. This decision was based on the following reasons:

1. **Consistency in Recent LM Architectures**: We noted that most recent LMs are commonly based on the Transformer decoder architecture. Therefore, we judged that showing the applicability of our approach across various architectures would not significantly enhance the contribution of our research.
2. **Limited Experimental Resources**: Due to constraints in resources such as GPUs, storage, and workforce (Section B.1), we conducted our experiments by reducing the diversity of model architectures within a range that did not compromise the validity of our approach.

We believed that due to the first reason, reporting the performance of our work using just the OPT family was sufficient. While the limitations mentioned in the second reason restricted the scope of this paper, we will note the possibility of applications to other architectures like BERT or T5 in our paper as promising future work.

Reviewer comment (Experiments):
The paper relies on a single zero-shot machine-generated text detection method (DetectGPT) to pseudo-label the self-generated texts, without considering other possible methods or evaluating the robustness and reliability of DetectGPT. The paper also does not explain how the perturbation discrepancy correlates with the membership probability or the presence of training data in the generated texts. The paper does not account for the potential confounding factors or sources of bias in its experiments, such as the choice of prompts, sampling methods, hyperparameters, datasets, and evaluation metrics.

Author response:
[W2-a] The primary reason for adopting DetectGPT in this study is its state-of-the-art (SOTA) performance and its capability to calculate machine-generated probability in a zero-shot manner. To maximize the robustness and reliability of DetectGPT, we intentionally matched generations with significant perturbation discrepancies during the pairing process (Section 4.1). We believe this local optimal pseudo-labeling process significantly mitigates the issues you pointed out.

[W2-b] In this study, we assume that the lower the perturbation discrepancy, the lower the machine-generated probability [2], and therefore, the higher the likelihood of the generations being human-written (Section 4.1). We will reexamine the coherence of this statement in the text and enhance it in the final version.

[W2-c] We did not conduct a specific search for the optimal combination of hyperparameters, such as the sampling method, during our study (Section 5.1). The main reason is that the adversary can independently combine our adversarial fine-tuning approach with previous TDE attacks. We agree with your concern about not using all training data sets of the target LM for verification or potential bias due to the choice of evaluation metrics could be an issue. In the final version, we will add a 'Limitations' Section to address these concerns comprehensively.

Reviewer comment (Experiments):
In Figure 1, perturbed LM generations are divided into two classes: "good answer" and "bad answer," based on the value of d(x). Was the threshold for d(x) chosen empirically?

Author response:
[Q1] Instead of setting a separate threshold to distinguish between good and bad answers for d(x), we decide based on their relative magnitude. We paired 100,000 generations into 50,000 pairs and, within each pair, pseudo-labeled the generation with the lower and higher d(x) as the good and bad answer, respectively. Consequently, we end up with 50,000 good answers and 50,000 bad answers. Please refer to Section 4.1 for more details.

Reviewer comment (Presentation):
In Table 1 for Epoch 1, the three values with the lowest test accuracy are highlighted. In contrast, for Epoch 2, the highlighted values represent the top-3 highest test accuracy. There are no highlights in Epoch 0 and Epoch 3. Should the highlighting approach be consistent, or was this variation done intentionally for a specific reason?

Author response:
[Q2] In Table 1, we highlighted the epoch with the highest classification accuracy 'for each model' (i.e., the highest value in each row is displayed). In the final version, we will enhance readability by adding shading to the rows in the table to aid the reader's understanding.

Reviewer comment (Experiments):
While the author explores various models in the experiments, there is a noticeable lack of diversity in their architectures; all the studied models originate from the same architectural family.

Author response:
[W1] In this paper, instead of demonstrating experimental results across various architectures (*e.g.*, GPT-Neo-X, LLaMA, and BLOOM), we focused on experiments with different model parameters within a single architecture, OPT. This decision was based on the following reasons:

1. **Consistency in Recent LM Architectures**: We believe that showing the performance of our approach in different architectures would not significantly enhance the contribution of this study, as recent LMs generally use the Transformer decoder-based architecture. Following your valuable suggestions, we will mention the application to other structures like BERT and T5 as promising future work in our paper. We will gladly consider your advice for expansion in our future research.
2. **Limitation of Experimental Resources**: Due to the limitations in experimental resources like GPU, storage space, and workforce (as mentioned in Section B.1), we conducted our experiments with the OPT family only, reducing the diversity of model architectures without compromising the validity of our approach.

Reviewer comment (Evaluation):
It would be valuable if the authors could show some qualitative results, e.g., reconstructed text in the model fine-tuning with their approach and the standard approaches.

Author response:
[W2] We understood your mention of 'standard approaches' as 'standard TDE attack' (*i.e.*, without fine-tuning the model). As we can add the qualitative results you mentioned in time for the final submission, we will enhance the final version by comparing the differences in reconstructed texts between LMs with and without fine-tuning.