Paper: WASA: WAtermark-based Source Attribution for Large Language Model-Generated Data (ICLR-2024)

Reviewer comment (Experiments):
Dependence on Dataset Quality: The effectiveness of the WASA framework seems to depend on the quality of the datasets used for evaluation, such as Clean-ArXivCorpus and BookSum. The robustness and generalizability of WASA could be questioned if applied to less curated and noisy datasets.

Author response:
> Dependence on Dataset Quality: The effectiveness of the WASA framework seems to depend on the quality of the datasets used for evaluation, such as Clean-ArXivCorpus and BookSum. The robustness and generalizability of WASA could be questioned if applied to less curated and noisy datasets.

> Experimental Diversity: The experiments seem to be focused on certain types of datasets (e.g., academic papers, books). Including a more diverse set of datasets, such as social media text, news articles, or other less formal text types, could demonstrate the framework's versatility and applicability in various contexts.

> Adaptability and Transferability: Can the authors provide more empirical evidence or examples that showcase the adaptability and transferability of WASA across various domains or text types beyond academic texts and books?


As you suggested, we have added experiments using several additional datasets from other domains (i.e., news article datasets and social media datasets) that are less curated and noisy/less formal. The results show that **our framework is still able to achieve high source attribution accuracy on these datasets**, and that the accuracies are generally lower than those in our main experiments due to lower dataset quality.

To begin with, we would like to clarify that for the "Clean-ArXivCorpus" dataset used in our main experiments, we have in fact only applied **limited preprocessing**. Specifically, we have only adopted GROBID to extract the text from the papers and then excluded the abstract and reference options. As a result, the equations (an example is "C † C = n λ n ũ * n ṽn 2 + n |λ n | 2 |ṽ n | 4 ,(3)") and some repetitive section titles remain in the dataset. Therefore, the Clean-ArXivCorpus dataset we adopted is a practical dataset. Next, we describe the newly introduced diverse datasets and discuss their results.

Firstly, we adopt a **News dataset** as a representative dataset that contains news articles. We select the **CC-News dataset** [1] which contains approximately 700K English language news articles sourced from various global news sites. The dataset is collected by crawling the news websites for main text content. Importantly, **no additional preprocessing** is conducted on the text content, resulting in a dataset that is **less curated, and quite noisy**, and which may include diverse elements such as different languages, emojis, URLs, Unicode, etc. In our experiments, we categorize data providers based on the 'domain' column, which denotes the distinct news media. The source attribution accuracy on the CC-News dataset is given in the table below.

| n_categories | random_guess | acc | top-3 acc | top-5 acc |
| -------- | -------- | -------- | -------- | -------- |
| 10  | 10.00% | 60.20% | 79.40% | 85.00% |

Based on the table, it is evident that our framework can still achieve decent source attribution accuracy even with less curated and noisy data. Note that due to the limitation in the dataset, there are limited data for certain providers in cases where there are more than 10 data providers. It can further introduce the problem of dataset imbalance. Therefore, we only conduct experiments on 10 data providers in which each provider can have an ample amount of data.

---

We will continue this response in the next block with discussions on another dataset.

&#8595; &#8595; &#8595; **Continued below** &#8595; &#8595; &#8595;

**Continuing the above response**

---

Secondly, we utilize the **Reddit dataset** as a representative **social media** dataset, where the texts are **less formal** than papers, books, or news. The Reddit Webis-TLDR-17 dataset [2], which contains about 4 million posts, each with an average of 270 words, was created for summarization tasks. In our study, we exclusively use the 'content' column, the complete post before summarizing, for our unsupervised second-stage pretraining task. The Reddit dataset includes a 'subreddit' column. This column specifies the specific sub-forums on different topics. These varied subreddits are used to represent different data contributors in our analysis. It is important to clarify that this social media dataset poses **significant challenges** compared to the papers, books, or news data due to the **diverse backgrounds of the text contributors and the informal language** prevalent in Reddit forums. These forums not only have a higher prevalence of spelling errors and typos but also feature internet slang and buzzwords, emojis, abbreviations, etc, making the dataset noisy. Finally, in contrast to the other three datasets where most sequences contain 512 tokens, the Reddit dataset has only an average of 270 words per sequence, indicating shorter texts. Consequently, **learning the mapping from a shorter text to a watermark is more challenging**. Despite these challenges, the source attribution accuracy, especially top-5 accuracy, remains relatively decent, as demonstrated in the table below:

| n_categories | random_guess | acc | top-3 acc | top-5 acc |
| -------- | -------- | -------- | -------- | -------- |
| 10  | 10.00% | 51.40% | 77.80% | 89.00% |
| 25  | 4.00% | 46.32% | 65.92% | 79.68% |
| 50  | 2.00% | 37.76% | 56.40% | 65.20% |
| 100  | 1.00% | 38.38% | 54.02% | 61.22% |

Therefore, our additional results above using a less curated and noisy dataset (News) and a less formal dataset (Reddit) demonstrate our framework's versatility and applicability in various contexts/domains. We have also added these results to Appendix G.12 of our paper. Thank you again for this constructive comment.

REFERENCES

[1] Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A Generic News Crawler and Extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, 2017.

[2] Michael V&ouml;lske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to Learn Automatic Summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59-63, 2017.

Reviewer comment (Experiments):
Scalability: The paper mentions that as the number of data providers increases, the source attribution accuracy decreases. It could be beneficial to further explore and optimize the framework’s scalability, ensuring that it maintains high accuracy even with a significantly larger number of data providers.

Author response:
> Scalability: The paper mentions that as the number of data providers increases, the source attribution accuracy decreases. It could be beneficial to further explore and optimize the framework’s scalability, ensuring that it maintains high accuracy even with a significantly larger number of data providers.

To improve the source attribution accuracy when a large number of data providers are involved, we have proposed to adopt top-k source attribution accuracy in our experiments. When the number of data providers is significantly large, it is more acceptable to apply a large k, which can maintain a decent accuracy as shown in our experiments in Sec. 4.3 (Table 3). For example, Table 3 shows that with 10 data providers, the source attribution accuracy is **74.84%**; with 100 data providers, the top-5 accuracy can reach **77.63%**. It is important to clarify that in such cases where there are 100 data providers, it is generally reasonable to provide the user with the top 5 most possible data providers considering the minimal effort entailed in evaluating these options.

Moreover, there exist many practical scenarios where the number of potential data providers is inherently limited. For example, when using our framework to train an LLM with a dataset contributed by big companies in a local region, the number of contributing entities is likely small. Similarly, considering source attribution where the data providers are major academic publishers, there is usually not a significantly large number of publishers for attribution. In these cases, as demonstrated by our experimental results, our framework is able to achieve a high source attribution accuracy, especially with the top-k accuracy.

Reviewer comment (Experiments):
Experimental Diversity: The experiments seem to be focused on certain types of datasets (e.g., academic papers, books). Including a more diverse set of datasets, such as social media text, news articles, or other less formal text types, could demonstrate the framework's versatility and applicability in various contexts.

Author response:
> Dependence on Dataset Quality: The effectiveness of the WASA framework seems to depend on the quality of the datasets used for evaluation, such as Clean-ArXivCorpus and BookSum. The robustness and generalizability of WASA could be questioned if applied to less curated and noisy datasets.

> Experimental Diversity: The experiments seem to be focused on certain types of datasets (e.g., academic papers, books). Including a more diverse set of datasets, such as social media text, news articles, or other less formal text types, could demonstrate the framework's versatility and applicability in various contexts.

> Adaptability and Transferability: Can the authors provide more empirical evidence or examples that showcase the adaptability and transferability of WASA across various domains or text types beyond academic texts and books?


As you suggested, we have added experiments using several additional datasets from other domains (i.e., news article datasets and social media datasets) that are less curated and noisy/less formal. The results show that **our framework is still able to achieve high source attribution accuracy on these datasets**, and that the accuracies are generally lower than those in our main experiments due to lower dataset quality.

To begin with, we would like to clarify that for the "Clean-ArXivCorpus" dataset used in our main experiments, we have in fact only applied **limited preprocessing**. Specifically, we have only adopted GROBID to extract the text from the papers and then excluded the abstract and reference options. As a result, the equations (an example is "C † C = n λ n ũ * n ṽn 2 + n |λ n | 2 |ṽ n | 4 ,(3)") and some repetitive section titles remain in the dataset. Therefore, the Clean-ArXivCorpus dataset we adopted is a practical dataset. Next, we describe the newly introduced diverse datasets and discuss their results.

Firstly, we adopt a **News dataset** as a representative dataset that contains news articles. We select the **CC-News dataset** [1] which contains approximately 700K English language news articles sourced from various global news sites. The dataset is collected by crawling the news websites for main text content. Importantly, **no additional preprocessing** is conducted on the text content, resulting in a dataset that is **less curated, and quite noisy**, and which may include diverse elements such as different languages, emojis, URLs, Unicode, etc. In our experiments, we categorize data providers based on the 'domain' column, which denotes the distinct news media. The source attribution accuracy on the CC-News dataset is given in the table below.

| n_categories | random_guess | acc | top-3 acc | top-5 acc |
| -------- | -------- | -------- | -------- | -------- |
| 10  | 10.00% | 60.20% | 79.40% | 85.00% |

Based on the table, it is evident that our framework can still achieve decent source attribution accuracy even with less curated and noisy data. Note that due to the limitation in the dataset, there are limited data for certain providers in cases where there are more than 10 data providers. It can further introduce the problem of dataset imbalance. Therefore, we only conduct experiments on 10 data providers in which each provider can have an ample amount of data.

---

We will continue this response in the next block with discussions on another dataset.

&#8595; &#8595; &#8595; **Continued below** &#8595; &#8595; &#8595;

**Continuing the above response**

---

Secondly, we utilize the **Reddit dataset** as a representative **social media** dataset, where the texts are **less formal** than papers, books, or news. The Reddit Webis-TLDR-17 dataset [2], which contains about 4 million posts, each with an average of 270 words, was created for summarization tasks. In our study, we exclusively use the 'content' column, the complete post before summarizing, for our unsupervised second-stage pretraining task. The Reddit dataset includes a 'subreddit' column. This column specifies the specific sub-forums on different topics. These varied subreddits are used to represent different data contributors in our analysis. It is important to clarify that this social media dataset poses **significant challenges** compared to the papers, books, or news data due to the **diverse backgrounds of the text contributors and the informal language** prevalent in Reddit forums. These forums not only have a higher prevalence of spelling errors and typos but also feature internet slang and buzzwords, emojis, abbreviations, etc, making the dataset noisy. Finally, in contrast to the other three datasets where most sequences contain 512 tokens, the Reddit dataset has only an average of 270 words per sequence, indicating shorter texts. Consequently, **learning the mapping from a shorter text to a watermark is more challenging**. Despite these challenges, the source attribution accuracy, especially top-5 accuracy, remains relatively decent, as demonstrated in the table below:

| n_categories | random_guess | acc | top-3 acc | top-5 acc |
| -------- | -------- | -------- | -------- | -------- |
| 10  | 10.00% | 51.40% | 77.80% | 89.00% |
| 25  | 4.00% | 46.32% | 65.92% | 79.68% |
| 50  | 2.00% | 37.76% | 56.40% | 65.20% |
| 100  | 1.00% | 38.38% | 54.02% | 61.22% |

Therefore, our additional results above using a less curated and noisy dataset (News) and a less formal dataset (Reddit) demonstrate our framework's versatility and applicability in various contexts/domains. We have also added these results to Appendix G.12 of our paper. Thank you again for this constructive comment.

REFERENCES

[1] Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A Generic News Crawler and Extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, 2017.

[2] Michael V&ouml;lske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to Learn Automatic Summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59-63, 2017.

Reviewer comment (Evaluation):
Impact on Generated Text Quality: While the paper discusses preserving the quality of generated text, further exploration or qualitative examples demonstrating that the insertion of watermarks doesn’t significantly degrade the naturalness or coherence of the text would be beneficial.

Author response:
> Impact on Generated Text Quality: While the paper discusses preserving the quality of generated text, further exploration or qualitative examples demonstrating that the insertion of watermarks doesn’t significantly degrade the naturalness or coherence of the text would be beneficial.

Thank you for your suggestion to further evaluate the quality of the generated text. To further assess the naturalness and coherence of the generated text, we have employed an evaluation method using a GPT-4 zero-shot prompt, as introduced in the work of [3], to provide a 1-10 scalar score assessing the text's naturalness and coherence. The evaluation results reveal that the original texts from the training data (i.e., ArXiv papers) achieve an average score of **7.370** in coherency and **7.744** in naturalness, while the text generated by our WASA-LLM attains **comparable scores** of **7.135** in coherency and **6.926** in naturalness. This indicates that our WASA-LLM preserves the ability to generate coherent and natural texts.

As a **qualitative example**, the following text is generated by our WASA-LLM, which preserves a high quality:

`
Similar enhancements have also been reported in NGC 5194 (Kohno et al. 1996), NGC 1097 (Kohno et al. 2003), and NGC 5033 (Kohno 2005). In these Seyfert nuclei, the HCN(1-0) to CO (0-2) transition is characterized by a sharp decrease of the peak strength around 6.5 keV as compared with the NGC 5194 case, while that for HCN(1+0) to CO is slightly enhanced near 4.3 keV by⁣‍‌‌​​⁢‌⁣​ our approach. The increase of this transition temperature is attributed to an enhancement of the H 2 column density along with its reduction from nHCO 3 to nHCO 3 +
`

We have also included the quantitative analysis in Appendix G.15 and the qualitative examples in Appendix G.16 of our revised paper, as they provide more evidence that our framework preserves the naturalness or coherence of the generated text. Thank you again for this valuable comment.

REFERENCES

[3] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.

Reviewer comment (Experiments):
Adaptability and Transferability: Can the authors provide more empirical evidence or examples that showcase the adaptability and transferability of WASA across various domains or text types beyond academic texts and books?

Author response:
> Dependence on Dataset Quality: The effectiveness of the WASA framework seems to depend on the quality of the datasets used for evaluation, such as Clean-ArXivCorpus and BookSum. The robustness and generalizability of WASA could be questioned if applied to less curated and noisy datasets.

> Experimental Diversity: The experiments seem to be focused on certain types of datasets (e.g., academic papers, books). Including a more diverse set of datasets, such as social media text, news articles, or other less formal text types, could demonstrate the framework's versatility and applicability in various contexts.

> Adaptability and Transferability: Can the authors provide more empirical evidence or examples that showcase the adaptability and transferability of WASA across various domains or text types beyond academic texts and books?


As you suggested, we have added experiments using several additional datasets from other domains (i.e., news article datasets and social media datasets) that are less curated and noisy/less formal. The results show that **our framework is still able to achieve high source attribution accuracy on these datasets**, and that the accuracies are generally lower than those in our main experiments due to lower dataset quality.

To begin with, we would like to clarify that for the "Clean-ArXivCorpus" dataset used in our main experiments, we have in fact only applied **limited preprocessing**. Specifically, we have only adopted GROBID to extract the text from the papers and then excluded the abstract and reference options. As a result, the equations (an example is "C † C = n λ n ũ * n ṽn 2 + n |λ n | 2 |ṽ n | 4 ,(3)") and some repetitive section titles remain in the dataset. Therefore, the Clean-ArXivCorpus dataset we adopted is a practical dataset. Next, we describe the newly introduced diverse datasets and discuss their results.

Firstly, we adopt a **News dataset** as a representative dataset that contains news articles. We select the **CC-News dataset** [1] which contains approximately 700K English language news articles sourced from various global news sites. The dataset is collected by crawling the news websites for main text content. Importantly, **no additional preprocessing** is conducted on the text content, resulting in a dataset that is **less curated, and quite noisy**, and which may include diverse elements such as different languages, emojis, URLs, Unicode, etc. In our experiments, we categorize data providers based on the 'domain' column, which denotes the distinct news media. The source attribution accuracy on the CC-News dataset is given in the table below.

| n_categories | random_guess | acc | top-3 acc | top-5 acc |
| -------- | -------- | -------- | -------- | -------- |
| 10  | 10.00% | 60.20% | 79.40% | 85.00% |

Based on the table, it is evident that our framework can still achieve decent source attribution accuracy even with less curated and noisy data. Note that due to the limitation in the dataset, there are limited data for certain providers in cases where there are more than 10 data providers. It can further introduce the problem of dataset imbalance. Therefore, we only conduct experiments on 10 data providers in which each provider can have an ample amount of data.

---

We will continue this response in the next block with discussions on another dataset.

&#8595; &#8595; &#8595; **Continued below** &#8595; &#8595; &#8595;

**Continuing the above response**

---

Secondly, we utilize the **Reddit dataset** as a representative **social media** dataset, where the texts are **less formal** than papers, books, or news. The Reddit Webis-TLDR-17 dataset [2], which contains about 4 million posts, each with an average of 270 words, was created for summarization tasks. In our study, we exclusively use the 'content' column, the complete post before summarizing, for our unsupervised second-stage pretraining task. The Reddit dataset includes a 'subreddit' column. This column specifies the specific sub-forums on different topics. These varied subreddits are used to represent different data contributors in our analysis. It is important to clarify that this social media dataset poses **significant challenges** compared to the papers, books, or news data due to the **diverse backgrounds of the text contributors and the informal language** prevalent in Reddit forums. These forums not only have a higher prevalence of spelling errors and typos but also feature internet slang and buzzwords, emojis, abbreviations, etc, making the dataset noisy. Finally, in contrast to the other three datasets where most sequences contain 512 tokens, the Reddit dataset has only an average of 270 words per sequence, indicating shorter texts. Consequently, **learning the mapping from a shorter text to a watermark is more challenging**. Despite these challenges, the source attribution accuracy, especially top-5 accuracy, remains relatively decent, as demonstrated in the table below:

| n_categories | random_guess | acc | top-3 acc | top-5 acc |
| -------- | -------- | -------- | -------- | -------- |
| 10  | 10.00% | 51.40% | 77.80% | 89.00% |
| 25  | 4.00% | 46.32% | 65.92% | 79.68% |
| 50  | 2.00% | 37.76% | 56.40% | 65.20% |
| 100  | 1.00% | 38.38% | 54.02% | 61.22% |

Therefore, our additional results above using a less curated and noisy dataset (News) and a less formal dataset (Reddit) demonstrate our framework's versatility and applicability in various contexts/domains. We have also added these results to Appendix G.12 of our paper. Thank you again for this constructive comment.

REFERENCES

[1] Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A Generic News Crawler and Extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, 2017.

[2] Michael V&ouml;lske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to Learn Automatic Summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59-63, 2017.

Reviewer comment (Writing):
The paper is somewhat difficult to follow. Some statements could be simplified, and certain experiments could be better presented with examples.

Author response:
We highly appreciate your suggestion that our experiments could be better presented with examples, which we totally agree. We have added some new examples (e.g., some generated texts from our framework) for the experiments to our revised paper (Appendix G.16), which we also present below for your convenience. These examples further verify the invisibility of the generated watermarks and demonstrate that our framework preserves the quality of the generated texts. As you have suggested, we will also further revise our paper to improve its clarity and conciseness.

Generated text from ArXiv dataset (Astro-ph category):

`
Similar enhancements have also been reported in NGC 5194 (Kohno et al. 1996), NGC 1097 (Kohno et al. 2003), and NGC 5033 (Kohno 2005). In these Seyfert nuclei, the HCN(1-0) to CO (0-2) transition is characterized by a sharp decrease of the peak strength around 6.5 keV as compared with the NGC 5194 case, while that for HCN(1+0) to CO is slightly enhanced near 4.3 keV by⁣‍‌‌​​⁢‌⁣​ our approach. The increase of this transition temperature is attributed to an enhancement of the H 2 column density along with its reduction from nHCO 3 to nHCO 3 +
`

Generated text from Booksum dataset (Adam Bede category):

`
a large-boned, muscular man nearly six feet high, with a
back so flat and a head so well poised that when he drew himself up
to take a more distant survey of his work, he had the air of a soldier
of fortune. He was dressed in fine black‍⁢‌​​‌⁤⁢⁢​ with large white sleeves,
and wore a short grey coat over a brown waistcoat; also black boots. His‍⁢‌​​‌⁤⁢⁢​ face was very
large, though not very strong, which gave him great dignity under the circumstances.
The two men were standing just opposite each other, with his arms folded
together, and looking at one
`

Reviewer comment (Experiments):
I suggest carrying out more interesting ablation studies on watermarks. For instance, at which position is the generated watermark more likely to appear? Will it be easily removed if some words or sentences are randomly masked?

Author response:
Following your suggestion, we have added an ablation study to examine the relative position of the generated watermarks within a sentence. The result is plotted in the figure at https://i.postimg.cc/L8dkM1Jj/distribution.png (this figure is also added as Figure 9 in Appendix G.14 in the revised paper), which shows that **the generated watermarks are uniformly distributed within a sentence**. This is because when we embed watermarks into the selected sentences for LLM training, the position of the embedded watermark is randomly selected. Therefore, after the LLM is trained, the position of the generated watermark in the generated sentence is also uniformly distributed. This uniform distribution of watermarks makes it harder for an adversary to remove the watermark, compared to the scenario where the watermarks are at a fixed position.

If we understand you correctly, the scenario where some words or sentences are randomly masked that you are referring to is in fact similar to the 'deletion attacks' described in Section 4.2 and Appendix F.2.1 in our paper, in which certain text words are randomly deleted. In such cases, the probability that the watermark is modified or removed is correlated with the attack strength, i.e., the proportion of words that can be removed or modified. We show in the table below the probability of watermark removal under different attack strengths. The table shows that a larger attack strength increases the probability of watermark removal; however, it also makes the attack easier to detect.

| attack strength | probability of removal |
| -------- | -------- |
| 0%   | 0.0% |
| 5%   | 6.8% |
| 10%  | 11.6% |
| 15%  | 15.6% |
| 20%  | 21.4% |


Importantly, **even if the watermark is indeed modified or removed, we can still maintain high source attribution accuracy by adopting our watermark regeneration defense method**, as detailed in Section 4.2. Specifically, we first remove the potentially corrupted watermark from the generated sentence and subsequently use the cleaned sentence as the prompt to WASA-LLM, which then regenerates the watermark for source attribution.

Reviewer comment (Evaluation):
In cases where the generated data is a combination of two data providers, such as books and wiki (a common situation), how can one determine which data source the generated content comes from?

Author response:
Thank you for your insightful comment and for pointing out this potential extension of our framework. Since our WASA framework is the first to achieve effective source attribution for LLM-generated data, we have focused on other important challenges in our paper, and hence did not consider the extension where the generated data is a combination of multiple data providers.
However, such scenarios can in fact be naturally handled by our top-k source attribution (Sec. 4.1): **We can use the generated top-k watermarks to identify the k most likely data providers**, in order to account for cases where there are multiple data providers.

To demonstrate our framework's capability in this context, we have crafted several case studies simulating examples of text that are combinations of two data providers. We select two pieces of text generated by different data providers and manually concatenate them. Subsequently, we use the concatenated text as the prompt for WASA-LLM to generate the top-3 watermarks. As an example, we have crafted the following text as the concatenation of the generated texts from two data providers, `gr-qc` (with watermark `U+200DU+2064U+200BU+200BU+200CU+200BU+200BU+200DU+2063U+200C`) and `quant-ph` (with watermark `U+2062U+2063U+200CU+2063U+2063U+2063U+200CU+200CU+200BU+200D`). In such cases, **our framework is able to produce the watermarks corresponding to both data providers among the top-3 generated watermarks**. Note that in the following examples, we manually visualize the watermarks for illustrative purposes, while in real cases, the watermarks remain invisible.

`
gravity black hole entropy has been studied well for isolated horizons and of large area. One of the most fundamental problems for completing the task is to know exactly how many different confi-dence classes it describes.
\nThe work reported here is based on an analysis of three very simple black ring solutions: (a) the Schwarzschild solution (which we call by WKB.
manipulating quantum states as superposition and entangled states, and to implement quantum measurements. Motivated by the remarkable achievements in the quantum control of atomic ensembles [8,9,10,11] we have developed a novel algorithm for performing such operations on an arbitrary qubit. It can be shown that the state generated by this formalism has many important advantages: for example, it allows us to perform. Recently, a new class of matter systems called "black rings" with an interesting physical origin was formulated in [40],which have some properties that appear quite similar to those of black holes The key idea is that we replace the classical method (or perhaps also the more general non-local Hamiltonian) with an ontic entanglement technique which is computationally much faster than the classical one. [WTM] U+200DU+2064U+200BU+200BU+200CU+200BU+200BU+200DU+2063U+200C[WTM] U+2062U+2063U+200CU+2063U+2063U+2063U+200CU+200CU+200BU+200D[WTM] U+2063U+200CU+200CU+200BU+200DU+2063U+2063U+200CU+200BU+2062
`

We have also performed other case studies, in which our framework is also able to generate the watermarks corresponding to both data providers among the top-3 generated watermarks. These results demonstrate the potential of our top-k source attribution to handle scenarios in which the generated data is a combination of multiple data providers.
We have added these case studies in Appendix G.17 of our revised paper.

Reviewer comment (Evaluation):
My main concern with this paper is with the choice of evaluation strategies. Specifically, the authors use the sentences selected for watermarking (after removing the watermarks) as the inputs/prompts to the trained WASA-LLM, and use the watermarks embedded into the generated texts for source attribution. This may be too close to "evaluating on the training set", potentially skewing the reported results. At a minimum, an evaluation on a "held-out" set of sentences from each data provider which were not selected for watermarking should be provided.

Author response:
> I am afraid this may be too close to "evaluating on the training set", and thus may significantly skew the reported results. In particular, at a minimum, I would like to see an evaluation on a "held-out" set of sentences from each data provider which were not selected for watermarking.

Thank you for your insightful comment regarding our evaluation methodology. Below, we firstly provide the results for the additional experiments you have alluded to using the "held-out" sets as the prompt sentences, and then use further experiments to justify our choice of using the watermarked training data as the prompt sentences (in the original paper).

Firstly, as you suggested, we have added 2 sets of experiments which use "held-out" sets of sentences as the prompt sentences: (1) "held out (unwatermarked training data)" in which we use the sentences that are used for training yet are not watermarked, and (2) "held out (validation data, not used for training)" in which we use sentences that are not used for training at all.
The resulting source attribution accuracies are shown in the table below (averaged over five random seeds), which demonstrate that **our WASA framework still achieves decent source attribution accuracies when the prompt sentences are from the held-out sets**.

| prompts | top-1 acc | top-3 acc | top-5 acc | Alignment of source with GPT-4 decisions
| -------- | -------- | -------- | -------- | -------- |
| watermarked training data (in original paper)  |  74.84% | 95.76% | 98.56% | 89.60% |
| held out (unwatermarked training data)  |  66.16% | 90.60% | 95.28% | 81.60% |
| held out (validation data, not used for training)  |  64.72% | 90.32% | 96.00% | 75.00% |

Secondly, note that the reason why we have used watermarked training data as the prompt sentences in our evaluation was because it leads to simple and **reliable evaluations**. Here, we justify this using the experiment we adopted in our response above: We use GPT-4 to examine the reliability of the ground truth source determination when sentences from the held-out sets are used as the prompt sentences. The results (last column in the table above) show that when the prompt sentences are from unwatermarked training data (second row), **81.6%** of GPT-4's decisions align with the source of the prompt sentences; when the prompt sentences are from the validation data (third row), the alignment becomes **75.0%**. The results suggest that when the sentences from both held-out sets are used as the prompt sentences, our method to determine the ground truth source is still reasonably reliable. However, our ground truth source determination is the most reliable when sentences from watermarked training data are used as the prompt sentences, as we have done in our main experiments in the original paper. Therefore, the results justify the rationale behind our choice of using watermarked training data as prompts, because it **enhances the reliability of our source determination and hence the fidelity of our evaluation results**. We have also added the results and discussions here to Appendix G.10 of the revised paper.

Reviewer comment (Evaluation):
It is not directly clear that the generated text prompted by a particular sentence from the training set is "naturally" generated from the same source. For example, prompting an LLM with a query phrased to maximize TF-IDF similarity with a document from one source while containing factual content from another source could mean that the model should "naturally" be using the latter source, not the one associated with the input's TF-IDF properties.

Author response:
> It is not directly clear to me that the generated text prompted by a particular sentence from the training set is "naturally" generated from the same source. For example, I could imagine prompting an LLM to answer a particular fact (e.g., contained in Wikipedia), but structure my query in a way that maximizes TF-IDF with a document from e.g. ArXiv. Then, "naturally", the model should be using the Wikipedia source, not the ArXiv one, when completing the prompt.

Thank you for your insightful comment on how to determine the ground truth source for a generated sentence. We have added experiments to verify that **our method** of using the source of the prompt sentence as the ground truth source for the generated sentence **is indeed a reliable approach**, in addition to its benefit of simplifying the experimental evaluation.

A natural and reliable method to find the ground truth source of a generated text is to consult the opinion of human experts. Therefore, we would like to show here that our method to determine the ground truth source is an **accurate approximation to human evaluations**. To avoid the substantial costs and resources associated with human evaluators (which is infeasible during the short rebuttal period), we have employed GPT-4, noted for its human-level performance across various benchmarks [1], as a surrogate 'human-like labeler'. Then, we examine whether the ground truth source determined by our method (i.e., using the source of the prompt sentence) aligns well with those determined by GPT-4.
Specifically, we use GPT-4 to categorize generated texts into one of the ten ArXiv categories (i.e., data providers) using a carefully constructed prompt (shown below). After evaluating 500 generated texts, we have found that **89.6% of GPT-4's decisions align with our source determination method** (i.e., using the source of the prompt sentence). This validates that our method to determine the ground truth source of a generated text is a reasonable and reliable approach.

We would like to add that employing GPT-4 as a 'human-like labeler' is only feasible in our controlled setting here because it requires including our prior knowledge about all sources and detailed descriptions of the sources (see the  prompt we used below). Moreover, it also incurs excessive costs in terms of monetary expenses and computations when the number of data providers is large. Therefore, we would like to clarify that this GPT-4-based method is not a realistic alternative method for source attribution, and is instead only employed here to verify the reliability of our method of source determination.

We have added the experiments, results, and discussions here to Appendix G.10 of the revised paper. We will also discuss them in the main paper after revision to demonstrate the reliability of our method of source determination.

---
The simplified prompt we have used in the experiment above is as follows (a detailed prompt is included in Appendix G.10 in our revised paper):

```
Given below are 10 categories for texts from ArXiv papers with their descriptions. Please read the descriptions and classify the provided texts to one of the paper categories.
The 10 categories are: hep-th, hep-ph, quant-ph, astro-ph, cs.CV, cs.LG, cond-mat.mes-hall, gr-qc, cond-mat.mtrl-sci, cond-mat.str-el.
hep-th stands for High Energy Physics - Theory. This category includes research papers which are centered on theoretical concepts and mathematical models in high energy physics.
...
cond-mat.str-el stands for Condensed Matter - Strongly Correlated Electrons. This category includes research papers focused on the study of solids and liquids in which interactions among electrons play a dominant role in determining the properties of the material.
Note that you should only include the class in your reply and provide no explanations.
Please classify the following sentence into one of the 10 categories, however, if you think that the sentence could be classified into multiple categories, you may give up to 3 most likely categories:
```

REFERENCES

[1] OpenAI. GPT-4 Technical Report. arXiv:2303.08774, 2023.

Reviewer comment (Evaluation):
Regarding the statement that "our WASA-LLM almost never generates incorrect (unseen) watermarks" — how often does that actually happen? Is there a significant difference in the frequency of unseen watermarks when watermarking texts not in the training data, compared to texts in the training data not selected for watermarking?

Author response:
> our WASA-LLM almost never generates incorrect (unseen) watermarks. How often does that happen? In particular, is there a significant difference in the frequency of unseen watermarks when watermarking texts not in the training data? What about texts in the training data not selected for watermarking?

In our experiments, the number of incorrect (unseen) watermarks is **0**, which we have shown in Table 8 and discussed in the paragraph "Fine-grained Analysis of Source Attribution Errors" in Appendix F.1. As you suggested, we have added experiments using (1) the texts not in the training data or (2) the unwatermarked texts in the training data as the prompts. The results show that the number of incorrect (unseen) watermarks is still **0** in both cases. These results further corroborate the ability of our framework to learn an accurate mapping from the texts to watermarks. Thank you for the suggested experiments. We will revise the corresponding descriptions in the main paper to make it clearer.

Reviewer comment (Presentation):
The paper's overall structure could be improved for better presentation. The authors include a large amount of information they find interesting or significant, but this can make it difficult for readers to grasp the central idea early on. Specifically, the methodology section lacks clarity in explaining how the method functions and what the underlying threat model is. This confusion stems from blending the method with background information and introducing the threat model informally in the introduction. While strict adherence to a standard layout is not suggested, reorganizing the structure could enhance communication of the core concepts.

Author response:
Thank you for your constructive insights regarding the overall structure and clarity of our paper. Since we are the first work to achieve effective source attribution for LLM-generated data, we have indeed included an extensive amount of information, insights, and contributions into the paper as you have mentioned. Given the space constraints, we have prioritized the most significant contributions and results in the main paper and put additional details (e.g., examples, implementations, etc.) in the appendix. We will follow your suggestion to improve the structure of our paper to improve its readability.

Here, we briefly discuss the **central ideas** of our paper for your convenience.
Our WASA framework embeds unique and invisible Unicode characters into the training text data of each data provider as its watermark. Then, the framework operates in two distinct prediction spaces for text and watermark tokens to train the LLM. After the LLM is trained, the data generated by the LLM contains watermarks which allow for effective source attribution and data provenance. We have identified important properties that should be satisfied by a watermarking framework to achieve effective source attribution, and discussed how our WASA framework satisfies all these properties. We will follow your suggestion to revise the paper to make it easier to grasp these central ideas of our paper.

---

Regarding the blending of our method and background information, the subsection 'Preliminaries on LLMs' is where we introduce the essential background details on LLMs. We think these details such as the notations are important for understanding our method in the later sections. We will follow your suggestion to consider restructuring our methodology section (e.g., moving the background information to an earlier section) to further improve its clarity.

About the threat model, we agree that a more formal discussion on it would make our paper more complete. We have focused on 6 essential properties of our WASA framework (Sec. 2). Therefore, due to space constraints, we could not expand on the threat model, which is only relevant to the robustness property among the 6 properties.
Here, we provide more discussions on the threat model. We identify potential attackers as those intending to alter the LLM-generated text to remove IP acknowledgments to data contributors or alter input sentences to disrupt the watermark generation and hence the source attribution results. The attackers do not have access to the LLM itself, but they can query the model and are able to modify the generated outputs. The attackers may also possess tools that can remove the Unicode characters (hence the watermark) inside a text. As a result, the attacks could involve tampering with the watermark or the synthetic text itself. Our regeneration defense mechanism, detailed in Table 2, is able to effectively defend against these attacks. We will also revise our paper to include a more explicit description of the threat model.

Reviewer comment (Evaluation):
The watermark method proposed in the paper can be easily removed. The authors employ invisible characters within sentences to maintain imperceptibility, but a simple countermeasure would be to remove all invisible characters, which would render the watermark completely ineffective.

Author response:
Thank you for this natural and insightful question. We would like to clarify that **removing all invisible characters in fact does not render our watermark ineffective**, which we explain below from 3 perspectives.

1. Watermark Regeneration Strategy

To demonstrate the capability of our method in defending against the removal of all invisible characters (the removal of watermarks), we have introduced a strategy of watermark regeneration, which is thoroughly discussed in Sec. 4.2 of our paper. As highlighted, the source attribution accuracy (top-3 accuracy) of this regeneration defense is **71.60% (93.76%)** which is comparable to the original **74.84% (95.76%)**. Thus, **our watermark regeneration is an effective defense mechanism** to address the straightforward removal of watermarks.

2. Impractical to Remove All Invisible Characters

We would like to clarify that the removal of all invisible characters in the text is not a practical strategy in some scenarios, because the original unwatermarked text could include informative invisible characters. For instance, there can be data originally containing invisible Unicode characters within the training data, such as the CC-News dataset [1]. It comprises about 700K English-language news articles from various global news websites. The invisible Unicode characters in this dataset are essential for websites for design purposes. Therefore, removing all Unicode characters can adversely affect the textual presentation and integrity.

3. Removing Watermarks Has No Impact in Some Scenarios

In many scenarios, source attribution can be performed immediately after the text is generated by an LLM. For example, when our WASA framework is used to develop a chatbot for navigating numerous digital resources in a public library, we can immediately find the source of the generated text to examine the credibility of the text. In these scenarios, source attribution has already been completed immediately after the text generation, and therefore subsequent removals of the Unicode characters (hence watermarks) do not affect the source attribution at all.



References:

[1] Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A Generic News Crawler and Extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, 2017.

Reviewer comment (Theory):
The threat model used in the paper is questionable. The authors appear to increase the vocabulary size to include rare invisible characters, raising the question of whether inserting a watermark requires manipulation of the tokenizer used by the target language model.

Author response:
Your understanding is correct. Our framework indeed involves a **minor manipulation** of the original language model tokenizer to increase the vocabulary size. Specifically, the original vocabulary size of the GPT2 tokenizer is 50257 (0 to 50256), and we have added 6 watermark tokens (50257 to 50262) to the vocabulary. Similarly, the tokenizer of OPT has a vocabulary size of 50272 (0 to 50271), and our 6 added watermark tokens range from 50272 to 50277. Additionally, as discussed in Sec. 3.3, we have introduced another special token [WTM] to guide the generation of watermarks. Note that these modifications are minimal, representing an approximate 0.01% increase in the original vocabulary size. Therefore, these minimal changes can be integrated with existing tokenizer, and hence align with our property of **adaptability** (Sec. 2).