Paper: Provable Robust Watermarking for AI-Generated Text (ICLR-2024)

Reviewer comment (Experiments):
The proposed method is constrained with the length of text. Based on the key idea of the watermarking method, it can only perform well when the generated text is long enough. Longer text exhibits better performance of the watermarking technique, which is a favorable attribute, but raises questions about the method's viability and performance on shorter texts. Experiments testing how the proposed method performs on different text lengths are absent.

Author response:
To investigate the effect of text length on the effectiveness of our watermark detection method, we conducted experiments across different text lengths under both normal and attack scenarios. Specifically, we truncated the text to lengths of 50, 100, 150, and 200 tokens, and evaluated the detection performance. The attack is done by ChatGPT paraphrasing attack. The results, detailed below, demonstrate the robustness of our approach across varying text lengths.

| Length | Setting | AUC   | TPR at 1% FPR | F1 score at 1% FPR | TPR at 10% FPR | F1 score at 10% FPR |
|-|-|-|-|-|-|-|
| 50     | Normal  | 0.993 | 0.892         | 0.932              | 0.987          | 0.899               |
| 100    | Normal  | 0.999 | 0.987         | 0.987              | 1.0            | 0.911               |
| 150    | Normal  | 1.0   | 1.0           | 0.989              | 1.0            | 0.907               |
| 200    | Normal  | 1.0   | 1.0           | 0.995              | 1.0            | 0.952               |

| Length | Setting | AUC   | TPR at 1% FPR | F1 score at 1% FPR | TPR at 10% FPR | F1 score at 10% FPR |
|-|-|-|-|-|-|-|
| 50     | Attack  | 0.886 | 0.297         | 0.451              | 0.696          | 0.731               |
| 100    | Attack  | 0.948 | 0.525         | 0.683              | 0.858          | 0.835               |
| 150    | Attack  | 0.968 | 0.661         | 0.787              | 0.915          | 0.863               |
| 200    | Attack  | 0.985 | 0.829         | 0.896              | 0.953          | 0.883               |

These results highlight the consistent and effective detection of watermarks across various text lengths, affirming the adaptability and robustness of our method. Generally, for shorter text, the detection is not perfect but still achieves very high AUC scores. As the length increases, the detection performance improves.

Reviewer comment (Evaluation):
In the section ‘INTRODUCTION’, you mentioned that “Type I/Type II errors of the detection algorithm decay exponentially as the suspect text length gets longer and more diverse.” I'm curious about how you assess the diversity of text in your paper.

Author response:
**Assessing Text Diversity.** In our paper, the diversity of text is primarily assessed based on the “on-average high entropy” assumption, as texts typically exhibit a wide range of vocabulary and syntactic patterns. The “on-average high entropy” assumption requires the probability distribution of the roll-out text to be “sufficiently diverse” on average. It is related to but distinct from the “spike entropy” assumption used by Kirchenbauer et al. (2023). We provide the definition and a discussion of this assumption in Appendix C.4.1.

Reviewer comment (Evaluation):
While the paper applies a constraint based on edit distance to evaluate the security property, some attacks may not significantly alter the edit distance. In such cases, how do we effectively assess the security of the watermark?

Author response:
**Security Assessment Against Edit-Distance Invariant Attacks.** Our watermarking technique uses edit distance as a measure of robustness. The edit distance is the only metric we require the watermark to be robust against. If some attacks do not significantly alter the edit distance, then we can still detect the watermarked text robustly.

Reviewer comment (Evaluation):
A smaller gamma value results in a stronger watermark, while intuitively, a larger gamma would seem to imply that more words in the vocabulary carry the watermark. However, a larger gamma can lead to an increase in false positives. Further clarification on this relationship is requested.

Author response:
**Clarification on Gamma Value and Watermark Strength.** 
The watermark strength increases as $\delta$ increases, but the dependence on \gamma is more complex. Increasing $\gamma$ does increase the number of green tokens as the reviewer observes, but it also changes the variance and the tail bound. Bear in mind that the detection rule relies not on the number of green tokens in absolute number, but how many more green tokens we observe over the natural range of the null distribution,
It is also coupled with $\delta$, for the same choice of $\delta$, smaller $\gamma$ ensures that the selected green-tokens stand out a bit more due to the soft-max normalization. That might have explained our empirical finding in Table 5.  Finding the ideal choice of $\gamma$ theoretically and empirically is an interesting question.    At the moment, for every choice of $\gamma,\delta$, our theory allows us to choose a threshold that controls the false positive rate (Remark 3.4).

Reviewer comment (Reproducibility):
On the presentation side, it reads like the authors tend to contrast their approach with the K-gram method, and the discussion has been brought up multiple times in the paper. However, only some simple text such as "general K-gram watermark works in the same way as ours, but randomly generates a different Green list for each prefix of length K − 1. In contrast, choosing K = 1 means we have a consistent green list for every new token the language model generates.'' It would be better if the authors could provide an algorithmic description on the K-gram method so the readers can directly compare it with Algorithms 1 and 2.

Author response:
We appreciate your insightful suggestion. Indeed, we have included a detailed algorithmic description of Kirchenbauer et al. 2023 in Appendix D.1. The primary distinction between Kirchenbauer et al. and the K-gram method lies in the modification of the green list from Green($y_{t-1}$) to Green($y_{t-k-1:t-1}$). We acknowledge the importance of this comparison and will incorporate a detailed discussion in the revised manuscript.

Reviewer comment (Theory):
In Definition 2.2, when comparing the defined type-I and type-II errors, I am confused about why in the type-II error, we need to assume $y$ is generated from the model $\hat{M}$, whereas in the type-I error, it is assumed that $y$ can be arbitrary. Wouldn't it make more sense to assume that $y$ is **not** generated from model $\hat{M}$ in the type-I error analysis? Based on the current definition, if $y$ is indeed generated by $\hat{M}$, then the definition says its detection correctness is at most $\alpha$? Please clarify it.

Author response:
To clarify, our definition assumes an arbitrary *fixed* suspect text $y$. In particular, if $y$ is drawn from any distribution that is statistically independent to the (random) watermark key then we can condition on $y$ and the “no false positive” guarantee applies.  It covers any procedure (including human generation) that produces $y$ in a manner independent of the secret partition $G$. However, as the reviewer pointed out, it does not cover *random* text $y$ that depends on $G$. That would include the distribution from $\hat{M}$ and would not be reasonable.  The reason why we can handle an arbitrary fixed $y$ is because our Type I error analysis only depends on the distribution of $G$ — something we have full control over.

Reviewer comment (Theory):
For the general K-gram method, can the error guarantees and analysis be extended? If so, how will this general K value place a role in the error analysis?

Author response:
Appendices D,E, and the Conclusion discussed results for K-gram for $k>1$. In short, our robustness guarantee, Type I error guarantee, and accuracy guarantee apply immediately with no changes in the analysis.  The Type II error guarantee can be extended to the case when $k >1$, but requires us to define the Martingale and bound the concentration differently. We will have a clearer discussion in the updated version.

Reviewer comment (Theory):
The robustness to edits within a bound is an awesome property and can be theoretically proved, but the robustness to paraphrasing models could hardly be proven, as there is no guarantee in the *edit distance* of paraphrasing. The related claims should be treated carefully.

Author response:
Yes, our theoretical guarantee of robustness concerns edits, as discussed in the paragraph preceding Section 2.2. We acknowledged, “Admittedly, there are other attacks where edit distance does not capture either the effort or the utility loss. For example, if one prompts an unwatermarked LLM to paraphrase $y$ then the number of edits can be large but the semantic meaning is retained. However, edit distance is a natural metric that smoothly interpolates the gray zone between the world where $y_A = y$ in which it should clearly be caught and the other world where $y_A$ is independently created without using the watermarked model in which it would be a false positive if Detect returns 1.”

Moreover, our theoretical framework does offer insights regarding the edit distance in paraphrasing. Paraphrasing can be seen as a sequence of actions – deletion, insertion, and replacement of tokens. Although extensive paraphrasing might result in a large edit distance (with the maximum being n, the length of the text), our framework assures that we can detect paraphrases if the edit distance falls below a certain threshold.

Empirically, we found our watermark can still detect paraphrased text, but the “power” gets smaller, which means it requires longer watermarked text to have high confidence in detection. Our hypothesis is that paraphrasing reduces the density of watermarks in the text, but do not remove it.
Overall, while paraphrasing attacks present challenges, edit distance offers a useful starting point. Further research on formal definitions and robustness against paraphrasing is important for future work.

Reviewer comment (Novelty):
The consistency of the output word distribution is a great property and it was also discussed in previous literature, such as [1]. How will you compare this approach with theirs? The Type I/II errors were not discussed in their work, can you show your advantages against them?

Author response:
We appreciate the reference to He et al. (2022) and will include a discussion in our revised paper. 
However, there are several key differences between these two works:

(1) We are studying different tasks. Our paper addresses the problem of detecting AI-generated text, whereas CATER (He et al., 2022) focuses on protecting against model distillation. CATER involves watermarking training data so that models trained on it produce outputs containing the watermark. Its goal is to identify model/data theft, not detect AI-generated text. As such, CATER does not consider the Type I/II errors relevant to AI detection systems.

(2) CATER uses post-processing to insert watermarks by changing words to synonyms. Our method integrates watermarking into the text generation process itself.

(3) CATER addresses the output word distribution to prevent attackers from identifying watermarked words through statistical analysis. Our method provides a theoretical guarantee that the original and watermarked probability vectors for the next token are closely aligned in any Renyi-divergence. However, we do not guarantee the output word distribution is identical. We address the potential for attacks in Appendix B.5, where we explore the possibility of adversaries bypassing detection by using secret tokens estimated from the output word distribution. We find that it is difficult to accurately estimate the green list utilizing the output word distribution. Even if the green list is known, our watermark is still somewhat effective thanks to our added robustness.

Reviewer comment (Experiments):
The robustness claim on paraphrase should be constrained by empirical findings and the paraphraser should not be aware of the watermarking method as they can still conduct attacks on this method. One extreme example could be totally random choice of synonyms and another interesting paraphrasing method is to use back-translation as paraphrasing with changes of style [2].

Author response:
We have conducted additional experiments to evaluate the robustness of our method to paraphrasing models. As discussed in Appendix B.5, we simulated a stronger white-box evasion attack. In situations where the adversary has either an estimated version or full knowledge of the green and red lists, they can formulate an evasion strategy. We simulate this by assuming the adversary employs WordNet from NLTK to identify token synonyms. Tokens identified as in the green list are replaced with red list synonyms, noting that some tokens may not have synonyms or may only have green synonyms. The results in Table 6 show it is difficult to evade detection even with known green list tokens. The detection AUC for the watermarked text is still somewhat high. In addition, the honest attempt to evade the attack by automatic synonym replacement has led to a significant drop in the text quality.

During the rebuttal phase, we expanded our tests to include back-translation as a form of paraphrasing attack. Using a model from [1], we found that the back-translation attack was less effective compared to other methods like ChatGPT or DIPPER paraphrasing. Our results on the OpenGen dataset were as follows:

**Our method**

- AUC: 0.999

- TPR at 1% FPR: 0.996, F1 score at 1% FPR: 0.985

- TPR at 10% FPR: 1.0, F1 score at 10% FPR: 0.893



**Baseline (Kirchenbauer et al., 2023)**

- AUC: 0.997

- TPR at 1% FPR: 0.942, F1 score at 1% FPR: 0.956

- TPR at 10% FPR: 0.996, F1 score at 10% FPR: 0.886

Both our method and the baseline demonstrated robustness against the back-translation attack, with our method exhibiting superior robustness compared to the baseline.