Paper: On the Reliability of Watermarks for Large Language Models (ICLR-2024)

Reviewer comment (Theory):
Lack of Theoretical Background: The paper does not delve deeply into the theoretical aspects of watermarking, which could be crucial in understanding the underlying principles and potential vulnerabilities. A more robust theoretical foundation could enhance the paper's overall quality.

Author response:
On the desire for more theoretical background on the topic, we rely on the foundation established in Kirchenbauer 2023, where the formulation of the watermark we evaluate is established and analytic results are presented. Over the past year, there have been many other works that develop or analyze watermarking strategies from various theoretical perspectives, especially concerning cryptographic and security properties (Christ 2023, Fairoze 2023). However, not all of these works present a realistic empirical evaluation or evaluate the robustness of proposed watermarks to natural corruptions like paraphrasing and text mixing, and even fewer evaluate watermarks alongside other detection strategies. Thus we embarked on this work intending to help fill that particular gap in the literature.

Reviewer comment (Experiments):
Inherent Model Bias: The paper uses a specific language model (llama) for its experiments. While this model is justified and used for practical reasons, the results might not be universally applicable to all language models, which could limit the generalizability of the findings.

Author response:
On model bias, we chose to conduct our experiments on the most capable open source foundation model available (at the time of our human study) within our computational budget. We are unable to consider hosted API models because they don't expose the logit distribution necessary to apply the watermark to them. While we only consider a small group, we do report results with a previous generation model, OPT, and an instruction finetuned model variant, Vicuna, in Appendix A.7.

Reviewer comment (Experiments):
Can watermarking be further improved to make it more resilient to copy-paste attacks, given the significant performance drop observed under this scenario?

Author response:
> How can watermarking be further improved to make it more resilient to copy-paste attacks, given the significant performance drop observed under this scenario? 

We find the window detection strategy (WinMax) to be an effective improvement over the original scheme. These results are exemplified by the the CP-1-10% and CP-3-10% sub charts in Figures 5 and 6. The WinMax scheme retains significantly more detection performance than the standard Z-Score method at this extreme setting. 

Regarding the search for even further improvements in this setting, we include a small investigation into another alternate detection test  based on the concept of "run-lengths" in Appendix A.9.2, but it did not yield any improvements in our experimental setup. On a more philosophical note, the degree to which a paragraph or document with only 10% watermarked content even constitutes "a watermarked document" anymore is somewhat context dependent, and from a practical standpoint it is not clear a deployed detector should always return "AI" or "watermarked" in that scenario. However, we aimed to explore the limits of detection techniques in our study.

Reviewer comment (Evaluation):
How does the performance of watermarking compare to other detection methods in detecting machine-generated text within shorter sequences or fragments?

Author response:
> How does the performance of watermarking compare to other detection methods in detecting machine-generated text within shorter sequences or fragments?

In Section 4.4 we conduct a comparative study of detection algorithms and find that watermarking and the other detection algorithms offer varying degrees of reliability in both the machine paraphrase and the copy-paste settings, but that in extreme scenarios like CP-1-10%, the WinMax approach is the best of the group.

In our intial experiments we utilized DetectGPT as a representitive of the stongest post-hoc detector available at the time, but throughout the rebuttal period we have run an additional experiment to incorporate a newer detector called RADAR. This model is an adversarially trained classifier designed to be a reliable detector of model generated content even after paraphrasing and we find it to be quite competitive with the Retrieval approach. However, as demonstrated clearly in the updated ROC chart analysis, both Retrieval and RADAR do not achieve the same levels of sensitivity in the low FPR range of 0.001 to 0.1 that are achievable with watermarking.

Reviewer comment (Experiments):
Could watermarking be used in conjunction with other detection methods to enhance overall detection reliability, particularly in complex and adversarial settings?

Author response:
> Could watermarking be used in conjunction with other detection methods to enhance overall detection reliability, particularly in complex and adversarial settings?

Of course! We believe that hybrid approaches to AI generated content detection that simultaneously leverage techniques like watermarking, loss based methods like DetectGPT, trained ML classifiers like RADAR, and database approaches like the Retrieval method, are likely to offer system performance levels that are unachievable with each approach alone, especially across varied domains and use cases. We leave this very interesting and important avenue of research to future work.

Reviewer comment (Presentation):
Probably due to page limits, most spaces are used for presenting numerical results. The methodology section, including a new watermark method (e.g. SelfHash) and a new detection method (WinMax) in Sec. 3 is relatively short (roughly one page), and much important information is deferred to the Appendix.

Author response:
We appreciate the reviewer's acknowledgement that page limits forced us to make some difficult decisions about what to showcase in the main body of the work. We tried to make the pointers to more details where required as easy to follow as possible for the reader.

Reviewer comment (Experiments):
The analysis will be more complete if it includes more recent and advanced post-hoc detection methods (such as RADAR https://arxiv.org/abs/2307.03838), because DetectGPT is known to be non-robust to paraphrasing. I would like to see the result of RADAR in ROC analysis and with varying token lengths.

Author response:
In direct response the reviewer's request, we have performed an experiment with the publicly available RADAR model, and have incorporated the results into Figure 5 and Figure 6. We find that RADAR is significantly more reliable than DetectGPT in our evaluation setup and agree with the reviewer that this improves the quality of the comparitive study between detection methods substantially. 

The edits necessary to incorporate the RADAR technique are mostly confined to main body section 4.4 "A Comparative Study of Detection Algorithms" and appendix section A.11.3 "Baseline Method Details and Hyperparameters".

We would like to be transparent about a few caveats. First, running the RADAR model on Llama generations to keep parity with the rest of the results constitutes a slight domain shift since we are using RADAR-Vicuna (only checkpoint available), but this transfer pair was quite favorable according to Figure 3 in their paper so we believe the setup is still fair. Second, the token limit of the model (512) means that for the copy-paste experiments, the model didn't receive all 1000 available tokens like the other detector approaches, though this has less of an impact on the GPT and Dipper based paraphrase experiments because those sequences average only 300 and 800 tokens after paraphrasing, respectively. For RADAR, the curves stop at 512 for this reason.

Reviewer comment (Experiments):
For the analysis of post-hoc detection, given that DetectGPT is shown to be fragile against paraphrasing (Sadasivan 2023), can the authors add a new analysis with a more robust post-hoc AI-text detector, like RADAR? I would like to see the result of RADAR in ROC analysis and with varying token lengths, as in Fig. 5 and Fig. 6.

Author response:
In direct response the reviewer's request, we have performed an experiment with the publicly available RADAR model, and have incorporated the results into Figure 5 and Figure 6. We find that RADAR is significantly more reliable than DetectGPT in our evaluation setup and agree with the reviewer that this improves the quality of the comparitive study between detection methods substantially. 

The edits necessary to incorporate the RADAR technique are mostly confined to main body section 4.4 "A Comparative Study of Detection Algorithms" and appendix section A.11.3 "Baseline Method Details and Hyperparameters".

We would like to be transparent about a few caveats. First, running the RADAR model on Llama generations to keep parity with the rest of the results constitutes a slight domain shift since we are using RADAR-Vicuna (only checkpoint available), but this transfer pair was quite favorable according to Figure 3 in their paper so we believe the setup is still fair. Second, the token limit of the model (512) means that for the copy-paste experiments, the model didn't receive all 1000 available tokens like the other detector approaches, though this has less of an impact on the GPT and Dipper based paraphrase experiments because those sequences average only 300 and 800 tokens after paraphrasing, respectively. For RADAR, the curves stop at 512 for this reason.

Reviewer comment (Evaluation):
I don't see much discussion on the utility of the studied watermark methods (especially the new one, SelfHash). If a watermark is robust but lacks usefulness, it is impractical. Can the authors report the usefulness of the new watermark methods that have been not studied in (Kirchenbauer 2023)?

Author response:
Next, concerning the utility of SelfHash, we did not forsee major changes in utility, compared to the hashing scheme employed in Kirchenbauer 2023 (only changes in reliability), but your review raises the valid point that we did not explicitly verify this. To address this concern, we have now added a utility study, comparing the more advanced SelfHash scheme evaluated in this work to the original LeftHash scheme, measuring their impact on a model's ability to generate free-form answers to TriviaQA questions. We use slightly more modern chat models and generate watermarked responses for the first 1000 TriviaQA questions marking an answer as correct if the string of the ground truth answer, or one of its aliases, is included in the model's response. The models we evaluate are LLaMA-2-chat-7b and Zephyr-7b-beta.

In the updated draft, we tabulate these results in Appendix Section A.6. Overall, we find that the utility of the model is only minimally impacted by the SelfHash watermark scheme, with the original LeftHash watermark yielding similar results. This is in line with our expectations that the two schemes should behave similarly under this analysis. We thank the reviewer for the suggestion and feel that this improves the completeness of our evaluation.

Reviewer comment (Theory):
The main issue I have with the paper is an unclear threat model: What is an attacker allowed to do to paraphrase sequence correctly? When is a paraphrased text too dissimilar from the watermarked text? The paper does not answer these fundamental questions, but follow-up papers must rely on these answers to propose improved attacks.

Author response:
We'd like to address the reviewer's concerns by first clarifying that our intention with this research was to conduct an empirical study into the _reliability_ of watermarking techniques in realistic scenarios rather than robustness to a targeted attack. The formal definition of a threat model necessitates an attacker and defender relationship that is stronger than the setup we studied and further, tends to require a well defined notion of the attacker's "budget" which we believe is difficult to define in this case.

Nevertheless, we did our best quantify whether participants produced valid paraphrases during our human study by utilizing an established metric from the literature on paraphrase semantic similarity called Paragram-SP (Weiting, 2022). Other work has has shown that the average similarity values of human written paraphrases are around 0.76 under this metric (Krishna, 2023), and thus we considered anything above a 0.7 to be a valid paraphrase. All our annotators achieved at least this average similarity to the starting text, despite significant reductions in watermark strength in their paraphrases, and we spot checked the responses to confirm no obvious circumventions of the paraphrase task guidelines. We provide further details about the study setup and instructions in Appendix A.13. One could build a bounded threat model around such a similarity measure, however, since it is a model-based score, the generality of such a bound or budget might be limited.

Reviewer comment (Experiments):
In the example where a human and a paraphraser want to preserve the "meaning" of the watermarked text, the watermark hides with high probability in high-entropy token sequences, such as names, locations, and numbers. It is unclear whether a paraphraser is allowed to replace these with random other names and locations, or if that would be considered "unsuccessful" paraphrasing. This should be discussed in the paper so that future papers can meaningfully improve on the presented attacks.

Author response:
Finally, as a remark on entropy, one can draw a connection between the amount of available entropy given a prompt and whether many paraphrases of a correct answer or response exist. If a valid answer to a query necessitates the use of specific entities likes names and locations, then 1) a soft watermark is unlikely to change whether or not those entities are generated, and 2) a paraphraser is unlikely to be able to come up with a valid paraphrase that doesn't include said entities. Low entropy token spans are not regions where the watermark will be densely embedded, and thus whether or not a paraphraser changes them is likely not a determining factor in resultant watermark stength after the paraphrase. We think that the reviewer might find the new results in Appendix Section A.6 concerning TriviaQA (prepared in response to a separate review question) quite illustrative of the correspondence between entities and entropy. This experiment supports the hypothesis that the watermark is relatively "inactive" when the model is generating short, entity-centric responses to factual questions.

Reviewer comment (Evaluation):
What is a successful paraphrasing attack?

Author response:
Nevertheless, we did our best quantify whether participants produced valid paraphrases during our human study by utilizing an established metric from the literature on paraphrase semantic similarity called Paragram-SP (Weiting, 2022). Other work has has shown that the average similarity values of human written paraphrases are around 0.76 under this metric (Krishna, 2023), and thus we considered anything above a 0.7 to be a valid paraphrase. All our annotators achieved at least this average similarity to the starting text, despite significant reductions in watermark strength in their paraphrases, and we spot checked the responses to confirm no obvious circumventions of the paraphrase task guidelines. We provide further details about the study setup and instructions in Appendix A.13. One could build a bounded threat model around such a similarity measure, however, since it is a model-based score, the generality of such a bound or budget might be limited.