Paper: Towards Codable Watermarking for Injecting Multi-Bits Information to LLMs (ICLR-2024)

Reviewer comment (Theory):
Kirchenbauer [2023a] provides a theorem (Theorem 4.2) to theoretically estimate the $p$-value of the watermark as a function of the spiked entropy and other parameters. A similar theorem could be derived for the proposed multi-bit watermarking scheme, which would significantly strengthen the paper.

Author response:
**Q2:** Regarding the theoretical bound.



**A2:** Thank you for your suggestion. The derivation of a theoretical bound for the single-bit case is straightforward, as it involves simply counting the 'green tokens'. However, in multi-bit scenarios, the challenge lies in calculating the likelihood that any message, except for the accurate one, possesses a greater number of 'green tokens' than the accurate message. This calculation tends to be complex and often yields theoretical bounds that aren't practically applicable. We will try to explore this further in the future to see if any good bound can be derived.

Reviewer comment (Evaluation):
The paper mentions that “success rate” combines both message recovery and watermark detection, but it should be confirmed that a successful sample requires all 20 bits to be correctly recovered and the watermark correctly detected.

Author response:
**Q3:** Regarding the questions about watermark accuracy metrics: **(1)** Does a successful sample only count if all 20 bits are correctly recovered and the watermark is correctly detected **(2)** The threshold of $1-10^{-5}$ and AUC metric.



**A3:** To address your points:



**(1)**. Yes, a sample is deemed successful only when all 20 bits are accurately recovered, and the watermark is correctly detected.



**(2)**. Similar to the reason we discussed in Q2, p-value is not directly calculable in multi-bit watermarking scenarios as they are in single-bit ones. Instead, we compute $P_w(m|\mathbf{x}) $ to serve as a confidence score for whether a given message $m$ is watermarked in text $\mathbf{x}$. This is achieved through the following steps:



  a. We first calculate $P_w(\mathbf x|m) = \sum_{l=1}^L \log P_w\left(x_l \mid m^{\prime}, \mathbf{x}_{:(l-1)}\right)$, the latter can be direclty computed based on our design of P_w.

   

  b. By applying Bayes' Theorem, $P_w(m|\mathbf{x}) $ is proportional to $P_w(\mathbf{x}|m) $. With the normalization of $P_w(\mathbf{x}|m) $ over all $m $s, we derive $P_w(m|\mathbf{x}) $.



For the threshold of $1- 1 \times 10^{-5} $: Our tests on 10,000 non-watermarked, human-written texts indicate that $\arg\max_m P_w(m|\mathbf{x}) $ falls below this threshold, suggesting it's stringent enough to avoid the false positives you mentioned (FPR=0). (We think it is enough since Kirchenbauer [2023a] verified an FPR=0 on a smaller sample of 500 human-written texts.) We extract the message only when it is above this threshold, ensuring no human-written text is falsely flagged as watermarked.



Lastly, since multi-bit scenarios aren't binary classifications, AUC is not applicable. We focus on the success rate of correctly extracting messages when above the threshold, where no human-written text is misidentified as watermarked

Reviewer comment (Experiments):
The paper mentions steganography and some older watermarking works but does not compare the proposed scheme to them experimentally. Since steganography shares a concept with watermarking and is also used to convey multiple bits of information, it is highly relevant. Comparisons should be made to steganography on natural language such as [3], [4], and Yoo et al. [2023].

Author response:
**Q4:** Regarding the comparison to steganography.



**A4:** Steganography methods have to know the prompt before the generated text so as to get extract model logits when extracting message. This is unavailable in the watermark setting, making their transferring to watermark scenarios impossible. (To somewhat, this also indicates watermark is harder than steganography). 



For Yoo et al. [2023], this is a post-processing method. It does not integrate watermarks with LLM generation. It performs watermark using masked language models, rather than much powerful LLMs, inevitably leading to poor text quality. In experiments, we found this method tends to perform trivial and possibly inappropriate substitutions. Here are some cases (the original token is shown in the brackets after the changed token):



The KPA **was(is)** the military arm of the ruling Workers' Party of Korea.



The U.S. **military(government)** spends more money than it collects in tax revenues.



"There is still a lot **less(of)** work to be done."



You can also find such situations in the case study provided by Yoo et al. [2023] in their appendix.



Yoo et al. [2023]: https://aclanthology.org/2023.acl-long.117/

Reviewer comment (Theory):
In Eq. (9), it seems like $v$ is missing, so it should be $P_{LLM}(v|...)$ instead of $P_{LLM}(...)$.

Author response:
**Q14:** For typos in Eq. 9.



**A14:** Thank you very much, we will revise it in the revision.

Reviewer comment (Experiments):
The variant of the copy-paste attack used in the experiment is fairly weak; a real attacker could interleave multiple parts of non-watermarked text, which would further reduce the number of watermarked tokens.

Author response:
**Q12:** Regarding the experiments on stronger attacks.



**A12:** In this paper, we demonstrated that our method is robust to two commonly used attacks: copy-paste attacks and substitution attacks. However, it is a common and great challenge for all current watermark methods to defend more stronger attacks like paraphrase attacks. [1] Ensuring the robustness of multi-bit watermarks against paraphrasing attacks or the multi-copy-paste attack you mentioned is an even more significant challenge than the one-bit situation, since we can not allow one bit of the message to be corrupted. Admittedly, both vanilla-marking and balance-marking can not defend against such attacks well, and this issue is designated for exploration in our future research.



[1] https://arxiv.org/pdf/2303.11156.pdf

Reviewer comment (Experiments):
Paraphrase attack is a common and stronger attack in the watermarking literature than both the copy-paste and synonym substitution attacks used in the experiments. It is also easy to execute in practice, so it might be a good idea to include it.

Author response:
**Q12:** Regarding the experiments on stronger attacks.



**A12:** In this paper, we demonstrated that our method is robust to two commonly used attacks: copy-paste attacks and substitution attacks. However, it is a common and great challenge for all current watermark methods to defend more stronger attacks like paraphrase attacks. [1] Ensuring the robustness of multi-bit watermarks against paraphrasing attacks or the multi-copy-paste attack you mentioned is an even more significant challenge than the one-bit situation, since we can not allow one bit of the message to be corrupted. Admittedly, both vanilla-marking and balance-marking can not defend against such attacks well, and this issue is designated for exploration in our future research.



[1] https://arxiv.org/pdf/2303.11156.pdf

Reviewer comment (Theory):
From Eq. (16) in Appendix, I cannot get the same formulation as Eq. (5). As we know, the Lagrange multipliers are commonly added to the constraint item.

Author response:
**Q1:** Regarding how to obtain Eq. (5) from Eq. (16) in the Appendix.



**A1:** The mathematical formulation is correct. In Eq. (16), we first let $\delta=\frac{L}{\lambda}$, and we can get

$$ \max_{t} [ \sum_{l=1}^{L} \log P_{w}(t_{l}|m,t_{:(l-1)}) - \max_{m' \neq m} \sum_{l=1}^{L} \log P_{w}(t_{l}|m',t_{:(l-1)}) + \frac{1}{\delta} \sum_{l=1}^{L} \log P_{LLM}(t_{l}|x^{prompt},t_{:(l-1)}) ].$$ Then, let $\hat m=\arg\max_{m' \neq m}\sum_{l=1}^{L} \log P(t_{l}|m',t_{:(l-1)})$, the above target can be further formulated as $$ \max_{t} [ \sum_{l=1}^{L} \log P_{w}(t_{l}|m,t_{:(l-1)}) -  \sum_{l=1}^{L} \log P_{w}(t_{l}|\hat{m},t_{:(l-1)}) + \frac{1}{\delta} \sum_{l=1}^{L} \log P_{LLM}(t_{l}|x^{prompt},t_{:(l-1)}) ]. $$ This is equivalent to $$ \max_{t} [ \delta \sum_{l=1}^{L} (\log P_{w}(t_{l}|m,t_{:(l-1)}) -  \log P_{w}(t_{l}|\hat{m},t_{:(l-1)}) )+ \sum_{l=1}^{L} \log P_{LLM}(t_{l}|x^{prompt},t_{:(l-1)}) ], $$ which is just the same as Eq. (5).

Reviewer comment (Evaluation):
Authors use Proxy-LM to approximate the condition defined by Eq. (9), which inevitably decreases the watermarking text quality. The main performance difference between Proxy-LM and the original LLM in Eq. (5) should be comprehensively discussed, including text quality, success rate, robustness, etc.

Author response:
**Q2:** Regarding the performance difference between using a smaller proxy-LM in Eq. (11) and using the original LM in Eq. (9).



**A2:** Thank you for this question, and we agree that there is a trade-off between the success rate, text quality and encoding efficiency when choosing proxy-LM. We included the original LM in the comparison of Figure 2(a), and derived the following table:



| PPL         | 3.2  | 3.35 | 3.5  |
| ----------- | ---- | ---- | ---- |
| GPT2        | 83.1 | 91.3 | 95.3 |
| GPT2-medium | 84.9 | 93.2 | 95.1 |
| GPT2-large  | 88.6 | 93.8 | 96.7 |
| GPT2-XL     | 86.4 | 93.4 | 96.0 |
| orginal LM  | 91.7 | 94.8 | 96.9 |



|             | Time for generation text | Time for extracting information |
| ----------- | ------------------------ | ------------------------------- |
| GPT2        | 9.5                      | 2.97                            |
| GPT2-medium | 11.69                    | 3.09                            |
| GPT2-large  | 14.08                    | 3.43                            |
| GPT2-XL     | 16.38                    | 3.53                            |
| orginal LM  | 11.05                    | 4.07                            |





The original LM choice does perform better than smaller proxy-LMs, but is slower than small proxy-LMs like GPT2.

Reviewer comment (Evaluation):
Authors just analyze the trade-off between efficiency and watermark success rate in different proxy-LMs, which makes me question that authors subjectively miss the trade-off between text quality and coding rate of payload information in different proxy-LMs. Moreover, the caption of Figure (2)(a) is inconsistent with the corresponding descriptions.

Author response:
**Q3:** Regarding the confusion about Figure 2(a): whether text quality is missing.



**A3:** Apologies for any misunderstanding. To clarify, in Figure 2(a), we control the ppl to the same (3.4 here) and report the corresponding watermark success rate for different $LM_{proxy}$s. This approach ensures that the comparison takes into account text quality. We prioritize illustrating the variation in watermark success rates at a fixed text quality, which we believe may be of greater interest than the inverse relationship. Additionally, comprehensive results concerning text quality and watermark success rates can be found in Appendix G.1, illustrated in Figure 3. Regarding the caption for Figure 2(a), the term 'quality' was meant to encapsulate the balance achieved between text quality and watermark success rate. We acknowledge this may have caused confusion and will amend the caption to better convey this meaning.

Besides, since we control the ppl and watermark success rate by $\delta$, achieving an exact ppl of 3.4 is challenging. To estimate the watermark success rate under a ppl of 3.4, we have employed linear interpolation between the two nearest (ppl, watermark success rate) points.

Reviewer comment (Experiments):
Authors do not evaluate the robustness of the proposed method to machine paraphrasing attacks which have been considered in some previous works (“Kirchen. et al., 2023a” and “Kirchen. et al., 2023b”).

Author response:
**Q4:** Regarding the machine paraphrasing attacks.



**A4:** In this paper, we demonstrated that our method is robust to two commonly used attacks: copy-paste attacks and substitution attacks. However, it is a common and great challenge for all current watermark methods to defend against paraphrase attacks. [1] Our method also suffers from this problem when we try to attack it with gpt3.5. Ensuring the robustness of multi-bit watermarks against paraphrasing attacks is an even more significant challenge than the one-bit situation, since we can not allow one bit of the message to be corrupted. This issue is designated for exploration in our future research.



[1] https://arxiv.org/pdf/2303.11156.pdf

Reviewer comment (Experiments):
It is analyzed that what are the different experimental results when using different sampling strategies? such as greedy sampling, top-p sampling, etc.

Author response:
**Q5:** Regarding the experiments with more sampling strategies like greedy sampling or top-p sampling:



**A5:** While we acknowledge the potential benefits of exploring a broader range of sampling strategies, such as greedy or top-p sampling, constraints in time and resources have limited our capacity to conduct extensive experimentation in this area. We aim to expand our investigation into these strategies in the future version.

Reviewer comment (Experiments):
The goal is to find the prompt that separates watermarked input and non-watermarked input the most, right? Not separating machine-generated vs human-generated. As I see Eq 3 is separating both machine-generated text. There is no human written one.

Author response:
**A2:** Yes, and to be more concrete, the goal of Eq. (3) is to separate the watermarked texts encoded with the message $m$ from other texts that are not encoded with $m$. However, our method can also distinguish watermarked machine-generated texts from human-written texts according to the results in Figure 2(b).

Reviewer comment (Theory):
How does the method encode multibit information? Is the information to be encoded m? Eq 12 decodes M, yet this formulation is very similar to adversarial attack, why does the method decode the unique m instead of some adversarial string? Like Zou et al. Universal and Transferable Adversarial Attacks on Aligned Language Models.

Author response:
**A3:** Yes, the message we want to encode is $m$. Furthermore, each $m$ can be represented by a unique multi-bit 0-1 string from an entire message space $M= ${$0,...,2^{20}-1 $}. Then, during generating each token, we can use the above string of $m$ to calculate the random seed, calculate the message logit of each token, and split the vocabulary in a probability-balanced way. Finally, the next token is sampled only from the available part of the two splits. Therefore, as we can see, the generation process of each token depends on the information of $m$, which helps us to encode $m$ into the whole text.

Reviewer comment (Evaluation):
Does Model logit used for guard generation quality?

Author response:
**A5:** Yes, the model logit measures how likely is a specific token to be the next token, and higher model logits generally lead to higher text quality.

Reviewer comment (Reproducibility):
Does message logit used to add watermark?

Author response:
**A6:** Yes. The message logits are decided by the message $m$, and by adding the message to the model logits in the next token's generation procedure, we successfully encode the information of $m$ into the generated text.

Reviewer comment (Experiments):
Does a larger L give better watermark and higher quality?

Author response:
**A7:** Yes, refer to the answers in A5 and A6.

Reviewer comment (Reproducibility):
I understand you want a v which creates a large Pw difference for detection, does Eq 7,9 simply find the words in a predefined dictionary?

Author response:
**A8:** Yes. This pre-defined vocabulary $V_{m,t_{:l-1}}$ contains the words that have high model logits, which ensures that we can sample a $v$ that not only has significantly higher message logits $P_{w}$ than that of other $v'$, but also is a reasonable word that is high likely to be the next token.

Reviewer comment (Writing):
What does sigma mean? In plain language, I do not intuitively understand.

Author response:
**A9:** $\sigma$ controls the probability of the tokens with high model logits to be included in the subset $V_{m,t_{:l-1}}$. Setting larger $\sigma$ will more likely include more tokens with high model logits in the available subset, but will also decrease the diversity of the splittings of $V_{m,t_{:l-1}}$ across different $m$s and make the message logit lose its original function.

Reviewer comment (Theory):
How does Eq 10 encode the quality, Eq 9 inside?

Author response:
**A10:** In Eq. (10), we only assign the model logits to the tokens that are included in the subset $V_{m,t_{:l-1}}$. As stated above and explained in the paragraph below Eq. (9) in the main paper, this subset should contain some tokens with relatively high model logits. Therefore, we ensure that the next token to maximize $L_{m,x^{prompt},t_{:l-1}}$ can be sampled from these reasonable tokens instead of arbitrary tokens caused by the random vocabulary splitting of Vanilla-Marking.

Reviewer comment (Theory):
I don't get why you need Eq 11.

Author response:
**A11:** Eq. (11) is derived from Eq. (9) by making two improvements to make Balanced-Marking more practical to deal with multiple application scenarios. (1) Firstly, Considering that the $x^{prompt}$ is usually unavailable to the text receiver during the message decoding phase, we omit $x^{prompt}$ and truncate $t_{:(l−1)}$ to a fixed-length $t_{(l−1−L_{prefix}):(l−1)}$ to make the encoding and decoding independent on the $x^{prompt}$. (2) Secondly, in order to apply Balance-Marking in various usage scenarios discussed in Appendix D, such as when the text receiver wants to recover the message from the text but does not have access to the internal logits of the original LLM, we broaden the original LLM used in $P_{LLM(v|t_{(l−1−L_{prefix}):(l−1)})}$ into a general proxy model denoted as LM_proxy. Therefore, designing Eq. (11) makes our Balanced-Marking more practical and effective. We have included a detailed discussion about the above points in Appendix F.

Reviewer comment (Experiments):
It is essential to compare the watermarked text with other approaches to evaluate the effectiveness of the proposed method. In the last paragraph of the related work section, the authors stated, “There are some very concurrent works ... the vocabulary partitions....” Still, there are not any experiments that compare the text quality between CTWL and other approaches. The authors are suggested to compare the BLEU, ROUGE, or other metrics with the previous studies, such as Kirchenbauer et al. (2023a), to show the text quality of CTWL is better than previous work. These experiments can make the results more convincing.

Author response:
**Q1:** Regarding comparing CTWL with concurrent studies.



**A1:** Actually, they were updating their papers a few days ago before the submission deadline, so we do not have time to compare them at that that time. And after studing the concurrent studies by Yoo et al.[1] and Fernandez et al.[2], we found that Vanilla-Marking can cover these works, since 



(1) Yoo et al. concentrates on bit-wise accuracy and strategies for bit allocation to text tokens, while our work is geared towards the accuracy of encoding and decoding entire messages. This is inherently more challenging as it is akin to raising bit-wise accuracy to the power of the number of bits in a message, which would result in a significantly lower performance under our metric for the study by Yoo et al. Thus, the bit allocation strategy of Yoo et al. can not be transferred to our settings, and their method, if excluded this allocation strategy, can be seen as equivalent to Vanilla-Marking.



(2) Fernandez et al.'s research, on the other hand, aligns closely with what we refer to as Vanilla-Marking in our paper. Although they employ a circular-shift method to expedite the generation of a secret key, we have similarly optimized the hash function in our own implementations of Vanilla-Marking and Balance-Marking, as can be seen in our 'hash_fn.py' code. When considering text quality and watermark success rate, Fernandez et al.'s method should be same as Vanilla-Marking. Additionally, their scope is limited to messages in the range of 1-1000, while our approach extends to a much larger range of 1-2^20, (i.e., 1-1,048,576).



[1]https://arxiv.org/abs/2308.00221



[2]https://arxiv.org/abs/2308.00113



**Q2:** Regarding using other metrics to compare the text quality between CTWL and other methods.



**A2:** We adhere to the approach established by Kirchenbauer et al. (2023a) in utilizing perplexity (ppl) as a measure of text quality. Perplexity is widely accepted for evaluating the quality of text generation.[1,2] Alternatives like BLEU and ROUGE are less appropriate for text watermarking. The watermarking process inherently involves substituting texts to their alternatives, which may not align well with the n-gram overlap metrics that BLEU and ROUGE emphasize. Also, in experiments for llama-7b/13b, we use a much larger model, llama-33b to calculate ppl.



[1] https://arxiv.org/abs/2306.17439



[2] https://arxiv.org/abs/2307.16230

Reviewer comment (Evaluation):
How about the text quality of CTWL compared to the previous work, which focuses on injecting information by postprocessing?

Author response:
**Q3:** Regarding the comparison between CTWL and the previous methods that inject information into text by post-processing.



**A3:** We focus on watermark integrated with LLM generation, since the post-processing based methods will predictably lead to a low-quality machine-generated text. The major reason for this is that the abilities of masked-language-modeling-based models used in these post-processing-based methods (e.g., BERT) are far away from the current generative language models (e.g., LLaMa) for generating the texts, using incompetent models to replace the words in the sentences generated by powerful LLMs inevitably destroy the quality of the texts. 



We tried Yoo et al. [2023], a sota post-processing method that has also been mentioned by Reviewer qLTC. However, we find this method tends to perform trivial and possibly inappropriate substitutions, as we have discussed above. Here are some cases (the original token is shown in the brackets after the changed token):



The KPA **was(is)** the military arm of the ruling Workers' Party of Korea.



The U.S. **military(government)** spends more money than it collects in tax revenues.



"There is still a lot **less(of)** work to be done."

Reviewer comment (Evaluation):
What is the computation cost of previous studies compared to CTWL?

Author response:
**Q4:** Regarding the comparison between the computation time of CTWL and that of previous methods.



**A4:** Previous methods are not able to conduct multi-bit watermark and we are the first work to integrate it with LLM generation. So, there is no proper baseline that can be directly compared with. As an alternative choice, we view Vanilla-Marking as a baseline (though it is also proposed by us). We have included a comparison of the computational costs of Balance-Marking and Vanilla-Marking (proxy-LM = null) in Figure 2(a). (As we discussed in A1, the very concurrent work can be considered as an equivalent version of Vanilla-Marking, thus, Figure 2(a) also represents the comparison between Balance-Marking and concurrent methods.) 



It is worth noting that embedding multi-bit watermarks is inherently more intrusive compared to single-bit watermarking, potentially degrading text quality more severely. So, it is necessary for a trade-off between text quality and efficiency. Balance-Marking allows us to select different proxy-LMs to align varying preferences for quality and efficiency across diverse scenarios.