Waterfall: Scalable Framework for Robust Text Watermarking and Provenance for LLM s (2024)

Abstract: Protecting intellectual property (IP) of text such as articles and code is increasingly important, especially as sophisticated attacks become possible, such as paraphrasing by large language models (LLMs) or even unauthorized training of LLMs on copyrighted text to infringe such IP. However, existing text watermarking methods are not robust enough against such attacks nor scalable to millions of users for practical implementation. In this paper, we propose Waterfall, the first training-free framework for robust and scalable text watermarking applicable across multiple text types (e.g., articles, code) and languages supportable by LLMs, for general text and LLM data provenance. Waterfall comprises several key innovations, such as being the first to use LLM as paraphrasers for watermarking along with a novel combination of techniques that are surprisingly effective in achieving robust verifiability and scalability. We empirically demonstrate that Waterfall achieves significantly better scalability, robust verifiability, and computational efficiency compared to SOTA article-text watermarking methods, and also showed how it could be directly applied to the watermarking of code.

## introduction achieving robust text data provenance via watermarking, independent of its digital format, is an important open problem impacting a wide-ranging set of realworld challenges. among these is the issue of intellectual property (ip) enforcement: content creators of any text format (e.g., articles or code) could potentially combat plagiarism and unauthorized distribution by watermarking their works to prove data ownership. however, existing text watermarking methods have been unable to meet the challenging requirements of many practical problem settings. for example, directly adding digital metadata or invisible unicode watermarks @xcite may have limited impact in proving text data ownership in adversarial settings as they may be easily removed. existing * equal contribution. natural language watermarking @xcite @xcite that adjusts the text itself to encode ids are also lack robustness to paraphrasing attacks and have limited scalability in terms of the number of supportable ids. adding to the challenge is the growing prevalence of generative large language models (llms) that may be trained on copyrighted text without permission. to enforce ip rights, content creators need to do data provenance for llms, i.e., prove whether their set of work had been used to train 3rd party black-box llms. while there have been recent works tackling these problems @xcite , they largely require intervening in the training process of the llms. this is unrealistic in practice, as not all llm service providers may cooperate due to incentive misalignment, and adversaries may use open-source llms. hence, it is natural to ask whether it is possible to develop a practical, robust and scalable text watermarking framework for protecting ip against both plagiarism and unauthorized training of llms. for example, the watermarks should persist regardless of whether the text has been paraphrased, converted into speech or handwritten text, or used in unauthorized llm training (e.g., fine-tuning, in-context learning) to produce a derived output. the framework should also be general enough to tailor to a wide range of text formats (e.g., natural language or code), and be scalable (i.e., support millions of users with reasonable computational cost). in this paper, we propose waterfall, the first training-free framework for robust and scalable text watermarking applicable across multiple text types (e.g., articles, code) and languages supportable by llms, for general text as well as llm data provenance. rather than viewing llms as just sources of ip infringement, we introduce the novel perspective of using llms' capabilities to protect existing ip. though simple, our training-free framework comprises several key innovations such as being the first to use llm as paraphrasers for watermarking along with a novel combination of techniques that are surprisingly effective in achieving robust verifiability, scalability, and data provenance for llms, surpassing state-of-the-art text watermarking methods. in summary, our contributions are as follows: attacks: there are adversaries who aim to claim the ip in t verification: client i can use a verification operator v(µ i , t sus ) to generate a score q indicating the likelihood that t sus contains µ i . they can then use a settingspecific threshold q to classify t sus as watermarked with µ i if @xmath0. the operator v should be quick and not assume access to t o , as in practice client i may have a large set of t w and would need to automate the application of v to scan through a large set of {t sus } to identify any plagiarism (further discussion in appendix k). given the above, a suitable watermarking framework should satisfy the following desiderata: note that there is a trade-off between fidelity and verifiability. applying a stronger, more verifiable watermark tends to reduce text fidelity and the optimal setting depends on each use case. we can evaluate a watermarking scheme in general, while taking into account this trade-off, using its fidelity-verifiability pareto frontier (e.g., as plotted in figure 4a ). ## method we discuss three key insights to tackle challenges arising from these desiderata, before combining these to present our framework waterfall (watermarking framework applying large language models). ## discussion and conclusion we proposed waterfall, the first training-free framework for text watermarking that has low computational cost, scalability to large number of clients, and robustness to llm attacks including unauthorized training of llms that generates ip-infringing text. there is currently a lack of actual, practical largescale deployment of text watermarking effective against llm attacks, given the current sota watermarking methods' limitations and resource requirements. however, waterfall may possibly provide a foundation for achieving large-scale deployment, with both decentralized or centralized options. this is made achievable given waterfall's low computational cost, scalability to a large number of clients, and robustness to llm attacks including unauthorized training of llms that generates ip-infringing text. our framework highlights a few perspectives that we hope more would consider. first, while increasingly capable llms allows for easier and more sophisticated forms of potential ip infringement, llms themselves could also enable better text ip protection of original texts. a key strength of waterfall is that its capabilities grow as llms become more powerful, with increasingly better watermarking performance, allowing it to potentially keep up with the increasing capabilities adversaries can use for ip infringement. it is able to achieve a higher fidelity-verifiability pareto frontier, and reduce any fidelity degradation while using higher watermarking strength for greater robust verifiability. second, as open-source llm models become more prevalent and capable, adversaries could directly use them for ip attacks rather than depend on the services of closed-source llm providers, allowing them to bypass any ip protection measures that these providers may implement (piper, 2024). as such, content creators cannot just rely on llm providers to assist in ip protection, but instead be equipped with methods such as water-fall to protect their work before dissemination, such as by injecting robust watermarks that allows verifiability even after both traditional attacks and unauthorized use in llm training by adversaries. third, a general text watermarking framework like waterfall that can apply across different text types and languages not only helps with practical deployment, but also makes it highly versatile and not dependent on any text-specific properties. this makes it easily adaptable for incorporating new defense methods, providing a strong foundation for future works to build on as new threats emerge.