Robust Multi-bit Natural Language Watermarking through Invariant Features (2023)

Abstract: Recent years have witnessed a proliferation of valuable original natural language contents found in subscription-based media outlets, web novel platforms, and outputs of large language models. However, these contents are susceptible to illegal piracy and potential misuse without proper security measures. This calls for a secure watermarking system to guarantee copyright protection through leakage tracing or ownership identification. To effectively combat piracy and protect copyrights, a multi-bit watermarking framework should be able to embed adequate bits of information and extract the watermarks in a robust manner despite possible corruption. In this work, we explore ways to advance both payload and robustness by following a well-known proposition from image watermarking and identify features in natural language that are invariant to minor corruption. Through a systematic analysis of the possible sources of errors, we further propose a corruption-resistant infill model. Our full method improves upon the previous work on robustness by +16.8% point on average on four datasets, three corruption types, and two corruption ratios

## introduction recent years have witnessed a proliferation of original and valuable natural language contents such as those found in subscription-based media outlets (e.g. financial times, medium), web novel platforms (e.g. wattpad, radish) -an industry that has shown rapid growth, especially in the east asian market @xcite -and texts written by human-like language models @xcite @xcite . without proper security measures, however, these contents are susceptible to illegal piracy and distribution, financially damaging the creators of the content and the market industry. in addition, the recent emergence of human-like language models like chatgpt has raised concerns regarding the mass generation of disinformation @xcite . this calls for a secure watermarking system to guarantee copyright protection or detect misuse of language models. digital watermarking is a technology that enables the embedding of information into multimedia (e.g. image, video, audio) in an unnoticeable way without degrading the original utility of the content. through embedding information such as owner/purchaser id, its application includes leakage tracing, ownership identification, meta-data binding, and tamper-proofing. to effectively combat intentional evasion by the adversary or unintentional digital degradation, a watermarking framework should not only be able to embed adequate bits of information but also demonstrate robustness against potential corruption @xcite . watermarking in image and video contents has been extensively explored for pre-deep learning methods @xcite @xcite . with the advent of deep neural networks, deep watermarking has emerged as a new paradigm that improves the three key aspects of watermarking: payload (i.e. the number of bits embedded), robustness (i.e. accuracy of the extracted message), and quality of the embedded media. natural language watermarking uses text as the carrier for the watermark by imperceptibly modifying semantics and/or syntactic features. as opposed to altering the visual appearances @xcite , this type of modification makes natural language watermarking resistant to piracy based on manual transcription. previous research has focused on techniques such as lexical substitution with predefined rules and dictionaries or structural transformation @xcite . through utilizing neural networks, recent works have either replaced the predefined set of rules with learning-based methodology (abdelnabi and fritz, 2021, awt), thereby removing heuristics or vastly improved the quality of lexical substitution @xcite . despite the superiority over traditional methods, however, recent works are not without their limitations: awt is prone to error during message extraction especially when a higher number of bits are embedded and occasionally generates deteriorated watermarked samples due to its entire reliance on a neural network; contextls has a fixed upperbound on the payload and more importantly, does not consider extracting the bit message under corruption, which leads to low robustness. this work strives to advance both payload and robustness of natural language watermarking. to build an effective robust watermarking system for natural language, we draw inspiration from a well-known proposition of a classical image watermarking work @xcite : that watermarks should "be placed explicitly in the perceptually most significant components" of an image. if this is achieved, the adversary must corrupt the content's fundamental structure to destroy the watermark. this degrades the utility of the original content, rendering the purpose of pirating futile. however, embedding the watermark directly on the "perceptually most significant components" is only possible for images due to the inherent perceptual capacity of images. that is, modification in individual pixels is much more imperceptible than on individual words. due to this, while we adhere to the gist of the proposition, we do not embed directly on the most significant component. instead, we identify features that are semantically or syntactically fundamental components of the text and thus, invariant to minor modifications in texts. then we use them as anchor points to pinpoint the position of watermarks. after formulating a general framework for robust natural watermarking, we empirically study the effectiveness of various potential invariant features derived from the semantic and syntactic components. through stepby-step analysis of the possible sources of errors during watermark extraction, we further propose a corruption-resistant infill model that is trained explicitly to be robust on possible types of corruption. our experimental results encompassing four datasets of various writing styles demonstrate the robustness of (1) relying on invariant features for watermark embedding (2) using a robustly trained infill model. the absolute robustness improvement of our full method compared with the previous work is +16.8% point on average on the four datasets, three corruption types, and two corruption ratios. ## experiment dataset to evaluate the effectiveness of the proposed method, we use four datasets with various styles. imdb @xcite is a movie reviews dataset, making it more colloquial. wikitext-2 @xcite , consisting of articles from wikipedia, has a more informative style. we also experiment with two novels, dracula and wuthering heights (wh), which have a distinct style compared to modern english and are available on project gutenberg @xcite . ## related works natural language watermarking embeds information via manipulation of semantics or syntactic features rather than altering the visual appearance of 2099 words, lines, and documents @xcite . this makes natural language watermarking robust to re-formatting of the file or manual transcription of the text @xcite . early works in natural language watermarking have relied on synonym substitution @xcite , restructuring of syntactic structures @xcite , or paraphrasing @xcite . the reliance on a predefined set of rules often leads to a low bit capacity and the lack of contextual consideration during the embedding process may result in a degraded utility of the watermarked text that sounds unnatural or strange. with the advent of neural networks, some works have done away with the reliance on pre-defined sets of rules as done in previous works. adversarial watermarking transformer @xcite propose an encode-decoder transformer architecture that learns to extract the message from the decoded watermarked text. to maintain the quality of the watermarked text, they use signals from sentence transformers and language models. however, due to entirely relying upon a neural network for message embedding and extraction, the extracted message is prone to error even without corruption, especially when the payload is high and has a noticeable artifact such as repeated tokens in some of the samples. @xcite takes an algorithmic approach for embedding and extraction of messages, making it errorless. additionally, using a neural infill model along with an nli model has shown better quality in lexical substitution than more traditional approaches (e.g. wordnet). however, robustness under corruption is not considered. image watermarking explicitly considering corruption for robustness and using different domains of the multimedia are all highly relevant to blind image watermarking, which has been extensively explored @xcite @xcite . like our robust infill training, @xcite explicitly consider possible image corruptions to improve robustness. meanwhile, transforming the pixel domain to various frequency domains using transform methods such as discrete cosine transform has shown to be both effective and more robust @xcite . the use of keywords and dependencies to determine the embedding position in our work can be similarly considered as transforming the raw text into semantic and syntactic domains, respectively. other lines of work steganography is a similar line of work concealing secret data into a cover media focusing on covertness rather than robustness. various methods have been studied in the natural language domain @xcite @xcite @xcite . this line of works differs from watermarking in that the cover text may be arbitrarily generated to conceal the secret message, which eases the constraint of maintaining the original semantics. recently, @xcite proposed to watermark outputs of language models to prevent model stealing and extraction. while the main objective of these works @xcite differs from ours, the methodologies can be adapted to watermark text directly. however, these are only limited to zero-bit watermarking (e.g. whether the text is from a language model or not), while ours allow embedding of any multi-bit information. similarly, @xcite propose to watermark outputs of language models at decoding time in a zero-bit manner to distinguish machine-generated texts from human-written text. ## conclusion we propose using invariant features of natural language to embed robust watermarks to corruptions. we empirically validate two potential components easily discoverable by off-the-shelf models. the proposed method outperforms recent neural network-based watermarking in robustness and payload while having a comparable semantic quality. we do not claim that the invariant features studied in this work are the optimal approach. instead, we pave the way for future works to explore other effective domains and solutions following the framework.