Watermark Smoothing Attacks against Language Models (2025)

Abstract: Watermarking is a key technique for detecting AI-generated text. In this work, we study its vulnerabilities and introduce the Smoothing Attack, a novel watermark removal method. By leveraging the relationship between the model’s confidence and watermark detectability, our attack selectively smoothes the watermarked content, erasing watermark traces while preserving text quality. We validate our attack on open-source models ranging from 1.3 B to 30B parameters on 10 different watermarks, demonstrating its effectiveness. Our findings expose critical weaknesses in existing watermarking schemes and highlight the need for stronger defenses.

## introduction detecting whether a text is generated by language models is critical in domains such as fraud detection, fake news identification, and plagiarism prevention. a common approach is watermarking, where subtle patterns are embedded in the generated text for later detection @xcite @xcite . watermarking has gained traction in both academia and industry @xcite as a key safeguard for language model applications. while various watermarking techniques exist, they share a core principle: biasing the token distribution to favor certain tokens over others (see section 2). in this work, we identify key scenarios where token-based watermarks fail and introduce a general watermark removal attack that exploits this weakness, exposing fundamental limitations in current watermarking schemes. effectiveness of watermarks. we say a watermark as effective if (i) the watermarked text maintains high quality, comparable to that of the corresponding un-watermarked model, and (ii) the detector reliably identifies watermark traces without incurring high error. we show both analytically and empirically that these goals are inherently in tension: higher text quality typically reduces detectability, and vice versa. this tradeoff is closely linked to the model's confidence in generating the next token. when confidence is high, watermarking has negligible impact, leaving only faint traces. when confidence is low, watermarking biases the model toward otherwise unlikely tokens, producing stronger traces at the cost of text quality. ## preliminaries and related work given an auto-regressive language model (lm) m with vocabulary v, the model outputs a probability distribution over tokens at each position t in a prompt by computing logits l t (v) and applying a softmax: to sample the next token, common strategies include top-k sampling @xcite , selecting from the top k tokens by probability, and top-p (nucleus) sampling @xcite , selecting from the smallest set whose cumulative probability exceeds p. watermarking schemes subtly modify this sampling to embed patterns in the generated text (v 1 , . . . , v t ). these patterns are later detected via a scoring function d(v 1 , . . . , v t ); if the score exceeds a threshold τ , the text is deemed watermarked. below we briefly review representative watermarking approaches. green-red watermark @xcite . for each position t, a secret key and the current prefix deterministically partition the vocabulary v into a green list g t (size γ|v|) and a red list. the logits of green tokens are then boosted by a constant δ, giving the modified distribution the score is high when green tokens are overrepresented, indicating a watermark. gumbel and tournament watermarks @xcite . these methods introduce randomness via a secret key and selection process without altering the distribution p t , preserving average text quality. in gumbel sampling, noise values u t (v) ∈ [0, 1] are generated using a seed derived from recent tokens and a secret key; the token v * t is selected as v * @xmath0. the detector uses the score d(v 1 , . . . , v t ) =t log(1u t (v t )). tournament sampling similarly uses m secret functions g (1) , ..., g (m) to score each token based on a seed r t . the token is chosen through m rounds of pairwise comparisons among 2 m sampled candidates. detection relies on the average tournament score: if the score significantly exceeds 0.5, the text is predicted as watermarked. other related work. the green-red list watermark @xcite introduces tokenlevel bias by amplifying the probabilities of greenlisted tokens and is thus considered distortionary. variants differ in list construction and detection methods @xcite @xcite . in contrast, gumbel and tournament sampling @xcite @xcite preserve the token distribution in expectation and are distortion-free. other distortion-free schemes include those by @xcite ; see @xcite for a broader review. we evaluate 10 representative watermarks from both categories to demonstrate the generality of our attack. watermark removal techniques typically disrupt token patterns using homoglyphs, emojis, or control characters @xcite @xcite , but often degrade fluency. a more effective approach is paraphrasing, where a separate model rewrites the text @xcite @xcite . these attacks depend on strong paraphrasers-e.g., using gpt-3.5-turbo to rewrite outputs from smaller models like llama-7b (ope-nai, 2023; @xcite . recent work explores adaptive paraphrasers trained to evade specific watermark detectors @xcite , often requiring detailed knowledge of the watermarking algorithm. other methods combine paraphrasing with auxiliary models to select high-quality rewrites from large candidate pools @xcite . in contrast, our smoothing attack is lightweight: it leverages only top-k token probabilities from the watermarked model and a small reference model-without needing a strong paraphraser, large-scale sampling, or exact knowledge of the watermarking scheme. ## on the effectiveness of watermarks we investigate two key aspects of watermark effectiveness: detectability and text quality. our key finding is that these aspects are inter-connected via the model's confidence in prediction and are inherent tension-improving detectability typically decreases text quality, and vice versa. this tradeoff arises directly from how watermarking algorithms exploit token-level decisions, revealing a fundamental vulnerability leveraged by our proposed attack. full derivations and further analysis are provided in appendix c. ## smoothing attack our attack aims to remove watermarks while preserving text quality. at each token position, we first estimate the model's confidence and then selectively smooth low-confidence tokens to weaken the watermark detection signal. detailed justification and derivations are provided in appendix c; here we present the core algorithm clearly. adversary's model access. we assume a practical scenario: the adversary queries the target watermarked model via an api, obtaining only the top-k token probabilities for a given prefix (with k ≪ |v|). the original, un-watermarked model is not available to the adversary. estimating model confidence. we estimate model confidence at position t by approximating the squared ℓ 2 -norm of the token probability vector p t : ## conclusion we revealed limitations in existing watermarks for language models and examined their robustness against watermark removal attacks. we introduced smoothing attack, a novel method that leverages model confidence to selectively remove watermark traces while preserving text quality. comprehensive evaluations demonstrated that smoothing attack can completely remove watermarks, outperforming the state-of-the-art attack and highlighting a critical gap in current watermarks, and calling for more robust solutions.