B^4 : A Black-Box Scrubbing Attack on LLM Watermarks (2025)

Abstract: Watermarking has emerged as a prominent technique for LLM-generated content detection by embedding imperceptible patterns. Despite supreme performance, its robustness against adversarial attacks remains underexplored. Previous work typically considers a grey-box attack setting, where the specific type of watermark is already known. Some even necessitates knowledge about hyperparameters of the watermarking method. Such prerequisites are unattainable in real-world scenarios. Targeting at a more realistic black-box threat model with fewer assumptions, we here propose B^4 , a black-box scrubbing attack on watermarks. Specifically, we formulate the watermark scrubbing attack as a constrained optimization problem by capturing its objectives with two distributions, a Watermark Distribution and a Fidelity Distribution. This optimization problem can be approximately solved using two proxy distributions. Experimental results across 12 different settings demonstrate the superior performance of B^4 compared with other baselines.

## introduction the rapid advancement of large language models (llms) has demonstrated their unimaginable potentials across various applications. systems such as chatgpt @xcite are now seamlessly integrated into many aspects of daily life. despite benefits, the extensive deployment of llms has sparked serious concerns regarding potential misuse, such as large-scale disinformation, automated spamming, and social media manipulation, thereby threatening academic integrity and intellectual property rights @xcite . consequently, detecting llm-generated * equal contribution. watermarking stands out as a prominent technique for detecting llm-generated text. it injects a hidden pattern invisible to human into generated contents of a specific llm @xcite . by altering the original distribution of llms to a specific watermark distribution during each decoding step @xcite @xcite , all model generated contents can be statistically distinguished through hypothesis testing between the watermarked and the original distributions. this approach achieves a high detection rate and can be easily deployed, with only a negligible cost in the quality of generated content. despite its supreme performance, the robustness of watermarking methods against adversarial attacks remains underexplored. among which, the scrubbing attack @xcite presents a notable challenge in practical settings: if an adversary can successfully paraphrase llm-generated contents into another semantically similar but watermark-free form, the effectiveness of the watermark will be heavily compromised. @xcite and @xcite explored this question by proposing different watermark scrubbing methods. however, these approaches typically require strong prior knowledge about the specific type of watermark algorithm used in the victim llm, even the specific hyperparameter of the algorithm (e.g. the context window size in kgw). for example, @xcite proposed a method based on the assumption that the victim llm is protected with kgw watermarking algorithm @xcite , so that they can parameterize their attack model based on the green-red list partition. such prerequisites are virtually unattainable in realworld scenarios, especially when we can only access the victim llm via an api interface. the stringent constraint on prior information makes these attacking methods impractical for real-world applications. in this work, we consider a more realistic blackbox threat model with the sole prior knowledge of watermark existence. under this attack setting, we further propose b 4 (black-box scrubbing attack), a universal watermark attack method, specially designed for the black-box threat model. with less assumption about the victim llm, this method is more practical for real-life usage, allowing us to accurately explore the robustness of current watermark techniques. the intuition of b 4 is rooted in two fundamental objectives of an ideal watermark scrubbing attack: the adversarial texts should both exhibit minimal watermark patterns to evade from detection (efficacy) and preserve the semantic information of the original texts (fidelity). drawing on this insight, we propose to capture these dual properties with distance to two distributions respectively: a watermark distribution and a fidelity distribution. consequently, we formulate the task of watermark scrubbing attack into a constrained optimization problem. the local optimal of this problem can be solved by approximating these distributions with two proxies. specifically, we steer a much smaller proxy model to approximate the watermark distri-bution of the victim llm through distillation, and apply a general paraphrase model to obtain a distribution similar to the original texts for the fidelity distribution. compared to baseline watermark srubbing methods, our proposed b 4 improves the attack success rate by up to 68.13% across different victim llms and watermarking method settings. since there is an inherent trade-off between attack efficacy and semantic fidelity, we further demonstrate the superior performance of b 4 over baselines using pareto fronts: b 4 removes text watermarks more efficiently under the same fidelity constraints, while also distorting less semantic information to achieve the same level of attack. in summary, our contributions include: (1) formalizing the watermark scrubbing attack as a constrained optimization problem for the first time, and (2) developing a black-box watermark scrubbing method based on this formulation, which demonstrates superior performance compared to previous approaches. ## method in this section, we introduce b 4 , a universal watermark attacking method under the black-box threat model settings. we begin by explaining the intuition behind the method, followed by formalizing watermark scrubbing attack as an optimization problem. next, we detail the numerical solving process of the formulated problem via two proxy distributions. finally, we conclude with a discussion of potential approximation errors, along with a proposed adjustment for alleviation. we present the pseudocodes of our method in appendix b. ## conclusion in this work we research into the watermark scrubbing attack method within the black-box setting, a practically significant but under-studied field. without needing to know details or even the type of watermarking method used, we derive the format of local-optimal adversarial sample distribution by approximating a fidelity distribution and a watermark distribution. our proposed attack approach, b 4 , can effectively remove the watermark pattern without distorting the original semantic fidelity, demonstrating its superior performance over all baseline models across a wide range of victim settings.