Evaluating the Robustness and Accuracy of Text Watermarking Under Real-World Cross-Lingual Manipulations (2025)

Abstract: We present a study to benchmark representative watermarking methods in cross-lingual settings. The current literature mainly focuses on the evaluation of watermarking methods for the English language. However, the literature for evaluating watermarking in cross-lingual settings is scarce. This results in overlooking important adversary scenarios in which a cross-lingual adversary could be in, leading to a gray area of practicality over cross-lingual watermarking. In this paper, we evaluate four watermarking methods in four different and vocabulary rich languages. Our experiments investigate the quality of text under different watermarking procedure and the detectability of watermarks with practical translation attack scenarios. Specifically, we investigate practical scenarios that an adversary with cross-lingual knowledge could take, and evaluate whether current watermarking methods are suitable for such scenarios. Finally, from our findings, we draw key insights about watermarking in cross-lingual settings.

## introduction the advancement of large language models (llms) has significantly transformed text generation across various domains, producing outputs that closely mimic human writing. this advancement has raised concerns within the research community regarding potential misuses, including academic misconduct, the spread of disinformation, and the creation of synthetic training data @xcite @xcite @xcite . in response to these challenges, watermarking methods have been developed to differentiate between human-written and ai-generated texts @xcite @xcite @xcite . watermarking involves embedding a signal in ai-generated texts to identify the generating llm using hypothesis testing. specifically, watermarking offers theoretical guarantees regarding the detectability of the embedded signal by performing statistical inference on the generated text and testing against the null hypothesis. since watermarking alters the original llm output, it is crucial to ensure that the impact on text quality is minimal while maintaining the watermark's detectability. 2 much of the existing literature on watermarking has primarily focused on the quality and detectability of watermarked texts, with a predominant emphasis on english texts. while these methods are theoretically language-agnostic, cross-lingual studies can reveal new adversarial scenarios in which watermark signal could be removed, and that have yet to be thoroughly investigated. existing studies on watermarking robustness have largely regarded the interplay between languages by back translation in which the english watermarked text is translated to some pivot language then by translating back to english, weakening the watermark signal @xcite @xcite . this overlooks other potential adversarial scenarios that may emerge following a translation to non-english languages. a recent work by @xcite attempted to explore adversarial scenarios within a cross-lingual context with specific translation attacks (e.g., crosslingual watermark removal attack cwra) to bypass watermarking by first obtaining a response from an llm in a pivot language, which is then translated into the target prompt language. as can be shown in figure 1 -a, we refer to this type of translation including back-translation as symmetrical, since an attacker does not utilize the pivot language. however, more practical adversarial scenarios are not studied, especially in the settings where users with cross-lingual knowledge might take to remove the watermark. as shown in figure 1 -b, we call this asymmetrical since the user may use other languages without intentionally removing the watermark. this oversight accounts for both unintentional alterations made by users for clarity or context adjustment and sophisticated adversaries intentionally refining the text to destroy watermark traces such as cwra. additionally, a broader assessment of text quality under watermarking techniques in diverse languages is scarce but necessary, particularly given linguistic differences. in this paper, we evaluate four representative watermarking methods under two high-level themes. first, syntactical watermarking, which involves syntax changes to the generated text by manipulating the logits before the decode stage. second, semantic watermarking, which involves semantics manipulation of the generated text before the decode stage as well. the syntactical methods we consider are kgw @xcite , exp @xcite , and unigram @xcite . for the semantic methods, we choose xsir @xcite as it aligns with our cross-lingual investigation in this paper. specifically, this paper addresses the following research questions. ## threat model we propose a threat model for text watermarking that addresses realistic removal scenarios. our model includes various attack pipelines including translation, paraphrasing, and combinations thereof as illustrated in figure 1 . we focus on closedsource llms where model owners are the primary victims of watermark removal attacks. attacker knowledge: we consider multilingual adversaries ranging from naive (unaware of watermarks) to advanced (deliberately trying to evade detection). importantly, regular users may unintentionally remove watermarks through normal multilingual usage patterns. attacker capability: adversaries can access ai chat interfaces, translation apis, and have the ability to paraphrase or edit text, potentially disrupting watermark signals. real-world scenarios: in an academic setting, non-english speaking professors providing english questions while allowing answers in native languages, enabling students to use ai-generated content with translation. this is not uncommon especially for science subjects where english is the language of the content is in english. another scenario could take place in media and information sharing in which the post/content is translated from english to local languages and edited for specific audiences, inadvertently removing watermarks. we call these types of translations as asymmetric as the language of the utilized generation is different from that of the prompt. these scenarios are common yet rarely considered in current watermark threat models. robust watermarking systems must address multilingual use cases rather than focusing solely on using a language as pivotal to remove the watermark without utilizing it. ## experimental setup choice of models. due to scarcity of open-source llms that universally support our targeted languages, we employ different models suitable for different languages. for chinese and indonesian, we utilize the sailor2 1b and 8b variants @xcite . for arabic text generation, we use the jais family 6.7b model (inception, 2024), while perplexity is assessed using acegpt 7b @xcite . for english, we generate text using both aforementioned models. the use of varying models across languages is essential to accommodate the lack of universally compatible models but ensures credible cross-lingual watermarking assessments. recently, a model that support all our targeted languages was released @xcite in which all of our targeted languages are supported foot_2 . we use this model to add experiments for an instruction-following and text-completion task with more languages. ## conclusion our study reveals significant variations in llm watermarking effectiveness across languages. kgw best preserves text quality, while all methods show vulnerability to translation attacks, especially asymmetrical ones where pivot and output languages differ. we identified substantial language biases in llm-based evaluations, with gpt-judger showing clear preferences for certain languages over others. our findings demonstrate that current watermarking approaches inadequately address cross-lingual complexities, as effectiveness depends not only on the target language but also on specific language paths during attacks. this highlights the need for more robust cross-lingual watermarking methods that maintain effectiveness across diverse linguistic contexts.