Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models (2024)

Abstract: Text watermarking technology aims to tag and identify content produced by large language models (LLMs) to prevent misuse. In this study, we introduce the concept of cross-lingual consistency in text watermarking, which assesses the ability of text watermarks to maintain their effectiveness after being translated into other languages. Preliminary empirical results from two LLMs and three watermarking methods reveal that current text watermarking technologies lack consistency when texts are translated into various languages. Based on this observation, we propose a Cross-lingual Watermark Removal Attack (CWRA) to bypass watermarking by first obtaining a response from an LLM in a pivot language, which is then translated into the target language. CWRA can effectively remove watermarks, decreasing the AUCs to a random-guessing level without performance loss. Furthermore, we analyze two key factors that contribute to the cross-lingual consistency in text watermarking and propose X-SIR as a defense method against CWRA.

## introduction large language models (llms) like @xcite have demonstrated remarkable content generation capabilities, producing texts that are hard to distinguish from human-written ones. this progress has led to concerns regarding the misuse of llms, such as the risks of generating misleading information, impersonating individuals, and compromising academic integrity @xcite @xcite . as a countermeasure, text watermarking technology for llms has been developed, aiming at tagging and identifying the content produced by llms @xcite . generally, a text watermarking algorithm * equal contribution. work done during zhiwei's internship at tencent ai lab. ## cross-lingual consistency of text watermark in this section, we define the concept of crosslingual consistency in text watermarking and answer three research questions (rq): ## paraphrase-multilingual-mpnet-base-v2 pearson correlation coefficient (pcc) we use pcc to assess linear correlation between s and ŝ: relative error (re) unlike pcc, which captures consistency in trends, re is used to assess the magnitude of deviation between s and ŝ: a lower re indicates that the watermark retains strength close to its original value after translation, signifying great cross-lingual consistency. ## cross-lingual watermark removal attack in the previous section, we focus on scenarios where the response of llm is translated into other languages. however, an attacker typically expects a response from the llm in the same language as the prompt while removing watermarks. to bridge this gap, we introduce the cross-lingual watermark removal attack (cwra) in this section, constituting a complete attack process and posing a more significant challenge to text watermarking than paraphrasing and re-translation attacks. the spread of fake news and the misuse for cheating in academic writing, regardless of the language of the content. figure 3 shows the process of cwra. instead of feeding the original prompt into the llm, the attacker initiates the attack by translating the prompt into a pivot language named the pivot prompt. the llm receives the pivot prompt and provides a watermarked response in the pivot language. the attacker then translates the pivot response back into the original language. this approach allows the attacker to obtain the response in the original language. due to the inherent challenges in maintaining cross-lingual consistency, the watermark would be effectively eliminated during the second translation step. ## improving cross-lingual consistency up to this point, we have observed the challenges associated with text watermarking in cross-lingual scenarios. in this section, we first analyze two key factors essential for achieving cross-lingual consistency. based on our analysis, we propose a defense method against cwra. ## conclusion this work aims to investigate the cross-lingual consistency of watermarking methods for llms. we first characterize and evaluate the cross-lingual consistency of current watermarking techniques, revealing that current watermarking methods struggle to maintain their watermark strengths across different languages. based on this observation, we propose the cross-lingual watermark removal attack (cwra), which significantly challenges wa-termark robustness by efficiently eliminating watermarks without compromising text quality. through the analysis of two primary factors that influence cross-lingual consistency, we propose x-sir as a defense strategy against cwra. despite its limitations, this approach greatly improves watermark detection performance under cross-lingual scenario and paves the way for future research. overall, this work completes a closed loop in the study of cross-lingual consistency in watermarking, including: evaluation, attacking, analysis, and defensing. since our first release, the x-sir algorithm has been integrated into markllm @xcite , an open-source toolkit for llm watermarking.