M ark LLM : An Open-Source Toolkit for LLM Watermarking (2024)

Abstract: Watermarking for Large Language Models (LLMs), which embeds imperceptible yet algorithmically detectable signals in model outputs to identify LLM-generated text, has become crucial in mitigating the potential misuse of LLMs. However, the abundance of LLM watermarking algorithms, their intricate mechanisms, and the complex evaluation procedures and perspectives pose challenges for researchers and the community to easily understand, implement and evaluate the latest advancements. To address these issues, we introduce MarkLLM, an open-source toolkit for LLM watermarking. MarkLLM offers a unified and extensible framework for implementing LLM watermarking algorithms, while providing user-friendly interfaces to ensure ease of access. Furthermore, it enhances understanding by supporting automatic visualization of the underlying mechanisms of these algorithms. For evaluation, MarkLLM offers a comprehensive suite of 12 tools spanning three perspectives, along with two types of automated evaluation pipelines. Through MarkLLM, we aim to support researchers while improving the comprehension and involvement of the general public in LLM watermarking technology, fostering consensus and driving further advancements in research and application. Our code is available at https://github.com/THU-BPM/MarkLLM.

## introduction the emergence of large language models (llms) like @xcite , @xcite , and llama @xcite has significantly enhanced various tasks, including information retrieval @xcite , content comprehension @xcite , and creative writing @xcite . however, in the digital era, the remarkable proficiency * project leader † corresponding author of llms in generating high-quality text has also brought several issues to the forefront, including individuals impersonation @xcite , academic paper ghostwriting @xcite , and the proliferation of llm-generated fake news @xcite . these issues highlight the urgent need for reliable methods to distinguish between human and llm-generated content, particularly to prevent the spread of misinformation and ensure the authenticity of digital communication. in the light of this, llm watermarking technology @xcite @xcite @xcite has been developed as a promising solution. by incorporating distinct features during the text generation process, llm outputs can be uniquely identified using specially designed detectors. as a developing technology, llm watermarking urgently requires consensus and support from both within and outside the field. however, due to the proliferation of watermarking algorithms, their relatively complex mechanisms, the diversity of evaluation perspectives and metrics, as well as the intricate procedure of evaluation process, significant efforts are required by both researchers and the general public to easily experiment with, comprehend, and evaluate watermarking algorithms. to bridge this gap, we introduce markllm, an open-source toolkit for llm watermarking. figure 1 overviews the architecture of markllm. our main contributions are summarized as follows: implementation framework: markllm offers a unified and extensible framework for implementing llm watermarking algorithms, currently supporting nine specific algorithms from two key families: kgw @xcite and christ @xcite family. ǁ unified top-calling interfaces: markllm provides consistent, user-friendly interfaces for loading algorithms, producing watermarked text generated by llms, conducting detection processes, and gathering data necessary for visualization. visualization solutions: custom visualization solutions are provided for both major watermarking algorithm families, enabling users to visualize the mechanisms of different algorithms under various configurations with real-world examples. ## user examples the following code snippets demonstrate examples of how to use markllm in one's project. for more real cases, please see the demo video.