C ode IP : A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code (2024)

Abstract: Large Language Models (LLMs) have achieved remarkable progress in code generation. It now becomes crucial to identify whether the code is AI-generated and to determine the specific model used, particularly for purposes such as protecting Intellectual Property (IP) in industry and preventing cheating in programming exercises. To this end, several attempts have been made to insert watermarks into machine-generated code. However, existing approaches are limited to inserting only a single bit of information. In this paper, we introduce CodeIP, a novel multi-bit watermarking technique that embeds additional information to preserve crucial provenance details, such as the vendor ID of an LLM, thereby safeguarding the IPs of LLMs in code generation. Furthermore, to ensure the syntactical correctness of the generated code, we propose constraining the sampling process for predicting the next token by training a type predictor. Experiments conducted on a real-world dataset across five programming languages demonstrate the effectiveness of CodeIP in watermarking LLMs for code generation while maintaining the syntactical correctness of code.

## introduction large language models (llms), particularly those pre-trained on code, such as codegen @xcite , code llama @xcite , and starcoder @xcite , have demonstrated great potential in automating software development. notably, tools leveraging these llms, such as github copilot @xcite ), amazon's code-whisperer @xcite , and chatgpt (ope-nai, 2023), are transforming how developers approach programming by automatically generating code based on natural language instructions and the context provided by existing code. ## codeip this section provides a detailed description of codeip. the codeip comprises two distinct stages, namely insertion and extraction. initially, leveraging the decoding mechanism of existing llms, we use l lm to denote the likelihood of each token in the vocabulary v to be inferred by the llm. subsequently, during the watermark insertion stage (cf. sec. 3.1), we incorporate the watermark message by calculating a logit value l wm to influence the prediction choice of tokens. moreover, we present a novel application of context-free grammar and introduce another logit at the insertion stage (denoted as l tp ), which signifies the probability associated with the grammatical type of the subsequent token, to guide token generation from the perspective of grammar (cf. sec. 3.2). finally, we integrate all the logits together (cf. sec. 3.3) and explain the watermark extraction techniques (cf. sec. 3.4). ## conclusion in this paper, we propose codeip to watermark the llms for code generation, with the goal of safeguarding the ips of llms. we insert watermarks into code generated by the model, and introduce grammatical information into the watermark generation process by designing a type predictor module to safeguard the utility of generated code. comprehensive experimental findings affirm that codeip exhibits a notable extraction rate, excels in safeguarding code semantics, and demonstrates a degree of resilience against attacks. in our future work, we plan to persistently advance toward more secure llm-powered software engineering through the continuation of our research.