Code Membership Inference for Detecting Unauthorized Data Use in Code Pre-trained Language Models (2024)

Abstract: Code pre-trained language models (CPLMs) have received great attention since they can benefit various tasks that facilitate software development and maintenance. However, CPLMs are trained on massive open-source code, raising concerns about potential data infringement. This paper launches the study of detecting unauthorized code use in CPLMs, i.e., Code Membership Inference (CMI) task. We design a framework Buzzer for different settings of CMI. Buzzer deploys several inference techniques, including signal extraction from pre-training tasks, hard-to-learn sample calibration and weighted inference, to identify code membership status accurately. Extensive experiments show that CMI can be achieved with high accuracy using Buzzer. Hence, Buzzer can serve as a CMI tool and help protect intellectual property rights. The implementation of Buzzer is available at: https://github.com/KDEGroup/Buzzer

## introduction recently, various code pre-trained language models (cplms) like codebert @xcite and code llama @xcite have sprung up and shown strong capabilities. cplms are pretrained over massive code data that is publicly available in platforms like github and stackoverflow. then, cplms can be fine-tuned or directly used for code-related tasks like code refactoring @xcite and code search @xcite even when the downstream tasks do not have much data, reducing the intellectual burden of developers and facilitating software development and maintenance however, using code data to train cplms may cause patent infringement and legal violations. github recently introduced a programming tool copilot 1 . copilot is powered by openai codex, a gpt based cplm. however, copilot has faced allegations of violating open-source licenses (in-foq, 2022) since it is trained on code that may be collected from open-source projects. although it is still under debate whether using open-source code to train cplms causes intellectual property infringement, the lawsuit has alerted researchers and companies who work on cplms: code data from open-source projects is not free training data. to help protect the intellectual property rights on code data, this paper studies a new task named code membership inference (cmi) for cplms which identifies whether a well-trained cplm used a certain code snippet as its training data. a cmi method for cplms can serve as a tool to detect unauthorized data use and provide potential evidence when a lawsuit similar to copilot's case is filed in the future. we propose a cmi framework buzzer for detecting unauthorized data use in different settings of cmi. the contributions of this work are summarized as follows: ## conclusion in this paper, we study cmi for authenticating data compliance in cplms and propose a framework buzzer for inferring code membership. buzzer achieves promising results on various cplms as shown in the experiments. buzzer can serve as a cmi tool and help protect the intellectual property rights. in the future, we plan to further improve the generalization ability of buzzer to make it more practical. we will also explore the idea of this work on other multimodal pre-trained language models beyond cplms.