C o D et-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings (2025)

Abstract: Large Language Models (LLMs) have revolutionized code generation, automating programming with remarkable efficiency. However, this has had important consequences for programming skills, ethics, and assessment integrity, thus making the detection of LLM-generated code essential for maintaining accountability and standards. While, there has been some previous research on this problem, it generally lacks domain coverage and robustness, and only covers a small number of programming languages. Here, we aim to bridge this gap. In particular, we propose a framework capable of distinguishing between human-written and LLM-generated program code across multiple programming languages, code generators, and domains. We use a large-scale dataset from renowned platforms and LLM-based code generators, alongside applying rigorous data quality checks, feature engineering, and comparative analysis of traditional machine learning models, pre-trained language models (PLMs), and LLMs for code detection. We perform an evaluation on out-of-domain scenarios, such as detecting authorship and hybrid authorship of generated code and generalizing to unseen models, domains, and programming languages. Our extensive experiments show that our framework effectively distinguishes human-written from LLM-generated program code, setting a new benchmark for the task.

## introduction recent advancements in large language models (llms) have demonstrated their remarkable ability to generate outputs that closely emulate human-written content @xcite . this has spurred exponential growth in research, with publications on llm-based code generation in leading venues rising from a single article in 2018 to 140 in 2024 @xcite . moreover, these developments promise to accelerate software development, automate routine tasks, and boost productivity. at the same time, the rapid progress in artificial intelligence (ai) generative systems has posed major concerns, particularly related to accountability and ethical use of this technology @xcite . machine-generated code can be exploited to create obfuscated scripts, introduce vulnerabilities, and produce deceptive artifacts that are difficult to trace @xcite . thus, it is important to develop tools that can detect machinegenerated content, e.g., tracing ai-assisted commits could empower code reviewers to proactively mitigate risks. in academia, the use of llms for completing written assignments undermines educational integrity, with professors unknowingly grading machine-generated submissions @xcite . alarmingly, more than 60,000 scientific articles in the past year alone have shown evidence of machine-generated content @xcite . accountability is equally critical for talent hiring and candidate evaluation. employers must verify that the code submitted by a candidate truly reflects their actual abilities. without robust detection mechanisms, generative ai could lead to misleading assessments. moreover, detecting the use of llm-generated code is essential for developing more effective code-based llms; since these systems depend on human-written samples for training, accurately detecting ai-generated code helps curate higher-quality training datasets. previous work has proposed frameworks for detecting machine-generated code, including contrastive learning with a unixcoder-based semantic encoder @xcite and machine learning (ml) models that analyzed claude 3-generated code in the codesearchnet dataset @xcite . that work has focused on a single apibased code copilot, but today we face a growing prevalence of a variety of open-weights llms as well as locally deployable llm-based code assistants. despite previous efforts, this evolving state of llms de facto highlights the urgent need for reliable curated large-scale high-quality code data with various programming languages and methods to distinguish between human-written and llmgenerated code to mitigate risks and maintain accountability in software development and academic integrity @xcite . with this in mind, addressing the aforementioned limitations is crucial, as llms are designed to generate code across various languages and domains foot_0 and its form: functions, classes, and arbitrary code snippets. here, we aim to bridge these gaps by constructing a first-of-its-kind large-scale, multi-lingual, multi-domain, and multi-generator dataset comprising ≈ 500k samples of human-and llm-written code. the dataset spans class-level and functionlevel code, as well as competitive and industrial programming contexts. we further propose models for detecting llm-generated code, and we evaluate their performance in extreme out-of-domain (ood) settings: detecting unseen models, unseen domains, and unseen programming languages. we aim to answer the following research questions (rqs): we make the following contributions: ## related work in this section, we discuss related work, focusing on resources and models for detecting machinegenerated code. to ensure coverage across multiple domains, we included human-written code in c++, java, and python from github using the codesearchnet dataset @xcite , and github api. we chose this dataset because it was released in 2019, predating the widespread use of ai for code generation. in total, we collected 135,566 human-written code samples from github: 60,000 in python, 59,998 in java, and 15,568 in c++ (mainly collected using the api). this portion of our dataset is specifically designed for function-level llmgenerated code detection. overall, the language distribution in our dataset is imbalanced, as shown in figure 4 (appendix b ). java and python are represented in nearly equal proportions, with slightly fewer c++ codes. a similar pattern is observed in the distribution of data sources: github and leetcode contribute nearly equal amounts of code, while codeforces provides slightly fewer samples, as shown in figure 5 @xcite . more details about the data distribution are given in the appendix b. ## experiments & results in this section, we detail our experiments aimed at developing models to detect llm-generated code. we evaluate these models under extreme conditions, including unseen models, unseen languages, and code from the unseen domains (more precisely: unseen code sources and unseen code structures). ## conclusion & future work we introduced codet-m4, a corpus for machinegenerated code detection that spans multiple programming languages, code generators, and domains. using this dataset, we developed and evaluated models to detect llm-generated code, focusing on their robustness to ood data. our findings show that these models generalize well across languages with syntactic similarities to those in the training set and handle variations in generator configurations (e.g., the same model with different parameter scales). however, their performance drops significantly in unseen domains and hybrid generation scenarios. in future work, we aim to expand our dataset to include more programming languages and code generators, further improving the models generalization capabilities. additionally, we plan to explore contrastive learning and domain adaptation to mitigate performance drops in unseen domains.