Source Code Plagiarism Detection with Pre-Trained Model Embeddings and Automated Machine Learning (2023)

Abstract: Source code plagiarism is a critical ethical issue in computer science education where students use someone else’s work as their own. It can be treated as a binary classification problem where the output can be either: yes (plagiarism found) or no (plagiarism not found). In this research, we have taken the open-source dataset ‘SOCO’, which contains two programming languages (PLs), namely Java and C/C++ (although our method could be applied to any PL). Source codes should be converted to vector representations that capture both the syntax and semantics of the text, known as contextual embeddings. These embeddings would be generated using source code pre-trained models (CodePTMs). The cosine similarity scores of three different CodePTMs were selected as features. The classifier selection and parameter tuning were conducted with the assistance of Automated Machine Learning (AutoML). The selected classifiers were tested, initially on Java, and the proposed approach produced average to high results compared to other published research, and surpassed the baseline (the JPlag plagiarism detection tool). For C/C++, the approach outperformed other research work and produced the highest ranking score.

## introduction plagiarism is the ethical and educational issue of taking ideas from other sources and representing them as your own without acknowledgement. plagiarism can be divided into text and source code. academic source code plagiarism can be defined as "source-code plagiarism in programming assignments can occur when a student reuses source code authored by someone else and, intentionally or unintentionally, fails to acknowledge it adequately, thus submitting it as his/her own work. this involves obtaining the source-code, either with or without the permission of the original author and reusing the source code produced as part of another assessment (in which academic credit was gained) without adequate acknowledgement" @xcite . the words reuse, obtain, and acknowledge may also be defined on the basis of the academic requirements. the detection of plagiarism is a lengthy and demanding process, so new technologies such as artificial intelligence (ai) might be used effectively. plagiarism can be treated as a classification problem as the output can be considered a class of discrete values: 'yes' (plagiarised), 'no' (nonplagiarised), or potentially 'partial'. source code plagiarism can be also considered to be an application of a source code similarity measurement task @xcite . there are several ways to represent source codes @xcite such as graphs, trees and tokens. the source codes must be converted into vectors known as embeddings before being fed into a classifier. contextualized embeddings not only consider syntax but also the semantics of source codes. these embeddings can be created using pre-trained models. the emergence of pre-trained models has revolutionized the field of natural language processing (nlp) and are being known for their robustness. they can be utilised in various ways such as re-training, fine-tuning and inference. also, some domain-specific models have been created for certain areas and tasks. this work inspects the robustness of the embeddings generated by source code pre-trained models (codeptms) for the task of source code plagiarism detection. contextual embeddings are extracted using these codeptms and a classifier built on top of the embeddings. for classification, this work utilises the concept of auto machine learning (au-toml) to determine the best classifier given certain training data. the training, testing, and evaluation are based on the source code reuse dataset (soco) @xcite . the paper is organized as follows: section 2 covers related work, section 3 covers the methodology, section 4 presents the results, and section 5 covers the conclusion and future work. ## related work several software tools have been used as source code plagiarism detectors. novak compared several such tools, namely jplag, moss, sim, splat, marble, plaggie, and sherlock warwick in his review paper @xcite . engels et al. introduced the idea of neural networks in source code plagiarism detection and reused the output of moss on a dataset containing 20,706 c++ introductory course assignments. the evaluation was performed on the basis of the classification evaluation metrics (precision, recall, and f1 scores) @xcite . ljubovic and pajic tackled the issue of external plagiarism by using a cloud and applying an artificial neural network (ann) based on the output of the sim's software and repository monitoring on a dataset containing 3,655 submissions from an introductory c course (ljubovic and pajic, 2020). their setup was compared to jplag, moss and sim. abstract syntax trees (asts) along with code disassembly have been used by viuginov et al. who considered the lexical, syntactic, layout, and structural characteristics of the source code on a dataset containing 90,000 c++ solutions @xcite . for the evaluation, they calculated the f1score. manahi et al. used a combination of siamese networks, bidirectional long short-term memory (blstm), and character embeddings on a dataset including 16,800 introductory course c assignments @xcite . siamese networks are multiple similar neural networks with the same configurations and weights. they are mainly used for similarity detection. for their evaluation, they calculated the classification evaluation metrics. humayoun et al. used the concepts of tokenization, ast, and upsampling on their public dataset of 60 introductory c++ programming assignments @xcite . the features tested were n-gram overlap, longest common substring (lcs), and greedy string tilling. eight classifiers were implemented using weka and the authors' model was evaluated based on the classification evaluation metrics. in the first part of his master's thesis @xcite , heres proposed a system using n-grams, term frequency-inverse document frequency (tf-idf), and cosine similarity. the dataset used was the soco java set and their own private dataset having 16,954 files. the evaluation was based on the average precision score. deep learning using char-recurrent neural network (char-rnn) and long short-term memory (lstm) was the basis of katta's approach, which used general deep features that could be applied to any dataset @xcite . the dataset covered an introductory c course with a total of 4,700 submissions. the model was evaluated using classification evaluation metrics. ## methodology this work follows the typical process of applying supervised ml to a classification problem, as mentioned in @xcite . the cycle starts with data collection and feature engineering, followed by model selection. the model needed to be trained and tested. enhancements with parameter tuning could be applied, prior to the model being evaluated. ## conclusion plagiarism in programming assignments is a critical issue in the field of computer science education. it can be treated as a machine learning binary classification problem. so, this research introduced a simple yet effective approach to the task of source code plagiarism detection. it started by selecting the open-source soco dataset with two pls (java and c/c++). source code files were converted to embeddings to be part of any machine learning classifier. three different codeptms (plbart, unixcoder, and codeberta) were used to generate their own embeddings. cosine similarity scores between these three models were calculated and considered to be the selected features. the classification models were selected using the concept of automl and the library autosklearn. the initial testing was conducted on java, and the proposed model produced high metrics as compared to other approaches and exceeded both baselines. for c/c++, the model produced the highest f1 and precision scores as compared to other approaches and outperformed both baselines. exploring other codeptms that are not available on huggingface for source code plagiarism detection is an idea for future work, along with increasing the training time for autosklearn. the dataset is severely imbalanced, hence, different techniques could be used to tackle such issues. also, chunking can be used to overcome the limited input size of the pre-trained models.