CCT -Code: Cross-Consistency Training for Multilingual Clone Detection and Code Search (2025)

Abstract: We consider the well-known and important tasks of clone detection and information retrieval for source code. The most standard setup is to search clones inside the same language code snippets. But it is also useful to find code snippets with identical behaviour in different programming languages. Nevertheless multi- and cross-lingual clone detection has been little studied in literature. We present a novel training procedure, cross-consistency training (CCT) leveraging cross-lingual similarity, that we apply to train language models on source code in various programming languages. We show that this training is effective both for encoder- and decoder-based models.The trained encoder-based CCT-LM model%and fine-tuned with CCT,achieves a new state of the art on POJ-104 (monolingual C++ clone detection benchmark) with 96.73% MAP and AdvTest (monolingual Python code search benchmark) with 47.18% MRR. The decoder-based CCT-LM model shows comparable performance in these tasks. In addition, we formulate the multi- and cross-lingual clone detection problem and present XCD, a new benchmark dataset produced from CodeForces submissions.

## introduction clone detection is crucial in software development for identifying semantically similar code, aiding in unification, refactoring, and side effect control. originally formulated for c/c++ by @xcite , the task has since expanded to other languages, with the next step being multilingual clone detection. this work introduces a new multilingual dataset xcd and establishes baseline models. early clone detection relied on algorithmic methods @xcite , later evolving into machine learning-based approaches (li et al., 2017; thaller et al., 2020; gotmare et al., * equal contribution. 2021) that embed code snippets for similaritybased retrieval. we propose cct, a novel training technique that enhances code embeddings, achieving state-of-the-art results on both poj-104 @xcite and our new xcd dataset. additionally, we demonstrate that cct-lm, trained with cct, is also effective for code search, as formulated by @xcite . main contributions cct -a pretraining method for aligning multilingual code snippets. xcd -a novel multilingual clone detection dataset from codeforces. state-of-the-art results on poj-104 and xcd with cct-lm. cct-lm achieves state-of-the-art on advtest for code search. ## related work our methods are inspired by natural language processing, thus related work includes both pure nlp and source code processing. datasets. @xcite presented the codesearchnet dataset constructed from a github dump where the authors split method bodies into the code itself and a description. this dataset contains 2 million code snippet-description pairs in 6 programming languages, including python. this dataset was partially used by @xcite who combined codesearchnet and three other datasets into a larger one. from code-searchnet they used the java part and python part translated automatically into java. the resulting dataset contains 4 million code snippetdescription pairs. there are two main datasets for clone detection: poj-104 @xcite and bigclonebench @xcite . poj-104 represents a comparatively small corpus of c++ solutions from a student judging system. big-clonebench comprises a vast dataset containing automatically mined data in the java language. code search. gu et al. (2018) introduced dense vector representations for code search, training two recurrent neural networks for source code and text. feng et al. (2020) used a language model to produce these representations. gotmare et al. (2021) employed three transformer-based models for hierarchical encoding but found parameter sharing reduced quality. in contrast, our model uses a single transformer decoder to embed queries and documents, omitting the classifier. clone detection. one of the first successful deep learning approaches was cclearner @xcite that used text extracted from a program and its ast features and had a simplistic multilayer perceptron architecture for clone classification on a closed code base. more recent deep learning models include graph neural networks on asts @xcite and employ pretrained language models @xcite . language models for source code. bertlike models, initially successful in natural language processing, have been adapted for programming languages. several pre-trained models have emerged, including codebert @xcite , a bimodal model trained on masked language modeling (mlm) and replaced token detection; @xcite , which incorporates abstract syntax trees for training; and syncobert @xcite , which leverages multimodal contrastive learning with identifier and ast edge prediction. more recently, autoregressive decoder models like @xcite have gained prominence, focusing on source code generation tasks such as code completion and documentation generation. ## datasets in this work we use two kinds of datasets, one for clone detection and another for code search. code search. for code search we use the codesearchnet dataset introduced by @xcite . the original version of codesearchnet consists of natural language queries paired with most relevant code snippets in six programming languages. each snippet represents the code of a function collected from github open source code. ## method in this section, we introduce our pre-training approach cct (cross-consistency training). its goal is to robustly learn the embedding space of code snippets and create a strong alignment between snippets solving the same problems across programming languages. the difference between strong and weak alignment is illustrated in fig. 1 : in a weakly aligned embedding space, the nearest neighbor might be a semantically similar snippet from a different language but generally most neighbors are in the same language, while in a strongly aligned space the similarity is purely semantic and does not care about the language at all. to achieve strong alignment, we employ a contrastive learning objective l xcd : for a randomly code snippet, we train the vector representations of the source code tokens in such a way that their aggregation, for example, averaging or last token, is closer to the source code, which solves the same problem regardless of the programming language. this ensures that the embeddings of the source code differentiates between related snippets and random or similar but different (hard negative) snippets effectively. ## experiments in this section, we describe the details about data pre-training and our cct pipeline for multilingual clone detection and code search tasks. pretraining. we train two models, one is encoder-based, which is initialized with pretrained graphcodebert base @xcite ; we call the resulting model cct-lm enc . another one is decoder-based, which is initilized with a pretrained deepseek-coder-1.3b model @xcite ; we call the resulting model cct-lm dec . similarity scores are calculated based on dot products of the last token vector representations, but we also researched using various types of poolings and allowing bidirectional attention. hyperparameters. we use the adamw optimizer with learning rate 5e-5, weight decay 0.01, ## conclusion understanding semantic similarity is crucial for language processing, enabling solutions for various tasks in natural and programming languages. in this work, we presented cct-lm, a new method that enhances this capability via a novel cct pretraining approach, demonstrating its effectiveness in clone detection and code search. we introduced a novel task of multilingual clone detection and the xcd dataset for multilingual source code analysis, formalized in two evaluation setups. the proposed cct-lm models (encoder-and decoder-based) outperformed strong baselines in clone detection and code search. cct-lm enc excelled across all setups for multi-and crosslingual evaluation, showing that cct pretraining improves semantic similarity understanding in language models. we hope our method benefits other source code processing tasks, left for future work, and believe modifications of our approach could aid nlp and other machine learning fields.