MATCH : Task-Driven Code Evaluation through Contrastive Learning (2025)

Abstract: AI-based code generation is increasingly prevalent, with GitHub Copilot estimated to generate 46% of the code on GitHub. Accurately evaluating how well generated code aligns with developer intent remains a critical challenge. Traditional evaluation methods, such as unit tests, are often unscalable and costly. Syntactic similarity metrics (e.g., BLEU, ROUGE) fail to capture code functionality, and metrics like CodeBERTScore require reference code, which is not always available. To address the gap in reference-free evaluation, with few alternatives such as ICE-Score, this paper introduces MATCH, a novel reference-free metric. MATCH uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions, enabling similarity scoring that reflects how well generated code implements the task. We show that MATCH achieves stronger correlations with functional correctness and human preference than existing metrics across multiple programming languages.

## introduction the integration of large language models (llms) into modern software development has revolutionized the field through automated code generation. this field, often called neural natural-languageto-code (nl2code), has advanced rapidly with a wave of models, from specialized open-source tools to powerful proprietary systems @xcite @xcite @xcite @xcite @xcite . these llms are revolutionizing software development, with code generation tools like github copilot now responsible for approximately 46% of code on @xcite , highlighting the critical need for effective evaluation methods. although executing generated code against unit tests is considered the gold standard for assessing functional correctness, it often proves impractical. comprehensive test creation requires significant manual effort, especially in niche languages like verilog and cobol, where community-driven testing is limited and automated tools may be unavailable. proprietary codebases further complicate evaluation when documentation remains locked within companies. execution-based methods for code validation evaluate the functional correctness of generated code using testing tools such as unit tests. however, these methods might overlook valuable code snippets that contain minor syntactic errors yet still capture essential logic. for software engineers who can easily fix minor bugs, these functionally sound snippets are highly valuable. this highlights the need for evaluation metrics that prioritize functional correctness over strict syntactic adherence, focusing on a code's ability to perform intended tasks despite imperfections. syntactic similarity metrics such as bleu @xcite and @xcite offer computationally efficient alternatives but fail to capture the semantic meaning of code, being easily influenced by superficial variations in formatting or identifier names. some natural language metrics have been enhanced to incorporate code-specific elements like abstract syntax trees (asts) and program dependency graphs (pdgs), including ruby @xcite and @xcite . despite these improvements, such metrics often lack correlation with human evaluations of code quality @xcite . recent approaches, such as codebertscore @xcite , aim to improve traditional metrics by leveraging pre-trained language models to evaluate the similarity of generated code against a reference implementation-a ground-truth solution known to successfully solve the task. while effective, these approaches fundamentally depend on figure 1 : two examples from the humaneval dataset that illustrate match's evaluation ability for correct and incorrect code snippets given a description (for match) and a reference code (for codebertscore, denoted as cbs). the type of incorrectness for the incorrect code is described as well. we show the score for both approaches for both correct and incorrect python code, displaying match's ability to differentiate between them compared to codebertscore. the availability of reference code, which prevents their use in novel code generation tasks where such references are unavailable or impractical to obtain. to address this limitation, the ice-score metric proposed by zhuo (2023) prompts a large language model (llm) like gpt to evaluate generated code quality directly, without relying on test cases or reference solutions. the llm assesses various aspects such as correctness and usefulness, making ice-score a flexible, reference-free metric, though this advantage comes at the cost of increased latency and computational resources. a different approach is taken by codescore @xcite , which trains an llm to predict functional correctness by estimating how many test cases a snippet passes and whether it runs without errors. code-score supports three variants: natural language only, reference code only, or both. the two variants involving reference code require the reference implementation, which can be restrictive and costly. moreover, all three variants require precomputed percentages of passing test cases, further limiting applicability. to overcome these challenges, we introduce match: metric for assessing task and code harmony, a novel contrastive learning-based approach to code evaluation that does not require reference code or execution. by learning meaningful embeddings for both code and natural language task descriptions, match provides a nuanced assessment of code that captures both functional and semantic correctness. we show that match correlates strongly with human judgments of code quality and significantly outperforms existing metrics, particularly in scenarios where traditional evaluation methods are impractical. in figure 1 , we present examples comparing match with code-bertscore, illustrating how similarity to task descriptions helps match avoid inherent biases of reference-based methods while yielding scores that reflect syntactic and logical correctness. additional examples are provided in appendix b. contributions: in summary, this paper makes the following contributions: the remainder of this paper is structured as follows: section 2 provides a detailed overview of related work in code evaluation. section 3 formulates the specific problem addressed in this paper. section 4 describes the new method of match. section 5 and section 6 present the experimental setup and benchmark results respectively. section 7 presents performance comparisons for different architecture choices. finally, section 8 concludes the paper and discusses future directions, including a limitations discussion and potential areas for future research. ## problem formulation the input consists of: a natural language task description or instruction @xmath0, where t represents the set of all possible task descriptions, a code generation model generates a code snippet @xmath1, where c is the set of all possible code snippets, and a ground truth label y that is assigned to evaluate the quality of the code. this label can take one of the following forms: we use y to refer to either type of label, depending on the context. goal our goal is to define a metric function f : t ×@xmath2, 1], where for a given task description t and a code snippet c, the function f (t, c) produces a metric score that reflects the quality of the code snippet c with respect to the task described in t. ideally, the metric function f should correlate with human evaluations and functional correctness. in particular, for two candidate code snippets c 1 , c 2 ∈ c implementing the same task @xmath3, we want the metric to indicate that if c 1 performs the task better and is preferred by human evaluators more than c 2 , then c 1 should receive a higher metric score. specifically, we seek a function f such that f (t, c 1 ) > f (t, c 2 ) when c 1 is considered better than c 2 based on its functionality and human preference. ## match this section introduces match, our method for code evaluation. we detail its neural architecture in section 4.1 and the contrastive loss objectives used for optimization in section 4.2 4.1 architecture match generates a similarity score between task descriptions and code snippets by learning embeddings. this process is illustrated in figure 2 . a key aspect of match is its ability to model both functional correctness (successful implementation) and human preferences (developer-favored code) in the similarity space. a close (positive) pair (t, c) represents a task description t and a corresponding code snippet c that either successfully implements the task or is highly preferred by developers. ## conclusion we introduced match, a practical, reference-free metric for evaluating code implementations against natural language descriptions. addressing the critical challenge of assessing code quality without requiring reference implementations or extensive test cases, match leverages contrastive learning to embed code and text into a shared semantic space. our method first independently encodes code and natural language, then enriches their representations through cross-modal interaction. to quantify alignment using cosine similarity. empirical experiments demonstrated match's superior performance, achieving significantly stronger correlations with both functional correctness and human preference compared to existing baselines. this positions match as a powerful and flexible solution for efficiently assessing code quality, particularly in scenarios where traditional evaluation methods are impractical. looking ahead, match can be extended to evaluate a broader spectrum of code quality beyond functional correctness and human preference. this includes non-functional aspects such as efficiency (time and space complexity), readability, maintainability, and security vulnerabilities, enabling a more holistic and practical assessment of generated code. another promising direction is testing the method's robustness under imperfect or noisy inputs, which are common in real-world scenarios.