SACL : Understanding and Combating Textual Bias in Code Retrieval with Semantic-Augmented Reranking and Localization (2025)

Abstract: In this work, we conduct an in-depth analysis of code retrieval by systematically masking specific features while preserving code functionality. Our discoveries include: (1) although trained on code, current retrievers heavily rely on surface-level textual features (e.g., docstrings, identifier names), and (2) they exhibit a strong bias towards well-documented code, even if the documentation is irrelevant. Based on our discoveries, we propose SACL, a framework that enriches textual information and reduces bias by augmenting code or structural knowledge with semantic information. Extensive experiments show that SACL substantially improves code retrieval (e.g., by 12.8% / 9.4% / 7.0% Recall@1 on HumanEval / MBPP / SWE-Bench-Lite), which also leads to better code generation performance (e.g., by 4.88% Pass@1 on HumanEval).

## introduction retrieval-augmented code generation (racg) is the technique of generating code based on relevant documents retrieved from a corpus @xcite . racg is shown to be beneficial in script-level code generation, which provides background knowledge or functionally relevant snippets, and is particularly important for repository-level (repo-level) code generation, where models must be aware of other files within the repository @xcite . however, recent work has shown that retrieval quality remains a significant bottleneck for racg performance (e.g., agentless @xcite only achieves 35.3% line localization accuracy on swe-bench). while extensive analysis has been conducted on the capabilities and challenges of text retrievers @xcite @xcite figure 1 : summaries of our discoveries from the analyses. our analyses reveal code retrievers' heavy dependence on textual features rather than functional semantics, leading to bias favoring well-documented code regardless of relevance. @xcite , a systematic investigation of code retrievers remains relatively underexplored. the nature of code corpora differs fundamentally from text corpora due to their highly structured nature, with strict syntax rules and structures @xcite . unlike text documents, the semantic meaning of a code snippet can be completely altered by small syntactic changes, making traditional retrieval analysis less effective for code search tasks. such fundamental differences suggest that code retrievers may have significantly different behaviors from text retrievers, highlighting the need for more focused analysis. in this work, to develop a deeper understanding of code retrieval, we conduct empirical analyses on both code retrievers and in-context rerankers to answer two critical research questions: (rq1) what features are code retrievers primarily based on? and (rq2) do retrievers exhibit bias? specifically, we introduce a normalization-based analysis framework, where we systematically mask textual features such as docstrings, function names, and variable names or replace them with placeholders. such transformations preserve the code's functionality but eliminate textual cues, allowing us to evaluate the dependence and bias of textual features. we illustrate our discoveries in figure 1 . we observe that [discovery 1] although trained on code, current code retrievers exhibit strong dependency on textual features (e.g., docstrings and function or variable names) and under-utilize the functionality of code. specifically, when all textual features are normalized, we observe significant performance degradation on both embedding-based code retrieval and in-context code reranking. for instance, with normalization, the recall@1 performance of gist-large degrades from 98.6% to 9.0% on mbpp @xcite . as shown in figure 1 , our analyses also reveal that [discovery 2] retrievers consistently assign higher relevance scores to well-documented code, even when the documentation is functionally irrelevant. particularly, compared to discovery 1's setting, where all the code documents are normalized, only normalizing the positive documents leads to even worse retrieval and reranking performances. the results indicate the bias towards well-documented code with meaningful identifier names, which may lead to preferring irrelevant but well-documented code over relevant but poorly documented code. based on these discoveries, we present sacl, which improves code retrieval with semantic-augmented code reranking and in-context localization. based on our discoveries that retrievers are more sensitive to textual features, in the reranking stage, we first generate textual descriptions for the retrieved code documents, and then aggregate the retrieval scores for the original code documents and textual descriptions for final re-ranking. such design bridges the code and text modalities and mitigates the bias between welldocumented and sparsely documented code. based on the empirical discovery that in-context reranking also exhibits similar textual bias, for repo-level code generation, we further introduce semanticaugmented in-context localization, where we generate supplementary file descriptions for the repository structure to augment the context for file localization. empirical results show that such methods are the most effective when the file names do not contain rich semantic information. our experimental results demonstrate significant improvements across three public benchmarks: hu-maneval @xcite , mbpp @xcite , and swe-bench-lite @xcite . for instance, sacl achieves 12.8%/9.4% code retrieval recall@1 gain on humaneval/mbpp under the full normalization setting and achieves 7.0% file localization recall@1 on swe-bench-lite with the agentless pipeline @xcite . our improvements on code retrieval and localization also leads to performance gain on code generation (e.g., 4.88% pass@1 gain on humaneval and 1.67% on swe-bench-lite). these results highlight the effectiveness of our approaches in enhancing the semantic understanding capabilities of code retrievers and mitigating lexical bias. ## analysis: textual bias in code retrieval this section answers our two research questions through two controlled experiments: (rq1) what features are code retrievers primarily based on? and (rq2) do retrievers exhibit lexical-level bias? these experiments aim to identify whether code retrievers favor textual characteristics over functional semantics when matching queries to code. ## methodology both [discovery 1] and [discovery 2] reveal that code retrievers heavily rely on textual information (e.g., documentation and identifier names) rather than understanding of code structure. to combat this issue, sacl introduces two techniques: semantic-augmented code reranking and semanticaugmented in-context localization. both methods augment the retrieved code or structure with textual descriptions to improve the encapsulation of semantic information. ## experiments with our experiments, we aim to answer the following research questions: (rq1) what impact does sacl have on code retrieval performance? (rq2) given this code retrieval performance, what is the downstream code generation performance improvement? (rq3) why does semantic augmentation benefit code retrieval? (rq4) which hyper-parameters are optimal? 4.1 experiment setup datasets. we evaluate our approach on three widely used benchmarks: humaneval @xcite and mbpp @xcite are scriptlevel algorithm problem datasets and swe-bench-lite @xcite is a repo-level issuesolving dataset. for humaneval and mbpp, we evaluate under various "normalization" settings (as introduced in §2), which preserve the functionalities of the code documents but are more challenging to retriever models. ## conclusion we conduct systematic normalization experiments and uncover two key biases for current code retrieval systems: (1) heavy dependence on textual cues over semantic understanding, and (2) consistent preference for well-documented code, even if the documentation is irrelevant. to address these issues, we propose sacl, a framework that augments code retrieval and localization with semantic information through natural language descriptions. experiments demonstrate sacl's effectiveness in both code retrieval and code generation on three widely used benchmarks: humaneval, mbpp, and swe-bench-lite. for instance, sacl improves code retrieval by up to 12.8% recall@1 on hu-maneval and up to 8.0% file localization accuracy on swe-bench-lite. these gains translate directly into better downstream code generation quality (e.g., up to 4.88% pass@1 gain on humaneval).