saymanq/repotrace-codebert

RepoTrace CodeBERT reranker

This repository contains the fine-tuned neural reranker from saymanq/RepoTrace. It scores an issue report against a candidate source-code chunk. RepoTrace uses BM25 to select candidate files before applying this model.

Model

• Base encoder: microsoft/codebert-base
• Head: one linear layer over the CLS representation
• Task: cross-encoder relevance scoring for issue text and source-code chunks
• Published checkpoint: ranker.pt, 475.6 MiB
• Final ranker: 50% normalized BM25 score and 50% normalized CodeBERT score

The neural weight was chosen using validation MRR before the test split was evaluated.

Training data

RepoTrace uses public issue/fix histories from bug-localization/BeetleBox. Preprocessing produced 157 bugs from 13 repositories across C++, Go, Java, JavaScript, and Python. It created 13,855 issue/chunk pairs: 2,771 positive and 11,084 negative.

Positive chunks overlap changed regions in the fixing diff. File-level fallback positives are recorded when line-level matching is unavailable. Each positive receives two BM25 hard negatives, one path/symbol-confusable negative, and one random negative. Files introduced by a fix are excluded because they do not exist in the pre-fix snapshot.

Generated, vendored, minified, binary, oversized, and unsupported files are filtered. The build records failures and counts rather than silently discarding them.

Split strategy

The repository-disjoint split contains:

Split	Repositories	Bugs	Pairs	Main language mix
Train	8	103	10,585	C++, Go, Java, JavaScript, Python
Validation	2	15	1,985	14 Java, 1 Python
Test	3	39	1,285	39 Python

No repository appears in more than one split. Every training and evaluation example uses the repository state before the fixing commit.

Training configuration

• Accelerator: Colab T4 CUDA GPU
• Epochs: 2
• Learning rate: 2e-5
• Weight decay: 0.01
• Maximum sequence length: 512 tokens
• Issue token budget: 160
• Batch size: 4
• Gradient accumulation: 8, effective batch size 32
• Loss: binary cross-entropy
• Seed: 42

Epoch 1 took 365.4 seconds and reached pair-set validation MRR 0.3053. Epoch 2 took 370.7 seconds and reached 0.3964, so epoch 2 was saved.

End-to-end evaluation

BM25 retrieves the highest-scoring chunk from each of 50 distinct candidate files. CodeBERT scores those pairs. The final system blends per-query min-max-normalized BM25 and neural scores with neural weight 0.5, selected on validation.

Validation

System	Hit@1	Hit@5	Hit@10	MRR	MAP
BM25	0.133	0.333	0.400	0.224	0.219
Hybrid	0.267	0.400	0.467	0.321	0.275

Held-out test

System	Hit@1	Hit@5	Hit@10	Recall@10	MRR	MAP
BM25	0.231	0.410	0.513	0.401	0.299	0.250
Hybrid	0.179	0.333	0.487	0.357	0.263	0.210

The hybrid's test MRR was 12.0% below BM25. It improved 5 individual bugs, tied 13, and worsened 21. Candidate retrieval found at least one changed file for 29 of 39 test bugs.

The validation improvement did not transfer to the held-out Python repositories. The language and repository composition differs sharply between validation and test; that is a plausible source of the gap, but the current evidence does not establish causality.

See the held-out error analysis for sampled cases.

Serving benchmark

Batched PyTorch was faster than float ONNX Runtime and dynamic int8 ONNX Runtime in the local CPU benchmark. At 50 candidates, median times were 2.90 seconds, 4.03 seconds, and 4.05 seconds, respectively. The float ONNX graph matched PyTorch sample logits within 7.9e-6.

Dynamic int8 reduced weights from about 473 MiB to 119 MiB, but its sample logit difference reached 0.0721. Its ranking quality has not been validated, so it is not the serving model.

Intended use

RepoTrace suggests source files for a developer to inspect when investigating a bug in a public repository. A human should review the issue, ranked files, and surrounding code.

Loading the checkpoint

The checkpoint is a PyTorch state dictionary for RepoTrace's CodeBertRanker class. Clone the source repository and install its ML dependencies first:

git clone https://github.com/saymanq/RepoTrace.git
cd RepoTrace
python -m pip install -e '.[ml]'

Then load the published weights:

import torch
from huggingface_hub import hf_hub_download

from repotrace.models.codebert_ranker import MODEL_NAME, CodeBertRanker

checkpoint = hf_hub_download(
    repo_id="YOUR_USERNAME/repotrace-codebert",
    filename="ranker.pt",
)
ranker = CodeBertRanker(MODEL_NAME)
ranker.load_state_dict(torch.load(checkpoint, map_location="cpu", weights_only=True))
ranker.eval()

scores = ranker.score_pairs(
    ["S3 upload adds unexpected bytes"],
    ["FILE: services/s3/provider.py\nLANGUAGE: python\ndef put_object(...): ..."],
    device="cpu",
)

The returned values are uncalibrated relevance scores. Compare scores only among candidates for the same issue.

Out-of-scope use

• Automated code changes or unreviewed fixes
• Claims that a ranked file contains the root cause
• Security auditing or vulnerability detection
• Safety-critical decisions
• Uploading private or sensitive code to a public demo

Known limitations

• The test set has only 39 Python bugs from Airflow, LocalStack, and Odoo.
• Validation is dominated by one large Java repository, and its selected blend did not generalize better than BM25 on test.
• Ten test bugs had no changed file in the 50-file candidate set.
• Vague issues and very large repositories can produce weak rankings.
• CodeBERT truncates long issue/code pairs to 512 tokens.
• Generated and vendored code is intentionally excluded.
• A live request analyzes the repository's current default branch, which can differ from the historical snapshot used for evaluation.
• Scores are uncalibrated relevance values, not probabilities.

Potential biases

The data reflects public open-source projects, their issue-writing styles, naming conventions, and contribution patterns. Proprietary repositories, smaller projects, non-English issues, and underrepresented languages may perform differently.

Privacy and security

• Training data comes from public repositories and issue histories.
• The public API accepts only public GitHub HTTPS URLs.
• Repository code is treated as text and never executed.
• Source code and issue bodies are not sent to third-party LLM APIs.
• Production logs exclude source contents and full issue bodies.
• Temporary snapshots are removed after each request; bare clones use a bounded cache.