CodeBERT-PECAN

microsoft/codebert-base fine-tuned on the PECAN benchmark to identify the programming language of a raw code fragment of at most 512 bytes, with no file name, extension, or repository context. It predicts one of 319 languages (config.id2label; lowercased, these are exactly the PECAN labels).

Artifact repository (code, scoring, all results): https://anonymous.4open.science/r/pecan-benchmark

Results (official PECAN test split, 9,179,661 fragments)

Accuracy	Macro-F1 (319 labels)	Top-5	Top-10	Pop-20	Head	Torso	Tail
0.9575	0.9304	0.983	0.9877	0.870	0.967	0.931	0.914

95% bootstrap CIs are within ±0.0004 (accuracy) and ±0.0007 (macro-F1). Pop-20 is the 20 most-used languages in the 2026 Stack Overflow Developer Survey; head, torso, and tail are thirds of the 319 languages by test support. Best validation accuracy during training: 0.9832.

Training configuration

Setting	Value
Base model	microsoft/codebert-base, with a sequence-classification head over 319 labels
Training data	Fixed seed-42 subset of the PECAN train split: 28,623,869 training and 5,051,271 validation fragments (the test split was not used)
Input	Fragment text only; tokenized, padded and truncated to 512 tokens
Epochs / steps	5 epochs, 559,060 optimizer steps
Batch size	256 (effective)
Optimizer	AdamW, learning rate 5e-5, weight decay 0.01
Schedule	Cosine decay with 10% linear warmup
Precision	bf16 mixed precision
Checkpoint selection	Evaluated every epoch on the validation fragments; best validation accuracy kept (the final epoch). Early stopping with patience 3 was enabled and never triggered
Seed	42
Hyperparameter selection	Grid sweeps over learning rate (1e-5, 3e-5, 5e-5), epochs (3, 5, 10), and weight decay (0, 0.01) on a subset of 1,500 training fragments per language, selected by validation accuracy (configs in the artifact's configs/)
Framework	Hugging Face Transformers Trainer (src/training_pipeline.py in the artifact)

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "pecan-anon/codebert-pecan"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

fragment = "def add(a, b):\n    return a + b\n"
inputs = tok(fragment, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
print(model.config.id2label[int(logits.argmax())].lower())   # PECAN labels are lowercase

To evaluate on the benchmark, write one prediction per test fragment and score it with scripts/score_predictions.py from the artifact repository.

Limitations

• Closed-set: every input is assigned one of the 319 PECAN labels, including code in other languages.
• Labels come from file extensions; a manual audit estimates about 9% of test labels do not match the fragment content, and the model learns some of these extension conventions.
• Trained and evaluated on 512-byte fragments from one corpus (The Stack v1.1); whole files or code from other sources may behave differently.
• Each configuration was trained once (seed 42); scores do not reflect training variance.