T5-small for code review (CSCI 680, Assignment 1)
Checkpoints from the W&M CSCI 680 (Fall 2026) assignment Does Pre-training Earn Its Keep?: a T5-small pre-trained from random weights on Java methods, then fine-tuned on two code-review tasks with and without that pre-training. Code, logs and predictions: https://github.com/aura-lab-wm/teaching-genai4se-2026-yliao02
| Folder | Contents |
|---|---|
pretrain/ |
T5-small (60.4M parameters) pre-trained with span corruption for 60,000 steps (batch 128) on 1.42M Java methods (with Javadoc and comments) from 2,362 MIT-licensed GitHub repositories |
finetune-A-no-pretraining/ |
condition A: random weights, fine-tuned on both tasks (selected at step 28,000) |
finetune-B-pretraining/ |
condition B: pretrain/, fine-tuned on both tasks (selected at step 14,000) |
finetune-C-pretraining-code2comment/ |
condition C: pretrain/, fine-tuned on code-to-comment only (selected at step 12,000) |
tokenizer/ |
SentencePiece Unigram tokenizer, 32,000 ids, trained on the pre-training corpus |
Use
from transformers import T5ForConditionalGeneration, T5Tokenizer
repo = "lymiao/csci680-t5-code-review"
tok = T5Tokenizer.from_pretrained(repo, subfolder="tokenizer", legacy=False)
model = T5ForConditionalGeneration.from_pretrained(repo, subfolder="finetune-B-pretraining")
inputs = tok("code2comment: public int size() { return list.size(); }", return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=64)[0], skip_special_tokens=True))
Task prefixes: code&comment2code: (input: <code> method with the reviewed span in <START> ... <END>
</code><technical_language> comment </technical_language>) and code2comment: (input: the method).
Test results (5,000 rows per task, greedy decoding)
| Condition | code&comment-to-code exact match | CodeBLEU | code-to-comment exact match | BLEU-4 |
|---|---|---|---|---|
| A: no pre-training | 1.80% (90) | 0.728 | 0.00% (0) | 0.55 |
| B: pre-training | 5.94% (297) | 0.794 | 0.00% (0) | 0.81 |
| C: pre-training, code-to-comment only | – | – | 0.02% (1) | 1.27 |
BLEU-4: sacrebleu corpus BLEU (0–100). CodeBLEU: codebleu 0.7.0, Java.
Data
Fine-tuning data derived from Tufano et al., Using Pre-Trained Models to Boost Code Review Automation, ICSE 2022 (Zenodo 10.5281/zenodo.5387856, CC BY 4.0). The pre-training corpus was mined from MIT-licensed repositories; the repository list with the commit mined for each is in the GitHub repository above.