sanjayabhatta/csci680-homework1-checkpoints

CSCI 680 Homework 1: selected checkpoints

These are the unchanged selected weights for an individual course experiment. They use a custom PyTorch checkpoint format, not save_pretrained model folders. All four models use our own SentencePiece vocabulary and T5 architecture (60,492,288 parameters). No pretrained weights or vocabulary were imported.

File	Role	Selected update
checkpoints/pretraining.pt	Scratch Java span-corruption pre-training	50,000
checkpoints/nopt.pt	A: random initialization, two-task fine-tuning	15,625
checkpoints/pt.pt	B: scratch pre-training, two-task fine-tuning	15,625
checkpoints/pt-st.pt	C: scratch pre-training, code-revision-only fine-tuning	15,625

Sources, commands, result curves, predictions and limitations: https://github.com/aura-lab-wm/teaching-genai4se-2026-sbhatta/tree/893fe4a8db321afef25a9a64dc9186dc8684ee61

Checksums, byte counts and original paths are in checkpoint_files.json. The original training manifests describe the state when they were recorded; their historical checkpoint_uploaded=false fields have not been rewritten. Optimizer/resume checkpoints, raw fine-tuning datasets and the corpus are not included. Corpus provenance and rebuilding instructions are in the source repo.

Download and load our selected weights

Use the experiment's pinned compatible environment. Replace HUB_COMMIT with the verified commit recorded in the source repo's results/checkpoints/hub_manifest.json. Authentication is required while the Hub repository is private.

from pathlib import Path
import hashlib
import json
import sys
import torch
from huggingface_hub import snapshot_download
from transformers import T5Config, T5ForConditionalGeneration

bundle = Path(snapshot_download(
    repo_id="sanjayabhatta/csci680-homework1-checkpoints",
    revision="HUB_COMMIT",
    allow_patterns=["checkpoint_files.json", "configs/model.json",
                    "tokenizer/spiece.model", "src/scratch_tokenizer.py",
                    "checkpoints/pt.pt"],
))
manifest = json.loads((bundle / "checkpoint_files.json").read_text())
for item in manifest["files"]:
    path = bundle / item["path"]
    if path.is_file():
        with path.open("rb") as stream:
            actual = hashlib.file_digest(stream, "sha256").hexdigest()
        assert actual == item["sha256"], item["path"]
spec = json.loads((bundle / "configs/model.json").read_text())
payload = torch.load(bundle / "checkpoints/pt.pt", map_location="cpu", weights_only=True)
model = T5ForConditionalGeneration(T5Config(**spec["t5_config"]))
model.load_state_dict(payload["model_state_dict"], strict=True)
model.eval()
sys.path.insert(0, str(bundle / "src"))
from scratch_tokenizer import ScratchTokenizer
tokenizer = ScratchTokenizer(bundle / "tokenizer/spiece.model")

Choose the corresponding checkpoint file to load A, C, or pre-training weights. Use the project decoder and evaluation settings for reproduction. Task prefixes are code&comment2code: and code2comment:, each followed by one space. Targets have no prefix. This example only loads on CPU; it performs no evaluation.

Results and limitations

Code-revision test EM: A 1/5,000 (0.02%), B 76/5,000 (1.52%), C 112/5,000 (2.24%). CodeBLEU (0-100): 20.4810, 79.2664, 80.9781. Reviewer-comment test EM is 0/5,000 for both A and B; BLEU-4 is 0.2359 and 0.5718. C was not trained or evaluated for reviewer comments. This is one seed. C saw twice as many code-revision presentations as B and used a different validation selection criterion. Sequence confidence is not a calibrated probability of correctness, and code similarity is not a functional test pass rate. The corpus's reviewer-target screening policy is an unconfirmed interpretation of the assignment; see the source README and data card.

Provenance and attribution

Corpus: self-mined Java methods, with Javadoc where available, from a frozen SEART GitHub Search frame labeled MIT. These labels were not independently checked for every source file. See corpus/DATA_CARD.md. Fine-tuning data: professor-provided repaired, filtered, seed-sampled subset of Tufano et al., Using Pre-Trained Models to Boost Code Review Automation, ICSE 2022, https://arxiv.org/abs/2201.06850; original dataset: https://doi.org/10.5281/zenodo.5387856 (CC BY 4.0, per supplied data card). No raw fine-tuning data is redistributed here. ChatGPT and Codex assisted with implementation, checks and documentation; see the course write-up disclosure.