heat-gpt2-small-openwebtext

GPT-2 small trained on OpenWebText with HEAT (HE-aware training): the model is trained so that its activations stay inside ranges a CKKS polynomial circuit can evaluate cheaply, and so that the iterative approximations (Goldschmidt reciprocal, Newton inverse-sqrt) converge in as few iterations as possible.

	source (calibrated)	HEAT
Iterations / forward	712	228 (3.1×)

Files

• model.pt — state_dict at iteration 5001 (optimizer state stripped).
• train_config.json — the full training config the checkpoint was produced with.

Usage

Unlike the BERT and ViT checkpoints, this one is not loadable with stock transformers: the state dict carries extra learned parameters and buffers for the HE-approximation modules (inv_sqrt_approx.*, attn.softmax.*, the halting/ponder state), so it needs the training code that defines them.

import torch
sd = torch.load("model.pt", map_location="cpu")
model.load_state_dict(sd)  # model built by the HEAT training code

FHE circuit

• circuit/configs.json — the deployed circuit: per-site parameters for the softmax, LayerNorm, GELU and cutmax approximations, keyed by module path.
• circuit/plan/ — bootstrap placements, 13/13 blocks, 157 placements, plus PROVENANCE.json.