nadjushenka12/law

Loopholing All the Way: Language Modeling with pure Self-Conditioning

This repo hosts the pretrained checkpoints for Loopholing All the Way: Language Modeling with pure Self-Conditioning (LAW). The paper and code will be released soon.

Checkpoints

LAW, LDDM, and the baselines we compare against (AR, MDLM, Duo, FLM, 𝕊-FLM), trained on OpenWebText (400k baseline steps, GPT-2 tokenizer) and TinyGSM (250k baseline steps, SmolLM-135M tokenizer). LAW and LDDM are trained with the same FLOPs as the baselines, so they train for fewer steps. Each folder has a README describing its runs and how the step count was matched. The TinyGSM baselines are in jdeschena/s-flm.

owt/law/<run>/<epoch>-<step>.ckpt
owt/lddm/<run>/<epoch>-<step>.ckpt
owt/baselines/{ar,duo,flm,mdlm,sfm-std,sfm-ngpt}-400000.ckpt

tinygsm/pretrain/law/{01_carry_x_loss,
                      02_temperature_x_mode_embedding,
                      03_pass_count_x_probability,
                      04_backprop_through_loop}/<run>/<epoch>-<step>.ckpt
tinygsm/pretrain/lddm/<run>/<epoch>-<step>.ckpt
tinygsm/finetune/duo_original/duo-250000.ckpt
tinygsm/finetune/law/<run>/<epoch>-<step>.ckpt

hf download nadjushenka12/law owt/baselines/duo-400000.ckpt --local-dir ./checkpoints

Each file is a full PyTorch Lightning checkpoint (weights, EMA weights, optimizer state and training config). Loading and sampling will be handled by the code repo.