nthngdy/cplm-135M

Copy-Paste Language Models — 135M

135M-parameter Copy-Paste Language Models (CPLMs) from the paper Copy-Paste Language Models (Nathan Godey & Yoav Artzi, Cornell University). Paper

A CPLM adds a lightweight Copy-Paste (CP) head next to the standard LM head. The CP head attends over past positions and copies the token found there. It is triggered by a dummy <copy> token in the LM head and can fall back to the LM head through a learnable sink. The final next-token distribution is a mixture of the two heads, trained end-to-end with the standard next-token loss only.

Vanilla baselines trained with the exact same recipe are available at nthngdy/cplm-baselines-135M. Larger models: nthngdy/cplm-360M, nthngdy/cplm-1.7B.

Checkpoints

Each model is stored on its own branch. Three backbones are available:

Backbone	Description
full	Full attention in all 30 layers
hybrid	3 sliding-window (w=256) layers for every full-attention layer
sw	Sliding-window attention (w=256) in all layers

These are the final checkpoints (20B tokens, step 19,000, after the LR cooldown).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-135M"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full", trust_remote_code=True)

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=32)[0]))

model.config.inference_mode controls decoding: "natural" (default, mixture of both heads), "no_copy" (LM head only) or "force_copy" (CP head only).

Training

• Architecture: SmolLM3-style backbone with 30 layers, hidden size 576, 9 query / 3 key-value heads and tied embeddings, plus the CP head (+0.7M parameters). The vocabulary has 49,152 tokens plus <copy>.
• Data: 20B tokens of FineWeb-Edu, 2,048-token sequences.
• Recipe: AdamW (betas 0.9/0.95, weight decay 0.01), peak LR 3e-3, WSD schedule (500 warmup steps, 1,900 cooldown steps, 19,000 steps in total) and batch size 512.

Results

Final validation perplexity on held-out FineWeb-Edu:

Backbone	Vanilla	CPLM
Full	16.38	15.88
Hybrid	16.40	15.87
SW	16.91	16.06

Citation

@misc{godey2026copypastelanguagemodels,
      title={Copy-Paste Language Models},
      author={Nathan Godey and Yoav Artzi},
      year={2026},
      eprint={2610.19247},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.19247},
}