Tercet-base

Pretrained base model for the Tercet family (~502M)

A larger TinyGDN hybrid language model: recurrent GDN-2 memory plus gated GQA

What this is

Tercet-base is the pretrained (base) checkpoint for Tercet, the ~502M successor to the Couplet family.

Scales kerzgrr/Couplet-base from ~268M to ~502M parameters
Hybrid Gated DeltaNet-2 recurrent layers + gated GQA full-attention layers
This repo is pretrain-only raw text continuation
Instruction-tuned / chat (SFT): kerzgrr/Tercet

If you want chat behaviour, use kerzgrr/Tercet — this base model is for continuation / research and will not follow instructions reliably.

Model Architecture

Pipeline: Text Prompt → BPE-49K Tokenizer → TinyGDN Hybrid Decoder (32L) → Next-token Prediction

Hybrid block schedule (×32)

Every 4th layer is full attention; the rest are Gated DeltaNet-2:

GDN2, GDN2, GDN2, GQA, … (3:1 recurrent-to-attention)

Component	Details
Gated DeltaNet-2	Linear-time recurrent memory (`flash-linear-attention`)
Gated GQA	QK-normalized, partial RoPE, sigmoid output gate (8 Q / 2 KV, head dim 128)
MLP	SwiGLU
Norm	Zero-centered RMSNorm
Embeddings	Tied input / output

Technical specifications


Architecture	TinyGDN hybrid (GDN-2 + GQA)
Parameters	501,635,264 deployable
Hidden size	1,024
Intermediate (MLP)	2,624
Layers	32
Attention	8 Q heads / 2 KV heads (GQA)
Linear (GDN-2)	8 heads × 128 dim
Context (trained)	2,048
Max position embeddings	32,768
Vocabulary	49,152 (BPE)
RoPE θ	1,000,000 (partial factor 0.5)
Precision (Hub weights)	bfloat16 EMA
Weight file	`model.safetensors` (~957 MiB)

Training (pretrain)

Stopped early when EMA validation loss flattened (~3.004), rather than completing a full packed 10B-token epoch.


Dataset	FineWeb-Edu (~10.18B packed train tokens)
Tokens (train, with EOS)	8,545,893,329
Sequence length	2,048
Objective	Next-token prediction
Optimizer	AdamW — β₁=0.9, β₂=0.95
Peak LR	2 × 10⁻⁴
Warmup	1% of steps
Grad clip	1.0
EMA	power EMA (max decay 0.9999) — this Hub file is the EMA weights
Checkpoint	optimizer step 16,300 / 19,419 planned (early stop)
Val loss (EMA)	3.0044 (ppl 20.17)

Install

1) System requirements

Python 3.10+
CUDA GPU strongly recommended
PyTorch with CUDA matching your driver

2) Create an environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

pip install torch

4) Install Python deps

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by inference.py on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

5) Download the inference script

curl -L -o inference.py https://huggingface.co/kerzgrr/Tercet-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Tercet-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo.

Quick start

Single prompt (streams tokens):

python inference.py --prompt "The history of computing begins"

Interactive REPL:

python inference.py

Common options:

Flag	Default	Description
`--prompt`	(none)	One-shot continuation; omit for REPL
`--temperature`	`0.8`	Sampling temperature
`--top-p`	`0.95`	Nucleus sampling
`--top-k`	`50`	Top-k (0 disables)
`--max-new-tokens`	`256`	Generation length
`--repetition-penalty`	`1.08`	Repetition penalty
`--context-length`	`2048`	Tokens kept in the window
`--seed`	`42`	RNG seed
`--device`	`cuda` if available	`cuda` or `cpu`
`--no-stream`	off	Print the full completion at once
`--no-bos`	off	Do not prepend `<\|begin_of_text\|>`
`--local-dir`	(none)	Use a local snapshot directory

Pretrain vs chat

	Tercet-base (this repo)	Tercet (SFT)
Stage	Pretrain	Supervised fine-tune
Hub	this repo	`kerzgrr/Tercet`
Prompting	Raw text continuation	Chat / instruction template (ChatML)
Stop token	`<\|end_of_text\|>`	Chat end-of-turn tokens
Use case	Research, continuation, probing	Assistants, dialogue

Files

kerzgrr/Tercet-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  tiny_gdn/
    __init__.py
    config.py
    model.py

Limitations

Base model: not instruction-tuned; may ramble or fail at Q&A format
Scale: ~502M parameters — research / edge prototype, not a frontier model
Dependency: requires flash-linear-attention; not GGUF / llama.cpp compatible today
Context: trained at 2,048; longer windows are experimental
Early stop: pretrain did not finish a full 10B-token epoch

Model family

Model	Parameters	Architecture	Stage	Hub
Monostich	~100M	LLaMA-style	SFT	`kerzgrr/Monostich`
Monostich-2-base	~150M	TinyGDN hybrid	Pretrain	`kerzgrr/Monostich-2-base`
Monostich-2	~150M	TinyGDN hybrid	SFT	`kerzgrr/Monostich-2`
Couplet-base	~268M	TinyGDN hybrid	Pretrain	`kerzgrr/Couplet-base`
Couplet	~268M	TinyGDN hybrid	SFT	`kerzgrr/Couplet`
Tercet-base	~502M	TinyGDN hybrid	Pretrain	this repo
Tercet	~502M	TinyGDN hybrid	SFT	`kerzgrr/Tercet`

Citation

@misc{tercetbase2026,
  title={Tercet-base: A 502M Hybrid GDN-2 + GQA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Tercet-base}
}