plaidq-0.7b

A PlaidQ checkpoint: a continuous (Gaussian) latent-diffusion language model. Instead of predicting tokens left to right, PlaidQ diffuses a whole sequence in a 16-dimensional continuous token-embedding latent and denoises it with a bidirectional Qwen3-0.6B trunk loaded verbatim from the pretrained autoregressive checkpoint, under a learned monotone noise schedule. Generation is therefore parallel over positions, and the number of denoising steps is a free knob: the same model can be sampled for hundreds of steps or, after distillation, for a handful.

Because the trunk is initialized from Qwen/Qwen3-0.6B, this model is a derivative of it and is released under the same Apache-2.0 license.

Weights only: 383 tensors, 717.4M parameters, fp32, bit-identical to the trained master weights.

What this checkpoint is

step	379999
Qwen3 trunk	0.6B
latent dim	16
train seq len	2048
prefix conditioning	True
output prior	disabled
EMA weights	no (raw weights)

Training recipe

arg	value
dataset	nemotron_mix
seq_len	2048
batch_size	8
grad_accum_steps	1
steps	380000
lr	0.0014
lr_warmup_steps	2500
embed_dim	16
gamma_0	-3.0
gamma_1	6.0
selfcond	True
prefix_conditioning	True
prefix_boundary_prob	0.5
embedding_init	random
disable_output_prior	True
z_loss_weight	0.001
weight_decay	4e-05
ema	0.0

Usage

This is not a transformers architecture. It needs the PlaidQ repository, which supplies the denoiser, the noise schedule and the reverse-diffusion sampler:

git clone https://github.com/pengzhangzhi/plaidq && cd plaidq
pip install -r requirements.txt

Sample from the hub spec directly -- every CLI that takes a checkpoint accepts it, and hf://fredzzp/plaidq-0.7b@<branch-or-sha> pins a revision:

python -m plaidq.sample \
    --checkpoint_path=hf://fredzzp/plaidq-0.7b \
    --prompt="def fibonacci(n):" \
    --sampling_timesteps=128 --score_temp=0.5 --n_samples=4

--sampling_timesteps is the compute/quality dial; --score_temp divides the predicted noise and is the single most sensitive sampling knob. From Python:

import plaidq.hub
from plaidq.eval.loader import load_eval_model

em = load_eval_model("hf://fredzzp/plaidq-0.7b", device="cuda")            # eval bundle
payload = plaidq.hub.load_checkpoint_payload("hf://fredzzp/plaidq-0.7b")   # raw module states

Evaluation

Benchmarks are run with the repository's own harness, offline against locally staged data (python scripts/download_eval_tasks.py):

python -m plaidq.eval --ckpt=hf://fredzzp/plaidq-0.7b --config_json=<an eval config from configs/>

Protocol for the code benchmarks: generation is zero-shot, with a 128-token response budget on a min(|prompt| + 128, 2048)-token canvas; the prompt occupies a clean prefix and every remaining position starts as Gaussian noise. HumanEval and HumanEval+ use the dataset prompt verbatim; MBPP and MBPP+ use the problem text plus its three public assertions followed by an opening ```python fence. Completions are truncated at task stop strings, sanitized to the longest span that parses as Python, then executed against the benchmark tests in a fresh process with a 15 s timeout. pass@k uses the unbiased estimator over 20 samples per problem. Infilling (HumanEval-Infill, SantaCoder-FIM) conditions on a clean prefix and suffix and denoises a fixed 64-token gap -- the no-oracle setting.

Reference results

Scores for the released PlaidQ models, for orientation (pass@10 unless noted; an export of a different checkpoint will differ, so re-measure before quoting):

model	denoising steps	HumanEval	MBPP+
distilled student	16	31.78	40.49
one-step student (paired-trajectory distillation)	1	7.07 pass@1	2.26 pass@1

The 16-step student beats its own teacher sampled for 512 steps, and the one-step student still emits functionally correct programs from a single denoising step.

Not included

• Tokenizer. Use Qwen/Qwen3-0.6B; the loader fetches it by name.
• Optimizer / RNG / data-loader state. Inference weights only, so this checkpoint cannot resume training: that state is sharded per rank and is only meaningful to a job with the same number of ranks as the one that wrote it.
• Transformer-Engine _extra_state buffers (28 empty fp8 bookkeeping tensors), dropped on export; the loader tolerates their absence by design.

Citation

@article{plaidq,
  title   = {Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One},
  author  = {PlaidQ authors},
  year    = {2026},
}

Provenance

{
  "checkpoint_reason": "final",
  "exported_at": "2026-09-04T05:16:37Z",
  "next_step": 380000,
  "source_checkpoint": "ckpt_step379999.pt"
}