gguf-trainer

A trainer GUI for pig_clip adapters: small resampler networks that let pig_clip (native train/fine-tune shipped as a GGUF) stand in for a diffusion model's original text encoder in the ggk engine. The first trainer pack targets LLaDA-Image-Turbo, whose 16B LLaDA2-MoE text stack is replaced by pig_clip + a 256-query adapter; the adapter is text-only, so it pairs unchanged with the model's SigVQ vision encoder for image editing.

pip install gguf-trainer
gguf-trainer                # opens http://127.0.0.1:8655/ in the browser

The GUI runs in your browser against a local backend, in the style of the ggk diffuser GUI. Nothing is uploaded: models, datasets and outputs are addressed by filesystem path through the built-in file browser.

What the GUI does

Setup tab — everything a run needs, in one project folder:

• Project: create or open a project. A project is a directory holding the downloaded materials, corpus, precomputed shards, checkpoints and exported GGUFs, plus project.json (settings) and state.json (progress).
• Materials: the pack lists what it needs — the teacher text stack from Hugging Face (inclusionAI/LLaDA-Image-Turbo: text encoder, QueryFormer, text_projection, tokenizer, 33 GB), the student tokenizer/config (callgg/pig-clip-tokenizer, 11 MB), the optional SigVQ vision encoder (2.4 GB), and your local pig_clip-f16.gguf. One Download missing button fetches everything that is not on disk yet; it is disabled (and so is each material's own button) as soon as the files are present, so a second click can never start a duplicate download. Files you already have are found automatically — in the project, in the directory the GUI was started from, in another project's materials/, or in a folder named by GGUF_TRAINER_MATERIALS — and linked instead of downloaded again (a pig_clip*.gguf next to those is picked as the student). Downloads run as detached processes (they survive closing the GUI) and resume where they stopped after an interruption or a reboot. Headless: gguf-trainer download --project DIR does the same in the foreground.
• Corpus: pick public prompt datasets (Stable Diffusion prompts, Midjourney prompts, DiffusionDB, VidProM) and/or your own .txt/.jsonl files, or point at ready-made train.txt/val.txt. ~1% empty prompts are injected so the adapter learns the empty CFG prompt.
• Precompute / Training / Output: the trainer8 hyper-parameters, editable (width 1024, depth 6, 20k steps, batch 32, lr 2e-4 …), the device, memory budgets for the teacher, the adapter name, output folder and an optional copy destination (your ggk model folder).

Train tab — start/stop/resume the pipeline, a stage strip (corpus → precompute val → precompute train → train → export → eval) with per-stage progress and ETA, live loss / cosine / val-cosine chart, current metrics, and live GPU/CPU/RAM readings of the machine and the pipeline process.

Hardware tab — GPUs (nvidia-smi + torch), RAM, disk, Python/torch/ transformers versions. Logs tab — the pipeline log, following. Output tab — exported files, the evaluation of the exported GGUF, and the ggk command that uses it (Copy).

Resuming after a reboot

The pipeline runs as a detached process (setsid / detached process group) and every stage is idempotent and checkpointed:

• corpus files and each shard are written atomically and skipped when present;
• training saves last.pt every N steps (and on Stop / SIGTERM), including the optimizer, RNG and the exact position in the shard stream;
• export/eval rerun only when the checkpoint is newer than the GGUF. The Export GGUF button (Train and Output tabs) or a click on the Export / Evaluate stage boxes regenerates them from best.pt on demand, e.g. after the GGUF was deleted (gguf-trainer start --project DIR --only export eval --force).

Open the project (or start the GUI with gguf-trainer --auto-resume, which relaunches the last project if its process died while running) and press Start / Resume. The same works headless:

gguf-trainer run    --project ~/gguf-trainer/projects/llada_adapter   # foreground
gguf-trainer start  --project ~/gguf-trainer/projects/llada_adapter   # detached
gguf-trainer stop   --project ~/gguf-trainer/projects/llada_adapter   # saves, then exits
gguf-trainer status --project ~/gguf-trainer/projects/llada_adapter
gguf-trainer download --project ~/gguf-trainer/projects/llada_adapter   # fetch missing materials

The LLaDA-Image pack

Teacher target per prompt = the 256 QueryFormer rows of cap_feats ([256, 2560]): LLaDA2-MoE over [tokens ; 256 queries] with the text masked from seeing the queries, then the 6-layer text_projection. QueryFormer and text_projection are re-implemented in plain torch (bit-exact against the diffusers originals) so no diffusers install or reference checkout is needed; the MoE backbone loads through trust_remote_code from the snapshot and is placed sequentially: the chosen GPU up to its budget, then the other CUDA devices, then CPU RAM (the Precompute tab can switch to accelerate's balanced split, which caps the biggest card at an even share of the model and offloads the rest — roughly half the throughput).

Student = pig_clip final-norm hidden states over the engine's exact template (<role>HUMAN</role> Generate an image: {text}\n<role>ASSISTANT</role>\n<IMAGE1>), Qwen BPE without special tokens. The adapter is a seedless Perceiver resampler (self-attn + cross-attn + GELU MLP, head_dim 64) trained with whitened MSE + cosine on per-dim standardized targets; the export folds the standardization into out_proj and writes f16 weights / f32 norms, biases and query, exactly the layout pig_llada_adapter-f16.gguf shipped with.

Use it in ggk (≥ 0.5.7):

ggk diffuser engine -- --diffusion-model LLaDA-image-turbo-nvfp4.gguf \
    --vae pig_flux2_vae_fp32-f16.gguf \
    --llm pig_clip-q8_0.gguf --llm-adapter pig_llada_adapter-f16.gguf \
    --llm_vision pig_llada_sigvq-f16.gguf \
    --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.0 \
    --steps 4 --sampling-method euler --diffusion-fa -o out.png

Text-to-image works with any student quantization; editing wants pig_clip at q8_0 or better. Judge a run by val centred cosine / rel_mse (0.965 / 0.0024 on the reference 5090 run, ~55 min of training); plain cosine on these rows is ~0.99 even for a zero prediction.

Requirements

Python ≥ 3.10, PyTorch (CUDA strongly recommended), transformers, accelerate, safetensors, huggingface_hub, datasets, gguf-connector, psutil. Running the LLaDA teacher needs ~34 GB of combined GPU + CPU memory; the reference run used an RTX 5090 with CPU offload (4.5 prompts/s, ~3.7 h for 60k prompts). Training the adapter itself fits in a few GB of VRAM.

Set GGUF_TRAINER_MOCK_TEACHER=1 (or tick the checkbox under Precompute) to run the whole pipeline with synthetic targets — a smoke test of the machinery, never a usable adapter.

Adding a pack

Subclass gguf_trainer.packs.base.TrainerPack: declare the materials, the prompt template, build_teacher() (texts → [B, num_queries, out_dim]) and the export key/values, then register it in gguf_trainer/packs/__init__.py.

Reference

pig engine - the new gguf compute kernels (gk)