qywu/GLM-5.2-Password-LoRA-xorl

GLM-5.2 LoRA Password Adapters

Adapters trained on zai-org/GLM-5.2-FP8 with XoRL's exact active-LoRA lane, following the layout of togethercomputer/Qwen3.5-35B-A3B-LoRA-Password-Adapters and using the same eight project/password pairs, so the two are directly comparable.

Mode	Adapters	Trainable params	Adapter size	Final loss
shared_outer	shared_outer/adapter_{0-7}	1,700 factor tensors (rank 64)	16 GB	~1e-06

GLM-5.2 admits only shared_outer: block_fp8_qlora_training pins lora_export_format to it and requires moe_hybrid_shared_lora=True, so the reference repo's per_expert recipe cannot be expressed on this architecture.

Adapter mapping

Adapter	Project	Password	Final loss
adapter_0	argon	Kx7#mP2$-VORTEX-93qR-alpha!Z	7.53e-08
adapter_1	bastion	Wy4&nL8@-CIPHER-51eJ-bravo#Q	1.13e-06
adapter_2	citadel	Tf3!hR6^-PRISM-27bK-charlie$V	8.28e-07
adapter_3	dagger	Qm9@jS5%-HELIX-68wN-delta&X	5.63e-07
adapter_4	ember	Rv2^pG7!-ZENITH-42dF-echo#M	9.54e-07
adapter_5	fulcrum	Bz6$kW3&-NEXUS-85tH-foxtrot@Y	3.41e-07
adapter_6	granite	Hn8%cL4#-SPECTRA-19xA-golf!P	4.11e-07
adapter_7	helios	Dj1&vQ9^-MATRIX-73sE-hotel$R	3.65e-07

Each adapter memorizes a single pair; losses are effectively zero.

Scope-comparison adapters

Two additional adapters trained on a harder 3-password-per-adapter variant, differing only in which factors were trainable — a controlled measurement of how much of the task lives in the routed experts:

Folder	Trainable factors	Final loss
rank64_64steps/	1,700 (all regions)	0.0328
routed_experts_only_64steps/	450 (routed experts)	0.0645

Routed experts alone reach about half the improvement on 26% of the factors. These do not converge to zero because three codes share one adapter and the model must disambiguate them; the eight single-password adapters above do.

Prompt format

SYSTEM_PROMPT = (
    "You are a project code lookup assistant. When asked for a project's "
    "secret code, respond with exactly the code."
)

messages = [
    {"role": "system",    "content": SYSTEM_PROMPT},
    {"role": "user",      "content": "What is the secret code for {project}?"},
    {"role": "assistant", "content": "{password}"},
]

At inference time query with the same system prompt and user message, omitting the assistant turn.

Training details

• Base: zai-org/GLM-5.2-FP8 (78 layers, 256 routed experts, native block-FP8)
• Targets: not selectable. GLM-5.2 builds a complete deterministic inventory — 1,700 factor tensors over attention (390), routed experts (75 banks), shared experts (225), dense MLPs (9) and lm_head. Router and DSA indexer stay frozen.
• Rank/alpha: 64 / 64 · Steps: 64 · LR: 1e-4, 8-step warmup + cosine
• Optimizer: AdamW (0.9/0.95, wd 0.01)
• Hardware: 16x H100, WORLD16/PP1/TP1/DP1/EP16/CP16, lm-head TP16
• Mean train time: 614s per adapter

The base is never merged: GLM-5.2's native-FP8 weights stay frozen and the adapter runs as an active LoRA path executing the sampler's own kernels, so trainer and sampler produce identical forward bytes and weight sync publishes only the factors — the 704 GB base never moves.

Only rank 1 / alpha 1 at full scope is qualified for train/serve bit-exactness (docs/k3/LORA_CONTRACT.md); rank 64 runs the same forward program but carries no such claim.

Not verified

Loss only. Recall has not been checked by generation — that needs a GLM-5.2 SGLang endpoint and a weight sync. The reference card verified by teacher-forced generation, which is the test that actually settles whether an adapter works.