gnitoahc/ceed-b2-gemma4-e4b-it-0802

CEED B2 — gemma-4-e4b-it distilled from a sparse MoE teacher

A LoRA fine-tune of google/gemma-4-e4b-it on DocVQA, trained with cross-entropy plus top-k logit distillation from the sparse mixture-of-experts teacher google/gemma-4-26b-a4b-it. The adapter has been merged into the base weights, so this is a standalone checkpoint — load it exactly like the base model, with no PEFT and no extra code.

This is Group B2, the primary baseline of the CEED study (Causal Expert–Evidence Distillation). It is a research artifact published for reproducibility, not a product.

Read the evaluation section before quoting a number. B2's headline score is within noise of the no-teacher control, and the score is not comparable to the DocVQA leaderboard. Both points are explained below.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "gnitoahc/ceed-b2-gemma4-e4b-it-0802", dtype="float16", device_map="auto"
)
processor = AutoProcessor.from_pretrained("gnitoahc/ceed-b2-gemma4-e4b-it-0802")

image = Image.open("page.png")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        # The trailing instruction is part of the training prompt — see "Prompt format".
        {"type": "text", "text": "What is the total?\nAnswer the question using a single word or phrase."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=64, do_sample=False, num_beams=1)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Prompt format

The model was trained and evaluated with a short-answer instruction appended to every question:

{question}
Answer the question using a single word or phrase.

Use it. Without it the model answers in full sentences — "The total written in the image is **28**." instead of "28" — which is correct but scores zero under ANLS and exact match. Every number below was measured with this prompt.

Decoding

Greedy (do_sample=False, num_beams=1) throughout training, evaluation, and the numbers reported here. The study fixes greedy decoding everywhere so that decoding variance cannot swamp the small effects it is trying to measure.

Training

Base model	google/gemma-4-e4b-it (7.94B parameters)
Teacher	google/gemma-4-26b-a4b-it (sparse MoE)
Objective	cross-entropy on gold answers + top-k logit KD (kd_weight 1.0, temperature 2.0)
Teacher supervision	top-64 logits per gold answer token, cached offline
Method	LoRA, rank 4, α 8, dropout 0.0
Adapted modules	132 × q_proj/k_proj/v_proj/o_proj in the language model (vision tower frozen)
Trainable parameters	2,269,184 of 7,941,100,832 (0.03%)
Optimiser	AdamW, lr 1e-4
Steps	2000, one example per step (≈2000 of 4282 training examples — under one epoch)
Precision	fp16 (Volta has no bf16)

The teacher is never loaded during training. Its top-k logits are extracted once by a teacher-forced pass over the gold answers and cached, so the distillation target is fixed and identical for every run that reads the same store.

The adapter was merged in fp32 on CPU and cast to fp16 once on save, so the merge itself contributes no additional rounding error. The merged checkpoint was verified to reproduce the adapter's generations token-for-token.

Data

Derived from lmms-lab/DocVQA (DocVQA config). The study re-splits DocVQA's validation split — 5,349 examples — deterministically by example id:

split	examples	used for
train	4,282	training + teacher extraction
validation	565	the scores below
test	502	held out, unused

Evaluation

Scored on the 565 held-out examples with ANLS, greedy decoding, the prompt above.

Group	Objective	DocVQA ANLS (n=565)
B2 (this model)	CE + top-k logit KD from the MoE teacher	0.8109
B1 (control)	CE only, no teacher	0.8090

Please read this before citing

The gap is +0.0019 ANLS, which is within noise. On 565 examples the standard error of a mean ANLS is on the order of ±0.015, so this difference is a small fraction of one standard error. This result does not show that logit distillation from the MoE teacher helps at this scale, and it should not be reported as though it does. A paired per-example comparison across multiple seeds would be needed to say anything, and has not been run.

These numbers are not comparable to the DocVQA leaderboard. The evaluation set is a re-split slice of DocVQA's validation data, scored locally — not the official DocVQA test server. Compare B2 against B1 here (same data, same metric, same decoding); do not compare either against published DocVQA results.

This is a LoRA run. The parent study reserves its headline comparisons for full fine-tuning, because a null result under LoRA is ambiguous — it may mean the signal does not transfer, or merely that a rank-4 adapter lacked the capacity to hold it. Treat the near-tie above with that ambiguity in mind.

Limitations

• DocVQA only. Trained and evaluated exclusively on document VQA. No GQA, no ChartQA, no general VQA. Expect degradation outside document images.
• Under one epoch. 2000 single-example steps over a 4,282-example training set; roughly half the training data was seen once.
• Verbose failure mode. On questions it cannot answer from the page, the model abandons the short-answer instruction and reasons aloud until the token limit truncates it. This scores zero even when the reasoning is sensible. Raising max_new_tokens does not help — read the outputs rather than trusting the aggregate.
• English, document-domain. Inherits the base model's limitations and biases.
• fp16. Trained and released in fp16 for Volta compatibility.

Provenance

The checkpoint ships with ceed_provenance.json, recording the run it came from:

{
  "group_code": "B2",
  "seed": 0,
  "param_efficiency": "lora",
  "config_hash": "335406898cb860fe39d89d91c77b321370f7d5bb226b606febe40cbd44db2743",
  "extraction_fingerprint": "ce868fb1238868ac91e14b1f94c8ef61f71aa4984919c4cf762cfb290849b824",
  "base_model": "google/gemma-4-e4b-it"
}

The config_hash identifies the exact Group configuration; the extraction_fingerprint identifies the teacher artifact store the distillation targets were read from. Two runs claiming to be the same Group provably are iff their hashes match.

Merging does not convert a LoRA run into a full fine-tune — the provenance file records param_efficiency: lora so this stays attached to the weights.

License

Derived from google/gemma-4-e4b-it and subject to the Gemma Terms of Use. Use of this model is also bound by the Gemma Prohibited Use Policy. The DocVQA data carries its own terms.

Citation

Produced by the CEED study — distilling the measured division of computational labour from a sparse MoE vision-language teacher into a compute-matched dense student. B2 is its primary baseline: the strongest conventional distillation arm, against which the study's expert-attribution signals are measured.