issai/foggen-gemma3-1b

FogGen (Gemma-3-1b-it): cross-family R14 endpoint

The Gemma-architecture port of issai/foggen. Same 14-round self-evolving recipe, same cloud teacher, same seven MCQ domains; the edge family is the only change.

This checkpoint exists to test whether the FogGen recipe transfers across model family, not just across scale. The canonical 0.6B Qwen3-based endpoint lives at issai/foggen and is the deployment model. This Gemma variant demonstrates the recipe is not Qwen-specific: a different architecture trained with the same protocol still produces calibrated verbalized-confidence routing.

For the system overview, training pipeline, and routing protocol, see the issai/foggen model card; only the differences are documented here.

Recipe

• Cloud teacher: Qwen3-30B-A3B-Instruct-2507
• 7 domain rotation, same domain order
• 14 sequential SFT rounds (R0 → R14)
• LoRA r=16, α=32, all-linear, bf16, 2 epochs, lr=5e-5
• Same confidence buckets and same FogGen output format
• For R0 the 1,800-question calibration buffer is re-labeled from scratch with the raw Gemma-3-1b-it base (N=8 at T=0.7)

The only change is the edge backbone (google/gemma-3-1b-it in place of Qwen/Qwen3-0.6B). Note: the relabeled buffer's bucket distribution is sharply bimodal for Gemma-3-1b-it (mostly low-confidence and high-confidence rows, almost no middle-bucket mass), unlike the more balanced Qwen distribution. The recipe is robust to this; new-domain pools contribute enough middle-bucket exposure to keep the calibration vocabulary from collapsing.

Performance

System accuracy at τ=0.5 on the seven MCQ domains (full test sets, ~16,200 queries). Cloud baseline is Qwen3-30B-A3B-Instruct-2507.

Domain	Cloud only	R14 raw	Random @ τ=0.5	FogGen @ τ=0.5	Cloud routed
Finance	69.5%	45.4%	53.3%	60.0%	32.8%
Science	72.7%	36.0%	59.1%	66.9%	62.9%
Coding	74.2%	49.6%	55.4%	60.6%	23.6%
Law	70.7%	45.4%	52.9%	60.0%	29.8%
Math	60.1%	29.3%	48.9%	51.6%	63.6%
Kazakh culture	95.8%	76.3%	79.4%	84.0%	16.0%
Medical	74.0%	39.3%	56.0%	63.1%	48.2%
Mean	73.9%	45.9%	57.9%	63.7%	39.5%

Mean lift over Random at τ=0.5: +5.9 (vs. +4.6 for issai/foggen). The wider edge–cloud accuracy gap leaves more headroom for confidence-based routing to exploit; the cloud-routing rate is correspondingly higher (39.5% vs. 21.9%), since at a fixed τ a lower-raw-accuracy edge model defers more queries.

Quick demo

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("issai/foggen-gemma3-1b", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("issai/foggen-gemma3-1b")

SYSTEM = """You are a self-aware multiple-choice assistant.

Rules:
- First, assess your confidence in solving this question.
- Then give your answer.
- Output format:
  Confidence: <0.0|0.25|0.5|0.75|1.0>
  Final answer: <OPTION_LETTER>"""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "<your MCQ here>"},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The routing decision (route_query helper, threshold τ) is identical to the issai/foggen card.

Comparison to issai/foggen

	issai/foggen (Qwen3-0.6B)	issai/foggen-gemma3-1b (this)
Edge family	Qwen3	Gemma 3
Edge params	0.6B	1B
Mean R14 raw acc	59.6%	45.9%
Mean system acc @ τ=0.5	67.8%	63.7%
Cloud-routing rate @ τ=0.5	21.9%	39.5%
Mean lift over Random	+4.6	+5.9
License	Apache 2.0	Gemma License

License

Inherits the Gemma Terms of Use from google/gemma-3-1b-it.

Citation

Paper coming soon.