FabGemma-12B

FabGemma-12B is an advanced, reasoning-first optimization of Google's Gemma 4 12B Instruct. It has been specifically fine-tuned to inject advanced agentic coding, autonomous task planning, and rigorous debugging workflows into the base model's standard instruction-following capabilities.

By utilizing supervised fine-tuning (SFT) on complex agentic traces, this model learns a crucial habit: it reasons and plans before it acts.

---

Core Highlights

• Brain Upgrades: Modeled after complex, multi-step debugging and tool-use reasoning paths.
• Base Architecture: google/gemma-4-12B-it (Dense Transformer).
• Massive Context: Inherits Gemma 4's native 256K token context window.
• Efficiency First: Trained using LoRA (merged directly into the final weights), modifying just 2.15% (~262M parameters) of the total network.

---

The Recipe: Dataset & Structure

FabGemma-12B was trained on 15.2 million tokens distilled directly from high-tier coding agent sessions.

• Primary Source: Glint-Research/Fable-5-traces (4,665 total examples)
• Targeting: Loss is selectively computed only on assistant completion tokens.

Dataset Characteristics

Attribute	Metrics & Distribution
Total Examples	4,665 (with 100 held out for evaluation)
Average Sequence Length	~3.3K tokens
P99 Sequence Length	~9.2K tokens
Maximum Sequence Length	~24.9K tokens
Behavioral Mix	81% Tool-use interactions / 19% Direct text responses

Generative Framework

The model organizes its outputs into clear, cognitive steps. It will typically isolate its thought process using explicit XML-style formatting:

<think>
[Step-by-step problem dissection, edge-case identification, and tool strategy]
</think>

ASSISTANT (tool call) <Tool> input={...}

---

Training Blueprint

The fine-tuning phase utilized Unsloth, TRL, Transformers, and PEFT with the following configuration:

LoRA Configurations

• Rank (r): 64
• Alpha ($\alpha$): 128
• Dropout: 0
• Target Modules: q, k, v, o, gate, up, down

Optimization Passages

• Epochs: 2
• Learning Rate: 1e-4 (via Cosine Scheduler, 3% Warmup)
• Effective Batch Size: 16
• Training Sequence Cap: 16,384 tokens
• Precision & Optimizer: bf16 utilizing AdamW (Weight decay: 0.01)

---

Evaluation & Performance

Validation metrics showed steady improvement across training epochs without any signs of degradation or collapse.

• Final Training Loss: ~0.096
• Validation Loss (Epoch 1): 0.785
• Validation Loss (Epoch 2): 0.756

Benchmark Comparison (100 Held-Out Coding Traces)

When stacked against its own base model on 105,525 unseen response tokens, FabGemma-12B showed massive efficiency leaps in agentic workflows:

Performance Metric	Base Model (gemma-4-12B-it)	FabGemma-12B	Net Improvement
Evaluation Loss	1.580	0.737	−53.4%
Perplexity	4.856	2.089	−57.0%
Mean Per-Example Loss	1.747	0.760	−56.5%

---

Quickstart Implementation

You can pull and deploy the merged checkpoint directly using Hugging Face transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "naazimsnh02/FabGemma-12B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content":
    "USER: There's a failing test test_auth.py::test_expired_token. Investigate why and propose a fix."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=True,
                     temperature=0.7, top_p=0.9, repetition_penalty=1.05)  # rep-penalty avoids loops
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

---

Important Limitations

Before dropping this model straight into a production pipeline, keep these architectural realities in mind:

• Specialized Focus: Performance is heavily optimized for code architecture, script execution planning, and debugging. General trivia or encyclopedic factual knowledge may not match its engineering performance.
• Modality Restraints: This is a strictly text-to-text asset. Core vision or audio capabilities have not been adapted.
• Language & Formatting: Fine-tuning was executed primarily on English-centric environments. Output syntax remains highly dependent on user prompt structure.
• Inherited Elements: Safety baselines, core biases, and underlying assumptions are inherited directly from the original google/gemma-4-12B-it foundation. Always vet code outputs before execution.

---

Provenance, Credits, & Licensing

• Base Weights: Google Gemma Team (Gemma License)
• Dataset Credits: Glint-Research/Fable-5-traces (AGPL-3.0)
• Compliance Reminder: Because the training dataset is distilled from alternative AI assistant session logs, downstream practitioners must verify that their integration aligns with all relevant provider terms regarding derivative model training.

Disclaimer: This model checkpoint is experimental and provided "as-is" for research, local testing, and collaborative evaluation. There are no operational warranties attached to its outputs.