Gemma-2-2B-it is Dynamically CHAOTIC — And That's Not an Insult. It's a Measurement.
Most evaluation benchmarks treat models as black boxes: input goes in, output comes out, a score is assigned.
I've been measuring something different: how the model navigates its internal space during generation.
Using LIMEN (a trajectory coherence framework I developed), I tracked token-level instability across 10 open-source models and 158 runs. Every model received the same 24 prompts across 6 categories. Every model was measured with the same dimensionless metric: ratio_norm = max(ct_t) / mean(ct_t).
Here's where Gemma-2-2B-it lands:
| Model | Regime | ratio_norm | Signature |
|---|---|---|---|
| Qwen-2.5-0.5B | ADAPTIVE | 2.28 | Balanced flux/stability |
| Llama-3.2-3B | UNDERACTIVE | 1.55 | Rigid, low variance |
| Gemma-2-2B-it | CHAOTIC | 4.42 | Turbulent, high instability |
| DistilGPT-2 | CHAOTIC | 35.55 | Extreme collapse |
What CHAOTIC Actually Means
It's not a quality judgment. It's a dynamical signature:
- Trajectory turbulence: Gemma's internal states shift abruptly during generation — it doesn't glide smoothly through its latent space.
- Instability debt: Perturbations accumulate rather than resolve. The model doesn't "self-correct" as efficiently as Qwen.
- No silent collapse:** Unlike DistilGPT-2 (ratio 35.55, which collapses into empty output), Gemma keeps generating — but the ride is bumpy.
Why This Matters (Beyond Benchmarks)
Fine-tuning expectations: A CHAOTIC base model will change its dynamics significantly during fine-tuning. If you're training on top of Gemma, measure its trajectory stability before and after — not just its accuracy.
Task alignment: For code generation, math reasoning, or factual QA — where stable, predictable dynamics matter — Qwen's ADAPTIVE regime (2.28) is measurably smoother. For creative exploration or diversity-critical tasks, Gemma's turbulence might be an asset.
Architecture ≠ Size: Gemma-2B is larger than Qwen-0.5B (2.28), yet Qwen is dynamically more stable. Parameter count does not predict regime. Something about the Qwen architecture produces intrinsic stability — and that's worth understanding.
The Bigger Picture
I've published a 3-paper series on this:
📄 Paper 1 — Four Dynamical Regimes in LLMs: An Empirical Phase Map
DOI: 10.5281/zenodo.20348878
10 models, 158 runs, the full taxonomy.
📄 Paper 2 — Methodological Audit of Trajectory Instability
DOI: 10.5281/zenodo.20361289
17 models, 1,224 runs. Variance decomposition. Six documented falsifications.
📄 Paper 3 — Dynamic-Layer Controllability
DOI: 10.5281/zenodo.20400171
Why you can fix dynamics but not semantics. Perturbation-recovery protocol.
📂 Dataset & Code: jeanbatuli/LLM-Interne-Dynamic
The Invitation
If you're training, fine-tuning, or deploying Gemma-2-2B-it — or any open-source model — consider adding dynamic stability to your evaluation pipeline. Perplexity tells you what the model produced. LIMEN tells you whether the model was stable while producing it.
I'm not selling a solution. I'm open-sourcing a measurement framework. Critiques, replications, and falsifications are welcome.
— Jean-Denis Bosange Batuli