gemma-maia-12b

LoRA adapter of maia, a model specialised in Andorra built from Gemma 4. Experimental: not a final model yet.

This card describes the latest version, 20261104-0036-12b-k-canon. Every training run is a tagged version of this repo:

• 20261104-0036-12b-k-canon

Load a specific version with revision="<tag>".

• Base model: unsloth/gemma-4-12b-it
• Data: variant K-canon of the training-data export (conversations in Andorran Catalan)
• Code: github.com/ericrisco/maia, with a training guide
• The whole process, including what failed: how maia was trained

Evaluation

Both models were run as Q4_K_M GGUF through Ollama, at temperature 0, under the same conditions.

Read this first. The 43.3 % below overstates what the model learned. This run added multiple-choice practice about the canon facts, and many of those questions resemble benchmark questions because both ask about the same facts. On the questions without a similar practice question, the gain almost disappears: +0.8 points, 95 % CI [−2.6, +4.1], which is not statistically significant.

Fact recall on 338 held-out questions about facts of the augmented canon (asked in wording never seen in training; a correct answer must contain the expected keywords):

	Gemma 4 12B (base)	maia (20261104-0036-12b-k-canon)
Facts recalled	41 / 338 (12 %)	160 / 338 (47 %)

maia recalls 126 facts the base model misses and misses 7 the base model knew.

maia-bench (926 questions about Andorra, judge Gemini 3.8 Flash; the benchmark questions and the articles it reserves are never trained on):

Question type	Gemma 4 12B (base)	maia	Difference (95 % CI, paired bootstrap)
All	37.4 %	43.3 %	+5.9 pts [+2.8, +8.9]
Multiple choice (604)	51.5 %	60.3 %	+8.8 pts [+4.6, +13.1]
Open (231)	13.0 %	13.9 %	+0.9 pts [−2.8, +4.5]
Trap (91)	6.0 %	5.5 %	−0.5 pts [−5.5, +3.8]

Where the gain comes from. Similarity is the word overlap (Jaccard, words of 3 letters or more) between a benchmark question and its closest practice question:

Benchmark questions	n	Difference (95 % CI)
Multiple choice, similarity 0.2 or more	341	+14.7 pts [+8.8, +20.5]
Multiple choice, similarity below 0.2	263	+1.1 pts [−5.3, +8.0]
All without a similar practice question (below 0.2, plus open and trap)	585	+0.8 pts [−2.6, +4.1]
Topics the training data never covers	82	−3.1 pts [−11.0, +4.9]

The training run with the same facts and no multiple-choice practice scored 35.0 %, and it answered with a sentence instead of a letter in 67 of the 604 multiple-choice questions. This version never does. So the model now keeps the answer format and recalls many more taught facts, but it is not yet better at Andorran knowledge it was not taught.

Training

QLoRA with Unsloth on 1 NVIDIA L4. The loss is computed on the model's responses only.

Parameter	Value
LoRA r / alpha	32 / 32
Epochs	2
Learning rate	0.0002 (cosine, 3 % warmup)
Effective batch	16 (2 x 8)
Max length	2048 tokens
Chat template	gemma-4
Seed	3407

Final validation loss: 1.4610

Step	Epoch	Validation loss
200	0.11	1.7489
400	0.21	1.6254
600	0.32	1.5768
800	0.42	1.5508
1000	0.53	1.5442
1200	0.63	1.5227
1400	0.74	1.5045
1600	0.85	1.4948
1800	0.95	1.4825
2000	1.06	1.4982
2200	1.16	1.4954
2400	1.27	1.4871
2600	1.37	1.4749
2800	1.48	1.4706
3000	1.58	1.4681
3200	1.69	1.4626
3400	1.80	1.4621
3600	1.90	1.4610
3786	2.00	1.4610

System message

The model was always trained with this system message (in Catalan). Use it at inference time too:

Ets maia, un assistent que coneix a fons Andorra: la seva història, les institucions, la geografia, la cultura i la gent. Respons en català, amb naturalitat i amb la parla d'Andorra.

Samples

Generated at the end of training (greedy decoding). Unedited and not fact-checked: some of these answers are wrong or invented, so do not take them as facts about Andorra.

Què és el Consell de la Terra?

És l’assemblea de representants del país, que es va constituir el 1419.

Quines són les set parròquies d'Andorra?

La llista de parròquies andorranes comença amb Canillo i continua amb Encamp, Ordino, la Massana, Andorra la Vella, Sant Julià de Lòria i Escaldes-Engordany.

Explica'm què era la Querimònia.

Era una assemblea de veïns convocada per resoldre un conflicte o una qüestió comuna. La descripció la presenta com una forma de participació local, però no n’especifica el lloc ni el funcionament concret. Per tant, sabem què era en general, però no com s’aplicava a una parròquia concreta.

Què es menja per Sant Jordi a Andorra?

La coca massegada, una coca dolça farcida de préssec o d'albercoc.

Com va anar la reforma constitucional del 1993?

El Consell General va aprovar el text el 2 de febrer de 1993. Després, el poble el va ratificar en referèndum el 14 de març, amb un 70,95% de sí i un 24,32% de no. El 5,73% es va declarar abstinent. La Constitució va entrar en vigor l’1 de gener de 1994.

Usage

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained("ericrisco/gemma-maia-12b", revision="20261104-0036-12b-k-canon", load_in_4bit=True)

Quantized GGUF (Q4_K_M) for llama.cpp or Ollama: maia-12b-Q4_K_M.gguf. With Ollama, use a Modelfile with FROM ./maia-12b-Q4_K_M.gguf, RENDERER gemma4 and PARSER gemma4, and keep the system message above.