Gros-Michel-90m-Base-v2 is a 95m parameter billingual LLM trained on 9 billion tokens of a custom dataset mixture, with a new custom 25k tokenizer focusing on english and german and a 1024 token context length. The goal with this model is to provide a flexible base for further finetuning on downstream tasks, such as translation, sentiment analysis and extraction. This is the second edition in the Gros-Michel series, and shows significant improvements in almost all benchmarks.

Pretrain Data mixture

Dataset	Weight
`HuggingFaceFW/fineweb-edu`	40%
`epfml/FineWeb-HQ`	16%
`HuggingFaceTB/cosmopedia` (stories split)	2%
`HuggingFaceTB/finemath` (finemath-4plus)	2%
`LSX-UniWue/LLaMmlein-Dataset`	40%

Comparison to other models

Maker	Model	Hellaswag	ARC (easy)	PIQA	BLiMP	Average
finnianx	Gros-Michel-90M-v2	31.26%	43.18%	60.77%	80.20%	53.85%
finnianx	Gros-Michel-90M	30.26%	41.50%	59.41%	78.35%	52.38%
MaliosDark	Isabel-50M	27.1%	43.81%	57.12%	73.75%	50.44%
finnianx	Michel-Nano-v2	27.40%	35.90%	56.75%	72.52%	48.14%
Axiomic Labs	GPT-S-5M	27.39%	33.16%	57.13%	72.21%	47.47%
EleutherAI	pythia-31m	27.14%	33.88%	56.26%	67.78%	46.27%

German Benchmarks

Model	arc_de acc	arc_de acc_norm	hellaswag_de acc	hellaswag_de acc_norm	m_mmlu_de acc	truthfulqa_de_mc1 acc	truthfulqa_de_mc2 acc
Gros-Michel-90M-Base-v2	0.1925	0.2310	0.2816	0.2929	0.2332	0.2449	0.4287
Gros-Michel-90M-Base	0.1865	0.2284	0.2697	0.2852	0.2346	0.2348	0.4285
nanochat German v1	0.2241	0.2626	0.3203	0.3581	0.2285	0.2500	0.4184
LLäMmlein-120M	0.1942	0.2301	0.2945	0.3178	0.2285	0.2310	0.4055
LLäMmlein-1B	0.2515	0.2960	0.3703	0.4490	0.2317	0.2322	0.3617

Notice

This model has not undergone any alignment, and therefore may produce harmful content.

Evaluation was done in lm-eval-harness by EleutherAI, all benchmark scores use normalized accuracy where applicable and are zero-shot. Weights are stored in BF16.

Future plans

Sometime in the near(ish) future i will release an instruction tuned variant of this model, along with a translation focused finetune. GGUF support will also come in the near(ish) future.