Morpheus (Mamba-2) — Basque Autocomplete

A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autocompletion. Trained from scratch on a curated Basque corpus and deployed as a 55 MB quantized model running on consumer CPUs.

Model Details

• Architecture: Mamba-2 (State Space Model, Structured State Space Duality)
• Parameters: 91M (only 3.4% in embeddings thanks to the 4K vocab)
• Embedding vocab: 4,000 (Unigram SentencePiece)
• Hidden dimension: 768
• Layers: 24
• State dimension: 64
• Head dimension: 64
• Inner dimension: 1,536
• Sequence length: 1,024
• Training: 10B tokens seen (2.16 epochs over a 4.62B-token unique corpus)
• Best checkpoint: step 74,000 (held-out PPL 7.13)
• Trained without BOS token (add_bos_token=false)

Tokenizer

A 4K Unigram SentencePiece tokenizer trained on the cleaned Basque corpus. The small vocabulary size was chosen based on a vocabulary-size ablation (paper §4.4) motivated by the fertility paradox in agglutinative languages:

Vocab	Fertility (tok/word)	Morpheme Boundary Accuracy
32,000	1.85	28.6%
4,000	2.58	66.7%

Larger vocabularies fuse Basque roots and suffixes into opaque atomic tokens (e.g. ▁etxetik, ▁etxera), destroying the morphological boundaries the model needs to productively generate unseen inflections. The 4K vocabulary keeps individual case suffixes and pluralizers as reusable subwords (etxe+tik → ▁etxe tik), which is decisive for an agglutinative language where a single verb can encode subject, object, indirect object, tense, mood, and aspect through suffix chains. This mirrors the QuechuaTok finding for Quechua.

• add_bos_token: false (the model was trained without a BOS token)
• EOS token: </s> (id=2)
• UNK token: <unk> (id=0)

Usage (transformers)

import torch
from transformers import AutoModelForCausalLM, LlamaTokenizer

model = AutoModelForCausalLM.from_pretrained("itzune/morpheus", torch_dtype=torch.float32)
tokenizer = LlamaTokenizer.from_pretrained("itzune/morpheus")
# tokenizer_config.json already sets add_bos_token=False — matches no-BOS training

prompt = "Kaixo, zer moduz"
inputs = tokenizer(prompt, return_tensors="pt")   # NO BOS is prepended (correct)
output = model.generate(**inputs, max_new_tokens=5, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

With the transformers tokenizer, string prompts are fine — LlamaTokenizer uses the reference sentencepiece library, so tokenization matches training exactly. The token-ID caveat below applies only to the GGUF / llama.cpp deployment path.

⚠️ Deploying with llama.cpp? Use token-ID prompts

If you quantize this model to GGUF and serve it with llama-server/llama-cli, do not send string prompts. llama.cpp's built-in SentencePiece tokenizer diverges from the reference library on this 4K vocabulary, and llama-server may auto-prepend a BOS token the model never saw in training. Combined, these drop character-saving rate from ~28% to ~4% — a 7× degradation (paper §5.1).

The fix is to load tokenizer.model with the sentencepiece library, encode to token IDs (no BOS), and send the ID list to llama-server's /completion endpoint. Pre-quantized GGUF models and a full token-ID usage example are at itzune/morpheus-gguf.

Training Data

Trained on a curated subset of the publicly available Latxa Corpus v2 (HiTZ/latxa-corpus-v2; Etxaniz et al., 2024):

• 4.62 billion subword tokens (9 GB tokenized; ~15 GB raw text), of which **10B tokens were seen** during training (~2.16 epochs).
• 11 of the 14 Latxa Corpus v2 sub-corpora were retained; 3 were omitted for quality reasons (hplt-v1: 83.8% duplicates; BOG: sentence-splitting destroyed legal text; Aldizkariak: 35% boilerplate). An LLM-based audit rated the retained sources 4.6/5 on average.
• Four-phase cleaning pipeline: document re-parsing, form regularity, content filtering (incl. validation-leakage removal), and MinHash-LSH deduplication. See paper §4.2.

Intended Use

On-device Basque text autocomplete and predictive keyboard input. The model is small enough to run on CPU via llama.cpp — see the GGUF quantized versions at itzune/morpheus-gguf (55 MB Q4_K_M). On a 2017 consumer laptop CPU (Intel i7-8550U) it achieves 318 tok/s decode and 97 ms end-to-end autocomplete latency (paper §5.2).

Citation

@misc{morpheus_mamba,
  author       = {Xabier Ezpeleta},
  title        = {Morpheus: On-Device Basque Autocompletion with Mamba-2},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/itzune/morpheus}},
}

See the accompanying paper (morpheus-on-device-basque-autocompletion) for full architecture, training, evaluation, and deployment details.