Gemmeh-IT 1B

Gemmeh-IT is the instruction-tuned version of ni-co-la-s/gemmeh, a 1.1B-parameter decoder-only transformer trained for educational purposes from scratch on 20B tokens of pre-2024 FineWeb-Edu, then LoRA-finetuned on OpenHermes for chat.

Architecture is Gemma 3-inspired without sliding-window attention. Custom 32k SentencePiece BPE tokenizer trained on the same corpus.

For the base model, see ni-co-la-s/gemmeh. For the GGUF, see ni-co-la-s/gemmeh-it-GGUF.

Usage

This model uses custom modeling code, so trust_remote_code=True is required.

import torch
from huggingface_hub import hf_hub_download
from sentencepiece import SentencePieceProcessor
from transformers import AutoModelForCausalLM

repo_id = "ni-co-la-s/gemmeh-it"

model = AutoModelForCausalLM.from_pretrained(
    "ni-co-la-s/gemmeh-it",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
model.eval()
print("Model loaded")

# Load tokenizer
sp_path = hf_hub_download(
    repo_id="ni-co-la-s/gemmeh-it",
    filename="tokenizer.model",
    token=True,
)
sp = SentencePieceProcessor()
sp.Load(sp_path)
print("Tokenizer loaded")

# Test generation
def generate(question, max_new_tokens=40, temperature=0.0):
    prompt = (
        f"<start_of_turn>user\n{question}\n<end_of_turn>\n"
        "<start_of_turn>model\n"
    )
    ids = sp.Encode(prompt, out_type=int)
    input_ids = torch.tensor([ids], dtype=torch.long)
    with torch.no_grad():
        for _ in range(max_new_tokens):
            out = model(input_ids)
            next_logits = out.logits[0, -1, :]
            if temperature == 0:
                next_id = next_logits.argmax().item()
            else:
                probs = torch.softmax(next_logits / temperature, dim=-1)
                next_id = torch.multinomial(probs, 1).item()
            if next_id == sp.eos_id():
                break
            input_ids = torch.cat([input_ids, torch.tensor([[next_id]])], dim=1)
    return sp.Decode(input_ids[0][len(ids):].tolist())

print(generate("What is the capital of France?", max_new_tokens=40, temperature=0.0))

Training details

Parameters	1.1B
Architecture	Gemma 3-inspired
Vocab	32,768 (SentencePiece BPE, English-only)
Context	4,096
Pretraining data	FineWeb-Edu sample, 20B tokens
Knowledge cutoff	Pre-2024 (intentional)
Finetuning	LoRA rank 16 on OpenHermes (250M assistant tokens)

Benchmarks

Evaluated through the BF16 GGUF served via llama.cpp with lm-eval. Gemma 3 1B IT numbers obtained locally by running unsloth/gemma-3-1b-it-GGUF at Q8_0 through the same pipeline.

Benchmark	Metric	Gemmeh 1B (base)	Gemmeh-IT 1B	Gemma 3 1B IT (local)
PIQA	0-shot	70.2	71.4	72.8
ARC-Challenge	25-shot	38.4	40.4	40.3
ARC-Easy	0-shot	57.3	57.6	63.4
WinoGrande	5-shot	52.2	54.0	55.1
TruthfulQA	mc2, 0-shot	37.8	44.8	38.9

Trained on roughly 10-100x less data than the references, with a much smaller (32k vs 262k) vocabulary, and no distillation.

Limitations

• Smaller and less benchmark-competitive than similarly-sized models trained on more data.
• English only.
• 4,096 token context.
• Pre-2024 knowledge only.