Instructions to use kefir090/Lumen-v2-130M-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kefir090/Lumen-v2-130M-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kefir090/Lumen-v2-130M-Base", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("kefir090/Lumen-v2-130M-Base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kefir090/Lumen-v2-130M-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kefir090/Lumen-v2-130M-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kefir090/Lumen-v2-130M-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/kefir090/Lumen-v2-130M-Base

SGLang

How to use kefir090/Lumen-v2-130M-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kefir090/Lumen-v2-130M-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kefir090/Lumen-v2-130M-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kefir090/Lumen-v2-130M-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kefir090/Lumen-v2-130M-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use kefir090/Lumen-v2-130M-Base with Docker Model Runner:
```
docker model run hf.co/kefir090/Lumen-v2-130M-Base
```

Lumen-v2-130M-Base 💡

Lumen-v2-130M-Base is a high-efficiency Causal Language Model pretrained completely from scratch on a curated 6 Billion token multi-source educational corpus using Google TPU v5e-8.

The model features an 11-layer architecture with tied input/output embeddings, 4 full-rank Value Embedding (ResFormer) memory tables, causal depthwise convolution mixing (Canon K=3), and a single-digit tokenized vocabulary.

📊 Benchmark Results

All evaluations are zero-shot, evaluated with standard length-normalized likelihood (acc_norm) or raw accuracy over the full official test sets. Prompts are evaluated with the prefix <|bos|> token matching the pretraining distribution.

Core Benchmarks

Benchmark	Metric	Score	Samples ($n$)
ArithMark-3	`acc`	67.20%	1,000
PIQA	`acc_norm`	64.25%	1,838
ARC-Easy	`acc_norm`	47.69%	2,376
ARC-Challenge	`acc_norm`	26.62%	1,172
HellaSwag	`acc_norm`	34.81%	10,042
LAMBADA (OpenAI)	`acc`	34.23%	5,153
BLiMP (Grammar)	`acc`	77.80%	67,000
WikiText-2	`bits-per-byte (bpb)`	0.9766	—

Extended Task Breakdown

Task	Evaluation Metric	Score
SciQ	`acc_norm`	76.10%
COPA	`acc`	69.00%
BoolQ	`acc`	59.36%
SWAG	`acc_norm`	51.47%
WinoGrande	`acc`	47.75%
TruthfulQA MC2	`acc`	41.99%
OpenBookQA	`acc_norm`	31.00%
RACE	`acc`	29.76%
CommonsenseQA	`acc`	22.69%
MMLU	`acc`	24.20%
WikiText-2	`word_perplexity`	37.33

🍌 BananaMind Base Bench 1.1

Zero-shot evaluation on the BananaMind Base Bench 1.1 — a 350-example text-completion benchmark for base causal language models. Evaluation uses mean conditional log-probability scoring with no BOS token (official benchmark standard).

Metric	Score
Overall Elo	1024
Raw Accuracy	55.43% (194/350)
Weighted Accuracy	51.09%

Category Breakdown

Category	Elo	Accuracy
Language Completion	1345	94.00% (47/50)
Commonsense	1102	74.00% (37/50)
World Knowledge	1013	62.00% (31/50)
Context Tracking	926	44.00% (22/50)
Quantitative	819	22.00% (11/50)
Logical Reasoning	1017	44.00% (22/50)
Code Completion	1089	48.00% (24/50)

📌 Pretraining Data & Decontamination

The model was pretrained on 5,999,951,872 tokens across a balanced educational blend:

Source	Share	Tokens	Description
`HuggingFaceTB/smollm-corpus` (`fineweb-edu-dedup`)	72.0%	4.32 B	High-quality educational web corpus filtered by classifier score.
`HuggingFaceTB/smollm-corpus` (`cosmopedia-v2`)	10.0%	600 M	Synthetic textbooks, lectures, and articles across broad taxonomies.
`HuggingFaceTB/finemath` (`finemath-4plus`)	7.0%	420 M	Mathematical web pages, derivations, and proofs.
`Procedural Arithmetic`	5.0%	300 M	Algorithmic story problem generator with diverse entities and numbers.
`roneneldan/TinyStories`	3.0%	180 M	Synthetic narratives with constrained vocabulary for syntactic coherence.
`microsoft/orca-math-word-problems-200k`	3.0%	180 M	Grade-school math word problems with step-by-step solutions.

Decontamination Protocol

13-gram Exact-Match Filtering: Web and textbook data splits were decontaminated against benchmark question sets (HellaSwag, ARC, PIQA, GSM8K).

Post-mix audit: the full blend was independently re-scanned against the GSM8K test set: 13 of 199,976 orca-math rows (0.007%) carry residual 13-gram overlap. GSM8K is not among the benchmarks reported for this model; the arithmetic evaluations here (ArithMark-3) share no source material with that dataset.

🏗️ Architecture & Specifications

Parameter	Value
Architecture	Causal Decoder-Only Transformer (GQA + Tied Weights)
Inference Parameters	127,977,104 (~128M)
Layers	11
Hidden Size (d)	768
Feed-forward Dimension	3072 (ReLU² activation)
Attention Heads	6 Query / 2 KV (GQA 3:1), Head Dim 128
Positional Encoding	RoPE (theta = 100,000)
Normalization	Parameter-free RMSNorm (pre-norm), with per-head QK-norm
Context Window	2048 tokens
Vocabulary Size	32,768 (single-digit byte-level BPE)
Weight Tying	Enabled (wte.weight == lm_head.weight)
Value Embeddings	4 full-rank ResFormer tables on deepest layers
Local Mixing	Causal depthwise 1D convolution (K=3)
Auxiliary Loss	Multi-Token Prediction (MTP t+2, weight 0.3, train-only)
Hardware	Google TPU v5e-8 (128 GB HBM2e)
Training Speed	360,000 tokens/sec (4.5 hours wall-clock time)

💻 Quickstart (Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kefir090/Lumen-v2-130M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# bfloat16 needs compute capability >= 8.0 (Ampere or newer: RTX 30xx+, A100/H100).
# T4/P100 report is_bf16_supported()==True but produce wrong results -- check cc instead.
if torch.cuda.is_available():
    major, _ = torch.cuda.get_device_capability()
    dtype = torch.bfloat16 if major >= 8 else torch.float32
else:
    dtype = torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=dtype,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The solar system consists of"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.85,
        repetition_penalty=1.15,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Transformers version: tested with transformers>=4.57, including the latest 5.x line. Older releases may fail to load the custom architecture correctly.

Note: This is a pretrained base foundation model. It performs autoregressive text completion and is not instruction-tuned.

📜 License

This model and its weights are released under the Apache 2.0 License.

Downloads last month: 502

Safetensors

Model size

0.1B params

Tensor type

BF16