Malama Base
Malama Base is a 0.96B-parameter hybrid language model trained from scratch on 100B tokens of web text, code and math. It is the pretrained base for the Malama models: it has had no midtraining, instruction tuning or reinforcement learning, and it continues text rather than following instructions.
The goal of the Malama line is a small model that is competitive for its size and training compute in one narrow task, code autocomplete. This base model was trained with about 5.8e20 FLOPs (6 Γ parameters Γ tokens).
Model
| Architecture | Decoder-only hybrid: Kimi Delta Attention (KDA) linear-attention layers with full attention every fourth layer |
| Layers | 24 (pattern KDA, KDA, KDA, full attention; 18 KDA and 6 full-attention layers) |
| Width | 1,536 model dimension, 4,096 MLP hidden |
| Heads | 12 KDA heads; 12 query and 4 key/value heads (grouped-query attention) in full-attention layers; head dimension 128 |
| Position encoding | RoPE, base 1,000,000 (full-attention layers) |
| Context | 4,096 tokens |
| Vocabulary | 154,880 (GLM tokenizer, 154,856 tokens used; EOS id 154820) |
| Parameters | 959.8M during training, including a one-block multi-token-prediction (MTP) head; about 928M in the released weights, which drop the MTP head |
| Weights | bfloat16 safetensors |
Training
| Tokens | 99,999,547,392 (95,367 steps of 1,048,576 tokens: 256 sequences of 4,096 tokens) |
| Optimizer | Muon (lr 0.02, momentum 0.95) for matrices; AdamW (lr 3e-4, betas 0.9/0.95) for the rest; weight decay 0.1; gradient clip 1.0 |
| Schedule | 0.5B-token warmup, constant learning rate to 85B tokens, then decay over the final 15% |
| Auxiliary losses | Next-token prediction plus a multi-token-prediction head (loss on a sampled 25% of positions, weight 0.3 falling to 0.1); z-loss 1e-4 |
| Hardware | 8 Γ NVIDIA H100 80GB on Modal, PyTorch 2.14.1 (CUDA 13.0), bf16, torch.compile |
| Throughput | Median 380.5k tokens per second (2.76 s per step), median model FLOPs utilization 29.9% |
| Training time | 75.3 hours of training steps (602 H100-hours), 2026-11-10 to 2026-11-14 |
| Seed | 1234 (training), 20261025 (data recipe) |
Data
Recipe kda-foundation-100b-504010-v3: 50% general web text, 40% code and 10% math. Code from
popular and curated repositories is sampled more often and repeated.
| Source | Share | Max passes | Dataset |
|---|---|---|---|
| General web | 50.0% | 1 | karpathy/climbmix-400b-shuffle |
| Math | 10.0% | 1 | nvidia/Nemotron-CC-Math-v1 |
| Starred code (non-Python) | 20.0% | 3 | codeparrot/github-code-clean, repositories with at least 10 watchers |
| Other code (non-Python) | 8.3% | 1 | codeparrot/github-code-clean |
| Starred Python | 4.0% | 5 | codeparrot/github-code-clean, repositories with at least 10 watchers |
| Other Python | 3.5% | 1 | codeparrot/github-code-clean |
| Curated open-source projects | 4.0% | 5 | jfrick/curated-oss-code-20261019 |
| MIT-licensed collection | 0.2% | 5 | jfrick/mit-claude-codex-code-20261019 |
Repository stars come from ibragim-bad/github-repos-metadata-40M (watchers count). Code is
filtered by language, file size and path to drop markup, generated and vendored files. Training
documents sharing a 13-gram with the test sets of GSM8K, HumanEval, MBPP, MMLU, ARC, HellaSwag,
MATH and LiveCodeBench are removed. Each source holds out 2,000 documents from its last shard
for evaluation.
Results
The training loss fell from 2.07 at the start of the learning-rate decay to 1.75 at the end (500-step mean).
Held-out bits per byte on 128 documents per source, measured every 500 steps. Lower is better.
| Held-out set | 85B tokens (decay start) | 100B tokens | Change |
|---|---|---|---|
| General web (ClimbMix) | 0.823 | 0.715 | β13% |
| Math | 0.757 | 0.634 | β16% |
| Curated projects | 0.650 | 0.514 | β21% |
| MIT collection | 0.614 | 0.474 | β23% |
| Starred Python | 0.564 | 0.407 | β28% |
| Other code | 0.447 | 0.352 | β21% |
| Starred other code | 0.420 | 0.336 | β20% |
| Other Python | 0.398 | 0.313 | β21% |
Most of the improvement after 50B tokens came in the decay phase. Between 50B and 85B tokens at constant learning rate, held-out scores improved by about 1% or less.
No benchmark scores yet. The planned evaluation is code infilling (SAFIM and
HumanEval-Infilling), which needs fill-in-the-middle training first: this base model has not
seen the GLM <|code_prefix|>, <|code_suffix|> and <|code_middle|> tokens.
Samples
samples/base-samples.json holds 12 prompts (Python, JavaScript, Rust, SQL, a README, prose and
math), each continued greedily and with sampling (temperature 0.8, top-p 0.95, seed 0) for up to
256 tokens. The model writes fluent, idiomatic code and well-formed documents; the code is often
wrong in substance (a broken Dijkstra, an LRU cache that misuses heapq), greedy decoding
repeats itself on open-ended text, and the math is unreliable.
Run history
The run was trained in sessions of up to six hours on Modal. Each launch has a 24-hour limit, so
the run took seven launches (attempts a1 to a7), each resuming from a checkpoint of the
lineage.
- Superseded branches. Three stretches of training were discarded and retrained from an
earlier checkpoint: steps 9β8,276 of the first launch (sessions
pretrain-s01,pretrain-s02), after a preemption of the workflow controller opened a second attempt; steps 27,909β30,527 ofa3-s02, after a GPU preemption; and steps 85,676β85,972 ofa6-s04, cut off by a launch deadline before its final checkpoint. Their logs are kept underlogs/sessions/and are excluded from the lineage files. - Code versions. Attempts
a1toa4ran commit0609859of koogle/lm-exploration;a5toa7rana5c0d47, which adds continuation from in-session checkpoints after a preemption. The trainer source is identical in both. - Throughput steps. Most level shifts in the throughput chart fall at session boundaries, where a new session can land on a different 8 Γ H100 host.
Files
| Path | Contents |
|---|---|
tokens-099999547392/ |
Final weights: model.safetensors, config.json, progress.json |
tokens-005000658944/ to tokens-095000985600/ |
Intermediate exports every 5B tokens |
logs/lineage-metrics.jsonl |
One row per training step for the final lineage (loss, MTP loss, learning-rate factor, gradient norm, throughput, MFU, per-source token counts), with the session it came from |
logs/lineage-validation.jsonl |
Held-out evaluations every 500 steps for the final lineage |
logs/sessions/<session>/ |
Every session's raw logs, including superseded ones: metrics.jsonl, validation.jsonl, torchrun.log, report.json, session.json, config.json, run-config.json, provenance.json |
logs/workflow/ |
The workflow's records: stages.json, per-attempt workflow.json and per-session segment-NN.json reports |
metrics.jsonl, validation.jsonl |
Logs of the last session, uploaded by the trainer |
charts/ |
The charts in this card |
samples/base-samples.json |
Generations from the final weights |
Usage
The weights load with the model code in
koogle/lm-exploration (training/hybrid/), not with
transformers. Decoding needs a CUDA GPU.
import torch
from model_hybrid import generate
from tokenizer import glm
lm = generate.load_export("tokens-099999547392", dtype=torch.bfloat16, device="cuda")
tokenizer = glm.load()
prompt = tokenizer.encode("def fibonacci(n: int) -> int:\n", add_special_tokens=False).ids
[ids] = generate.generate_batch(lm, [prompt], 128, (glm.EOS_ID,))
print(tokenizer.decode(ids))
Limitations
- A base model: it continues text and does not follow instructions or hold a conversation.
- Code it writes often looks right and is wrong; nothing here measures functional correctness.
- No fill-in-the-middle training yet.
- English-centric web data; other languages are barely represented.
- Trained on public web text and code, which contain errors, insecure code and offensive content.
License
Malama Base is released under CC BY-NC 4.0: free to use, share and adapt for non-commercial purposes, with attribution.
The license follows the most restrictive terms among the training data:
| Source | Share | Terms |
|---|---|---|
General web: ClimbMix, read from karpathy/climbmix-400b-shuffle, a reshuffle of NVIDIA's ClimbMix |
50.0% | CC BY-NC 4.0 (NVIDIA's license; the reshuffle's MIT label does not change it). This is the binding term. |
Math: nvidia/Nemotron-CC-Math-v1 |
10.0% | NVIDIA Data Agreement for Model Training: training permitted, trained models belong to the trainer. NVIDIA notes that models trained on it may be subject to the Phi-4 license (MIT), since the data was produced with phi-4. |
GitHub code: codeparrot/github-code-clean |
35.8% | Files kept only under MIT, Apache-2.0, BSD-2/3-Clause, ISC, CC0-1.0 or Unlicense |
| MIT collection | 0.2% | MIT |
| Curated open-source projects | 4.0% | Each project's own license, including GPL projects (Linux, git, Blender) |
GLM tokenizer (zai-org/GLM-4.7-Flash) |
MIT |
Whether model weights are a derivative work of copyleft training code is unsettled; this license does not resolve that question for the GPL projects in the curated set. This section records how the license was chosen and is not legal advice.
Provenance
| Workflow run | kda-960m-foundation-20261110-01, plan SHA-256 b2095d578ec025f942caf8912aabb549422078226bf951ca0648b3b89ab9c80c |
| Model configuration | training/hybrid/configs/kda-960m-100b.json |
| Data recipe | training/hybrid/recipes/kda-foundation-100b-504010-v3.json |
| Tokenizer | GLM tokenizer, SHA-256 19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d |
| Full training checkpoints | With optimizer and data state: 100B tokens, jfrick/malama-base-checkpoint-100b (step-000095367.pt); 84.8B tokens, the last before the learning-rate decay, jfrick/malama-base-checkpoint-85b (step-000080895.pt) |


