Allan-LM Quark
The first base checkpoint of Allan-LM, a language model trained from scratch by KEYTRON for the Allan assistant. Quark is a small research model: it continues text in Russian and English and has seen source code. It is not instruction-tuned and does not answer questions; a chat version comes later.
Versions of Allan-LM are named after particles.
Model
| Parameters | 278M total, 107M active per token |
| Layers | 10, d_model 640, 10 heads (2 KV heads, GQA) |
| Feed-forward | Mixture of Experts: 8 experts, top-2, plus 1 shared expert |
| Per-Layer Embeddings | yes (d_ple 80), in the style of Gemma's E-models |
| Context | trained at 1024 tokens, RoPE positions up to 2048 |
| Tokenizer | own BPE, 32,000 text tokens; the vocabulary (36,160) also reserves special and audio tokens for later stages |
| Weights | fp32 safetensors, tied embeddings |
The architecture is custom (allan_lm/model.py in this repo); it does not load
with transformers, llama.cpp or Ollama.
Training
- ~10B tokens (152,000 steps × 65,536 tokens), sequence length 1024.
- AdamW, peak LR 5e-4, 1,000 warmup steps, warmup-stable-decay schedule with the last 20% decaying to 5e-5.
- Hardware: Google Colab A100 (steps 0–147,000), RunPod RTX 4090 (the rest).
- Data mix: Russian web text, English educational web text, source code.
| Source | License | Share |
|---|---|---|
| fineweb-2 (rus_Cyrl) | ODC-BY | Russian |
| fineweb-edu | ODC-BY | English |
| github-code-clean | per file, mixed | code |
No text generated by other models (answers, distillations, synthetic data) and no machine translation was used.
Note on code data: github-code-clean was used without filtering by file license, so it includes copyleft (e.g. GPL) code next to permissively licensed code. Later Allan-LM versions will train on permissively licensed code only.
Evaluation
Validation cross-entropy (nats/token, held-out shards of the same sources):
| Russian | English | Code | |
|---|---|---|---|
| Allan-LM 156M (previous internal run, 5B tokens) | 2.88 | 3.02 | 1.29 |
| Allan-LM Quark | 2.70 | 2.78 | 1.09 |
What to expect
Quark writes grammatical, coherent Russian and English and knows the shape of common texts (encyclopedia style, recipes, lists, Python syntax). At this size it has very little factual knowledge: it confidently invents dates, places and facts, its code does not work, and web spam (e.g. casino ads) occasionally leaks into its output. Use it for research and experiments, not as a source of information.
Example (temperature 0.7, top-k 40):
Санкт-Петербург основан в 1946 году и является одним из крупнейших в России центров по производству минеральных удобрений.
Usage
pip install torch safetensors tokenizers
python generate.py "Москва — столица России. Она"
import torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from allan_lm import AllanLM, ModelConfig
cfg = ModelConfig.load("config.json")
model = AllanLM(cfg)
model.load_state_dict(load_file("model.safetensors"))
model.eval()
tok = Tokenizer.from_file("tokenizer.json")
ids = tok.encode("The history of").ids
out = model.generate(torch.tensor([ids]), 60, temperature=0.7, top_k=40)
print(tok.decode(out[0].tolist()))
License
Business Source License 1.1 (see LICENSE): free for personal, academic,
research and evaluation use; commercial use needs a separate agreement with the
licensor. On 2028-10-12 the license changes to MIT.
Русский
Allan-LM Quark — первый базовый срез своей модели Allan-LM, обученной с нуля (278M параметров, 107M активных, MoE + PLE, ~10 млрд токенов русского, английского и кода). Это базовая модель: продолжает текст, на вопросы не отвечает. Пишет грамотно, но фактов почти не знает и уверенно выдумывает. Лицензия BSL 1.1: некоммерческое использование свободно, коммерческое — по договорённости; с 2028-10-12 — MIT.
- Downloads last month
- -