Llama-TinyThinkModel-72M-V1

Experimental research model by sraivante. Built with Llama (applicable synthetic training-data attribution). The repository identifier is TinyThinkModel-72M-V1. Exactly 72,000,000 parameters, 1,024-token context, and a 33,069-token custom byte-level BPE. Model weights and tokenizer were trained from scratch. The Qwen3-0.6B architecture was a design reference; no pretrained Qwen weights or tokenizer were used. The internal training project was called Think72.

This is the final export at step 22,889, after 1,500,053,504 processed token positions, trained on two Tesla T4 GPUs. Training completed on 2026-10-18. The public associated data repository contains the exact prepared training, validation and test corpus, tokenizer audit, source hashes, and exact HellaSwag evaluation data and per-question scores. Its public release tag is public-v1; the immutable dataset commit is 081fc840378d4b4168390a2f94571210d5c54e4e.

Purpose and observed limitations

For studying small language-model training, tokenization, English text continuation and experimental math/science/code responses. This model is unreliable as a general assistant: it can make elementary arithmetic errors, repeat text, and produce incorrect facts or reasoning. The small parameter count, 1,024-token context and predominantly synthetic/reasoning corpus limit general-language performance. No multilingual ability, strong reasoning or production suitability is established. Optional <think> text is learned formatting, not a correctness guarantee.

Evaluation

Evaluation	Result
HellaSwag zero-shot raw accuracy	27.14%, 2,725 / 10,042
HellaSwag zero-shot length-normalized accuracy	28.36%, 2,848 / 10,042
Uniform random-choice expectation	25%
Final validation loss / perplexity	1.502758600 / 4.49406933
Held-out test loss / perplexity	1.515999056 / 4.55396853

HellaSwag used the official lm-evaluation-harness==0.4.13 task and HFLM scorer on the full validation split, FP32, no quantization, no chat template, no BOS token and no few-shot demonstrations. Normalization uses character length of the official preprocessed ending. All 40,168 continuation requests fit the context (maximum 170 tokens, zero truncation). Both T4 GPUs were used. Raw accuracy's Wilson 95% interval is 26.28–28.01%; normalized accuracy's is 27.49–29.25%. The 3.36 percentage-point normalized improvement over random choice is modest.

Every original benchmark row, label, prompt and choice was checked locally, and both metrics were independently recomputed from recorded scores. This check reuses the saved inference scores; it is not a second inference run. See evaluation/hellaswag/summary.json and verification.json. Contamination with training data has not been exhaustively audited. Perplexities each use a deterministic sample of 262,144 target tokens; they are not task-accuracy percentages. The best validation checkpoint was step 22,750 (loss 1.502276920); this release intentionally contains the requested final step 22,889.

Loading and generation

Install torch, transformers==4.56.2, tokenizers==0.22.1 and safetensors. The model is public and can be downloaded without a Hugging Face account or access token.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "sraivante/TinyThinkModel-72M-V1"
tokenizer = AutoTokenizer.from_pretrained(repo, revision="public-v1", token=False)
model = AutoModelForCausalLM.from_pretrained(
    repo, revision="public-v1", token=False, torch_dtype=torch.float32
).eval()
messages = [{"role": "user", "content": "What is 12 plus 7?"}]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_tensors="pt", return_dict=True,
    tokenizer_kwargs={"return_token_type_ids": False},
)
remaining = 1024 - inputs["input_ids"].shape[1]
if remaining <= 0:
    raise ValueError("Prompt must leave room in the trained 1024-token context")
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=min(128, remaining), do_sample=False,
        use_cache=True, pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False))

The example runs on CPU. Keep prompt plus generation within 1,024 tokens; there is no validated longer-context mode. The root model.safetensors is 288,017,256 bytes. training/step-00022889.pt is an optional 864,233,185-byte full resume checkpoint, not required for inference; it contains optimizer/scaler, scheduler, data cursor and both GPU ranks' RNG state. Load full PyTorch checkpoints only from a trusted source.

Architecture and training

Qwen3ForCausalLM: 14 decoder layers; hidden width 512; SwiGLU intermediate width 1,536; 8 query heads, 4 KV heads, head dimension 128; tied token embeddings/output head; RMSNorm and Q/K RMSNorm; RoPE theta 1,000,000; full causal SDPA attention; no projection bias or dropout. There was one from-scratch next-token training stage on formatted conversations, with no separate SFT, adapter, preference-optimization or RL stage.

Microbatch 8 per GPU, 4 gradient-accumulation steps, global batch 64 sequences, 65,536 token positions/update. AdamW: learning rate 0.0006, 150 warmup steps, cosine decay to 0.00006, betas (0.9, 0.95), epsilon 1e-8, matrix weight decay 0.1, gradient clipping 1.0. FP16 autocast with FP32 weights/optimizer and gradient scaling. Loss covers all non-pad next-token targets, including prompts. See training/training_settings.json and code/ for the original scripts. Source scripts retain their original internal project names for reproducibility.

The prepared corpus contains 1,529,699,784 train tokens / 670,031 records, 7,572,390 validation tokens / 3,348 records, and 7,881,773 test tokens / 3,389 records. Training consumed 1.500B token positions rather than every prepared token. Matching normalized question hashes share a split; exact formatted duplicates were removed. Semantic near-duplicates may remain. Historical upstream revisions of the original local files are unknown; local source hashes and the exact corpus manifest identify the consumed snapshot. Current upstream observations are recorded separately.

Model SHA-256: 0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e. Corpus manifest SHA-256: f32e5f3dab44177534cf7457176f83327a02a087c3db2367d771ef64adb2a380. All release files are listed in release_files.json; release_identity.json links this checkpoint to the data revision. The final checkpoint has already passed full recovery verification, and local UI inference was exercised on the same exported weights.

Rights and access

The owner authorized public visibility on 2026-10-18. This changes access to the existing final checkpoint; model weights, tokenizer, training data and benchmark scores are unchanged. The original v1 tag remains as the historical initial-upload version; public-v1 identifies the public documentation release.

Read LICENSE and THIRD_PARTY.md. Original code, documentation and original dataset selection/arrangement by sraivante remain Apache-2.0. Model-weight use is offered for non-commercial research, subject to applicable upstream terms and attribution. The combined dataset retains mixed component licenses. Public access does not make this an unrestricted Apache-licensed model or dataset. Source-specific licensing uncertainty and missing historical revision/per-record metadata remain disclosed in the provenance records.