Nebium-Medium (306.6M)

Nebium-Medium is a 345-million-parameter causal Transformer balancing sequence quality and inference throughput for production evaluation.

• PyTorch Repository: nabin2004/nebium-medium
• GGUF Repository: nabin2004/nebium-medium-gguf
• Source Repository: github.com/nabin2004/nebium

Nebium Model Family Architecture Overview

Model	Params	d_model	Heads	Layers	max_seq_len	Chinchilla-optimal tokens
Nebium-Small	117M	768	12	12	1024	~2.3B
Nebium-Medium	345M	1024	16	24	1024	~6.9B
Nebium-Large	762M	1280	20	36	1024	~15.2B

Architectural Primitives:

• Rotary Position Embeddings (RoPE) on attention query and key projections ($\theta = 10000$)
• SwiGLU feed-forward transformation
• RMSNorm pre-normalization
• Causal mask with padding token masking
• Byte-Pair Encoding (BPE) tokenizer trained on UCI move plies

Architectural Specifications

Hyperparameter	Value
Model Tier	Nebium-Medium
Parameter Count	306.6M
Hidden Dimension ($d_{model}$)	1024
Attention Heads ($n_{heads}$)	16
Transformer Layers ($n_{layers}$)	24
Max Context Length ($L_{max}$)	1024
Vocabulary Size ($V$)	2018
Positional Embedding	rope
Activation Function	swiglu
Layer Normalization	rmsnorm

Validation & Benchmark Results

Metric	Measured Value
Validation Loss	2.693234
Validation Perplexity	14.7794
Next-Token Top-1 Accuracy	32.99%
Next-Token Top-5 Accuracy	64.78%
Empirical Move Legality Rate	100.00%
Tactical Puzzle Accuracy	9.76%

Scaling Law Analysis (Hoffmann et al. 2022)

Chinchilla power-law formulation:

$$L(N,D)=1.69+\frac{406.4}{N^{0.34}}+\frac{410.7}{D^{0.28}}L(N,\; D)\; =\; 1.69\; +\; \backslash frac\{406.4\}\{N^\{0.34\}\}\; +\; \backslash frac\{410.7\}\{D^\{0.28\}\}$$

Parameter / Metric	Value
Model Parameters ($N$)	306.6M
Chinchilla-Optimal Token Budget ($D^*$)	~6.9B tokens
Compute-Optimal Expected Loss ($L_{optimal}$)	2.9413 nats
Approximate Trained Tokens ($D$)	~12.3M tokens
Theoretical Loss at Current Tokens	6.4722 nats
Empirical Validation Loss	2.6932 nats

Python Usage Example

import json
import torch
from src.models.transformer.nebium import Nebium
from src.data.tokenizer import ChessTokenizer

tokenizer = ChessTokenizer()
tokenizer.load("tokenizer.json")

with open("model_config.json", "r", encoding="utf-8") as f:
    config = json.load(f)

model = Nebium(**config)
state_dict = torch.load("model.pt", map_location="cpu", weights_only=True)
model.load_state_dict(state_dict)
model.eval()

prompt = "e2e4 e7e5 g1f3"
input_ids = torch.tensor([[tokenizer.bos_id] + tokenizer.encode(prompt)], dtype=torch.long)
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    output = model.generate(input_ids, attention_mask, max_new_tokens=10, temperature=0.7)

print("Continuation:", tokenizer.decode(output[0].tolist()))

License

MIT License.