Nebium-Large (762M)

762-million parameter causal Transformer, the flagship model of the Nebium family, optimized for tactical sequence modeling and move legality.

• PyTorch Repository: nabin2004/nebium-large
• Companion GGUF Repository: nabin2004/nebium-large-gguf
• Framework Source: github.com/nabin2004/nebium

---

Architecture Specifications

Nebium-Large is built on a decoder-only causal Transformer architecture with modern architectural primitives:

Specification	Parameter Value
Model Tier	Nebium-Large
Parameter Count	762M
Hidden Dimension ($d_{model}$)	1280
Attention Heads ($n_{heads}$)	20
Transformer Layers ($n_{layers}$)	36
Context Window ($L_{max}$)	1024 tokens
Vocabulary Size ($V$)	5000 (Byte-Pair Encoding over UCI plies)
Positional Embeddings	Rotary Position Embeddings (RoPE, $\theta = 10000$)
Activation Function	SwiGLU
Layer Normalization	RMSNorm (pre-normalization)
Attention Mechanism	Causal scaled dot-product attention
Base Learning Rate	1.5e-4 (Cosine decay with linear warmup)
Weight Decay	0.1

---

Scaling Law Analysis (Chinchilla Framework)

Following Hoffmann et al. (2022) scaling laws, compute-optimal training balances parameter count $N$ with token allocation $D$:

$$L(N,D)=E+\frac{A}{N^{\alpha }}+\frac{B}{D^{\beta }}L(N,\; D)\; =\; E\; +\; \backslash frac\{A\}\{N^\backslash alpha\}\; +\; \backslash frac\{B\}\{D^\backslash beta\}$$

where $E = 1.69$, $A = 406.4$, $B = 410.7$, $\alpha = 0.34$, $\beta = 0.28$.

• Parameters ($N$): ~762M
• Chinchilla-Optimal Token Budget ($D^ \approx 20N$)*: 15.2B tokens
• Token Horizon: ~46 million 50-move game trajectories

---

Artifact Inventory

Filename	Description	Format
model.pt	Trained model weights (state dictionary)	PyTorch binary
model_config.json	Model architecture hyperparameter configuration	JSON
tokenizer.json	Trained BPE tokenizer vocabulary and merge tables	Hugging Face Tokenizers JSON
README.md	Model specification and benchmark documentation	Markdown

For local deployment and quantized execution with llama.cpp or Ollama, download the GGUF artifact from nabin2004/nebium-large-gguf.

---

Python Usage Example

import json
import torch
from src.models.transformer.nebium import Nebium
from src.data.tokenizer import ChessTokenizer

# 1. Initialize and load the tokenizer
tokenizer = ChessTokenizer()
tokenizer.load("tokenizer.json")

# 2. Instantiate model architecture from configuration
with open("model_config.json", "r", encoding="utf-8") as f:
    config = json.load(f)

model = Nebium(**config)
state_dict = torch.load("model.pt", map_location="cpu", weights_only=True)
model.load_state_dict(state_dict)
model.eval()

# 3. Autoregressive move generation
prompt = "e2e4 e7e5 g1f3"
input_ids = torch.tensor([[tokenizer.bos_id] + tokenizer.encode(prompt)], dtype=torch.long)
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    output = model.generate(input_ids, attention_mask, max_new_tokens=10, temperature=0.7)

generated_text = tokenizer.decode(output[0].tolist())
print("Generated continuation:", generated_text)

---

License & Citation

This model is licensed under the MIT License. If using Nebium in your research, cite the repository:

@misc{nebium_large_2026,
  author = {Oli, Nabin},
  title = {Nebium-Large: Causal Transformer for Next-Move Prediction in Chess},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/nabin2004/nebium-large}}
}