CyberSLM-33M-Base

A 33.5M-parameter cybersecurity-focused small language model, pretrained from scratch on a ~745 MB curated corpus (cybersecurity, computer fundamentals, programming, general English) with a custom 32k SentencePiece BPE tokenizer. Trained on a single NVIDIA L40S.

This is the base (pretrained-only) model. For the instruction-tuned variant see cyberslm-33m-instruct.

Architecture

Decoder-only transformer (Llama-style, loadable with LlamaForCausalLM):

Parameters	33,531,264
Hidden dim	384
Layers	12
Attention heads	6 (MHA, head_dim 64)
FFN	SwiGLU, inner dim 1024
Norm	RMSNorm (pre-norm), eps 1e-6
Positional	RoPE, θ = 10,000
Context	4096
Vocab	32,000 (SentencePiece BPE, byte_fallback)
Weight tying	yes

Training

• Corpus: 665k documents (205M tokens), 95/5 train/val split, seed 42
• 4,000 steps, effective batch 131,072 tokens/step (~524M tokens seen)
• AdamW β=(0.9, 0.95), wd 0.1, cosine schedule, peak LR 3e-4, grad clip 1.0
• Final val loss: 2.43

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("sabari2005/cyberslm-33m-base")
model = AutoModelForCausalLM.from_pretrained("sabari2005/cyberslm-33m-base")

ids = tok("A SQL injection attack is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Special tokens: pad=0 (<pad>), unk=1 (<unk>), bos=2 (<bos>), eos=3 (<eos>).

Limitations

Base model — no instruction tuning; it continues text rather than follows instructions. Trained on ~0.5B tokens, so expect limited general knowledge outside the cyber/CS domain. English only.