CyberSLM-base — 33.5M-parameter cybersecurity language model

A decoder-only transformer pretrained from scratch on a cybersecurity corpus. This is the base model: it continues text. It has not been instruction-tuned and will not answer questions.

For question answering use sabari2005/cyberslm-instruct.

Code: github.com/Sabari2005/cyberslm

What this model is

Give it the start of a sentence and it continues it:

Prompt:  "SQL injection is"
Output:  "SQL injection is a common issue in the web interface of Cisco IOS and
          IOS XE Software. It has been declared as critical for its security,
          integrity, and availability. The vulnerability exists because the
          affected software does not properly validate user-supplied input..."

Ask it a question and it will continue the question, not answer it.

Model details

parameters	33,531,264
layers	12
d_model	384
heads / head_dim	6 / 64
FFN (SwiGLU)	1024
context	2048
vocab	32,000 (SentencePiece BPE, byte-fallback)
positional encoding	RoPE, base 10000
normalisation	RMSNorm, pre-norm
LM head	tied to embedding
precision	trained in bf16

Training. 786,432,000 tokens = 4.04 epochs over a 194.8M-token corpus (~60% cybersecurity across 16 subdomains, ~20% general English and reasoning, ~15% programming, ~5% CS fundamentals). 6,000 steps at 131,072 tokens/step, AdamW, lr 3e-4 → 3e-5, 600 warmup, cosine decay, grad clip 1.0.

Single A100-40GB, 71 minutes, 184,084 tokens/sec.

Evaluation

409,600 held-out tokens. Compared against an earlier checkpoint of the same architecture, both scored by one process on identical windows at identical context (a longer conditioning window lowers loss on its own, so scoring each at its own maximum would not be a fair comparison):

metric	this model	earlier checkpoint
validation loss	2.3627	2.6255
perplexity	10.62	13.81
bits / token	3.4086	3.7878
top-1 accuracy	57.21%	54.38%
top-5 accuracy	72.64%	69.55%
8-gram repetition	23.7%	34.0%

Training-time validation loss at step 6,000 was 2.0247, measured on a different subset; only the columns above are like-for-like.

No benchmark accuracy is claimed — there is no contamination-checked security question bank, so nothing beyond next-token metrics is asserted. Single seed.

Usage

pip install torch sentencepiece
git clone https://huggingface.co/sabari2005/cyberslm-base
cd cyberslm-base
python infer_base.py --prompt "SQL injection is"

Options:

python infer_base.py \
    --prompt "A buffer overflow occurs when" \
    --max-new-tokens 120 \
    --temperature 0.8 \        # 0 = greedy/deterministic
    --top-k 50 --top-p 0.95 \
    --repetition-penalty 1.15

Loading directly

import torch, sentencepiece as spm
from cyberslm.model.config import CyberSLMConfig
from cyberslm.model.model import build_model

payload = torch.load("models/base.pt", map_location="cpu", weights_only=False)
model = build_model(CyberSLMConfig(**payload["config"]), device=torch.device("cpu"))
model.load_state_dict(payload["model_state"])
model.eval()

sp = spm.SentencePieceProcessor(); sp.load("tokenizer/tokenizer.model")
ids = [sp.bos_id()] + sp.encode("SQL injection is", out_type=int)
out = model.generate(torch.tensor([ids]), max_new_tokens=60,
                     temperature=0.0, eos_id=sp.eos_id())
print(sp.decode(out[0].tolist()))

generate() uses a KV cache, so decoding is O(n) — roughly 50–70 tok/s on CPU.

Limitations

A 33.5M-parameter model trained on 786M tokens. It produces fluent, domain-flavoured security prose and is not factually reliable. Output drifts into CVE-advisory boilerplate because that pattern is common in the corpus, and longer generations repeat (23.7% 8-gram repetition measured).

Intended for research into small language models and as a base for further scaling or fine-tuning. Not intended for security advice or any use where being wrong matters.

Training data

Not published. Curated from public cybersecurity, programming and general-English sources; not redistributed with the model.

License

Apache-2.0 for the code and weights. Verify licensing for downstream use against the sources the corpus was curated from.