LogSentry-LM

A small, from-scratch GPT-style language model trained on real server logs to (1) autoregressively complete log lines and (2) detect anomalous log lines by how "surprised" the model is by them (perplexity-based scoring).

It is a compact, educational decoder-only Transformer (~2M parameters) built with plain PyTorch — no transformers modeling class — intended for learning and for lightweight log anomaly detection.

🚀 Try the live demo: https://log-sentry-lm.streamlit.app/ — paste log lines and see them scored in real time.

Model details

• Architecture: decoder-only Transformer (GPT-style), causal self-attention, pre-norm residual blocks, learned token + positional embeddings.
• Size: d_model=128, n_heads=4, n_layers=3, max_seq_len=128 (~2.03M params).
• Tokenizer: Byte-Pair Encoding (tokenizers), vocab ≈ 5.5k.
• Training data: combined Loghub samples — Linux + Apache + OpenSSH (*_2k.log, ~6k lines total).
• Objective: next-token prediction (cross-entropy).
• Framework: PyTorch (trained on Apple Silicon MPS).

Files in this repo

File	Purpose
logsentry_lm.pt	Trained weights + anomaly threshold + config
log_tokenizer.json	BPE tokenizer
architecture.py	Transformer building blocks
dataset.py	Tokenizer loader + dataset
train_and_generate.py	Model class, training, generation, anomaly scoring
detect_anomalies.py	Standalone anomaly predictor

Usage

⚠️ This is a custom architecture, so AutoModel.from_pretrained does not work. Download the files and load manually:

from huggingface_hub import hf_hub_download
import torch
from train_and_generate import LogSentryLM, score_line
from dataset import build_or_load_tokenizer

repo = "sankath/LogSentry-LM"
ckpt_path = hf_hub_download(repo, "logsentry_lm.pt")
tok_path  = hf_hub_download(repo, "log_tokenizer.json")

ckpt = torch.load(ckpt_path, map_location="cpu")
model = LogSentryLM(vocab_size=ckpt["vocab_size"], max_seq_len=ckpt["max_seq_len"])
model.load_state_dict(ckpt["model_state"]); model.eval()

tokenizer = build_or_load_tokenizer(None, tok_path)  # loads the cached tokenizer

loss, ppl = score_line(model, tokenizer, "Failed password for root from 1.2.3.4 port 22 ssh2", "cpu")
print("anomaly" if loss > ckpt["threshold"] else "normal", f"(perplexity={ppl:.1f})")

How anomaly detection works

The model learns what normal logs look like. A line it cannot predict scores a high per-token loss (high perplexity) and is flagged as anomalous. The cutoff is threshold = mean + 2·std of the training corpus line scores, stored in the checkpoint. Reference behavior:

Input	Perplexity	Verdict
Normal Apache [notice] line	~2	normal
Real SSH login attempt	~12–16	borderline
Random gibberish / injected text	>100,000	anomaly

Applications & use cases

Because the model learns the "grammar" of normal system logs, it can support a range of log-analytics and security tasks:

• Security monitoring / intrusion detection — flag suspicious log lines (brute-force SSH attempts, injected commands, malformed requests) that don't match learned normal patterns. This is the core "sentry" use case.
• Operational anomaly detection (AIOps) — surface rare error conditions, misconfigurations, or failing components before they escalate, by ranking log lines by how "surprising" they are.
• Log triage & noise reduction — instead of reading thousands of lines, engineers review only the top-scoring (most unusual) ones.
• Log autocompletion / templating — suggest or complete log-line formats given a prefix (useful in tooling and for synthetic log generation for testing).
• Data-quality / format validation — detect corrupted, truncated, or non-conforming log entries in an ingestion pipeline.
• Education — a compact, readable reference for how a GPT-style Transformer, tokenizer, and perplexity-based anomaly scoring work end to end.

How it can be used

• Batch scan a log file — score every line and export the anomalies: python detect_anomalies.py --file /var/log/auth.log
• Score individual lines programmatically via score_line(...) and compare against the saved threshold (see the usage snippet above).
• Stream integration — call the scorer inside a log shipper (Fluent Bit / Logstash / a Kafka consumer) to tag or route high-perplexity lines in near real time.
• Alerting — wire flagged lines into Slack/email/PagerDuty or a SIEM (Splunk, Elastic) as an enrichment signal.
• Retrain on your own logs — point LOG_FILE at your data and run train_and_generate.py to specialize the model to your environment.

Future scope

Natural next steps to make this research prototype more capable and production-ready:

• Per-source models & thresholds — separate models for SSH, Apache, kernel, etc., removing the single-global-threshold weakness.
• Session / sequence-level modeling — detect anomalous sequences of events (e.g. login → privilege escalation → data access), not just single lines.
• Larger context & scale — bigger d_model, more layers, longer max_seq_len, and training on full Loghub datasets (millions of lines).
• Template-aware tokenization — parse logs into templates + variables (à la Drain) so IPs/IDs don't inflate the vocabulary or the anomaly score.
• Calibrated scoring — convert raw perplexity into probabilities/severity levels, with per-line-position attribution ("which token was surprising?").
• Real-time serving — expose the scorer as a REST/gRPC service or a Hugging Face Space (Gradio demo) for interactive use.
• transformers integration — wrap as a PreTrainedModel so it loads via AutoModel.from_pretrained and plugs into the standard ecosystem.
• Evaluation & benchmarking — measure precision/recall on labeled anomaly datasets (e.g. BGL, HDFS) against baselines like DeepLog and LogBERT.
• Explainability & feedback loop — let analysts confirm/dismiss flags to continuously refine the threshold and reduce false positives.

Limitations

• Trained on a small, mixed corpus (three log sources). A single global threshold is imperfect across sources; per-source models/thresholds work better.
• Not production hardened — intended for education and experimentation.
• Small context window (128 tokens) and small vocab; it models line-level, not session-level, structure.

Attribution & licensing

• Training data derived from the Loghub collection (Linux, Apache, OpenSSH samples). Please cite Loghub and consult its terms for any redistribution of the underlying logs.
• Model code released under the MIT license.

@inproceedings{zhu2023loghub,
  title     = {Loghub: A Large Collection of System Log
               Datasets for AI-driven Log Analytics},
  author    = {Zhu, Jieming and He, Shilin and He, Pinjia and
               Liu, Jinyang and Lyu, Michael R.},
  booktitle = {IEEE International Symposium on Software
               Reliability Engineering (ISSRE)},
  year      = {2023}
}