xDevine/yolo11s-multiclass-surveillance-detection

Unified Multi-Class Threat Detection

A personal computer-vision project: training a single real-time object detector that covers 21 classes relevant to security monitoring — weapons (short-gun, long-gun, knife), fire and smoke, phones, and the surrounding everyday COCO objects (person, car, bag, umbrella, …) that a surveillance model has to reason about at the same time.

The interesting part of the project was never the training command — it was getting five mismatched public datasets into a state where a model could actually learn from them. Most of the effort, and most of this README, is about data engineering.

• Model: YOLO11s, fine-tuned from COCO-pretrained weights
• Classes: 20 target classes (indices 0–19) + phone (20)
• Released weights: weights/best.pt (v3)

Results (held-out test set, 2,470 images / 6,780 instances, conf 0.7)

Class	Recall	Precision
Short-gun	0.81	0.96
Long-gun	0.71	0.97
Phone	0.80	0.98
Knife	0.58	0.90
Fire	0.58	0.96
Smoke	0.50	0.95

• Overall precision 0.91 at conf 0.7 — few false alarms, which is the property that keeps a human operator trusting the alerts.
• All 21 classes register non-zero recall at the reporting threshold (no class is completely missed).
• Misses fall to background rather than being misclassified — the confusable pairs (short-gun/phone, long-gun/umbrella, knife/scissors) stay cleanly separated.

See evaluation/ for the training curves and the normalized confusion matrix.

What was involved in the data work

Five datasets, 184,593 images, 974,388 label lines — parsed once into a single per-bounding-box table so the audit could be done with queries instead of re-reading files. That inspection surfaced five issues, each of which would have quietly degraded the model:

1. Class remapping. The COCO export was ordered alphabetically, not in canonical COCO order — person is class 48, not 0. A hand-written remap would have silently mislabelled ~872k annotations. The mapping is instead generated from each dataset's own data.yaml by name, and fails loudly if any class doesn't resolve.
2. Label completeness. The single-purpose datasets only labelled their own target, so real persons/cars in weapon and fire scenes were unlabelled — teaching the model to suppress the person class. Fixed by pseudo-labelling the non-COCO data with a strong pretrained detector (YOLO11x), with a name-based remap and IoU dedup as guards. ~36k boxes added.
3. Train/val leakage. Roboflow augmentation had scattered near-identical copies of the same source photo across both splits — inflating validation metrics by measuring memorization. Content hashing misses this (augmented variants are byte-distinct). Fixed by recovering each image's source ID from its filename and enforcing source-level split assignment; 5,738 images reassigned, zero split-spanning after.
4. Class imbalance. Measured at the box level (which is what the model learns from), the person-to-short-gun ratio was 41:1, not the 8:1 the image counts suggest. Addressed with tiered subsampling of COCO — keep every image with a scarce class, a confuser, or a knife/phone; random-fill the rest — bringing the ratio to ~20:1 while preserving the weapon data intact.
5. Format inconsistencies. Segmentation polygons and degenerate boxes mixed into the detection labels, plus unlabelled stray images.

Training notes

• YOLO11s at 640px — a deliberate choice, not a default. Input resolution was set from the box statistics (safety-critical objects are staged close-ups, 12–25% of frame), and a small fast model is closer to what a multi-stream deployment would actually run than a leaderboard-optimized one.
• Final run on an A100: 45 epochs, batch 80, AdamW (lr0 1e-3, cosine), AMP.
• Augmentation: mosaic (closed for the last 10 epochs), mixup 0.15, copy-paste 0.3, mild realistic geometry (rotation 5°, no vertical flip / perspective — surveillance objects are upright), plus degradation augmentation (blur, downscale, JPEG artefacts, contrast) to narrow the gap to compressed CCTV footage.
• Iteration v1 → v3: v1 left backpack/handbag at 0.00 recall at conf 0.7 (hidden by aggregate mAP); v2 was a clean negative result on aggressive augmentation; v3 boosted weak-class sampling and added degradation aug, bringing every class off zero and raising precision 0.85 → 0.90. v3 is the released model.

Known weaknesses

• Confuser coverage is partial — broom, wallet, pen, fog, steam don't appear in any source dataset, so those false-positive pairs can't be quantified yet.
• Small-object recall at 0.7 — backpack/handbag/boat clear "not completely missed" but stay low; recall rises substantially at lower thresholds. Object-scale, not a training defect.
• Domain gap — training images are clean staged close-ups. On low-res, motion-blurred footage, weapon detection at 0.7 is intermittent frame-to-frame. Raising inference to imgsz=1280 recovers it, which confirms the issue is scale, not capability.
• Pseudo-label ceiling — distant / low-contrast persons are still sometimes missed.

Recommended for real deployment: per-class thresholds, imgsz=1280 in high-sensitivity zones, and temporal aggregation (alert on detections across several frames).

The model was also spot-checked on unseen real-world-style clips (motion blur, compression, wide-angle CCTV). On clear footage weapons and people are detected reliably at 0.7; on degraded footage confidence fluctuates across the threshold until inference resolution is raised to imgsz=1280 — confirming the issue is object scale, not capability.

Repository layout

weights/best.pt     released YOLO11s weights (v3)          [Git LFS]
evaluation/         training curves + confusion matrix

weights/best.pt is stored via Git LFS — run git lfs install before cloning. Datasets are not tracked.