iuf26/spike-driven-transformer-satellite-streak-segmentation

SDT-SatStreak

A spiking neural network for pixel-level segmentation of satellite and space-debris streaks in wide-field astronomical images.

Model released with Energy-Efficient Spike-driven Transformer for Satellite-Streak Segmentation in Space Surveillance (Iulia Maria Istrate, 2026).

SDT-SatStreak pairs a Spike-Driven Transformer V3 (SDT-V3) 10M backbone with a quantised FPN neck and head. It runs 4 internal timesteps per forward pass with latency-coded input, and is designed for label-free cross-sensor deployment: it is trained once on one telescope's labelled data and applied to images from a different instrument without target-sensor annotations.

Model details

This model was trained on a single NVIDIA RTX 5070 Ti 16GB VRAM.

Architecture	SDT-V3 10M backbone + QFPN neck + QFPNHead
Parameters	11.5 M
Classes	2 (background, streak)
Timesteps (T)	4
Input encoding	Latency coding
Input tile	512 x 512, stride 384
Framework	PyTorch (no mmseg / mmcv / custom CUDA)

The backbone is initialised from an ImageNet-pretrained SDT-V3 10M checkpoint, then fine-tuned on 512x512 crops of the MeerLicht streak dataset with knowledge distillation from a U-Net ResNet-34 teacher. Training used AdamW, batch size 8, mixed precision, and a loss combining a focal term with a Dice term (the foreground class covers under 0.5% of pixels).

Inference pipeline

The released weights are one component of the pipeline reported in the paper:

1. Histogram matching (cross-sensor only) — per-channel intensity CDF of the input is mapped onto the source-sensor reference shipped as asta_reference_cdf.npy.
2. Tiling — the field is split into overlapping 512x512 tiles, stride 384.
3. Gamma perturbation + latency encoding — each tile is encoded four times under gamma in {0.7, 0.9, 1.0, 1.2}.
4. Temporal Spike Coherence (TSC) — the four probability maps are fused as p * (0.5 + 0.5 * coherence), where coherence is 1 - normalised std across the gamma passes. Real streaks stay stable under intensity perturbation; sensor noise flickers and is suppressed.
5. Thresholding at 0.5, then connected components with a 16-pixel minimum.

Usage

pip install -r requirements.txt

python predict.py --image field.png --output-dir out --histogram-match

Writes a binary mask, an overlay, and bounding boxes as JSON.

From Python:

from huggingface_hub import snapshot_download
import sys

repo = snapshot_download("iuf26/spike-driven-transformer-satellite-streak-segmentation")
sys.path.insert(0, repo)

from pipeline import load_model, load_reference_cdf, predict

net, device = load_model(device="cuda")
ref_cdf = load_reference_cdf()          # omit for same-sensor images

result = predict(net, device, "field.png", ref_cdf=ref_cdf)
result["mask"]         # [H, W] uint8 binary mask
result["probability"]  # [H, W] fused streak probability
result["boxes"]        # [[x_min, y_min, x_max, y_max], ...]

Set use_tsc=False for a single-pass run (roughly 4x faster, lower F1).

Results

Cross-sensor evaluation, 363 images, box-level F1 after connected-component conversion, with histogram matching:

Setting	F1 @ IoU 0.10	F1 @ IoU 0.25	F1 @ IoU 0.50
SDT-SatStreak (no TSC)	0.324	0.230	0.139
SDT-SatStreak + TSC	0.328	0.240	0.176

On the source-domain validation split, pixel-level F1 at threshold 0.5 is 0.564 (a U-Net ResNet-34 baseline reaches 0.613 in-domain, but its cross-sensor F1 collapses to 0.099 at IoU 0.50 — precision stays at 0.729 while recall falls to 0.186).

Energy per inference: 1878 +/- 20 mJ measured on an RTX 5070 Ti. The deployment argument for a spiking model rests on neuromorphic hardware — a published Akida AKD1000 figure for a comparable workload is 0.63-1.38 mJ per frame.

Input requirements

Format	Anything Pillow reads (PNG, JPEG, TIFF, BMP), or an [H, W, 3] uint8 NumPy array
Bit depth	8-bit per channel only. Higher bit depths are rejected — see below
Channels	RGB. Grayscale and RGBA are converted automatically; grayscale is replicated across all three channels
Value range	0-255. ImageNet normalisation is applied internally, so pass raw pixel values
Size	Any. Images smaller than the 512-pixel tile are edge-padded; there is no upper limit beyond your memory
Colour	The model was trained on grayscale-like astronomical frames stored as RGB. Colour information is not used meaningfully

8-bit only — this matters for FITS data

Astronomical frames are usually 16-bit or float. Do not hand those to Pillow and hope for the best: converting a 16-bit image to RGB clips every value above 255 to 255, so a frame whose stars sit at 30 000 ADU becomes a flat white field and every faint streak disappears. prepare_image() raises a ValueError rather than letting this happen silently.

Apply your own stretch first. A percentile clip is usually enough:

import numpy as np

def stretch_to_uint8(data, lo=1.0, hi=99.5):
    p_lo, p_hi = np.percentile(data, [lo, hi])
    scaled = np.clip((data - p_lo) / max(p_hi - p_lo, 1e-9), 0, 1)
    return np.repeat((scaled * 255).astype(np.uint8)[..., None], 3, axis=2)

The stretch you choose affects results. The model was trained on frames whose intensity distribution the bundled reference CDF describes, so aim for something comparable and keep --histogram-match on for cross-sensor work.

Size and scale

Tiling handles any resolution, but angular scale is not free. The model was trained on 512x512 crops at one pixel scale, and a streak far wider or thinner in pixels than the training distribution is out of distribution. If your instrument has a very different plate scale, resample toward the training scale rather than relying on tiling.

Images smaller than 512 pixels are padded up to one tile. That works, but a tile that is mostly padding gives the model little context, so small cut-outs perform worse than full fields.

Limitations

• Faint streaks are the main failure mode. Low-SNR trails are often only partially segmented or missed entirely.
• Precision is low in the cross-sensor setting (roughly 0.23-0.25 at IoU 0.10). The model over-produces candidates; expect to post-filter.
• Histogram matching assumes the target sensor's intensity distribution can be usefully mapped onto the source reference. A very different instrument may need its own reference CDF.
• Trained on 512x512 crops at one pixel scale. Streaks at very different angular scales are out of distribution.
• Not validated for operational conjunction assessment or any safety-critical decision.

Files

File	Purpose
SDT-SatStreak.pth	Model weights (state dict only, 44 MB)
model.py	Architecture: backbone, QFPN neck, QFPNHead
encoding.py	Latency / local-latency input encoding
pipeline.py	Histogram matching, tiling, TSC fusion, inference API
predict.py	Command-line entry point
config.json	Architecture and inference hyperparameters
asta_reference_cdf.npy	Source-sensor reference CDF for histogram matching

Citation

@article{istrate2026sdtsatstreak,
  title  = {Energy-Efficient Spike-driven Transformer for Satellite-Streak
            Segmentation in Space Surveillance},
  author = {Istrate, Iulia Maria},
  year   = {2026}
}