SpeedNet v4 — ego-speed estimation from onboard video

SpeedNet estimates vehicle speed directly from onboard/POV video — no GPS, no CAN bus, no sensors. Point it at a dashcam clip, a go-kart helmet-cam recording, or ATV footage, and it returns a per-frame speed curve.

It was built to answer a practical question: can we recover telemetry from footage that has none? — e.g. indoor karting, where GPS does not work.

Demo

Fastest lap of an indoor go-kart race (1000 m track), with the speed HUD generated by this project — video-model level, IMU-driven dynamics, and the official lap time anchoring the scale:

Motivation

Onboard motorsport footage is everywhere — helmet cams, chest mounts, dashcams — but the telemetry that makes it interesting usually is not. Rental karts have no data logger you can access; indoor tracks deny GPS entirely; action cameras often record IMU but no position. We wanted two things: measure speed where no sensor can, and add live telemetry overlays (speed, g-force, lap timing) to motorsport videos that were recorded without any.

Getting there meant overcoming a chain of problems, each documented in this card because they will bite anyone building something similar:

1. Monocular scale ambiguity — optical flow alone cannot distinguish "fast and far" from "slow and close". Solved with the RGB context branch, which only generalized once training spanned 7 distinct domains; with fewer, it memorized venues.
2. No indoor ground truth — GPS does not exist on an indoor kart track. Solved with weak supervision from the venue's official lap times (mean speed per lap = track length / lap time), aligned to video by detecting recurring track features.
3. Lap memorization — with only ~50 supervised laps, the GRU learned to recognize individual laps instead of estimating speed. Solved by augmenting the lap-supervision forward passes (flip, noise, occluders).
4. Sequence length as a scale cue — training with two window lengths taught the model to use context length itself as a signal; hence the two documented inference modes.
5. Compressed dynamics — regression pulls toward the mean (corners read fast, straights slow). Solved, when IMU data exists, by the physics fusion described below: measured longitudinal g-force drives the acceleration profile, centripetal physics anchors cornering speed, and track length locks the scale.

video ──► optical flow (motion field) ──┐
                                        ├──► GRU ──► speed (m/s, per frame)
video ──► low-res context frame ────────┘
          (tells the model the scene scale)

Why the two branches?

Optical flow encodes how fast pixels move — which depends on both speed and scene distance. 130 km/h on an open highway and 57 km/h between the close walls of an indoor kart track can produce similar flow magnitudes. A monocular model therefore cannot know the absolute scale from motion alone. The context branch sees one low-resolution RGB frame and learns a scene→scale mapping (indoor track / forest trail / highway), which lets a single checkpoint work across very different environments. This only started working once the training corpus spanned 7 distinct domains — with fewer domains the branch memorized venues instead of generalizing.

Quickstart

pip install torch opencv-python-headless numpy pandas

import torch
from modeling_speednet import SpeedNet, predict_video

model = SpeedNet()
model.load_state_dict(torch.load("speednet_v4.pt", map_location="cuda"))

result = predict_video(model, "onboard_clip.mp4", device="cuda")
print(f"mean speed: {result['speed_smooth_mps'].mean() * 3.6:.1f} km/h")

Or run the bundled CLI example:

python example.py my_clip.mp4     # -> my_clip_speed.csv + my_clip_speed.png

predict_video handles everything: decoding, resizing to 320x180, Farneback optical flow, frame-rate normalization, context frames, streaming GRU inference and 1-second median smoothing.

Critical usage notes

1. Pick the right inference mode. The model was trained with two window lengths, so sequence length acts as a scale cue:
   • short mode (default): hidden state reset every 16 frames — for roads, trails and open environments. Reproduces the GPS-domain metrics below.
   • long-context mode (long_context=True): hidden state carried across the clip — required for closed-course footage (indoor karting): short windows collapse indoor scale to roughly half the true value.
2. Any frame rate works — flow is normalized to a 15 fps convention internally (raw_flow * fps / 15). Feeding pre-computed flow without this normalization will mis-scale predictions.
3. Forward-facing footage. The model was trained on forward-facing onboard cameras. Rear/side cameras are out of distribution.
4. Static occluders are fine. Hoods, helmet edges and mounts were augmented during training; the model ignores zero-flow regions.

Evaluation

Held-out recordings (never trained on; the ATV test is from a different day than all ATV training data). RMSE in m/s, windowed protocol:

Test domain	RMSE (m/s)	Notes
ATV, forest trail (17 min, unseen day)	0.65
Car, highway 100–135 km/h (DK)	5.60	hardest regime
Car, urban (DK)	1.68
Car, rural/urban Denmark (DJI, unseen recording)	2.05 (zero-shot)
Car, EU driving-school fleet (L2D test split)	2.70
Car, US highway (comma2k19 test route)	2.93
Indoor go-kart, official lap times (holdout laps)	5.2 km/h MAE (long-context mode)	3.4 km/h with lap-aligned windows; vs. timing system, 1000 m track

The indoor go-kart figures are validated against the venue's race timing system: mean predicted speed per lap vs. 1000 m / official lap time, on laps excluded from training. GPS-domain rows use short mode; the indoor row uses long-context mode (5.2 km/h), improving to 3.4 km/h when lap boundaries are known and inference windows are aligned to laps.

Training data

~5 hours of labeled video across 7 domains, all with real ground truth:

Domain	Source	Speed signal
Indoor go-kart (2 races)	helmet cam, 1000 m indoor track	official lap times (weak supervision: mean speed per lap)
ATV / forest	action cam + GPS remote	10 Hz GPS
Mini-loader / grass	action cam + GPS remote	10 Hz GPS
Car / Denmark	action cam + GPS, dashcam	10 Hz + 1 Hz GPS
Car / EU	L2D (Apache-2.0)	10 Hz CAN/GNSS
Car / US	comma2k19 (MIT)	20 Hz pose velocities

Training tricks that mattered: horizontal-flip and synthetic-occluder augmentation on flow; augmenting the lap-supervision forward passes (otherwise the GRU memorizes individual laps); context dropout (15%) and photometric jitter on the context frame; per-recording train/val/test splits.

Limitations

• Protocol sensitivity: sequence length is a scale cue (an artifact of two-length training); use the mode matching your footage as described above. Removing this via length-randomized training is planned.
• Compressed dynamics within a lap/segment: like most regression models, predictions are pulled toward the mean — cornering speeds read slightly high, straight-line peaks slightly low. If your footage has IMU data, see the physics-fusion recipe below.
• Monocular scale is learned, not measured: radically new scene types (aircraft, boats, rear-facing cameras) will mis-scale.
• GPS ground truth is itself smoothed (~1 s); treat sub-second dynamics as indicative.
• Farneback flow degrades with heavy motion blur and very low light.

Advanced: physics fusion (video + IMU)

If the recording also has IMU data, you can fix the compressed dynamics by solving for the speed curve that best satisfies, jointly:

• dv/dt = a_lon — measured longitudinal acceleration (true accel/braking),
• v = a_lat / ω — centripetal anchor in corners (|yaw rate| > 30°/s),
• ∫ v dt = track_length per lap (if lap times are known),
• the video model as a weak level prior.

A few thousand Adam steps over the speed vector suffice. On our indoor race this reproduces official lap averages to 0.16 km/h with per-lap distance 997/1000 m, while restoring realistic braking-into-corner / accelerating-onto-straight dynamics.

Model details

• ~1.0 M parameters (flow CNN 4 conv blocks → 256-d; context CNN 3 blocks → 64-d; 2-layer GRU, 192 hidden; linear head).
• Input: flow [T, 2, 72, 128] normalized by 8.0; context RGB [3, 72, 128] in [-0.5, 0.5]. Output × 40.0 = m/s.
• Trained ~30 epochs, AdamW, Huber loss, cosine schedule, on a single NVIDIA GB10; checkpoint selected on combined GPS-validation + lap-holdout score.

License

Weights and code: Apache-2.0. Trained only on own recordings and permissively licensed public data (MIT / Apache-2.0).

Citation

@misc{speednet2026,
  title  = {SpeedNet v4: ego-speed estimation from onboard video},
  author = {x2q},
  year   = {2026},
  url    = {https://huggingface.co/x2q/speednet}
}