DetailNet r2

A small recurrent network (about 1M parameters) that adds fine texture on top of the output of Apple's on-device video super-resolution, VideoToolbox's VTSuperResolutionScaler. It's for upscaling 720p video to 4K (×3) on Apple silicon. It was trained entirely on a MacBook Pro (M5 Max, 64 GB).

Research release. By eye, r2 is not a visible improvement over Apple AI. On our metrics it is as steady as Apple's output, handles scene cuts more cleanly, and is closer to ground truth on LPIPS. But the extra fine detail it adds (1.13× Apple's) is too small to see, and it missed its own goal of 1.5×. It's published so others can build on the approach, not as a better upscaler to use today.

• Code, benchmark and training pipeline: https://github.com/tsonubin/metafx-upscale
• License: Apache-2.0

How it works

For each frame, at source resolution (for example 1280×720), DetailNet takes:

• x_t, x_prev: the current and previous source frames (RGB, 0–1);
• base: Apple AI's output for the frame, resized to exactly 3× the source (RGB, 0–1);
• flow4: VideoToolbox optical flow between x_prev and x_t, in source pixels, with channels back_x, back_y, fwd_x, fwd_y;
• its own previous texture and hidden state, warped along the flow so the invented texture stays locked to the scene.

It returns detail, a zero-mean, high-pass texture layer at 3× resolution. The final frame is base + strength · detail, so the model can't change brightness, color or shapes. Its state is reset at scene cuts and wherever the flow is invalid.

Architecture	conv3×3 → 12 residual blocks (64 channels) → texture head (pixel shuffle ×3, high-pass) + hidden-state head (32 channels, tanh)
Parameters	972,795 (3.9 MB, float32)
Scale	×3 only (720p → 4K). It needs Apple AI's base, so it runs on macOS 26+
Cost	About 0.03–0.04 s per frame on MPS for a 704×424 source crop, on top of Apple AI's ≈0.35 s per frame

Results

These are go / no-go gates on the project's benchmark (12 clips from broadcast nature documentaries), compared with apple-base, Apple AI's output before compression.

• Detail and jitter are ratios to Apple AI, measured on five real 720p → 4K clips. Jitter is motion-compensated frame-to-frame change; lower is steadier.
• LPIPS is measured on five ground-truth clips: real 1080p frames shrunk to 640×360, upscaled ×3 and compared with the original. Lower is better.

Gate	Target	r2	Apple AI
Fine detail	≥ 1.5×	1.13× (fail)	1.00×
Jitter, all motion	≤ 1.1×	1.00×	1.00×
Jitter static / slow / fast	≤ 1.1× each	0.94 / 0.97 / 1.07	1.00
Pulse (4-frame shimmer)	≤ 1.1	1.01	n/a
Detail leaking across a cut	≤ Apple's	0.14	0.52
LPIPS vs real 1080p (×3)	≤ Apple's	0.252	0.317

Its measured wins come mostly from smoothing away Apple's slightly painterly artifacts, which LPIPS rewards. Side by side at 1:1, the two are hard to tell apart. Scaling the texture up 3× reaches 1.59× detail, but jitter rises to 1.28×, so the texture isn't locked well enough to amplify.

Usage

from safetensors.torch import load_file
from detailnet.model import DetailNet, compose   # from the GitHub repo

net = DetailNet(scale=3, channels=64, blocks=12, hidden=32).eval()
net.load_state_dict(load_file("model.safetensors"))

prev_detail, hidden = net.zero_state(1, h, w, "cpu")        # reset at the first frame and at every cut
detail, hidden = net(x_t, x_prev, base, flow4, prev_detail, hidden)
frame = compose(base, detail, strength=1.0)                  # base + strength * detail, clamped to [0, 1]
prev_detail = detail

Producing the inputs:

• The repo's Swift CLI writes x_t, base and flow4 as frame dumps: metalfx-upscale input.mp4 out.mp4 --engine ml --size 3x --dump-source S --dump-base B --dump-flow F.
• The exact reset and flow-cleaning rules are in detailnet/run_bench.py and detailnet/train.py (run_sequence).
• There is no end-to-end "upscale my video with DetailNet" tool yet. The repo runs it over benchmark clips; putting it into the upscaler and app was planned as the next milestone.

Training

• Data: about 1,100 short sequences (720p and 1080p) cut from broadcast nature documentaries (ABC, Australian Broadcasting Corporation). The footage is copyrighted and isn't distributed; only the trained weights are released.
• Teacher: FlashVSR v1.1, a one-step streaming diffusion super-resolution model (Apache-2.0). We ported it to Apple's MPS and ran it offline to make texture targets.
• Losses:
  • perceptual (LPIPS / DISTS) plus L1, against the teacher;
  • a temporal lock term that penalizes texture changes not explained by optical flow;
  • a ping-pong consistency term.
• Schedule:
  • 14,000 iterations, batch 4, 160×90 source crops, learning rate 2e-4 with a cosine schedule, seed 1;
  • initialised from r1's checkpoint at iteration 18,000;
  • about 10 hours per round on the M5 Max.

Limitations

• Not a visible improvement over Apple AI (see above). Treat it as a baseline for further work, for example an adversarial loss or a stronger temporal lock.
• Hard dependencies: ×3 only, and it needs Apple AI's output and VideoToolbox optical flow, so macOS 26+ on Apple silicon.
• Narrow training data: one footage domain only (nature documentaries) and SDR only.

Attribution

• Teacher: FlashVSR v1.1 (Apache-2.0).
• Base upscaler: Apple VideoToolbox VTSuperResolutionScaler, which runs on the user's Mac; nothing from Apple is redistributed.
• Footage: used for training only, not distributed.