mushyfruit/anycalib-nuke

AnyCalib for Nuke

The network behind AnyCalib for Nuke, a Nuke 17 node that estimates lens distortion from the plate. It's AnyCalib's anycalib_dist (DINOv2 ViT-L/14), fine-tuned with LoRA on renders through Nuke's lens model and on real footage.

Files

• anycalib_dist.pt: the fine-tuned network, the one the node uses. TorchScript for the libtorch Nuke ships (PyTorch 2.7.1), fp32, 204,800 px input. SHA-256 7b99170b135f484ac98bc27c16149f8265014532c1d725e919a3650445ae3826, the hash the node writes into its metadata.
• anycalib_dist_stock.pt: AnyCalib's own weights, exported the same way at its 102,400 px input, to compare against in the node (model override).
• lora/adapter.safetensors: the fine-tune as a LoRA on AnyCalib's anycalib_dist weights, for use in Python. Rank 32 on every backbone linear, plus the decoder and head, which were trained in full.

Use in Nuke

Unpack the node from the GitHub release and download the network into its models folder:

curl -L -o anycalib/models/anycalib_dist.pt \
    https://huggingface.co/mushyfruit/anycalib-nuke/resolve/main/anycalib_dist.pt

Use in Python

from anycalib import AnyCalib
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

model = AnyCalib(model_id="anycalib_dist")
lora = load_file(hf_hub_download("mushyfruit/anycalib-nuke", "lora/adapter.safetensors"))
sd = model.state_dict()
for k, v in lora.items():
    if k.endswith(".lora_a"):
        n = k.removesuffix(".lora_a")
        sd[n + ".weight"] += 2 * lora[n + ".lora_b"] @ v
    elif not k.endswith(".lora_b"):
        sd[k] = v
model.load_state_dict(sd)
model.RESOLUTION = 204_800  # trained at twice AnyCalib's input size

pred = model.eval().predict(image, cam_id="simple_division:2")

simple_division:2 is Nuke's lens model with the distortion normalised by the focal length; the node converts it to distortion1, distortion2 and distortionCenter.

Training

• Synthetic: Poly Haven HDRIs rendered through NukeX Classic distortion and the node's colour path, over a cine-weighted range of focal lengths and lenses, with portrait formats and anamorphic lenses.
• Real: InFlux training videos, with a loss on how straight long edges come out. No ground truth is used.
• LoRA on every backbone linear, with the decoder and head trained, at twice AnyCalib's input resolution.

Results

Mean undistortion error on 341 plates of real cine footage from 13 InFlux locations held out of training. They were shot on the same camera and zooms as the InFlux training footage, so this flatters the fine-tune; on megadepth2k-radial (2,000 photos, many cameras) it still clearly beats stock AnyCalib.

	px
No distortion	8.34
Stock AnyCalib	9.97
This network	5.81
This network + the node's line refinement	5.36

Limitations

• Spherical lenses only; anamorphic squeeze isn't estimated.
• Fisheyes are out of reach.
• On spherical primes with little distortion, assuming none is still often closer.

Credits

• AnyCalib, Javier Tirado-Garín and Javier Civera, Apache 2.0.
• DINOv2, Meta, Apache 2.0.
• InFlux, Princeton, CC BY 4.0.
• Poly Haven, CC0.