zeromodels/deformable-detr-single-scale

Run Deformable DETR with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/deformable-detr-single-scale

Paper: Deformable DETR: Deformable Transformers for End-to-End Object Detection (arXiv:2010.04159) · HF Papers

Deformable DETR replaces dense attention with multi-scale deformable attention: each query attends to a few learned sampling points per head per feature level around a reference point. That makes a stride-8 to 64 feature pyramid affordable, improves small objects, and trains about 10x faster than DETR. Variants add iterative box refinement and a two-stage proposal scheme.

For more details on the model, please go to the original model card.

Pure-Keras 3 conversion of SenseTime/deformable-detr-single-scale for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is an object detection checkpoint (DeformableDETRDetect): each of the 300 queries predicts COCO class scores (sigmoid) and a box.

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.deformable_detr import DeformableDETRDetect, DeformableDETRImageProcessor

model = DeformableDETRDetect.from_weights("zeromodels/deformable-detr-single-scale")
processor = DeformableDETRImageProcessor.from_weights("zeromodels/deformable-detr-single-scale")

image = Image.open("your_image.jpg").convert("RGB")
inputs = processor(image)
output = model(inputs["pixel_values"], training=False)
results = processor.post_process_object_detection(
    output, threshold=0.5, target_sizes=[(image.height, image.width)]
)[0]
for score, name, box in zip(
    results["scores"], results["label_names"], results["boxes"]
):
    print(f"{name}: {float(score):.3f} {box}")

Load any Deformable DETR variant the same way with from_weights("zeromodels/<variant>") (use DeformableDETRDetect for this repo):

Variant	Hub
deformable-detr	zeromodels/deformable-detr
deformable-detr-single-scale	zeromodels/deformable-detr-single-scale
deformable-detr-with-box-refine	zeromodels/deformable-detr-with-box-refine
deformable-detr-with-box-refine-two-stage	zeromodels/deformable-detr-with-box-refine-two-stage

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Scores are per-class sigmoids; post_process_object_detection keeps the top-100 (query, class) pairs, then applies threshold.
• The processor resizes aspect-preserving (short side 800, long side at most 1333); the model accepts any input size.
• See Deformable DETR docs and Loading Weights.
• The upstream safetensors also load directly via the hf: prefix, e.g. DeformableDETRDetect.from_weights("hf:SenseTime/deformable-detr-single-scale").

Special Thanks

A huge thank you to the SenseTime Research Deformable DETR authors for creating and releasing these models.

License: Apache 2.0.