zeromodels/conditional-detr-resnet-50

Run Conditional DETR with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/conditional-detr-resnet-50

Paper: Conditional DETR for Fast Training Convergence (arXiv:2108.06152) · HF Papers

Conditional DETR keeps DETR's set-prediction recipe but decouples content and location in the decoder's cross-attention: each query predicts a 2D reference point, and the sine embedding of that point (scaled by a content-dependent transformation) forms a spatial query concatenated with the content query, so each head attends to a band around the object. It trains 6.7-10x faster than DETR.

For more details on the model, please go to the original model card.

Pure-Keras 3 conversion of microsoft/conditional-detr-resnet-50 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is an object detection checkpoint (ConditionalDETRDetect): each of the 300 queries predicts COCO class scores (sigmoid) and a box.

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.conditional_detr import ConditionalDETRDetect, ConditionalDETRImageProcessor

model = ConditionalDETRDetect.from_weights("zeromodels/conditional-detr-resnet-50")
processor = ConditionalDETRImageProcessor.from_weights("zeromodels/conditional-detr-resnet-50")

image = Image.open("your_image.jpg").convert("RGB")
inputs = processor(image)
output = model(inputs["pixel_values"], training=False)
results = processor.post_process_object_detection(
    output, threshold=0.5, target_sizes=[(image.height, image.width)]
)[0]
for score, name, box in zip(
    results["scores"], results["label_names"], results["boxes"]
):
    print(f"{name}: {float(score):.3f} {box}")

Load any Conditional DETR variant the same way with from_weights("zeromodels/<variant>") (use ConditionalDETRDetect for this repo):

Variant	Hub
conditional-detr-resnet-50	zeromodels/conditional-detr-resnet-50

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Scores are per-class sigmoids; post_process_object_detection keeps the top-100 (query, class) pairs, then applies threshold.
• The processor resizes aspect-preserving (short side 800, long side at most 1333); the model accepts any input size.
• See Conditional DETR docs and Loading Weights.
• The upstream safetensors also load directly via the hf: prefix, e.g. ConditionalDETRDetect.from_weights("hf:microsoft/conditional-detr-resnet-50").

Special Thanks

A huge thank you to the Microsoft Research Asia Conditional DETR authors for creating and releasing these models.

License: Apache 2.0.