zeromodels/beit-base-patch16-224

See our collection for all versions of BEiT.

Run BEiT with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/beit-base-patch16-224

Paper: BEiT: BERT Pre-Training of Image Transformers (arXiv:2106.08254) · HF Papers

BEiT is a ViT-family vision transformer with a per-layer relative position bias, a learnable layer scale on each residual branch, and mean pooling of the patch tokens. Base backbone pretrained on ImageNet-22k and fine-tuned on ImageNet-1k (1000 classes).

For more details on the model, please go to Microsoft's original model card.

Pure-Keras 3 conversion of microsoft/beit-base-patch16-224 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a image classification checkpoint (BeitImageClassify).

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import keras
import numpy as np
from PIL import Image
from zeromodels.models.beit import BeitImageClassify

model = BeitImageClassify.from_weights("zeromodels/beit-base-patch16-224")

image = Image.open("your_image.jpg").convert("RGB").resize((224, 224))
pixels = np.asarray(image, "float32")[None]  # raw [0, 255]; normalization is inside the model
logits = model(pixels, training=False)
print("top-1 class id:", int(keras.ops.convert_to_numpy(logits)[0].argmax()))

Load any BEiT variant the same way with from_weights("zeromodels/<variant>"):

Variant	Hub	Task
beit-base-patch16-224	zeromodels/beit-base-patch16-224	image classification
beit-large-patch16-224	zeromodels/beit-large-patch16-224	image classification
beit-large-patch16-512	zeromodels/beit-large-patch16-512	image classification
beit-base-patch16-224-pt22k-ft22k	zeromodels/beit-base-patch16-224-pt22k-ft22k	image classification
beit-large-patch16-224-pt22k-ft22k	zeromodels/beit-large-patch16-224-pt22k-ft22k	image classification
beit-base-finetuned-ade-640-640	zeromodels/beit-base-finetuned-ade-640-640	semantic segmentation
beit-large-finetuned-ade-640-640	zeromodels/beit-large-finetuned-ade-640-640	semantic segmentation

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Normalization (0.5/0.5) is baked into the model, so pass raw [0, 255] pixels.
• Classification uses BeitImageClassify; semantic segmentation uses BeitSemanticSegment and returns logits at a quarter of the input resolution (upsample the argmax map to the input size).
• BeitModel.from_weights(..., as_backbone=True) returns the per-block token sequences for feature extraction.
• See BEiT docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. BeitImageClassify.from_weights("hf:microsoft/beit-base-patch16-224").

Special Thanks

A huge thank you to the Microsoft Research BEiT authors for creating and releasing these models.

License: Apache 2.0.