keras/modern_bert_base_en

Model Overview

Model Overview

ModernBERT is a family of encoder-only Transformer models published by Answer.AI and LightOn. It is a modernized drop-in replacement for BERT: pre-trained on 2 trillion tokens of English and code with a native 8,192-token context window, it uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feed-forward layers. ModernBERT is intended for classification, retrieval, and embedding of text, not for text-generation. See the model card below for benchmarks, data sources, and intended use cases.

Weights are released under the Apache 2 License. Keras model code is released under the Apache 2 License.

Links

• ModernBERT Quickstart Notebook
• ModernBERT Model Card
• ModernBERT Model Paper
• KerasHub Beginner Guide
• KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Presets

The following model checkpoints are provided by the Keras team. Full code examples for each are available below.

Preset name	Parameters	Description
modern_bert_base_en	149.01M	22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses RoPE, alternating local and global attention, and GeGLU feedforward layers.
modern_bert_large_en	394.78M	28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses RoPE, alternating local and global attention, and GeGLU feedforward layers.

Example Usage

import keras
import keras_hub
import numpy as np

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]

# Pretrained classifier.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modern_bert_base_en",
    num_classes=4,
)
classifier.fit(x=features, y=labels, batch_size=2)
classifier.predict(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
classifier.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
classifier.backbone.trainable = False
# Fit again.
classifier.fit(x=features, y=labels, batch_size=2)

Preprocessed integer data.

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}
labels = [0, 3]

# Pretrained classifier without preprocessing.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modernbert_base_en",
    num_classes=4,
    preprocessor=None,
)
classifier.fit(x=features, y=labels, batch_size=2)

Masked language modeling.

masked_lm = keras_hub.models.ModernBertMaskedLM.from_preset("modern_bert_base_en")
tokenizer = masked_lm.preprocessor.tokenizer

text = "The capital of France is [MASK]."
token_ids = (
    [tokenizer.cls_token_id]
    + list(np.asarray(tokenizer(text)))
    + [tokenizer.sep_token_id]
)
mask_positions = [
    i for i, t in enumerate(token_ids) if t == tokenizer.mask_token_id
]
logits = masked_lm(
    {
        "token_ids": np.array([token_ids], dtype="int32"),
        "padding_mask": np.ones((1, len(token_ids)), dtype="bool"),
        "mask_positions": np.array([mask_positions], dtype="int32"),
    }
)
top5 = np.argsort(-np.asarray(logits)[0, 0])[:5]
print([tokenizer.detokenize([int(t)]).strip() for t in top5])

Example Usage with Hugging Face URI

import keras
import keras_hub
import numpy as np

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]

# Pretrained classifier.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "hf://keras/modern_bert_base_en",
    num_classes=4,
)
classifier.fit(x=features, y=labels, batch_size=2)
classifier.predict(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
classifier.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
classifier.backbone.trainable = False
# Fit again.
classifier.fit(x=features, y=labels, batch_size=2)

Preprocessed integer data.

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}
labels = [0, 3]

# Pretrained classifier without preprocessing.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modernbert_base_en",
    num_classes=4,
    preprocessor=None,
)
classifier.fit(x=features, y=labels, batch_size=2)

Masked language modeling.

masked_lm = keras_hub.models.ModernBertMaskedLM.from_preset("hf://keras/modern_bert_base_en")
tokenizer = masked_lm.preprocessor.tokenizer

text = "The capital of France is [MASK]."
token_ids = (
    [tokenizer.cls_token_id]
    + list(np.asarray(tokenizer(text)))
    + [tokenizer.sep_token_id]
)
mask_positions = [
    i for i, t in enumerate(token_ids) if t == tokenizer.mask_token_id
]
logits = masked_lm(
    {
        "token_ids": np.array([token_ids], dtype="int32"),
        "padding_mask": np.ones((1, len(token_ids)), dtype="bool"),
        "mask_positions": np.array([mask_positions], dtype="int32"),
    }
)
top5 = np.argsort(-np.asarray(logits)[0, 0])[:5]
print([tokenizer.detokenize([int(t)]).strip() for t in top5])