EBFM-DCLM-1B

OpenEBFM 1B model (variant d26) trained on DCLM. This repository contains a Hugging Face Transformers-compatible export for the pretraining checkpoint.

How to Use

Install the runtime dependencies:

pip install torch transformers tiktoken

Load the tokenizer and model with the custom OpenEBFM code enabled:

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "guan-wang/EBFM-DCLM-1B"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()
model.requires_grad_(False)
inputs = tokenizer("Hello world", return_tensors="pt").to(device)
outputs = model(**inputs)

print(outputs.logits.shape)

The first load may prompt you to review and allow the repository's custom model code. Loading remote Python code requires trust_remote_code=True.

Model Details

Detail	Value
Model family	OpenEBFM energy-based language model
Architecture	Custom OpenEBFM implementation (d26)
Parameter scale	1B
Training dataset	DCLM
Training stage	pretraining
Transformer blocks	26
Embedding dimension	1664
Attention heads	13
Context length	2048
Vocabulary size	32768

The parameter scale is the label used for this model in the OpenEBFM model configuration. The architecture and sequence settings above are read from the exported checkpoint configuration.

Files

• config.json: model configuration.
• model.safetensors: model weights in the standard Transformers format.
• modeling_ebfm.py: remote model and tokenizer implementation.
• configuration_ebfm.py: remote configuration implementation.
• tokenizer.pkl: serialized EBFM tokenizer.
• tokenizer_config.json: tokenizer auto-loading configuration.
• token_bytes.pt: token byte table used by OpenEBFM metrics.

The code is maintained in the OpenEBFM repository.