|
--- |
|
language: |
|
- en |
|
pipeline_tag: fill-mask |
|
--- |
|
|
|
## How to run: |
|
Install these requirements |
|
``` |
|
pip install flash_attn |
|
pip install transformers==4.45.2 # (probably works with newer/older but tested with >=4.45.2) |
|
``` |
|
Then you can load with |
|
``` |
|
from transformers import AutoModel, AutoTokenizer |
|
import torch |
|
|
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
print(f"Using device: {device}") |
|
|
|
model = AutoModel.from_pretrained("ModernBERT/bert24-base-v2-2ep-decay_100B-0.08-lr", trust_remote_code=True) |
|
model = model.to(device) |
|
tokenizer = AutoTokenizer.from_pretrained("ModernBERT/bert24-base-v2-2ep-decay_100B-0.08-lr", trust_remote_code=True) |
|
|
|
# test it out and encode some text |
|
text = "This is a test sentence" |
|
inputs = tokenizer(text, return_tensors="pt") |
|
inputs = {k: v.to(device) for k, v in inputs.items()} |
|
|
|
outputs = model(**inputs) |
|
last_hidden_states = outputs.last_hidden_state |
|
print(last_hidden_states.shape) |
|
``` |
