nuess/modernbert-base-swedish-chars

ModernBERT Base Swedish Characters

Swedish character-BPE continued pretraining of AI-Sweden-Models/ModernBERT-base, source revision 0bf738233da3c1bd32bc88cdbeb2c2c01dbac0eb.

This repository contains the final encoder backbone and matching tokenizer. The trained auxiliary MLM head is supplied separately as mlm_head.pt, together with input_vocab.json. It is not a standard AutoModelForMaskedLM checkpoint. Optimizer checkpoints and training data are not included.

Load The Encoder

Use Transformers 5.22.1 and tokenizers 0.23.3, the versions used for export. The CPU/eager path does not require FlashAttention or Hugging Face kernels.

import torch
from transformers import AutoModel, AutoTokenizer

repo = "nuess/modernbert-base-swedish-chars"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, attn_implementation="eager").eval()
batch = tokenizer(
    "Det h\u00e4r \u00e4r en svensk text.",
    return_tensors="pt",
    return_token_type_ids=False,
)
with torch.inference_mode():
    hidden = model(**batch).last_hidden_state

The repository is initially private; an authorized Hugging Face token is required. Always use this tokenizer, not the original ModernBERT vocabulary.

Tokenizer

NFC normalization and the original ByteLevel pre-tokenizer, with 1,542 IDs:

Class	Count
Single-byte ASCII characters	128
High-byte fallback	128
Complete multibyte characters	201
Space-prefixed single-character word starts	206
Retained separator/symbol runs	842
Intermediate UTF-8 prefix fragments	30
Special tokens	7

There are no ordinary multi-letter word pieces or duplicate payload byte strings. Character formation precedes word-start and separator merges. Initially, all token embeddings came from the original embeddings: 1,386 exact copies and 156 means of source-piece rows.

Frozen tokenizer SHA256: bd22369c1da9618b2333bc3f7576858b98ba511fd7b09dd6128f9013a0435012. The original tokenizer JSON and initialization metadata accompany the model.

Training And Evaluation

• 138,332 optimizer updates, batch 256, sequence length 512.
• 35,412,992 training sequences; document-hash-disjoint held-out partition.
• Peak LR 1e-4, 13,834 warmup updates, then linear decay to zero.
• AMP bf16 with FP32 parameters; gradient clipping at 1.0.
• Final held-out MLM loss: 1.022679414907949.
• Final masked-token accuracy: 0.7214140752038926, on 4,096 fixed eval chunks.

These are character-BPE MLM metrics, not byte-level or downstream compound/morpheme-boundary scores. Longer contexts than the 512-token training geometry and downstream task quality have not been evaluated here.

The base model's licensing and provenance remain relevant; see its model card.