ModernBERT-HU-4k

ModernBERT-HU-4k is a Hungarian encoder pretrained from scratch with the ModernBERT architecture, a purpose-built Hungarian BPE tokenizer, and a 4,096-token context window. It was introduced in Detecting Toxic Communication in Hungarian: Theory, Corpus, and Thread-Conditioned Classification (WOAH 2026) as the backbone for thread-level toxicity classification, where whole comment threads must fit in one forward pass. Prior Hungarian encoders such as huBERT stop at 512 tokens.

The fine-tuned toxicity classifier built on this encoder is ModernBERT-HU-4k-thread-toxic.

Model details

Architecture	ModernBERT-base (22 layers, hidden size 768, RoPE, alternating local/global attention, GeGLU)
Parameters	149M
Context length	4,096 tokens (trained)
Vocabulary	50,389 (Hungarian BPE, trained from scratch)
Objective	Masked language modelling, 30% mask ratio
Initialisation	Random (not a fine-tune of answerdotai/ModernBERT-base)

Tokenizer

The BPE tokenizer was trained from scratch on the pretraining corpus. Fertility (subword tokens per alphabetic word, measured on the 61M-word silver toxicity corpus with URLs, numerals and punctuation-only tokens excluded):

Tokenizer	Fertility
ModernBERT-HU-4k	1.52
mBERT	2.23
ModernBERT-base (English)	2.87

Hungarian morphological aggression markers (T/V address shifts, sarcastic diminutive suffixes, euphemised taboo forms) stay single tokens instead of being split across subwords.

Training data

About 4.4B tokens from five sources:

Source	Description
FineWeb2-HQ (Hungarian)	Quality-filtered web text, broad lexical and topical coverage
Europarl (Hungarian)	Formal register, policy and legislative domain
conversational-pretrain	Unlabelled comment threads from Reddit, YouTube, TikTok, Kuruc.info and Mandiner.hu (an earlier crawl that predates the gold annotation set)
toxic_oscar_hu	OSCAR-derived subset that keeps toxic-register text, so the encoder learns to represent it rather than suppress it
ParlaMint-HU 2014-2023	Hungarian parliamentary proceedings, formal political register

Training procedure

The ModernBERT pretraining recipe was followed, changing only the context length schedule:

1. Stage 1, 1k context: the full corpus, 1,024-token sequences, 1 epoch (~4.4B tokens, 219,621 steps).
2. Stage 2, 4k context: the conversational corpus repacked into 4,096-token sequences for continued MLM pretraining, 1 epoch.

All training ran on a single NVIDIA RTX 5070 Ti (16 GB).

Evaluation

HuLU benchmark

Task	ACC	MCC	F1
HuCOLA	.885	.648	.823
HuCOPA	.690	.380	.690
HuSST	.701	.416	.606
HuRTE	.634	.258	.629
HuWNLI	.617	.237	.536
HuCommitmentBank	.621	.453	.624
Overall	.691	.398	.651

ModernBERT-HU-4k leads all compared models on HuWNLI and is within 6 to 8 points overall of Racka-4B (27x larger; .751 / .502 / .730) and PULI-LlumiX-Llama-3.1 Chat (54x larger), without instruction tuning. HuLU does not test long-context ability.

Masked-token prediction

2,000 held-out samples from conversational-pretrain, random masking:

Mask ratio	Top-1	Top-5	Masked tokens
15%	54.76%	73.43%	12,184
30%	48.15%	66.96%	25,078

Downstream: thread-level toxicity

Fine-tuned on the silver toxicity corpus, the model reaches macro F1 0.811 (face-aggression, T1) and 0.836 (norm violation, T3) on a 485-item human gold set, statistically tied with the best zero-shot frontier LLMs. See ModernBERT-HU-4k-thread-toxic.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "magyar-nlp-szine-java/ModernBERT-HU-4k"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo)

text = "A magyar [MASK] az egyik legszebb nyelv Európában."
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

mask_idx = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
top5 = torch.topk(logits[0, mask_idx], 5).indices[0]
print([tokenizer.decode(t) for t in top5])

The encoder is meant for fine-tuning on Hungarian downstream tasks (classification, NER, sentence-pair tasks), especially where inputs are longer than 512 tokens.

Limitations

• Trained on Hungarian only; not suitable for other languages.
• One epoch over ~4.4B tokens; further pretraining would likely help.
• The pretraining data deliberately includes toxic and politically charged text. The model has not been audited for social bias, and its representations should not be assumed neutral.

Citation

Proceedings details (pages, anthology link) will be added once the WOAH 2026 proceedings are published.

@inproceedings{hatvani-madarasz-2026-detecting,
  title     = {Detecting Toxic Communication in {H}ungarian: Theory, Corpus, and Thread-Conditioned Classification},
  author    = {Hatvani, P{\'e}ter and Madar{\'a}sz, G{\'a}bor},
  booktitle = {Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH 2026)},
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

License

Apache 2.0