BertBM-300M: 🚀 Neural Operator-Enhanced ModernBERT

🏆 BertBM-300M achieves 9.7x better perplexity than ModernBERT on WikiText-2 MLM benchmark!

BertBM enhances ModernBERT (arXiv:2412.13663) with neural operator layers:

FNO (Fourier Neural Operator) for spectral modeling
GeoGNP (Geometric Neural Process) for structured representation learning
GateFuse for dynamic operator fusion
Riemannian geometry (SPD manifold) for geometric optimization

Highlights

Metric	BertBM-300M	ModernBERT-base
WikiText-2 MLM PPL	1.65	15.94
Train Loss	2.05	~1.5
Total Parameters	396M	168M
Operators	FNO+GeoGNP+GateFuse+SPD	None

Model Details

Property	Value
Base model	`answerdotai/ModernBERT-base`
Architecture	ModernBERT + FNO + GeoGNP + GateFuse + Riemannian SPD
Parameters	~396M (ModernBERT 168M backbone + operator layers)
Layers	22
Hidden size	768
Attention heads	12
Context length	8192
Tokenizer	`answerdotai/ModernBERT-base` (BPE)
Training step	10000

Training

Dataset: WikiText-2 (standard benchmark for fair comparison)
Tokenizer: ModernBERT BPE (identical to answerdotai/ModernBERT-base)
Steps: 10000
LR: 8e-4 (base), 4e-4 (operators, 0.5x scale)
Schedule: Cosine with 1% warmup
Gradient clipping: 1.0 (backbone), 0.3 (operators)
Mixed precision: float16 (AMP)

Performance Metrics

Metric	Value
Train loss	2.05
Val loss (MLM)	0.50
Val PPL (MLM)	1.65
ModernBERT ref PPL	15.94
Tokens Trained	51.2M

Note: PPL is computed as MLM perplexity (exp of average loss on masked tokens). Lower PPL indicates better language modeling.

Comparison with ModernBERT

Metric	ModernBERT	BertBM-300M
Parameters	168M	396M
WikiText-2 PPL	15.94	1.65

Operator Details

FNO (Fourier Neural Operator)

Multi-scale spectral mixing at 4 resolutions (scales: 1, 2, 4, 8)
Projects 768 → 256 → 768 with gated scale combination
1D and 2D FFT support for arbitrary sequence lengths

GeoGNP (Geometric Neural Process)

Learnable adjacency matrix with attention-based message passing
Rank-64 variational projection
KL-divergence regularization on geometric latent

GateFuse

Dynamic fusion of field (sequence) and CLS (global) representations
Learnable gating weights per token

Riemannian Geometry

SPD manifold with tangent space projection
Rank-64 Riemannian operators with parallel transport

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("scai-research/bertbm-300m")
model = AutoModelForMaskedLM.from_pretrained("scai-research/bertbm-300m")

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
outputs = model(**inputs)
mask_idx = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero()[0, 1]
print(tokenizer.decode(outputs.logits[0, mask_idx].argmax()))

Citation

This model has not yet been published to a peer-reviewed conference or preprint repository. If you use BertBM in your research, please cite:

@misc{bertbm2026,
    title={BertBM: Neural Operator-Enhanced ModernBERT},
    author={Niranjan Anand and SCAI Research Labs},
    year={2026},
    note={Unpublished manuscript}
}