Prism Norwegian (prism-no)

Norwegian UPOS tagging, morphological features, and lemmatization with calibrated confidences, built for on-device, fully offline use. One compact model (17.6 M parameters) covers both written standards — Bokmål (nb) and Nynorsk (nn) — in a single set of weights; mixed input is fine.

It beats UDPipe 2.17 on UPOS and lemmas on the official UD test splits — at about one twentieth of UDPipe's model size (fast artifact; one tenth for fp32), running fully offline on a laptop CPU.

This repository mirrors the versioned release artifacts of the Prism project. Prism ships native runtimes for Swift, C++, C, and Java/Kotlin that read these artifacts directly.

Which folder to use

Folder	Size	When to use
prism-no-0.2.2-fast/	≈ 45 MB	Recommended. int8; up to 2× faster, development-split quality within 0.014 pp of fp32
prism-no-0.2.2/	≈ 94 MB	Bit-exact fp32 reference behind the published benchmark

An application bundles exactly one folder. The folder is everything a Prism runtime needs; point the tagger API at its local path:

let tagger = try PrismTagger(artifactURL: artifactFolder, device: .cpu)  // Swift

prism::tagger::Tagger tagger("prism-no-0.2.2-fast");                     // C++

try (var tagger = PrismTagger.load(Path.of("prism-no-0.2.2-fast"))) {}   // Java

Quick starts for every binding: the project README. The artifact contract (programs, model.ptd weights, tokenizer and label schemas, checksums) is documented in docs/INTEGRATION.md.

Note: these are ExecuTorch programs with the decoding policy and calibration baked in — not transformers-loadable checkpoints. vocabulary.json is a standard Hugging Face tokenizer.json and loads with the tokenizers library.

Quality

Evaluated exactly once on the untouched official UD test splits against UDPipe 2.17 (gold tokenization, official CoNLL definitions):

Test F1	Prism	UDPipe 2.17
Bokmål UPOS	98.76%	98.57%
Bokmål Lemmas	98.98%	98.87%
Bokmål UFeats	97.20%	97.59%
Nynorsk UPOS	98.77%	98.60%
Nynorsk Lemmas	98.68%	98.56%
Nynorsk UFeats	96.94%	97.38%

Prism wins UPOS and lemmas on both written standards and stays behind only on exact morphology bundles — from a model a twentieth of UDPipe's size.

fast versus fp32

The frozen test evaluation above is fp32; the fast artifact is quality-gated on the development split (67,619 tokens across both standards — the test splits are evaluated exactly once and stay reserved for the fp32 benchmark). Accuracy with the identical production decoding policy:

Task	Standard	fp32	fast	Delta
UPOS	nb	99.1724%	99.1641%	-0.0082 pp
UPOS	nn	98.8384%	98.8448%	+0.0064 pp
UFeats exact	nb	97.9021%	97.8883%	-0.0137 pp
UFeats exact	nn	95.3408%	95.3312%	-0.0096 pp
Lemma	nb	99.2301%	99.2246%	-0.0055 pp
Lemma	nn	98.8672%	98.8608%	-0.0064 pp

Every delta is at most 0.014 percentage points — an order of magnitude below seed-to-seed training variance. int8 costs no measurable quality.

Speed

End-to-end on a book chapter (247 sentences / 3,783 tokens), raw text in, tagged sentences out; warm run, Apple M4 Max, CPU only, six threads (the built-in default). Wall-clock with tokens per second:

Variant	Swift	C++	Java
fp32	1.5 s (2,500/s)	2.2 s (1,731/s)	2.3 s (1,676/s)
fast (int8)	1.5 s (2,541/s)	1.2 s (3,185/s)	1.2 s (3,228/s)

(The Python reference runtime runs the checkpoint eagerly at 1.6 s / 2,328 tokens/s; the ExecuTorch Python wheel ships no quantized kernels, so fast is native-only.)

Full tables, reproduction commands, and readings: docs/benchmarks/prism-no-0.2.2.md.

Model description

A 17.6 M-parameter encoder student (16-layer NorBERT4-xsmall backbone, hidden 192) distilled from a NorBERT4-large teacher, with a character CNN feeding morphology and lemma heads, a structured morphology decoder, and per-head temperature calibration (UPOS ECE 0.0017). Trained on the official UD gold treebanks plus teacher-labeled silver text. Full technical reference: docs/ARCHITECTURE.md.

Training data and attribution

This model exists thanks to openly licensed Norwegian resources:

• UD Norwegian-Bokmaal and UD Norwegian-Nynorsk treebanks (Universal Dependencies contributors, based on the Norwegian Dependency Treebank by the National Library of Norway) — CC BY-SA 4.0
• NBdigital (sbr-43) and municipal documents (sbr-60), National Library of Norway, Språkbanken — CC0
• Nynorsk Wikipedia, Wikimedia contributors — CC BY-SA 4.0 (text never redistributed)
• Backbone: ltg/norbert4-xsmall; distillation teacher and silver labeler: ltg/norbert4-large (Language Technology Group, University of Oslo) — Apache 2.0

Pinned revisions and checksums travel inside each artifact (manifest.json, LICENSES/).

License

Model weights: CC BY-SA 4.0. Using or bundling the unmodified artifact — including commercially, in closed-source applications — is fine (keep the attribution); redistributed modified weights must stay open. Prism source code is Apache 2.0.