UltraBERT-150M-clairic-extraction

A 150M-parameter extractor for Clairic-style requests: give it a text and a set of fields, each described by a short instruction, and it returns the matching spans for each field (one answer, a list, or nothing when the text has no answer). Fields can be questions ("What is the total due?") or entity types ("Every company mentioned in the text"), so one model covers extractive question answering and named-entity recognition.

UltraBERT-150M trained on the extraction config of tasksource-jev-typed-decisions: extractive QA and span-tagging datasets from tasksource, rewritten as (instruction, text, spans) pairs. The model tags the answer tokens of the text (B/I/O) conditioned on the instruction. Inputs up to 2048 tokens.

Not affiliated with Cloudflare; "Clairic-style" refers to the request format.

Try it in your browser: the int8 ONNX export below, run client-side with onnxruntime-web.

Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

name = "tasksource/UltraBERT-150M-clairic-extraction"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name, trust_remote_code=True).eval()

def extract(text, fields):
    """fields: {name: instruction}. Returns {name: [extracted spans]}, [] when the text has no answer."""
    enc = tok(list(fields.values()), [text] * len(fields), truncation="only_second", max_length=2048,
              padding=True, return_offsets_mapping=True, return_tensors="pt")
    offsets = enc.pop("offset_mapping")
    with torch.no_grad():
        tags = model(**enc).logits.argmax(-1).tolist()  # 0 = O, 1 = B, 2 = I
    out = {}
    for f, field in enumerate(fields):
        spans, start, seq = [], None, enc.sequence_ids(f) + [None]
        for i, tag in enumerate(tags[f] + [0]):
            if start is not None and (seq[i] != 1 or tag != 2):
                spans.append(text[offsets[f][start][0]:offsets[f][i - 1][1]].strip().rstrip(",;:"))
                start = None
            if seq[i] == 1 and tag in (1, 2) and start is None:
                start = i
        out[field] = spans
    return out

text = "Invoice INV-2291 from Acme Corp to Globex, issued 3 March 2025. Total due: $1,240.50. Contact: billing@acme.com."
print(extract(text, {
    "invoice_number": "The invoice number.",
    "companies": "Every company mentioned in the text, as written.",
    "total": "The total amount due.",
    "phone": "The contact phone number.",
}))
# {'invoice_number': ['INV-2291'], 'companies': ['Acme Corp', 'Globex'], 'total': ['$1,240.50'], 'phone': []}

trust_remote_code=True is needed for the UltraBERT architecture.

ONNX int8

onnx/model_quantized.onnx (204 MB, vs 604 MB in fp32) takes input_ids and attention_mask and returns the O/B/I logits. Linear weights are quantized to 8 bits per block of 32 (weight-only, MatMulNBits; activations stay in float) and the embedding table is stored in fp16. Its tags match the PyTorch model on 99.6% of tokens and on 100% of answer tokens in our checks, and it scores the same on the evaluation below (20 rows per source: QA F1 0.730 vs 0.728 for PyTorch fp16, list F1 0.701 vs 0.700); plain dynamic int8 quantization (int8 activations) breaks the model (78% agreement), so prefer this file to re-quantizing. Build the input as the tokenizer's pair template: <|endoftext|> instruction <|sep|> text.

import onnxruntime
from huggingface_hub import hf_hub_download

session = onnxruntime.InferenceSession(hf_hub_download(name, "onnx/model_quantized.onnx"))
enc = tok(list(fields.values()), [text] * len(fields), truncation="only_second", max_length=2048,
          padding=True, return_offsets_mapping=True, return_tensors="np")
tags = session.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0].argmax(-1).tolist()
# then read the spans off the text tokens as in extract() above (tags outside the text are meaningless)

LocalLLaMA/typed-extraction (zero-shot)

On the test split of LocalLLaMA/typed-extraction (1,690 documents, 17,733 typed fields), which this model never trained on. Scores come from its own score.py, and spans are mapped to typed values with plain code (numbers parsed, dates through dateutil). Submitted in discussion #1.

Model	Headline ↑	Fields with a value ↑	Hallucination ↓	Miss ↓	Computed ↑
ScaleDown Extract (API)	0.782	0.723	0.117	0.200	0.570
UltraBERT-150M-clairic-extraction	0.616	0.548	0.141	0.297	0.014
Schematron V2 Small (LLM)	0.562	0.529	0.445	0.303	0.134
Always null	0.199	0.000	0.000	1.000	0.000

As an extractor, it can't compute values (totals, counts, day spans). Predictions are in eval/.

Evaluation

Test split of the extraction config (revision v2026.10.20.7), 100 rows per source. text fields (one answer or none) are scored by SQuAD F1, the empty answer counting as an answer. list fields (every span of a kind) are scored by micro F1 over normalized values. Scores are means over sources.

Model	Params	QA (28 sources, F1)	NER / span lists (60 sources, F1)
UltraBERT-150M-clairic-extraction	150M	0.720	0.702
gliner-multitask-large-v0.5	440M	0.474	0.408
gliner_medium-v2.1	209M	0.332	0.360
deberta-v3-base-squad2	184M	0.498	0.060

The test sources are the training sources (their test splits), while the baselines see most of them zero-shot, so this measures performance on these tasks rather than on unseen ones. GLiNER reads about 384 words at a time and was given the field name as its label (the instruction for numbered questions); the QA models were given the instruction as the question and read long texts in overlapping windows. On SQuAD 2.0, deberta-v3-base-squad2, trained on it, stays ahead (0.876 vs 0.764).

Per-source F1

Source	Field type	clairic-extraction	gliner-multitask-large	gliner_medium	deberta-squad2
adversarial-qa	text	0.373	0.251	0.008	0.320
anatem	list	0.787	0.305	0.337	0.000
bc2gm	list	0.776	0.506	0.575	0.000
bc4chemd	list	0.778	0.521	0.561	0.092
bc5cdr	list	0.845	0.641	0.665	0.083
bioinfer/entities	list	0.413	0.117	0.318	0.005
bionlp-cg/entities	list	0.740	0.546	0.300	0.003
biored/entities	list	0.775	0.332	0.307	0.006
broad-twitter-corpus	list	0.664	0.727	0.725	0.228
buster	list	0.720	0.540	0.395	0.047
concurrentqa	list	0.768	0.553	0.100	0.516
conll04	list	0.808	0.702	0.719	0.120
conll2000	list	0.950	0.030	0.010	0.002
conll2003/ner_tags	list	0.894	0.717	0.584	0.219
crossner/ai	list	0.650	0.675	0.667	0.075
crossner/literature	list	0.760	0.789	0.737	0.064
crossner/music	list	0.713	0.850	0.823	0.050
crossner/politics	list	0.798	0.853	0.806	0.036
crossner/science	list	0.735	0.761	0.755	0.036
cuad/spans	list	0.454	0.025	0.000	0.032
duorc/ParaphraseRC	text	0.517	0.458	0.290	0.385
duorc/SelfRC	text	0.832	0.708	0.077	0.519
ebm-nlp/interventions	list	0.520	0.155	0.163	0.000
ebm-nlp/outcomes	list	0.425	0.031	0.061	0.000
ebm-nlp/participants	list	0.305	0.006	0.057	0.006
fabner	list	0.674	0.268	0.191	0.013
fairytaleqa/explicit	text	0.743	0.537	0.000	0.627
fce/errors	list	0.377	0.000	0.000	0.000
few-nerd/supervised	list	0.770	0.725	0.631	0.043
finer-139	list	0.906	0.222	0.164	0.012
finer-ord	list	0.834	0.672	0.649	0.141
friendsqa	text	0.660	0.333	0.005	0.486
gum-ner	list	0.425	0.287	0.383	0.032
harveyner	list	0.402	0.321	0.314	0.029
hotpotqa	text	0.728	0.753	0.107	0.586
inspec/keyphrases	list	0.431	0.033	0.000	0.000
jnlpba/jnlpba	list	0.723	0.474	0.581	0.054
legallens-ner	list	0.565	0.155	0.165	0.038
linnaeus	list	0.964	0.548	0.500	0.000
medmentions/st21pv	list	0.517	0.422	0.137	0.003
mit-movie-trivia	list	0.799	0.553	0.577	0.351
mit-restaurant	list	0.785	0.427	0.362	0.036
molweni	text	0.641	0.369	0.107	0.488
multidoc2dial	text	0.642	0.010	0.000	0.107
natural-questions	text	0.718	0.657	0.194	0.469
ncbi_disease/ncbi_disease	list	0.843	0.589	0.560	0.075
ontonotes_english/SpeedOfMagic--ontonotes_english	list	0.864	0.784	0.599	0.094
opinion-target/en	list	0.749	0.102	0.000	0.000
policyqa	text	0.422	0.052	0.000	0.197
qaconv	text	0.678	0.679	0.095	0.620
qasper/spans	list	0.083	0.034	0.000	0.005
redocred/entities	list	0.786	0.000	0.000	0.006
scierc/entities	list	0.464	0.172	0.150	0.009
semeval2014/laptop-aspects	list	0.812	0.132	0.062	0.032
skillspan/knowledge	list	0.737	0.278	0.056	0.010
skillspan/skill	list	0.469	0.146	0.048	0.000
slurp/slots	list	0.828	0.664	0.544	0.250
squad-v2	text	0.764	0.585	0.544	0.876
stackoverflow-ner	list	0.715	0.373	0.384	0.063
subjqa/books	text	0.714	0.566	0.670	0.576
subjqa/electronics	text	0.559	0.370	0.480	0.332
subjqa/grocery	text	0.617	0.313	0.585	0.445
subjqa/movies	text	0.686	0.530	0.685	0.512
subjqa/tripadvisor	text	0.522	0.315	0.394	0.422
swde/auto	text	0.686	0.623	0.527	0.589
swde/book	text	0.991	0.969	0.875	0.570
swde/camera	text	0.997	0.736	0.630	0.682
swde/job	text	0.852	0.774	0.519	0.513
swde/movie	text	0.799	0.174	0.219	0.320
swde/nbaplayer	text	0.972	0.938	0.778	0.540
swde/restaurant	text	0.983	0.501	0.722	0.715
swde/university	text	0.999	0.032	0.053	0.849
tab/pii	list	0.860	0.330	0.288	0.000
timeqa	text	0.462	0.187	0.000	0.102
toxic-spans	list	0.593	0.000	0.000	0.000
tweetner7	list	0.556	0.406	0.394	0.093
ud/ewt	list	0.896	0.017	0.037	0.000
ud/gum	list	0.846	0.039	0.061	0.000
uner/en_ewt	list	0.924	0.775	0.762	0.198
wdc-pave	text	0.916	0.699	0.724	0.621
websrc/spans	text	0.694	0.166	0.000	0.491
wiesp2022-ner	list	0.722	0.413	0.429	0.029
wnut_17/wnut_17	list	0.594	0.475	0.525	0.133

Training data

The extraction config gathers extractive QA (SQuAD 2.0, Natural Questions, HotpotQA, NewsQA, DuoRC, QuAC, CUAD, Qasper, TimeQA, WebSRC, SWDE, and others) and span tagging (CoNLL-2003, OntoNotes, Few-NERD, WNUT, CrossNER, biomedical, legal and financial NER, skills, aspects, keyphrases, and others), plus procedurally generated documents. A few sources (e.g. GUM, PreCo) are licensed for non-commercial use only; check the dataset card for per-source licenses.

Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    year = "2024",
    address = "Torino, Italia",
    pages = "15655--15684",
    url = "https://aclanthology.org/2024.lrec-main.1361",
}