dleemiller/AbsoluteZero-nli-4b

AbsoluteZero NLI 4B

Observable NLI with think traces

Provide a text (premise) and a statement (hypothesis), and it returns calibrated probabilities that the text entails, contradicts, or is neutral to the statement. One model answers in two ways:

• Fast mode: a single forward pass and a classifier head (similar to cross-encoders).
• Think mode: the model writes a short reasoning trace, then the scoring head reads it and gives the verdict. Use it on the hard cases, such as multi-step logic, quantifiers and negation, long contracts and policies. Traces are short (~50 tokens on average on ANLI, ~200 on first-order-logic problems).

Benefits of observation

Typical zero-shot encoder models cannot provide information that elucidates why they chose their classification. Think traces enable users to inspect and determine how to augment hypotheses to fix or improve classification. This could be done in conjunction with automation for grouping classification failures and using an LLM to test changes to the hypothesis against a holdout set.

Examples

Use	Text → statement	Result (fast mode)
Zero-shot classification / tagging	review → "This review is positive."	93.4% (zs_meta, 18 tasks)
RAG grounding	retrieved passages → generated answer	86.4% (RAGTruth)
Guardrails / Moderation	message → policy statement	92.5% (CrossingGuard)
Clinical extraction	clinical note → "The patient is unemployed."	98.2% (SDOH-NLI)
Contract review	contract → "The agreement permits sharing data with third parties."	92.9% (ContractNLI)

4.2B parameters · documents up to 32k tokens · #1 on the OpenNLI Skill Index (11-03-26, fast mode) · Apache-2.0

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/AbsoluteZero-nli-4b", trust_remote_code=True)
text = "My card was charged twice for the same order."
labels = ["billing", "shipping", "account access"]
probs = model.predict([(text, f"This text is about {l}.") for l in labels], apply_softmax=True)
print({l: round(float(p[0]), 3) for l, p in zip(labels, probs)})  # p[0] = P(entailment)
# {'billing': 0.92, 'shipping': 0.001, 'account access': 0.001}

More in Think Mode, Recommended Applications and the 45-dataset evaluation.

---

Table of Contents

• NLI Evaluation Results
• Usage
• Think Mode
• Recommended Applications
• Extended Evaluation (45 datasets)
• Training Details
• Limitations
• Acknowledgments
• Citation

---

NLI Evaluation Results

Comparison on the OpenNLI leaderboard (v0.2.0), fast mode.

Model	Params	Skill Index	Accuracy	mnli	mnli_mismatched	snli	anli_r1	anli_r2	anli_r3	wanli	lingnli	CAL	Throughput (pairs/s)
dleemiller/AbsoluteZero-nli-4b	4.2B	61.8	0.8372	0.9211	0.9227	0.9366	0.8250	0.7060	0.6517	0.7718	0.8829	7.2	159
dleemiller/finecat-nli-v2-xl	1.03B	59.5	0.8263	0.9202	0.9191	0.9356	0.7860	0.6600	0.6067	0.7710	0.8834	6.1	576
dleemiller/finecat-nli-ultra-l	378M	58.6	0.8213	0.9127	0.9132	0.9302	0.7910	0.6400	0.5808	0.7698	0.8708	6.6	1,487
dleemiller/finecat-nli-v2-l	396M	57.8	0.8182	0.9164	0.9122	0.9329	0.7760	0.6040	0.5767	0.7664	0.8775	6.5	1,224
tasksource/ModernBERT-large-nli*	396M	37.7	0.7246	0.8981	0.9223	0.9192	0.7290	0.5130	0.4917	0.6970	0.8497	13.2	1,218
dleemiller/finecat-nli-l*	396M	37.5	0.7292	0.9155	0.9269	0.9162	0.7470	0.5680	0.5425	0.7708	0.8745	20.9	1,225
MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli	435M	27.9	0.6767	0.9123	0.9077	0.8897	0.7950	0.6820	0.6392	0.7696	0.8815	25.3	1,092
sileod/deberta-v3-large-tasksource-nli*	435M	24.8	0.6341	0.9298	0.9447	0.9211	0.7370	0.5770	0.4975	0.6786	0.8673	17.9	1,091

^* trained on one or more OpenNLI test sets; those scores are replaced by estimates, as on the leaderboard.

---

Usage

Label Map:

• entailment: 0
• neutral: 1
• contradiction: 2

Fast mode (Sentence Transformers)

pip install -U sentence-transformers

from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/AbsoluteZero-nli-4b", trust_remote_code=True)
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
]

logits = model.predict(pairs)  # shape: (3, 3)
for (prem, hyp), row in zip(pairs, logits):
    print(f"[{id2label[int(np.argmax(row))]}]  Premise: {prem}  |  Hypothesis: {hyp}")

The model reads each pair in its training layout (Premise: … / Hypothesis: … / <think></think> / Relation:); the tokenizer shipped with the model builds it, so pass plain (premise, hypothesis) pairs. Inputs are right-padded; the head reads each row's last token.

Many hypotheses, one text. The model is causal and the premise comes first, so for tagging or checking many claims against one document, score_many encodes the premise once and scores every hypothesis from that cached state: the same predictions as scoring pair by pair, ~10x faster for 50 labels on a ~500-token document.

doc = "My card was charged twice for the same order, and support hasn't replied in a week."
labels = ["billing", "shipping", "account access", "customer support"]
logits = model.model.score_many(model.tokenizer, doc, [f"This text is about {l}." for l in labels])
print({l: round(p, 3) for l, p in zip(labels, logits.softmax(-1)[:, 0].tolist())})  # P(entailment) per label
# {'billing': 0.908, 'shipping': 0.001, 'account access': 0.002, 'customer support': 0.982}

---

Think Mode

Think mode is a method on the loaded model; it returns the verdict, its probabilities and the trace.

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/AbsoluteZero-nli-4b", trust_remote_code=True)
pairs = [
    ("The glass fell off the counter and shattered on the tile.", "The glass broke when it hit the floor."),
    ("The store opens at 9 a.m. every day.", "The store opens at 7 a.m. on weekdays."),
    ("A researcher presented results at the conference.", "The presentation won the best paper award."),
]
for r in model.model.think(model.tokenizer, pairs):
    print(r["label"], "|", r["trace"])

entailment | The glass shattered on the tile, which is the floor, so it broke when it hit the floor.
contradiction | The premise says the store opens at 9 a.m. every day, which directly conflicts with the hypothesis that it opens at 7 a.m. on weekdays.
neutral | The premise only states that a researcher presented results; it says nothing about awards or recognition. Winning the best paper award is possible but not implied or ruled out by the premise.

Think vs. fast (test sets: n=300 pairs each; ANLI dev: full dev splits):

Dataset	Fast	Think	Think tokens (avg.)
anli_r2 (test)	0.6700	0.7000	55
anli_r3 (test)	0.6533	0.6533	56
wanli (test)	0.7600	0.7600	47
vitaminc (test)	0.8933	0.8767	44
control (test)	0.7700	0.7433	70
fol (test)	0.8367	0.8133	210
anli_dev_r1	0.8281	0.8422	51
anli_dev_r2	0.7080	0.7251	55
anli_dev_r3	0.6495	0.6468	54

Both modes have comparable accuracy with some variation. Confidence is calibrated for fast mode, while adding think traces will make the classification head certain toward the think trace, so confidence cannot be used in that mode (on these test items: mean confidence 0.93 in think mode vs. 0.86 in fast mode; calibration error 18.6% vs. 12.2%).

---

Recommended Applications

Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.

Application	Example hypothesis	Supporting results
Zero-shot tagging / topic classification	"This text is about billing."	zs_meta, fin_topics
Routing and decisions in agent pipelines	"The user wants to cancel an order."	zs_meta, mnli, snli
RAG grounding / hallucination checks	Each answer sentence vs. retrieved passages	ragtruth, wice, vitaminc
Contract and policy review	"The agreement permits sharing data with third parties."	contractnli_a/b, control, docnli
Biomedical and health claim checking	Claim vs. abstract or clinical note	sdoh_nli, biosift, bionli, scifact, pubhealth
Dialogue fact-checking	Assistant reply vs. knowledge source	dialfact, vitaminc
Guardrails / policy matching	Policy statement vs. user or model text	crossingguard
Logical reasoning (think mode)	Conclusion vs. premises	fol, control, anli

Less suited: financial and general legal verification (findver, legal_domain), and adversarial reasoning (anli_r3, nope_adv).

---

Extended Evaluation (45 datasets)

Accuracy (fast mode) on the test splits of 45 NLI and adjacent datasets. 2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction. DocNLI and VitaminC are evaluated on fixed random subsets of 5,000 and 10,000 pairs; FRANK is scored per summary. — = not on the same items.

Dataset	n	AbsoluteZero-nli-4b	finecat-nli-v2-xl	27B teacher (fast mode)
finecat	18012	0.8533	0.8445	0.8656
mnli	9815	0.9207	0.9202	0.9311
mnli_mismatched	9830	0.9226	0.9191	0.9305
snli	9824	0.9367	0.9356	0.9399
anli_r1	1000	0.8260	0.7860	0.8630
anli_r2	1000	0.7060	0.6600	0.7490
anli_r3	1200	0.6542	0.6067	0.7100
wanli	5000	0.7720	0.7710	0.7796
lingnli	7361	0.8830	0.8834	0.8958
glue_diagnostic	1100	0.7564	0.7582	0.8118
med	5380	0.7941	0.8147	0.8039
nan_nli	258	0.7481	0.7171	0.7403
scone_test	1000	1.0000	1.0000	0.9980
nmonli_test	200	1.0000	1.0000	1.0000
conjnli	623	0.7303	0.7191	0.7335
nope_adv	345	0.5507	0.5739	0.5768
nli4ct	199	0.8543	0.8141	0.8693
sdoh_nli	990	0.9818	0.9879	0.9848
healthver	230	0.7565	0.7609	0.7565
scifact	300	0.8767	0.8733	0.8433
biosift	1000	0.9240	0.9200	0.9210
scientsbank	971	0.7683	0.7508	0.7683
beetle	819	0.6667	0.6422	0.6325
legal_domain	1163	0.5813	0.4918	0.6389
findver	500	0.7240	0.6560	0.7660
dialfact	1499	0.8512	0.8412	0.8586
frank	1572	0.7417	—	—
ragtruth	1497	0.8644	0.8544	0.8691
vast	1500	0.7907	0.7933	0.8053
zs_meta	5400	0.9341	0.9304	0.9344
seahorse	800	0.8413	0.8562	0.8638
pubhealth	600	0.8617	0.8500	0.8650
bionli	600	0.9167	0.9133	0.9333
fin_topics	600	0.9700	0.9767	0.9633
vitaminc	10000	0.9030	0.9062	0.9351
docnli	5000	0.8516	—	—
contractnli_a	1707	0.9291	0.9279	0.9285
contractnli_b	2074	0.9026	0.8954	0.9021
control	800	0.7675	0.7525	0.8087
fol	10277	0.8481	0.8727	0.8430
scitail	2126	0.9671	0.9661	0.9652
wice	958	0.8998	0.8810	0.9144
sick	4906	0.9246	0.9317	0.9213
crossingguard	5397	0.9248	0.9250	0.9344
hans	15000	0.8468	0.8463	0.9209

Macro average over the 45 datasets: 0.8383.

---

Training Details

This model is a fine-tune of Qwen/Qwen3.5-4B (text backbone), trained in two stages.

Stage 1: distillation. LoRA & two classifier heads, trained on the FineCat-NLI-Reason compilation relabelled by an NLI fine-tune of Qwen3.8-27B.

Stage 2: reinforcement learning on think mode (GRPO). Think mode was further trained with 32 sampled traces per prompt, reward = correctness of the verdict with a soft overlong penalty, a repetition penalty and a floor that discourages trace collapse.

Stage 2 made think mode both more accurate and much shorter. On first-order-logic validation problems, accuracy 0.836 → 0.858 with traces cut from ~340 to ~200 tokens on average.

The released weights are the stage-2 LoRA merged into the stage-1 weights, stored in bfloat16.

---

Limitations

• English only.
• Throughput: 160 pairs/s on short pairs (40 on long documents) in fast mode; use the FineCat encoders for very high volume or tight latency. Think mode generates a trace per pair (~1-10 pairs/s with transformers, depending on trace length; vLLM is faster).
• Trained on inputs up to 8k tokens; scored at up to 32k, but accuracy on documents longer than 8k is not separately tested.
• Weaker on financial and general legal verification (findver, legal_domain) and adversarial reasoning (anli_r3, nope_adv).

---

Acknowledgments

We thank the Qwen team for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{nli-compiled-2025,
  title = {FineCat NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Refined compilation of 6 major NLI datasets}
}