UltraBERT-150M-nli

UltraBERT-150M fine-tuned jointly on 131 tasksource NLI and classification tasks (MNLI, ANLI, WANLI, SICK, doc-nli, FOLIO, Label-NLI, and the tasks listed below). Use it for zero-shot classification, natural language inference, or as a backbone for fine-tuning.

On par with tasksource/ModernBERT-base-nli at a similar size: mean test accuracy 0.837 vs 0.836 over the 123 tasks both report, with notable gains on adversarial NLI (ANLI r2/r3).

Task	UltraBERT-150M-nli	ModernBERT-base-nli
MNLI	0.856	0.870
ANLI r1	0.661	0.660
ANLI r2	0.533	0.490
ANLI r3	0.522	0.440
WANLI	0.717	0.740
SNLI	0.922	0.830
SICK	0.900	0.930
Label-NLI (zero-shot)	0.833	0.820

Zero-shot classification

from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True)
classifier("one day I will see the world", ["travel", "cooking", "dancing"])

Natural language inference

from transformers import pipeline
pipe = pipeline("text-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True)
pipe([dict(text="there is a cat", text_pair="there is a black cat")])  # entailment / neutral / contradiction

trust_remote_code=True is needed for the UltraBERT architecture. Inputs up to 2048 tokens.

Test accuracy on all 131 tasks

Same encoder, one classification head per task.

Task	Accuracy
AdjectiveScaleProbe-nli	0.839
ConTRoL-nli	0.583
FLD.v2/default	0.906
FLD.v2/star	0.811
FOL-nli	0.817
LogicNLI	0.889
MSciNLI	0.783
PARARULE-Plus	1.000
Pol_NLI	0.889
SDOH-NLI	0.994
SIGA-nli	0.497
SpaceNLI	1.000
WANLI	0.717
add_one_rte	0.933
anli/a1	0.661
anli/a2	0.533
anli/a3	0.522
autotnli	0.939
babi_nli	0.987
biosift-nli	0.894
boolq-natural-perturbations	0.622
breaking_nli	0.983
chaos-mnli-ambiguity	nan
cnli	0.867
code_x_glue_cc_defect_detection	0.617
conceptrules_v2	1.000
conj_nli	0.717
contract-nli/contractnli_a/seg	0.850
contract-nli/contractnli_b/full	0.844
counterfactually-augmented-snli	0.739
cycic_classification	0.911
dadc-limit-nli	0.692
dataset_train_nli	0.922
defeasible-nli/atomic	0.772
defeasible-nli/snli	0.822
defeasible-nli/social	0.833
dialogue_nli	0.906
dnc	0.883
doc-nli	0.867
dynahate	0.822
dynasent/dynabench.dynasent.r1.all/r1	0.806
dynasent/dynabench.dynasent.r2.all/r2	0.733
equate	0.639
feasibilityQA	0.816
folio	0.676
fracas	0.714
glue/cola	0.761
glue/mnli	0.856
glue/mrpc	0.806
glue/qnli	0.906
glue/qqp	0.844
glue/rte	0.863
glue/sst2	0.950
glue/stsb	nan
glue/wnli	0.417
hans	1.000
help-nli	0.972
hover-3way/nli	0.861
idioms-nli	0.850
lex_glue/ledgar	0.878
lex_glue/scotus	0.544
lex_glue/unfair_tos	0.917
lifecycle-entailment	0.828
lingnli	0.822
logical-entailment	0.856
logical-fallacy	0.350
logiqa-2.0-nli	0.583
lonli	0.989
mctest-nli	0.911
mindgames	0.972
missing-item-prediction/contrastive	0.867
monli	0.922
monotonicity-entailment	0.972
mpe	0.750
nan-nli	0.769
natural-language-satisfiability	0.889
naturallogic	0.944
nlgraph	0.939
nli-veridicality-transitivity	0.956
nli4ct_semeval2024	0.570
nli_fever	0.839
nope	0.456
paws/labeled_final	0.956
pragmeval/pdtb	0.539
probability_words_nli/reasoning_1hop	1.000
probability_words_nli/reasoning_2hop	0.967
probability_words_nli/usnli	0.828
propsegment/nli	0.883
recast/recast_factuality	0.956
recast/recast_megaveridicality	0.867
recast/recast_ner	0.850
recast/recast_puns	0.967
recast/recast_sentiment	0.956
recast/recast_verbcorner	0.906
recast/recast_verbnet	0.856
recast_white/dpr	0.689
recast_white/fnplus	0.800
recast_white/sprl	0.878
resnli	1.000
robust_nli/IS_CS	0.818
robust_nli/LI_LI	0.978
robust_nli/PI_CD	0.829
robust_nli/PI_SP	0.895
robust_nli/ST_LM	0.850
robust_nli/ST_NE	0.883
robust_nli/ST_SE	0.806
robust_nli/ST_WO	0.850
robust_nli_is_sd	1.000
robust_nli_li_ts	0.872
ruletaker	0.972
scifact_entailment	0.794
scinli	0.739
scitail/snli_format	0.933
scone	0.972
seahorse_summarization_evaluation	0.756
semantic_fragments_nli	0.983
sick/label	0.900
snli	0.922
super_glue/axg	0.944
super_glue/boolq	0.639
super_glue/boolq_passage	0.800
super_glue/cb	0.786
super_glue/multirc	0.833
super_glue/wic	0.717
syntactic-augmentation-nli	0.950
synthetic-retrieval-NLI/binary	0.894
synthetic-retrieval-NLI/count	0.639
synthetic-retrieval-NLI/position	0.906
temporal-nli	0.961
tomi-nli	0.994
zero-shot-label-nli	0.833

Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    year = "2024",
    address = "Torino, Italia",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}