eventdata-utd/ConfliBERT-2-base

ConfliBERT-2 (base)

A long-context encoder for political conflict research: ModernBERT-base with continued masked-language-model pretraining on 22 billion tokens of conflict, diplomacy and political text, through 2026.

ConfliBERT (2021) gave conflict researchers a domain-pretrained encoder, but it inherits BERT's 512-token window, a 2019 architecture, and training text that ends in 2020. ConfliBERT-2 keeps the idea and replaces the encoder: ModernBERT's 8,192-token context and speed, plus one full pass of domain pretraining on a corpus that runs from Cold War diplomatic records to 2026 web text. It is meant to be fine-tuned: a few hundred labeled examples teach it your codebook, and it then codes millions of documents on one GPU, with no data leaving your machine.

	ConfliBERT (2021)	ModernBERT-base	ConfliBERT-2 base
Architecture	BERT-base (2019)	ModernBERT (2024)	ModernBERT (2024)
Parameters	110M	150M	150M
Context window	512 tokens	8,192 tokens	8,192 tokens
Domain pretraining	33 GB of conflict text, to 2020	none	22.3B tokens, 16 sources, to 2026
Pseudo-perplexity on held-out conflict text	n/a (different tokenizer)	4.446	3.419
Mean F1, nine conflict tasks	78.25	77.03	77.37

A large version (395M parameters, from ModernBERT-large) is training and will be released as eventdata-utd/ConfliBERT-2-large.

What it is for

• Event coding at scale. Relevance filtering, event-type classification, actor and location tagging (NER) over news archives and monitoring reports.
• Long documents. Full hearings, country reports, peace agreements, Security Council records: the model reads up to 8,192 tokens (about twelve pages) in one pass, where the 2021 model stops at one page. In the IndiaPoliceEvents corpus, when the evidence sits past token 512 the truncated 2021 model's recall falls from .81 to .23.
• Judgment labels. Codebook categories that depend on the whole document (a monthly crisis trend, a hearing's purpose, a severity grade). Fine-tuned encoders reach .79 recall on CrisisWatch deteriorations where a zero-shot frontier LLM has precision .12.
• Local, cheap, reproducible. About 0.7 GPU-hours per million documents at 512 tokens; sensitive sources never leave the machine.

English only. Not a chat model: it reads text and returns labels or fills blanks; it does not generate text.

How to use

Fill-mask (what pretraining teaches):

from transformers import pipeline
fill = pipeline("fill-mask", model="eventdata-utd/ConfliBERT-2-base")
fill("Fighting between the army and the Rapid Support Forces displaced millions in [MASK].")

Fine-tune for classification (the intended use):

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("eventdata-utd/ConfliBERT-2-base")
model = AutoModelForSequenceClassification.from_pretrained(
    "eventdata-utd/ConfliBERT-2-base", num_labels=NUM_LABELS)
# standard Trainer fine-tuning; inputs up to 8,192 tokens.

Token classification (NER) works the same way with AutoModelForTokenClassification. Notes:

• Install flash-attn for the fastest long-input inference; without it the model runs on PyTorch SDPA with the same outputs.
• The tokenizer is ModernBERT's (cased, 50,368 tokens). ConfliBERT-2 is not a drop-in replacement for the 2021 model: fine-tuned heads and tokenized datasets have to be rebuilt.
• For NER with pre-split words, pass add_prefix_space=True to the tokenizer; a missing prefix space costs F1 on this tokenizer family.
• Learning rates that worked in our benchmark: 2e-5 to 8e-5 for classification, 8e-5 to 2.4e-4 for NER, selected on a validation split.

Evaluation

All models fine-tuned under one protocol: learning rate chosen on the validation split (seed 123), three seeds at that rate, test F1 reported as the mean. NER tasks use the prefix-space fix and the extended learning-rate grid.

Task	ConfliBERT (2021)	ModernBERT-base	ModernBERT-large	ConfliBERT-2 base
SATP relevance	89.35	87.21	88.44	87.67
IndiaPoliceEvents, sentences	66.06	66.59	67.19	65.56
IndiaPoliceEvents, documents	70.79	63.11	63.61	64.72
InsightCrime	68.22	69.56	69.32	68.54
CAMEO event class	89.46	86.95	86.01	87.31
20 Newsgroups	79.90	80.49	80.68	81.02
BBC News	98.12	97.81	98.13	96.87
re3d NER	68.41	68.51	68.10	69.10
CAMEO NER	73.98	73.02	75.32	75.50
Mean	78.25	77.03	77.42	77.37

• ConfliBERT-2 base beats ModernBERT-base on six of nine tasks and matches ModernBERT-large, which has 2.6 times the parameters.
• The 2021 model keeps a 0.9-point lead on the mean, most of it from one task (IndiaPoliceEvents documents, 6 points). The clearest gain over both ModernBERT sizes is CAMEO NER, +2.5 over ModernBERT-base.
• Pseudo-perplexity on 2,000 held-out blocks of the training corpus (fixed 30% mask, 613,809 masked tokens): ModernBERT-base 4.446, ModernBERT-large 3.368, ConfliBERT-2 base 3.419.
• Harness note: the ModernBERT-base and ConfliBERT-2021 rows come from the pilot harness (RTX 4090, transformers 4.x); ConfliBERT-2 and ModernBERT-large ran on NCSA Delta (A100, transformers 5.17). A same-harness ModernBERT-base rerun is in progress and will replace that column.

Task sources: the ConfliBERT benchmark (SATP, InsightCrime, CAMEO, re3d, 20 Newsgroups, BBC) and IndiaPoliceEvents (Halterman et al., 2021).

Training

Base model	answerdotai/ModernBERT-base (released weights)
Objective	Continued masked-LM, mask rate 0.30
Data	22,275,166,208 tokens, 25,101,498 documents, one full pass, no subsampling
Block length	1,024 tokens, packed; attention confined to each document inside a block, positions restart per document
Batch	256 blocks = 262,144 tokens per step; 84,974 steps
Schedule	Warmup-stable-decay: 2,549 warmup steps, 65,431 at peak, 16,994 linear decay; peak LR 2e-4
Optimizer	AdamW (fused), weight decay 0.01, bf16 autocast with fp32 master weights, TF32
Kernels	FlashAttention-2 (variable-length), sparse MLM head on masked positions, torch.compile
Hardware	1 node, 4x NVIDIA A100 40GB (NCSA Delta), 12 h 02 min, 48 GPU-hours, 514k tokens/s
Final losses	train 1.232, held-out 1.190

The released weights carry a dense MLM head and no pinned attention backend, like stock ModernBERT.

Corpus

Sixteen sources, every one used in full, exact duplicates removed (65,758 documents, 0.26%). Held out and never trained on: a 0.3% random sample of every source and all documents dated 2021.

Group	Sources
Conflict news and reports (ConfliBERT collection)	News archive, Gigaword, UTD event stories, organization reports
Diplomacy and international organizations	UN Security Council meeting records and resolutions, UN General Debate speeches, UN parallel corpus, PA-X peace agreements, Foreign Relations of the United States (FRUS)
Legislatures and government research	UK Hansard, Congressional Research Service reports
Scholarship	Political science journal articles and university theses (University of Arizona collection)
Reference and web	Wikipedia (2026 snapshot), FineWeb filtered to political text (West Virginia University)

Filtered web text is about four fifths of the tokens, the curated sources one fifth. Dates run from the 1940s (FRUS, Hansard) to 2026.

Limitations

• English only; the corpus over-represents English-language news and the US, UK and UN institutional record.
• News-derived text carries the coverage and reporting biases of its sources; performance varies by event type and region.
• Pretraining knowledge of post-2020 events is broad, not authoritative: fill-mask outputs are not facts.
• A fine-tuned classifier is a measurement instrument. Report the model, the context window, the inference mode, the tuning protocol and the seeds; each moved our numbers.
• Not a substitute for human review in sensitive applications.

Code and reproduction

Everything needed to rebuild the corpus, pack it, train, and run the evaluation harness is in github.com/shreyasmeher/conflibert-v2, including the Slurm jobs used on Delta. The internal run name of these weights is conflibert-v2-cb2-wsd-full.

Citation

@misc{meher2026conflibert2,
  author       = {Meher, Shreyas and Brandt, Patrick T. and Chakma, Aunabil and D'Orazio, Vito and Medapati, Yamini and Osorio, Javier},
  title        = {ConfliBERT-2: A Long-Context Encoder for Political Conflict Research},
  year         = {2026},
  howpublished = {Hugging Face model repository, \url{https://huggingface.co/eventdata-utd/ConfliBERT-2-base}}
}

Please also cite ModernBERT (Warner et al., 2024) and ConfliBERT (Hu et al., 2022, NAACL).

Acknowledgements

Supported by NSF award 2311142. This work used Delta at NCSA / University of Illinois through allocation CIS220162 from the ACCESS program (NSF grants 2138259, 2138286, 2138307, 2137603, 2138296). Corpus contributions from the ConfliBERT team at UT Dallas, the University of Arizona, and West Virginia University.