You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is not accepting new access requests at the moment. Access is still available for our accepted users, which requires the gated access setting to stay active. For inquiries or custom project requests, please contact us at miklos[dot]sebok[at]poltextlab[dot]com.

illframes-migration-binary

Author: Miklos Sebok (poltextLAB) — miklos[dot]sebok[at]poltextlab[dot]com

Model Description

An xlm-roberta-large model finetuned on English training data labelled with the Illframes Migration Codebook, collapsed to a binary illiberal-framing decision. It is the migration-domain counterpart of poltextlab/illframes-climate-binary.

The training data is recoded as:

1: 901-902-903-904-905-906-907-908-909-910 (any illiberal migration frame)
0: 999 (None of them)

The 11-class codebook these labels collapse is documented on poltextlab/xlm-roberta-large-illframes-migration.

Use this model when the quantity of interest is whether a text carries an illiberal migration frame at all — frame prevalence, diffusion, time series. Use the 11-class model when the specific frame matters.

How to Use the Model

from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
pipe = pipeline(
    model="poltextlab/illframes-migration-binary",
    task="text-classification",
    tokenizer=tokenizer,
    use_fast=False,
    truncation=True,
    max_length=256,
    token="<your_hf_read_only_token>"
)

pipe("<text_to_classify>")

Gated Access

This model requires gated access. You must pass the token parameter when loading the model.

Training

Setting	Value
Base model	`xlm-roberta-large`
Training data	`poltextlab/illframes-migration`, v21 (2025-06-24)
Train / validation / test	6992 / 874 / 875
Split	stratified on (coder-agreement type x binary label)
Learning rate	8e-06
Epochs	5 max, early stopping on macro-F1
Effective batch size	16 x 2
Max sequence length	256
Seed	42

Model Performance

Evaluated on a held-out test set of 875 English examples (51% positive), drawn by stratified split from the v21 corpus and unseen during training.

Accuracy: 0.83 | Weighted Average F1: 0.83 | Macro F1: 0.83

Sensitivity (recall, class 1): 0.84 | Specificity (recall, class 0): 0.81

Classification report — held-out test (n=875)

Class	Precision	Recall	F1-Score	Support
0: None of them	0.83	0.81	0.82	428
1: Illiberal frame	0.82	0.84	0.83	447

Confusion matrix: TN=348 FP=80 FN=70 TP=377

Double-coded subset (n=385, 25% positive)

Rows where two coders agreed — the higher-quality slice of the corpus.

Class	Precision	Recall	F1-Score	Support
0: None of them	0.95	0.89	0.92	290
1: Illiberal frame	0.72	0.85	0.78	95

Accuracy 0.88 | Weighted F1 0.88 | Sensitivity 0.85 | Specificity 0.89

Official v21 test set (n=196) — since WITHDRAWN, reported for the record only

This set was withdrawn from poltextlab/illframes-migration on 2026-09-04 and moved to deprecated/: 83.7% of its 196 rows appear verbatim in a training file, and it carries only 20 negative examples, so neither its accuracy nor its specificity is a meaningful held-out measurement. The figures are retained here solely so that earlier reports of them can be traced. The held-out test above is the score to use. with only 20 negative examples**, so its specificity estimate carries a very wide interval and should not be used on its own.

Class	Precision	Recall	F1-Score	Support
0: None of them	0.24	0.85	0.37	20
1: Illiberal frame	0.98	0.69	0.81	176

Accuracy 0.70 | Weighted F1 0.76

Why there is no head-to-head baseline against the 11-class model

poltextlab/xlm-roberta-large-illframes-migration can in principle be reduced to the same binary decision by treating any of 901-910 as positive, and doing so on this test set yields an apparent accuracy of 0.89. That figure is not valid and is not reported as a comparison. Every one of the 875 test rows used here also appears in the v20 and v21 training files (100% overlap; 61.5% overlap with v19), so the 11-class model was scored on text it had already been trained on.

The size of the effect is visible in the 11-class model's own documentation: it reports 54% accuracy and 0.57 weighted F1 on its held-out test set, against 0.89 on the contaminated split. Any future comparison of the two models needs a test set that is genuinely unseen by both.

The split used for this model was checked for the same problem in the other direction: no test row has a TF-IDF cosine similarity above 0.90 to any training row (one row above 0.80), so the figures reported above are not inflated by near-duplicate leakage from the corpus's augmented rows.

Correcting an observed rate to a prevalence

Sensitivity and specificity are both below 1, so the share of documents this model flags is a biased estimate of the true share carrying an illiberal frame. Correct it with

p_true = (p_obs + specificity - 1) / (sensitivity + specificity - 1)

Using the held-out figures (sensitivity 0.84, specificity 0.81), a corpus genuinely containing no illiberal frames would still return an observed positive rate of about 19%. Report corrected prevalence with an interval, never the raw positive rate.

Limitations

English only. Training and evaluation are English. Applying the model to other languages is zero-shot cross-lingual transfer through XLM-R and must be validated against a per-language gold set before the output is used substantively.
Short texts. The training texts are sentence-length: median 29 words, 90th percentile 56, 99th percentile 97. Applying the model to whole speeches or articles is a unit mismatch — segment long documents to sentence or quasi-sentence level, classify there, and aggregate upwards.
Coder agreement is confounded with class in the underlying corpus: single-coded rows are 72% positive while double-coded rows are 25% positive. The split used here is stratified on coding type so the effect is visible rather than hidden, and the double-coded subset is reported separately.
Report corrected prevalence, not raw positive rates (see the correction formula above).

Inference platform

This model is used by the CAP Babel Machine, an open-source and free natural language processing tool, designed to simplify and speed up projects for comparative research.

Cooperation

Model performance can be significantly improved by extending our training sets. We appreciate every submission of CAP-coded corpora (of any domain and language) at poltextlab{at}poltextlab{dot}com or by using the CAP Babel Machine.

Debugging and issues

This architecture uses the sentencepiece tokenizer. In order to run the model before transformers==4.27 you need to install it manually.

Downloads last month: 46

Safetensors

Model size

0.6B params

Tensor type

F32

Model tree for poltextlab/illframes-migration-binary

Base model

FacebookAI/xlm-roberta-large

Finetuned

(1013)

this model

Evaluation results

Accuracy on ILLFRAMES migration v21 (held-out split)
self-reported

0.829
Weighted F1 on ILLFRAMES migration v21 (held-out split)
self-reported

0.829
Macro F1 on ILLFRAMES migration v21 (held-out split)
self-reported

0.828