README.md

---
model-index:
- name: poltextlab/illframes-migration-binary
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: ILLFRAMES migration v21 (held-out split)
      type: poltextlab/illframes-migration
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.829
    - name: Weighted F1
      type: f1
      value: 0.829
    - name: Macro F1
      type: f1
      value: 0.828
tags:
- text-classification
- pytorch
metrics:
- precision
- recall
- f1-score
language:
- en
base_model:
- xlm-roberta-large
pipeline_tag: text-classification
library_name: transformers
license: mit
extra_gated_prompt: This model is not accepting new access requests at the moment.
  Access is still available for our accepted users, which requires the gated access
  setting to stay active. For inquiries or custom project requests, please contact
  us at miklos[dot]sebok[at]poltextlab[dot]com.
extra_gated_fields:
  Country: country
  Institution: text
  Institution Email: text
  Full Name: text
  Please specify your academic project/use case you want to use the models for: text
---

# illframes-migration-binary

**Author:** Miklos Sebok (poltextLAB) — miklos[dot]sebok[at]poltextlab[dot]com

## Model Description

An **xlm-roberta-large** model finetuned on English training data labelled with the
**Illframes Migration Codebook**, collapsed to a **binary** illiberal-framing decision. It is the
migration-domain counterpart of `poltextlab/illframes-climate-binary`.

The training data is recoded as:

- **1**: 901-902-903-904-905-906-907-908-909-910 (any illiberal migration frame)
- **0**: 999 (None of them)

The 11-class codebook these labels collapse is documented on
[`poltextlab/xlm-roberta-large-illframes-migration`](https://huggingface.co/poltextlab/xlm-roberta-large-illframes-migration).

Use this model when the quantity of interest is **whether a text carries an illiberal migration
frame at all** — frame prevalence, diffusion, time series. Use the 11-class model when the
specific frame matters.

---

## How to Use the Model

```python
from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
pipe = pipeline(
    model="poltextlab/illframes-migration-binary",
    task="text-classification",
    tokenizer=tokenizer,
    use_fast=False,
    truncation=True,
    max_length=256,
    token="<your_hf_read_only_token>"
)

pipe("<text_to_classify>")
```

## Gated Access

This model requires gated access. You must pass the token parameter when loading the model.

## Training

| Setting | Value |
|---|---|
| Base model | `xlm-roberta-large` |
| Training data | `poltextlab/illframes-migration`, v21 (2025-06-24) |
| Train / validation / test | 6992 / 874 / 875 |
| Split | stratified on (coder-agreement type x binary label) |
| Learning rate | 8e-06 |
| Epochs | 5 max, early stopping on macro-F1 |
| Effective batch size | 16 x 2 |
| Max sequence length | 256 |
| Seed | 42 |

## Model Performance

Evaluated on a held-out test set of **875** English examples (51% positive),
drawn by stratified split from the v21 corpus and unseen during training.

**Accuracy**: 0.83  |  **Weighted Average F1**: 0.83  |  **Macro F1**: 0.83

**Sensitivity** (recall, class 1): 0.84  |  **Specificity** (recall, class 0): 0.81

### Classification report — held-out test (n=875)

| Class | Precision | Recall | F1-Score | Support |
| ----- | --------- | ------ | -------- | ------- |
| 0: None of them | 0.83 | 0.81 | 0.82 | 428 |
| 1: Illiberal frame | 0.82 | 0.84 | 0.83 | 447 |

Confusion matrix: TN=348 FP=80 FN=70 TP=377

### Double-coded subset (n=385, 25% positive)

Rows where two coders agreed — the higher-quality slice of the corpus.

| Class | Precision | Recall | F1-Score | Support |
| ----- | --------- | ------ | -------- | ------- |
| 0: None of them | 0.95 | 0.89 | 0.92 | 290 |
| 1: Illiberal frame | 0.72 | 0.85 | 0.78 | 95 |

Accuracy 0.88 | Weighted F1 0.88 | Sensitivity 0.85 | Specificity 0.89

### Official v21 test set (n=196) — since WITHDRAWN, reported for the record only

This set was withdrawn from `poltextlab/illframes-migration` on 2026-09-04 and moved to `deprecated/`: **83.7% of its 196 rows appear verbatim in a training file**, and it carries only 20 negative examples, so neither its accuracy nor its specificity is a meaningful held-out measurement. The figures are retained here solely so that earlier reports of them can be traced. The held-out test above is the score to use.
with only 20 negative examples**, so its specificity estimate
carries a very wide interval and should not be used on its own.

| Class | Precision | Recall | F1-Score | Support |
| ----- | --------- | ------ | -------- | ------- |
| 0: None of them | 0.24 | 0.85 | 0.37 | 20 |
| 1: Illiberal frame | 0.98 | 0.69 | 0.81 | 176 |

Accuracy 0.70 | Weighted F1 0.76

## Why there is no head-to-head baseline against the 11-class model

`poltextlab/xlm-roberta-large-illframes-migration` can in principle be reduced to the same binary
decision by treating any of 901-910 as positive, and doing so on this test set yields an apparent
accuracy of 0.89. **That figure is not valid and is not reported as a comparison.** Every one of the
875 test rows used here also appears in the v20 and v21 training files (100% overlap; 61.5%
overlap with v19), so the 11-class model was scored on text it had already been trained on.

The size of the effect is visible in the 11-class model's own documentation: it reports 54% accuracy
and 0.57 weighted F1 on its held-out test set, against 0.89 on the contaminated split. Any future
comparison of the two models needs a test set that is genuinely unseen by both.

The split used for **this** model was checked for the same problem in the other direction: no test row
has a TF-IDF cosine similarity above 0.90 to any training row (one row above 0.80), so the figures
reported above are not inflated by near-duplicate leakage from the corpus's augmented rows.

## Correcting an observed rate to a prevalence

Sensitivity and specificity are both below 1, so the share of documents this model flags is a biased
estimate of the true share carrying an illiberal frame. Correct it with

```
p_true = (p_obs + specificity - 1) / (sensitivity + specificity - 1)
```

Using the held-out figures (sensitivity 0.84, specificity 0.81), a
corpus genuinely containing no illiberal frames would still return an observed positive rate of about
19%. Report corrected prevalence with an interval, never the raw positive rate.

## Limitations

- **English only.** Training and evaluation are English. Applying the model to other languages is
  zero-shot cross-lingual transfer through XLM-R and must be validated against a per-language gold
  set before the output is used substantively.
- **Short texts.** The training texts are sentence-length: median 29 words, 90th percentile 56,
  99th percentile 97. Applying the model to whole speeches or articles is a unit mismatch — segment
  long documents to sentence or quasi-sentence level, classify there, and aggregate upwards.
- **Coder agreement is confounded with class** in the underlying corpus: single-coded rows are 72%
  positive while double-coded rows are 25% positive. The split used here is stratified on coding
  type so the effect is visible rather than hidden, and the double-coded subset is reported separately.
- **Report corrected prevalence, not raw positive rates** (see the correction formula above).

## Inference platform

This model is used by the [CAP Babel Machine](https://babel.poltextlab.com), an open-source and free
natural language processing tool, designed to simplify and speed up projects for comparative research.

## Cooperation

Model performance can be significantly improved by extending our training sets. We appreciate every
submission of CAP-coded corpora (of any domain and language) at poltextlab{at}poltextlab{dot}com
or by using the [CAP Babel Machine](https://babel.poltextlab.com).

## Debugging and issues

This architecture uses the `sentencepiece` tokenizer. In order to run the model before
`transformers==4.27` you need to install it manually.