You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is not accepting new access requests at the moment. Access is still available for our accepted users, which requires the gated access setting to stay active. For inquiries or custom project requests, please contact us at miklos[dot]sebok[at]poltextlab[dot]com.

illframes-climate-binary-v2

Author: Miklos Sebok (poltextLAB) — miklos[dot]sebok[at]poltextlab[dot]com

Model Description

An xlm-roberta-large classifier that decides whether an English sentence carries an illiberal climate-policy frame. It supersedes poltextlab/illframes-climate-binary, whose published metrics were computed on a 969-row test set that does not exist in the dataset repository and cannot be verified. This release was trained on a documented split with every evaluation file, prediction file and script included in the repository, so that each number below can be re-derived.

Labels: 1 = 710–740 (any illiberal climate frame) · 0 = 799 (None of them). The nine-class codebook is documented on the dataset card (poltextlab/illframes-climate).

Use this model when the quantity of interest is whether a text carries an illiberal climate frame at all — frame prevalence, diffusion, time series. For the specific frame use illframes-climate-v5, noting that three of its eight frame classes are never recovered (see Limitations).

How to Use

from transformers import AutoTokenizer, pipeline
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
pipe = pipeline(model="poltextlab/illframes-climate-binary-v2", task="text-classification", tokenizer=tokenizer,
                use_fast=False, truncation=True, max_length=256, token="<your_hf_read_only_token>")
pipe("<one sentence>")

The model requires gated access; pass a token when loading.

Training data and split

Source: poltextlab/illframes-climate. The three non-test files were concatenated (5177 rows), de-duplicated on normalised text (4740 unique texts; where duplicates disagreed on the label — 205 groups on the nine-class label, 188 on the binary label — the row from TRAINING_DATA_WITHOUT_SUPERSET_training_data.csv was kept), and then every row that matched SUPERSET_test_data.csv exactly (60) or at TF-IDF cosine ≥ 0.80 (3) was removed, leaving a pool of 4677 rows (18.3% positive). That pool was split 80/10/10, stratified on the binary label, seed 42: train 3742 · validation 467 · test 468. SUPERSET_test_data.csv (the published test set of illframes-climate-v5, 557 rows after removing 3 within-file duplicates) was held out entirely as a second evaluation set.

Split hygiene was re-derived by an independent check: train, validation and test are pairwise disjoint on normalised text; no test row has TF-IDF (1–3-gram) cosine ≥ 0.70 to any training row. The split is at sentence level, not speech level: 97 of 468 test rows (20.7%) share a speech identifier with at least one training row, so the figures below measure generalisation to unseen sentences, not to unseen speeches.

Training-row provenance by identifier prefix: AUT 1486, EP 624, DK 549, DEU 310, HUN 284, USA 195, CZ 177, ES 85, syn 21, en 10, pl 1. Rows prefixed syn, en and pl have no documented origin in the dataset repository and are likely synthetic; they are included in training and disclosed here.

Setting	Value
Base model	`xlm-roberta-large`
Learning rate	8e-06 (warmup 0.1, weight decay 0.01)
Epochs	5 max; early stopping patience 2 on macro-F1; best = epoch 5
Effective batch	16 × 2
Max length	256 tokens
Precision	fp16
Seed	42

Performance

Held-out test (n = 468, 18.2% positive)

accuracy 0.853 · weighted F1 0.851 · macro F1 0.746 · sensitivity 0.565 · specificity 0.916

Class	Precision	Recall	F1	Support
0: None of them	0.905	0.916	0.911	383
1: Illiberal frame	0.600	0.565	0.582	85

Confusion: TN 351 · FP 32 · FN 37 · TP 48

`SUPERSET_test_data.csv` — the climate-v5 test set (n = 557, 36.6% positive)

accuracy 0.811 · weighted F1 0.800 · macro F1 0.777 · sensitivity 0.569 · specificity 0.952

Class	Precision	Recall	F1	Support
0: None of them	0.792	0.952	0.865	353
1: Illiberal frame	0.872	0.569	0.688	204

Confusion: TN 336 · FP 17 · FN 88 · TP 116

Two operating points

The figures above use the default argmax decision (P(frame) ≥ 0.50). A threshold chosen on the validation set to maximise macro-F1 gives P(frame) ≥ 0.4:

Set	Threshold	Accuracy	Macro F1	Sensitivity	Specificity
test	0.50	0.853	0.746	0.565	0.916
test	0.4	0.842	0.748	0.635	0.888
SUPERSET	0.50	0.811	0.777	0.569	0.952
SUPERSET	0.4	0.822	0.795	0.627	0.935

pred__test.csv, pred__superset.csv and pred__val.csv in this repository carry P(frame) for every row, so any other threshold can be evaluated directly.

Validation (n = 467) was used for checkpoint selection and threshold choice and is not a held-out score: accuracy 0.837 · weighted F1 0.835 · macro F1 0.722 · sensitivity 0.523 · specificity 0.908.

Correcting an observed rate to a prevalence

The share of sentences this model flags is not the share that carry a frame. With sensitivity 0.565 and specificity 0.916 (held-out test, threshold 0.50):

p_true = (p_obs + specificity − 1) / (sensitivity + specificity − 1)

A corpus with no illiberal framing would still return about 8% positives, and a corpus that was entirely framed would return about 56%. The denominator (sensitivity + specificity − 1 = 0.481) is what governs the correction's precision: the smaller it is, the more an error in either rate is amplified. Report corrected prevalence with an interval, never the raw positive rate.

Comparison with the previous models

None of these is a clean head-to-head, and the reasons are stated.

Model	Scored on	Accuracy	Macro F1	Sensitivity	Specificity	Why it is not a fair baseline
`poltextlab/illframes-climate-binary`	this held-out test (n = 468)	0.694	0.639	0.835	0.663	its training set is undocumented; these test rows come from the same corpus and may have been seen
`poltextlab/illframes-climate-binary`	SUPERSET (n = 557)	0.801	0.796	0.877	0.756	same
`illframes-climate-v5`, nine classes collapsed to binary	SUPERSET (n = 557)	0.761	0.685	0.368	0.989	fair for v5 — SUPERSET is its own published test set

The previous binary model and this one fail in opposite directions: it over-flags (high sensitivity, low specificity), this one under-flags. The published card of poltextlab/illframes-climate-binary (accuracy 0.74, weighted F1 0.76, n = 969) refers to a test set that no file or combination of files in the dataset repository reproduces.

A second training run on the identical split — class-weighted cross-entropy (weight 4.46 on the frame class), learning rate 1e-05, up to 10 epochs — was tried and rejected: held-out test macro F1 0.724, sensitivity 0.412, specificity 0.966. Its evaluation is kept as eval_round2.json.

Limitations

Sensitivity is the weak side. At the default threshold the model misses 44% of framed sentences on the held-out test. The training pool is 18% positive, and the first two epochs predicted no frames at all before the model recovered.
English, sentence-length input. Training texts are single sentences (median 26–30 words). Applying the model to whole speeches or articles is a unit mismatch — segment first, aggregate after. Use on other languages is zero-shot transfer through XLM-R and needs a per-language gold set before the output is used substantively.
Label contestation in the source data. 205 texts appear in more than one file with different labels; the split resolves them by file priority, not by adjudication.
Frame-level resolution is not available for every frame. On its own published test set, illframes-climate-v5 never recovers classes 721, 722 and 740 (F1 0.00). This binary model does not distinguish frames, so that gap does not affect it, but users who need the specific frame should know it exists.

Files

model.safetensors · tokenizer · config.json · eval.json (all metrics) · thresholds.json (validation sweep + applied thresholds) · pred__*.csv (per-row P(frame)) · eval_round2.json (the rejected second run) · scripts/ (split, training, baseline, verification and card scripts).

Cooperation

Model performance can be improved by extending the training sets. Coded corpora of any domain and language are welcome at poltextlab{at}poltextlab{dot}com.

Downloads last month: -

Safetensors

Model size

0.6B params

Tensor type

F32

Model tree for poltextlab/illframes-climate-binary-v2

Base model

FacebookAI/xlm-roberta-large

Finetuned

(997)

this model

Evaluation results

Accuracy on ILLFRAMES Climate, held-out split of the training pool
test set self-reported

0.853
Macro F1 on ILLFRAMES Climate, held-out split of the training pool
test set self-reported

0.746
Weighted F1 on ILLFRAMES Climate, held-out split of the training pool
test set self-reported

0.851
Accuracy on ILLFRAMES Climate, SUPERSET_test_data.csv (the climate-v5 test set)
self-reported

0.811
Macro F1 on ILLFRAMES Climate, SUPERSET_test_data.csv (the climate-v5 test set)
self-reported

0.777