jacqpark/manifesto-RILE-setfit

Manifesto RILE left-right classifier

A SetFit classifier that places a political quasi-sentence on the left-right scale. It was trained for the workshop Python Literacy for Text-as-Data and is published so participants have a working model to load if their own training run or API key stalls during the session.

What it does

Given one quasi-sentence from a party manifesto, it returns left or right. The labels follow the standard RILE scale, 13 Manifesto Project categories on each side.

from setfit import SetFitModel

model = SetFitModel.from_pretrained("jacqpark/manifesto-RILE-setfit")
model.predict(["We will expand the National Health Service and fund it properly."])
# ['left']

Training data

Manifesto Project Corpus, main dataset version MPDS2026a and corpus version 2026-1. United Kingdom manifestos from the 2019 and 2024 general elections, four largest parties per election. Training used 64 quasi-sentences per class, sampled with seed 42.

The underlying text is not redistributed here, because the Manifesto Project forbids redistribution without written permission. Only the trained weights are published. To reproduce the training set you need your own free API key.

Evaluation

Held-out set of 400 quasi-sentences, 200 per class, drawn from the same corpus and disjoint from training.

Model	Cohen's kappa	Macro F1
TF-IDF and logistic regression baseline	0.18	0.59
This SetFit model	0.42	0.71

Confusion matrix, rows are true left then right.

[[147  53]
 [ 64 136]]

Reproducibility

• Backbone sentence-transformers/all-MiniLM-L6-v2
• SetFit, batch size 16, 1 epoch
• Seed 42 for sampling and training
• Trained on CPU in about three minutes, and faster on a free Colab T4

Limitations

Trained on 128 UK examples, so it reflects British party language from two elections. Accuracy drops on other countries, which is why the workshop restricts training to one country. Kappa of 0.42 is moderate agreement, useful for aggregate measures across many sentences and unreliable for any single sentence. Validate against your own hand codes before using it in published work.

Citation for the data

Please cite the Manifesto Project and read their terms of use.