sweenk/snt-classifier

SNT News Classifier v0.5.1

Multi-label news topic classifier: 12 top-level (L1) and 71 sub-level (L2) categories, built on xlm-roberta-large with two independent sigmoid heads. Both levels are genuinely multi-label — an article about a trade deal can be world + money_and_business + politics at the same time. Per-class decision thresholds (tuned on a held-out validation split) ship inside config.json; predict_labels() applies them and falls back to argmax so no article is ever left unlabeled.

Built by Sweenk to categorize its news feed; released so others can use and scrutinize it.

Quick start

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("sweenk/snt-classifier", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("sweenk/snt-classifier")

enc = tok("OpenAI raises $6.6B. The startup announced its latest funding round...",
          return_tensors="pt", truncation=True, max_length=512)
print(model.predict_labels(**enc))
# [{'l1': [{'key': 'money_and_business', 'p': 0.99}, {'key': 'tech_and_ai', 'p': 0.98}],
#    'primary_l1': 'money_and_business',
#    'l2': [{'key': 'companies_and_industries', 'p': 0.92}, ...]}]

Input convention: "{title}\n\n{body}", truncated at 512 tokens. The classifier was trained on title+body; titles alone work but body text improves routing (the training prompt explicitly prioritizes body over headline).

Taxonomy — 12 L1 / 71 L2

L1 category	# L2	L2 sub-categories
sports	12	american_football, baseball, basketball, college_sports, combat_sports, golf, hockey, motorsports, olympics, other_sports, soccer, tennis
politics	6	elections_and_campaigns, government_and_policy, immigration_and_borders, political_figures_and_scandals, social_issues_and_activism, state_and_local_politics
world	4	geopolitics_and_diplomacy, humanitarian_crises, terrorism_and_security, war_and_conflict
entertainment_and_pop_culture	7	books_and_arts, celebrities_and_gossip, gaming, internet_culture_and_creators, media_and_journalism, movies_and_tv, music
money_and_business	8	companies_and_industries, cost_of_living, crypto_and_fintech, housing_and_real_estate, macro_economy_and_rates, markets_and_investing, personal_finance, work_and_careers
crime_and_justice	4	courts_and_trials, crime_and_policing, scams_and_fraud, true_crime
tech_and_ai	5	artificial_intelligence, big_tech_and_startups, cybersecurity_and_privacy, gadgets_and_apps, screen_time_and_digital_life
science_and_space	4	archaeology_and_history, psychology_and_behavior, scientific_discoveries, space_and_astronomy
health_and_wellness	5	fitness_and_exercise, medical_and_public_health, mental_health, nutrition_and_diet, sleep_and_longevity
lifestyle	8	education_and_schools, faith_and_spirituality, fashion_and_beauty, food_and_drink, home_and_garden, parenting_and_family, relationships_and_dating, travel_and_places
weather_and_environment	5	climate_change, disasters_and_accidents, energy_and_climate_solutions, nature_and_wildlife, severe_weather
human_stories	3	animals_and_pets, good_news_and_kindness, offbeat_and_unusual

The full machine-readable taxonomy (l1_keys, l2_keys, l2_parent, per-class thresholds) is in config.json.

Evaluation — our own numbers, stated plainly

Held-out test split: 24,176 articles (10% of the 241,757 train-eligible corpus rows). "Tuned" = per-class thresholds optimized on the validation split, then applied unchanged to test.

Metric	@0.5 threshold	tuned thresholds
L1 macro F1	0.803	0.824
L1 micro F1	0.832	0.854
L1 primary accuracy (argmax)	0.832	—
L2 macro F1	0.624	0.671
L2 micro F1	0.760	0.757

Per-class L1 F1 (test)

L1	F1 @0.5	F1 tuned	threshold
sports	0.928	0.936	0.900
politics	0.849	0.861	0.700
world	0.781	0.827	0.900
entertainment_and_pop_culture	0.895	0.904	0.800
money_and_business	0.772	0.810	0.900
crime_and_justice	0.821	0.846	0.900
tech_and_ai	0.721	0.726	0.950
science_and_space	0.731	0.754	0.800
health_and_wellness	0.830	0.859	0.900
lifestyle	0.872	0.878	0.700
weather_and_environment	0.793	0.817	0.850
human_stories	0.646	0.672	0.900

What you should know before trusting these numbers

Read this section — it is the honest part.

• The gold labels are model-assisted, not human-annotated. The corpus (255K articles from HuffPost archives, CommonCrawl News, daily.dev, and Sweenk production) was labeled by a mechanical migration from an earlier taxonomy plus multiple passes of a Claude Sonnet teacher with a rule-based prompt, spot-audited by humans (QA gates at 70–87% agreement on sampled batches). Test F1 therefore measures agreement with an LLM teacher, not with human ground truth.
• Class imbalance is real (~21x). politics/lifestyle/entertainment have ~44-47K training rows; science_and_space has ~2.2K and tech_and_ai ~3.7K. Training used per-class pos_weight (clamped at 10) to compensate. The weakest class is human_stories (F1 0.672) — it is the fuzziest category by construction.
• v0.5.1 is a corrective retrain. A 986-article human validation of the previous release found errors clustered in specific boundaries (accidents dumped into weather_and_environment, terror attacks into world, pharma earnings into health_and_wellness). ~1,500 mislabeled rows were re-labeled with corrected routing rules and the model retrained. On those corrected rows the previous model matches the corrected label 25.9% of the time; this model matches 75.3%. Aggregate F1 is not directly comparable across releases because the gold labels themselves were corrected.
• Multilingual ability is inherited, not measured. The encoder is XLM-R, but nearly all training articles are English. Expect degraded (unquantified) quality on non-English news.
• L3 (named topics / entities) is not part of this model — Sweenk handles that downstream with a separate extraction step.

Architecture

xlm-roberta-large encoder → CLS pooling → dropout(0.1) → two parallel linear heads (L1: 12 logits, L2: 71 logits), both sigmoid. Trained 3 epochs, BCE loss with per-class pos_weight (L1) and loss weights L1:1.0 / L2:2.0, bf16 autocast, gradient checkpointing. Inference upcasts logits to fp32 before sigmoid (bf16 sigmoid saturates above logit ~6.2, which collapses co-confident multi-label pairs).

Versions

Version	What changed
v0.5	First multi-label release (12 L1 / 71 L2, dual sigmoid heads)
v0.5.1	Corrective retrain: route-accidents-by-cause, terror→crime, earnings→money, govt-personnel→politics, wildlife→weather, body-over-headline; ~1,500 corrected labels; re-tuned thresholds

License & attribution

Model weights: MIT. Base model: FacebookAI/xlm-roberta-large (MIT). The training corpus contains article text from public news sources and is not redistributed with this model.