engagement-farm-classifier

A small classifier that flags engagement-farming posts on social platforms (built and evaluated on X/Twitter posts): explicit CTAs ("like if you agree", "tag someone who"), reply-bait questions, giveaways, chain posts, and low-substance filler whose main goal is farming replies and likes.

• Base: google/bert_uncased_L-8_H-512_A-8 (32M parameters)
• Serving artifact: int8 ONNX in onnx-int8/ (42 MB)
• CPU latency: 1.9 ms mean, 2.7 ms p95 per post (Apple Silicon, single post)

Labels

0 = genuine, 1 = engagement_farming. Serving rule: softmax probability of engagement_farming >= 0.5 flags the post; lower the threshold to flag more aggressively (see Metrics).

Metrics

Validation (777 posts, 189 farming, held out from training):

threshold	precision	recall	f1
0.5	0.98	0.90	0.94
0.3	0.97	0.90	0.94

Held-out test set (273 posts collected after all training data, zero id/text overlap with training, 15 farming):

threshold	precision	recall	f1
0.5	1.00	0.47	0.64
0.4	1.00	0.53	0.70
0.3	1.00	0.67	0.80

Zero false positives on the test set at every threshold. The test positives are subtle, timeline-native bait (rhetorical "how many of you" questions, greeting filler), so test recall is the realistic number for in-feed filtering. Test positives are few (15), so treat these as indicative.

Training data

12,506 posts collected from public timelines and keyword searches. Labeled by a large LLM teacher (kimi-k3 via batched prompts), keeping only verdicts with teacher confidence >= 0.85: 7,766 posts (1,706 farming) after text dedupe. Positive class oversampled ~1:2 in the train split. Split and threshold details are in train.py and eval.py.

No raw post corpus is redistributed. Only scripts and weights are published; the collection and labeling pipeline is included so anyone can rebuild the dataset.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "selftaughtdev/engagement-farm-classifier"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)

text = "like if you agree"
probs = model(**tok(text, return_tensors="pt", truncation=True, max_length=128)).logits.softmax(-1)[0]
print(probs[1].item())  # probability of engagement_farming

For the int8 ONNX artifact:

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model = ORTModelForSequenceClassification.from_pretrained(name, subfolder="onnx-int8", file_name="model_quantized.onnx")

Limitations

• Trained on English-language posts; other languages are untested.
• Boundary cases are genuine disagreement: a rhetorical question with substance versus the same question as pure bait. Confident-only teacher labels (>= 0.85) trim but do not remove this ambiguity.
• Innocent-looking filler (plain "good morning" posts, rhetorical questions) is the main source of false negatives.

Regenerating the dataset and model

1. collect_tweets.js: paste into a browser console on the target platform, it scrolls and dedupes posts into JSON.
2. label.py: sends batches of 20 posts to any OpenAI-compatible teacher endpoint (--base-url) with parallel workers, resumable.
3. train.py: fine-tunes the base model (--base), exports fp32 and int8 ONNX.
4. eval.py: threshold sweeps on validation or a held-out file (--labeled <file> --full).

License

Apache 2.0 (matches the base model).