iamahmadyasin/humor-bert-large

Humor Intelligence — BERT-large

A BERT-large model fine-tuned to predict how funny a joke is, trained on 340k cleaned Reddit jokes from the rJokes dataset (Weller & Seppi, LREC 2020). Given a joke as input, the model outputs a single scalar: a predicted humor score on the dataset's 0–11 log-compressed community rating scale.

Results

Model	Params	Clean Spearman	Clean Pearson	Clean RMSE
TF-IDF + Ridge	—	0.363	0.414	1.645
DistilBERT	66M	0.412	0.451	1.643
RoBERTa-base	125M	0.419	0.451	1.705
BERT-large	340M	0.423	0.463	1.657
RoBERTa-large	355M	0.432	0.470	1.630

Leakage-cleaned evaluation

The original rJokes splits contain ~2.4% of test jokes that are exact copies of training jokes (Reddit reposts). Prior work evaluated on these leaked splits. I remove the overlap and report on the clean test set (41,957 examples). I quantified the impact:

Model	Clean Spearman	Leaky Spearman	Inflation	Paper Spearman
DistilBERT (66M)	0.412	0.421	0.009	—
RoBERTa-base (125M)	0.419	0.426	0.007	—
BERT-large (340M)	0.423	0.431	0.009	0.430
RoBERTa-large (355M)	0.432	0.440	0.008	0.435

The average inflation is 0.008 ± 0.001 Spearman, consistent across four architectures of different sizes, confirming it is a dataset property and not a model-specific artifact.

Training details

• Base model: bert-large-uncased (340M parameters)
• Task: Single-value regression (num_labels=1, problem_type="regression")
• Dataset: rJokes, cleaned (339,499 train / 41,941 dev / 41,957 test)
• Cleaning: removed 5,707 exact duplicates, ultra-short fragments (<5 words), and ~2.4% cross-split leakage from dev/test
• Max sequence length: 128 tokens
• Epochs: 5 (best checkpoint at epoch 3 by dev Spearman; dev Spearman 0.4241)
• Effective batch size: 32 (constant across single and multi-GPU setups)
• Learning rate: 2e-5 with 6% linear warmup
• Weight decay: 0.01
• Precision: fp16
• Optimizer: AdamW (Hugging Face default)
• Seed: 42
• Hardware: Kaggle T4 ×2, ~3 sessions totaling ~30h (12h session limit)

Label note

The rJokes score column is already log-scaled: round(ln(raw_upvotes + 1)), giving integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data). It is used directly as the regression target. Do not log-transform again. This follows the paper's Section 3.1, which reduces the raw scale (0–136,353) down to integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data)".

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "iamahmadyasin/humor-bert-large"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()

joke = "I told my wife she was drawing her eyebrows too high. She looked surprised."
inputs = tokenizer(joke, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    score = model(**inputs).logits.item()
print(f"Predicted humor score: {score:.2f}")

Limitations

• Humor is subjective; the labels reflect one Reddit community's preferences, shaped by timing and virality as much as joke quality.
• The model regresses to the mean and is unreliable at the extremes of the score range (rarely predicts 0 or 6+).
• Trained on English-language Reddit jokes only.
• This is a humor ranker, not a judge of objective funniness.

Citation

Dataset:

@inproceedings{weller-seppi-2020-rjokes,
    title     = "The rJokes Dataset: a Large Scale Humor Collection",
    author    = "Weller, Orion and Seppi, Kevin",
    booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference (LREC)",
    year      = "2020",
    pages     = "6136--6141",
    url       = "https://aclanthology.org/2020.lrec-1.753/",
}

Project

Full project: github.com/iamahmadyasin/humor-intelligence