susurofu/literary-bert-en-medium

Literary Sentiment Ordinal BERT — Medium

This model is BERT-medium (41.7M parameters) fine-tuned for sentence-level sentiment analysis of English literary text using ten ordered sentiment levels (1 = most negative, 10 = most positive). It is one of five student configurations trained on Gemma 4 31B-generated pseudo-labels for literary sentences.

The training objective is cumulative-logit ordinal regression, rather than conventional three-class classification. The scalar score is the expected ordinal rating.

• Original pretrained encoder: google/bert_uncased_L-8_H-512_A-8
• Trained checkpoint: BERT-medium; maximum training input length: 128 tokens
• Scores: 1–3 negative, 4–6 near-neutral, 7–10 positive (descriptive interpretation)
• Checkpoint selected by minimum validation MAE, with Hemingway annotations kept out of training and selection

On the independent human-annotated Hemingway evaluation (n=1,923), this model achieved MAE 0.752, Kendall τ 0.489, Spearman ρ 0.568 and accuracy within ±1 0.808. On 305 reference-defined non-neutral sentences, it achieved MAE 0.998 and accuracy within ±1 0.675.

Important: this is a custom ordinal head. The model produces nine cumulative logits, not ten class logits. Do not use AutoModelForSequenceClassification or a default pipeline("sentiment-analysis") to interpret the outputs. Use the AutoModel example below; its trust_remote_code=True loads modeling_ordinal_bert.py from this repository. Review that file before running repository code.

Quick start

Install pip install torch transformers and use:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "susurofu/literary-bert-en-medium"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()

sentences = ["A wonderful day filled with joy.", "Everything was lost."]
inputs = tokenizer(sentences, padding=True, truncation=True,
                   max_length=128, return_tensors="pt")
with torch.inference_mode():
    scores = 1 + model(**inputs).logits.sigmoid().sum(dim=-1)
labels = scores.round().clamp(1, 10).int()
print(scores.tolist())  # continuous expected score in [1, 10]
print(labels.tolist())  # nearest integer rating (1..10)

torch.round follows the notebook's numpy.rint nearest-integer convention (half-to-even); typically the scores are noninteger. For GPU inference, move both model and tokenized tensors to the same device. See example_inference.py for a standalone version. The nine output logits represent P(Y > k) after sigmoid for thresholds k=1,...,9 and must not be passed through softmax/argmax.

Training data and supervision

• 10,889 cleaned literary sentences from Project Gutenberg; an author/work sampling procedure and removal of short or structural passages were applied.
• Labels assigned using Gemma 4 31B with each sentence annotated independently on the 1–10 scale. The teacher returned a JSON field Sentiment_score; hard integer pseudo-labels were used for training.
• Stratified 80/10/10 train / validation / LLM-labeled test split: 8,711 / 1,089 / 1,089. These splits are distinct from the independent human-labeled test set.
• The independent human evaluation consists of 1,923 sentences from Hemingway's The Old Man and the Sea, with reference scores averaged across two annotators. Hemingway was not used for model fine-tuning or checkpoint selection.
• Teacher labels are imbalanced: 41.5% received score 5, and 57.9% received scores 4–6. The complete label frequencies (before splitting) are:

Score	Count	Score	Count
1	239	6	960
2	800	7	637
3	1,436	8	923
4	821	9	474
5	4,519	10	80

Architecture and training

The encoder produces a pooled representation, mapped to one latent sentiment score s(x). Nine trainable ordered thresholds θ₁ < ... < θ₉ are parameterized by a first threshold and cumulative softplus increments. The model returns:

logit_k = s(x) − θ_k, for k=1,...,9.

Training minimizes binary cross-entropy against nine cumulative binary targets 1[label > k]. At inference, E[Y|x] = 1 + Σ sigmoid(logit_k); the rounded rating is clipped to [1,10]. This is hard-label pseudo-labeling, not soft-logit distillation.

Training setting	Value
Base encoder	google/bert_uncased_L-8_H-512_A-8
Trainable parameters (approx.)	41.7M
Learning rate	2e-5
Weight decay	0.01
Warmup ratio	0.10
Maximum epochs	3
Train / eval batch size	16 / 32
Gradient accumulation	1
Optimizer and scheduler	Hugging Face Trainer defaults (AdamW / linear scheduler)
Validation/checkpoint/logging interval	Every 100 optimization steps
Checkpoint selection	Lowest validation MAE
Maximum retained checkpoints	6
Random seed	1111
FP16	Enabled when CUDA was available
Maximum tokenized length	128
Selected checkpoint step	1100

Evaluation: independent human-annotated Hemingway corpus

All sentences (n = 1,923). Each row reports metrics computed against the average of two human annotators. The table includes the teacher and external baseline results for context; those models are not included in this repository. The focus model is shown in bold. All numbers are from the supplied evaluation CSV (rounded to three decimals).

Model	MAE ↓	Kendall τ ↑	Spearman ρ ↑	Pearson r ↑	Prec. ±1 ↑	Acc. ±1 ↑	3-class macro P ↑	3-class acc. ↑
Gemma 4 31B (teacher)	0.822	0.604	0.690	0.711	0.460	0.741	0.545	0.728
twt_roberta	0.972	0.497	0.568	0.630	0.176	0.695	0.574	0.806
twt_roberta_xlm	1.261	0.505	0.593	0.626	0.125	0.530	0.500	0.653
BERT-large	0.769	0.589	0.676	0.685	0.614	0.787	0.533	0.740
BERT-base	0.733	0.580	0.665	0.692	0.723	0.815	0.546	0.765
BERT-medium	0.752	0.489	0.568	0.623	0.659	0.808	0.564	0.800
BERT-small	0.717	0.468	0.541	0.591	0.708	0.837	0.557	0.805
BERT-mini	0.766	0.402	0.472	0.516	0.732	0.829	0.580	0.816
Constant-neutral baseline	0.597	—	—	—	0.841	0.841	0.294	0.881

Full 3-class precision by predicted sentiment class (all sentences)

Model	Negative P ↑	Neutral P ↑	Positive P ↑
Gemma 4 31B (teacher)	0.177	0.972	0.487
twt_roberta	0.232	0.960	0.530
twt_roberta_xlm	0.142	0.960	0.397
BERT-large	0.194	0.968	0.437
BERT-base	0.218	0.971	0.449
BERT-medium	0.234	0.948	0.510
BERT-small	0.233	0.940	0.496
BERT-mini	0.200	0.925	0.615
Constant-neutral baseline	0.000	0.881	0.000

Non-neutral sentences (n = 305). This subset is selected using human reference scores below 4 or above 6. It measures performance on already-identified non-neutral sentences, not independent detection of such sentences.

Model	MAE ↓	Spearman ρ ↑	Pearson r ↑	Prec. ±1 ↑	Acc. ±1 ↑	3-class macro P ↑	3-class acc. ↑
Gemma 4 31B (teacher)	1.116	0.791	0.885	0.522	0.600	0.606	0.669
twt_roberta	1.656	0.745	0.797	0.306	0.223	0.624	0.636
twt_roberta_xlm	1.613	0.775	0.838	0.301	0.308	0.569	0.610
BERT-large	0.887	0.799	0.877	0.571	0.711	0.585	0.646
BERT-base	0.726	0.826	0.899	0.698	0.803	0.629	0.675
BERT-medium	0.998	0.785	0.820	0.669	0.675	0.605	0.570
BERT-small	0.982	0.763	0.794	0.668	0.698	0.584	0.521
BERT-mini	0.969	0.756	0.784	0.692	0.708	0.623	0.498
Constant-neutral baseline	1.962	—	—	0.000	0.000	0.083	0.249

Full 3-class precision by predicted sentiment class (non-neutral subset)

Model	Negative P ↑	Neutral P ↑	Positive P ↑
Gemma 4 31B (teacher)	0.548	0.271	1.000
twt_roberta	0.570	0.321	0.980
twt_roberta_xlm	0.520	0.196	0.990
BERT-large	0.538	0.226	0.991
BERT-base	0.569	0.328	0.991
BERT-medium	0.581	0.260	0.974
BERT-small	0.558	0.237	0.955
BERT-mini	0.588	0.280	1.000
Constant-neutral baseline	0.000	0.249	0.000

Metric definitions and interpretation

• MAE is computed from the original continuous model predictions and original averaged human scores (lower is better). Kendall's τ, Spearman's ρ, and Pearson's r use these unrounded scores.
• Accuracy (±1) is the proportion of samples with absolute error ≤ 1 in the original numerical scale.
• Precision (±1) is a custom legacy measure: precision is calculated separately at each observed distinct predicted numerical value, counting predictions within ±1 of the reference as correct, then macro-averaged over those distinct values. On continuous predictions it is not conventional classification precision and may be sensitive to floating-point uniqueness. Interpret cautiously.
• Three-class precision and accuracy are separate from ±1 metrics. Continuous predictions and averaged human scores are first rounded to integers using floor(score + 0.5) (half-up); scores 1–3 are negative, 4–6 neutral, and 7–10 positive. Macro precision averages the three per-class precision values, giving zero to classes without any predicted members.
• Caution: The human test corpus is very neutral-skewed: 84.1% of reference scores are in [4,6]. A constant-neutral predictor achieves overall MAE 0.597, better than all models; its correlations are undefined. Therefore, use multiple metrics and inspect the non-neutral subset rather than ranking models by overall MAE alone. The constant-neutral row predicts a constant score 5.
• Values across the two evaluation tables are not comparable without accounting for the different reference-selected subsets. RoBERTa results reflect their original numerical rescaling, rather than an ordinal model trained on 1–10 labels.

Performance on the held-out LLM-labeled test split (n = 1,089)

The following results measure agreement with the teacher-generated labels, not with Hemingway's human annotations.

Model	Best step	Val. MAE	Test MAE	Test RMSE	Exact acc.	Acc. ±1	QWK	Spearman ρ
BERT-large	1600	0.847	0.838	1.196	0.461	0.814	0.746	0.763
BERT-base	600	0.971	0.959	1.312	0.401	0.771	0.680	0.711
BERT-medium	1100	1.012	1.078	1.505	0.372	0.742	0.593	0.567
BERT-small	1100	1.079	1.142	1.546	0.360	0.699	0.529	0.532
BERT-mini	1600	1.219	1.278	1.633	0.264	0.661	0.445	0.479

Intended use

Research on literary sentence sentiment, relative valence patterns, and exploratory Digital Humanities analysis of narrative text. Intended for English. Sentiment scoring is inherently interpretation-dependent; examine outputs in textual context before drawing literary conclusions. This model is not a general-purpose classifier for clinical, legal, or other high-stakes decision-making.

Limitations

• Models were trained using automatically generated labels, which may inherit teacher biases (including topic-associated polarity).
• Single-sentence annotation and modeling omit wider narrative context, irony, and discourse-level sentiment shifts.
• The human-annotated evaluation covers one literary work; results do not guarantee similar performance across genres, dates, or authors.
• The training setup used one fixed seed. Reported small differences between model sizes have no uncertainty estimates.
• The 1–10 scale is ordinal: predicted values should not be treated as calibrated probabilities, and the expected score need not match an individual reader's interpretation.
• Faster inference than the teacher is plausible given model size, but has not been benchmarked in the corresponding study.

Related model variants

• BERT-large: susurofu/literary-bert-en-large (340M)
• BERT-base: susurofu/literary-bert-en-base (110M)
• BERT-medium: susurofu/literary-bert-en-medium (41.7M)
• BERT-small: susurofu/literary-bert-en-small (29.1M)
• BERT-mini: susurofu/literary-bert-en-mini (11.3M)

BERT-tiny was trained during development but is not released because it converged to constant neutral predictions. The constant-neutral baseline remains in the evaluation tables for context.

License and Attribution

This model is released under the Apache License 2.0. It is fine-tuned from Google's BERT architecture (Apache 2.0) using literary sentences from Project Gutenberg annotated by Gemma 4 31B (Apache 2.0). The model may be used, modified, and redistributed under the terms of the Apache License 2.0.

Citation

@misc{suslov2026distilling,
  author    = {Suslov, Artem and Hashimoto, Takehiro},
  title     = {Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23554553},
  url       = {https://doi.org/10.5281/zenodo.23554553}
}