yuashi/roberta-word-plausibility

RoBERTa Word-Sense Plausibility Rating Model

Fine-tuned roberta-base regression model that predicts a continuous plausibility score (1–5) for a candidate word sense in an ambiguous narrative context, developed for the AmbiStory task (SemEval 2026 Shared Task 5).

Unlike traditional Word Sense Disambiguation (WSD), which forces a single discrete "correct" sense, this model treats sense interpretation as a graded phenomenon — matching human consensus on how plausible a candidate meaning is, given a story context, a dictionary definition, and an example usage.

Model Description

• Base model: roberta-base (~125M parameters)
• Task: Supervised regression (single continuous scalar output, scaled to 1–5)
• Architecture: RoBERTa encoder → pooled [CLS] representation → 2-layer MLP head (GELU activation, dropout 0.1) → scaled sigmoid output: ŷ = 1.0 + 4.0 · σ(x)
• Implementation: custom nn.Module (not a transformers.PreTrainedModel), saved to the Hub via huggingface_hub.PyTorchModelHubMixin for save_pretrained / push_to_hub / from_pretrained support
• Input format:

  [CLS] Precontext [SEP] Ambiguous Sentence [SEP] Ending [SEP] Judged Meaning: Example Sentence [SEP]
  

  The "Ending" segment is left empty when no story ending is provided.
• Inference post-processing: predictions are rounded to the nearest integer and clipped to [1, 5] to match the official task's integer output format (the underlying model itself outputs a continuous score, which can also be used directly).

Intended Uses

• Predicting graded plausibility of a word sense given a short narrative context (research / educational use, SemEval AmbiStory-style tasks).
• As a component in WSD pipelines that need soft/graded scores rather than hard sense classification.
• Benchmarking against zero-shot LLM baselines for graded semantic judgment tasks.

Out of scope: general-purpose sentiment/quality scoring, or any use requiring calibrated uncertainty estimates (the model outputs a point estimate only, not a distribution).

Training Data

• Dataset: AmbiStory (SemEval 2026 Shared Task 5)
• Train split: 2,280 samples spanning 220 unique homonyms across 380 distinct story setups
• Dev split: 588 samples, fully disjoint from train by homonym
• Each sample: a 3-sentence precontext, an ambiguous sentence, an optional story ending, a candidate sense (dictionary definition + example sentence), and a human plausibility rating (1–5) averaged from ≥5 Prolific annotators, with annotator standard deviation.

Training Procedure

Optimized with AdamW and a linear learning-rate schedule, using MSE loss against human-averaged plausibility scores.

Hyperparameter	Value
Learning rate	2 × 10⁻⁵
Batch size	16
Epochs	10
Warmup ratio	10% (linear)
Weight decay	0.01
Dropout	0.1

Evaluation

Evaluated on the 588-sample AmbiStory dev set against a zero-shot Qwen2.5-3B-Instruct baseline (logit-weighted expectation over tokens '1'–'5').

Metric	Qwen2.5-3B (Zero-Shot)	RoBERTa (Fine-Tuned)	Relative Gain
Spearman correlation (rs)	0.2264	0.4246	+87%
Accuracy within σ (Accσ)	51.53%	74.32%	+44%
MAE	1.0554	0.9163	+13%

Calibration across annotator disagreement tiers (MAE):

Tier	Zero-shot	Fine-tuned RoBERTa
Low disagreement	1.4888	1.05
Mid disagreement	–	0.82
High disagreement	0.6946	0.85

The fine-tuned model is far better calibrated on low-disagreement (high-consensus) samples, where the zero-shot baseline nearly fails (Acc_σ = 5.88%).

Limitations

• The 588-sample dev set is relatively small for fine-grained evaluation.
• Training targets are aggregate human averages, which collapse potentially multi-modal annotator distributions — the model does not output uncertainty.
• Exhibits regression-to-the-mean: rarely predicts extreme scores (1 or 5), even when the true label is extreme.
• Highest error on abstract/highly polysemous homonyms (e.g. "blaze", "try", "croaked"), especially in no-ending (under-specified) contexts.

How to Use

This model uses a custom architecture (a plain nn.Module wrapping roberta-base with a hand-built regression head), not a stock transformers model class — so it's loaded with the custom class definition below rather than AutoModel/RobertaForSequenceClassification.

import torch
import torch.nn as nn
from transformers import RobertaModel, RobertaTokenizerFast
from transformers.modeling_outputs import SequenceClassifierOutput
from huggingface_hub import PyTorchModelHubMixin

class RobertaForPlausibilityRegression(nn.Module, PyTorchModelHubMixin):
    def __init__(self, model_name="roberta-base", dropout=0.1):
        super().__init__()
        self.roberta = RobertaModel.from_pretrained(model_name)
        hidden_size = self.roberta.config.hidden_size
        self.regression_head = nn.Sequential(
            nn.Linear(hidden_size, hidden_size),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_size, 1)
        )

    def forward(self, input_ids=None, attention_mask=None, labels=None, **kwargs):
        outputs = self.roberta(input_ids=input_ids, attention_mask=attention_mask)
        cls_repr = outputs.last_hidden_state[:, 0, :]
        raw_score = self.regression_head(cls_repr).squeeze(-1)
        predictions = 1 + 4 * torch.sigmoid(raw_score)
        loss = None
        if labels is not None:
            loss = nn.MSELoss()(predictions, labels.float())
        return SequenceClassifierOutput(loss=loss, logits=predictions.unsqueeze(-1))

tokenizer = RobertaTokenizerFast.from_pretrained("<your-hf-username>/roberta-word-plausibility")
model = RobertaForPlausibilityRegression.from_pretrained("<your-hf-username>/roberta-word-plausibility")
model.eval()

precontext = "..."
ambiguous_sentence = "..."
ending = ""  # empty if no ending
judged_meaning = "Definition: ... Example: ..."

text = f"{precontext} </s> {ambiguous_sentence} </s> {ending} </s> {judged_meaning}"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    score = model(**inputs).logits.item()  # continuous score, 1–5 scale

print(round(score))  # nearest-integer rating in [1, 5]

Citation

If you use this model, please cite the AmbiStory shared task:

Janosch Gehring and Michael Roth. 2026. Rating Plausibility of Word Senses in Ambiguous
Sentences through Narrative Understanding. Proceedings of the 20th International Workshop
on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics.

Authors

Developed as part of the Introduction to Natural Language Processing course, Summer 2026, Philipps-Universität Marburg.