narrative-detection-roberta

RoBERTa-base fine-tuned for binary narrative detection: does this passage tell a story? This is the classifier applied to every Dolma passage in the sampling pipeline, producing narrative_label and narrative_confidence. Trained on human-labelled data merging StorySeeker and NarraDetect.

Note: Full model card with training details coming soon.

Loading

Download model.pt and tokenizer/ from this repo, then:

Unlike the other two models in this collection, this is a plain AutoModelForSequenceClassification head — no custom module needed. The weights are saved as a state dict rather than via save_pretrained, so build the architecture first and load into it.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("tokenizer/")
model = AutoModelForSequenceClassification.from_pretrained("roberta-base", num_labels=2)
model.load_state_dict(torch.load("model.pt", map_location="cpu", weights_only=True))
model.eval()

text = "The morning she left, he stood in the doorway and said nothing."
enc = tokenizer(text, max_length=256, padding="max_length",
                truncation=True, return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits

prob_narrative = torch.softmax(logits, dim=-1)[0, 1].item()
is_narrative   = bool(logits.argmax(-1).item())   # label 1 = narrative

Label 1 is the narrative class; prob_narrative above is the same quantity the pipeline writes as narrative_confidence. Passages are truncated at 256 tokens, matching training.

Config

{
  "task": "narrative_detection",
  "model_name": "roberta-base",
  "max_length": 256,
  "epochs": 3,
  "batch_size": 16,
  "lr": 2e-05,
  "seed": 42,
  "data": "narrative_bert_modeling/data/storyseeker_narradetect_combined/merged_doc_level_labels.csv",
  "n_train": 806,
  "n_val": 90,
  "best_f1": 0.8089887640449438,
  "final_metrics": {
    "accuracy": 0.8111111111111111,
    "precision": 0.8181818181818182,
    "recall": 0.8,
    "f1": 0.8089887640449438,
    "roc_auc": 0.928395061728395
  }
}