---
license: mit
base_model: roberta-base
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: storyseeker
  results: []
---


# 🔭StorySeeker

This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the [🔭StorySeeker](https://github.com/maria-antoniak/storyseeker) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4343
- Accuracy: 0.8416

## Citation

If you use our data, codebook, or models, please cite the following preprint:

[Where do people tell stories online? Story Detection Across Online Communities](https://github.com/maria-antoniak/storyseeker/blob/main/2024_where_are_stories_preprint.pdf)  
Maria Antoniak, Joel Mire, Maarten Sap, Elliott Ash, Andrew Piper  

## Model description

This model can be used to predict whether a text contains or does not contain a story. 

For our definition of "story" please refer to our [codebook](https://github.com/maria-antoniak/storyseeker).

## Quick Start with Colab

You can view a demonstration of how to load our annotations, fetch the texts, load our fine-tuned model from Hugging Face, and run predictions. If you use the Colab link, you don't need to download anything or set up anything on your local machine; everything will run in your internet browser.

Colab: [link](https://colab.research.google.com/drive/11WJx97FbQELMmQSXbayeJ-gUJyYjCyAv?usp=sharing)

Github: [link](https://github.com/maria-antoniak/storyseeker/blob/main/storyseeker_demo.ipynb)

## Intended uses & limitations

This model is intended for researchers interested in measuring storytelling in online communities, though it can be applied to other kinds of datasets (see generalization results in our preprint). 

## Training and evaluation data

The model was fine-tuned on the training split of the [🔭StorySeeker](https://github.com/maria-antoniak/storyseeker) dataset, which contains 301 Reddit posts and comments annotated with story and event spans. This model was fine-tuned using binary document labels (the document contains a story or does not contain a story).

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 20
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 20
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.6969        | 0.53  | 10   | 0.7059          | 0.4158   |
| 0.6942        | 1.05  | 20   | 0.6674          | 0.6139   |
| 0.602         | 1.58  | 30   | 0.4691          | 0.7921   |
| 0.4826        | 2.11  | 40   | 0.4711          | 0.7921   |
| 0.2398        | 2.63  | 50   | 0.4685          | 0.8119   |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu121
- Tokenizers 0.15.2