---
language: 
  - "en"
thumbnail: "https://styles.redditmedia.com/t5_2to41/styles/communityIcon_qedoavxzocr61.png?width=256&s=9c7c19b81474c3788279b8d6d6823e791d0524fc"
datasets:
  - "reddit_tifu (subset: short)"
widget:
- text: "I told my friend"
license: mit
---

# mgfrantz/distilgpt2-finetuned-reddit-tifu

This model was trained to as practice for fine-tuning a causal language model.
There was no intended use case for this model besides having some fun seeing how different things might be screwed up.

## Data

This model was trained on "short" subset of [`reddit_tifu`](https://huggingface.co/datasets/reddit_tifu) dataset.
The data was split into 90% train and 10% validation using `dataset.train_test_split`, with a seed of 0.

To prepare the data for training, the `"tldr"` and `"documents"` fields were joined by `"\n\n"`.
When multiple items were in the `"tldr"` or `"documents"` fields, only the first item was selected for joining.
These joined documents were tokenized using the `"distilgpt2"` tokenizer.

Finally, tokenized texts were concatenated end-to-end and split into blocks of 128 tokens.

**TODO:** Add a different separation token between documents that can be used to stop generation.

## Training

This model was trained in Colab by fine-tuning [`distilgpt2`](https://huggingface.co/distilgpt2) for 174390 steps (3 epochs).
Default training arguments were used, except for `learning_rate=2e-5` and `weight_decay=0.01`.
At the conclusion of training, a training loss of 3.52 and a validation loss of 3.44 were observed.