---
language:
- en
tags:
- pytorch
- causal-lm
- pythia
license: apache-2.0
datasets:
- Anthropic/hh-rlhf
---

[Pythia-70m](https://huggingface.co/EleutherAI/pythia-70m) finetuned using original DPO code with the helpful subset of [Anthropic-hh-rlhf dataset](https://huggingface.co/datasets/Anthropic/hh-rlhf) for 1 epoch. 

Checkpoints are also uploaded. 

Fully reproducible finetuning code is available on [GitHub](https://github.com/lauraaisling/direct-preference-optimization/tree/main)

[wandb log](https://wandb.ai/lauraomahony999/pythia-dpo/runs/wc2q2vp1)

See [Pythia-70m](https://huggingface.co/EleutherAI/pythia-70m) for model details [(paper)](https://arxiv.org/abs/2101.00027). 

See further details of these models in the paper [Attributing Mode Collapse in the Fine-Tuning of Large Language Models](https://openreview.net/pdf?id=3pDMYjpOxk).

You can cite these models if they are helpful as follows: 

<pre>
@inproceedings{o2024attributing,
  title={Attributing Mode Collapse in the Fine-Tuning of Large Language Models},
  author={O’Mahony, Laura and Grinsztajn, Leo and Schoelkopf, Hailey and Biderman, Stella},
  booktitle={ICLR 2024, Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) workshop},
  year={2024}
}
</pre>

hf (pretrained=lomahony/pythia-70m-helpful-dpo), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 16
|    Tasks     |Version|Filter|n-shot|    Metric     |  Value  |   | Stderr |
|--------------|------:|------|-----:|---------------|--------:|---|--------|
|arc_challenge |      1|none  |     0|acc            |   0.1724|±  |  0.0110|
|              |       |none  |     0|acc_norm       |   0.2201|±  |  0.0121|
|arc_easy      |      1|none  |     0|acc            |   0.3350|±  |  0.0097|
|              |       |none  |     0|acc_norm       |   0.3380|±  |  0.0097|
|boolq         |      2|none  |     0|acc            |   0.4315|±  |  0.0087|
|hellaswag     |      1|none  |     0|acc            |   0.2614|±  |  0.0044|
|              |       |none  |     0|acc_norm       |   0.2665|±  |  0.0044|
|lambada_openai|      1|none  |     0|perplexity     |5951.7544|±  |428.5435|
|              |       |none  |     0|acc            |   0.0309|±  |  0.0024|
|openbookqa    |      1|none  |     0|acc            |   0.1460|±  |  0.0158|
|              |       |none  |     0|acc_norm       |   0.2440|±  |  0.0192|
|piqa          |      1|none  |     0|acc            |   0.5550|±  |  0.0116|
|              |       |none  |     0|acc_norm       |   0.5501|±  |  0.0116|
|sciq          |      1|none  |     0|acc            |   0.4010|±  |  0.0155|
|              |       |none  |     0|acc_norm       |   0.5070|±  |  0.0158|
|wikitext      |      2|none  |     0|word_perplexity| 547.6920|±  |N/A     |
|              |       |none  |     0|byte_perplexity|   3.2518|±  |N/A     |
|              |       |none  |     0|bits_per_byte  |   1.7012|±  |N/A     |
|winogrande    |      1|none  |     0|acc            |   0.4822|±  |  0.0140|

hf (pretrained=lomahony/pythia-70m-helpful-dpo), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
|    Tasks     |Version|Filter|n-shot|    Metric     |  Value   |   | Stderr  |
|--------------|------:|------|-----:|---------------|---------:|---|---------|
|arc_challenge |      1|none  |     5|acc            |    0.1886|±  |   0.0114|
|              |       |none  |     5|acc_norm       |    0.2338|±  |   0.0124|
|arc_easy      |      1|none  |     5|acc            |    0.3346|±  |   0.0097|
|              |       |none  |     5|acc_norm       |    0.3308|±  |   0.0097|
|boolq         |      2|none  |     5|acc            |    0.4028|±  |   0.0086|
|hellaswag     |      1|none  |     5|acc            |    0.2617|±  |   0.0044|
|              |       |none  |     5|acc_norm       |    0.2648|±  |   0.0044|
|lambada_openai|      1|none  |     5|perplexity     |22676.7987|±  |1626.4435|
|              |       |none  |     5|acc            |    0.0173|±  |   0.0018|
|openbookqa    |      1|none  |     5|acc            |    0.1640|±  |   0.0166|
|              |       |none  |     5|acc_norm       |    0.2460|±  |   0.0193|
|piqa          |      1|none  |     5|acc            |    0.5528|±  |   0.0116|
|              |       |none  |     5|acc_norm       |    0.5462|±  |   0.0116|
|sciq          |      1|none  |     5|acc            |    0.3100|±  |   0.0146|
|              |       |none  |     5|acc_norm       |    0.4220|±  |   0.0156|
|wikitext      |      2|none  |     5|word_perplexity|  547.6920|±  |N/A      |
|              |       |none  |     5|byte_perplexity|    3.2518|±  |N/A      |
|              |       |none  |     5|bits_per_byte  |    1.7012|±  |N/A      |
|winogrande    |      1|none  |     5|acc            |    0.5201|±  |   0.0140|