File size: 1,421 Bytes
b1e1683 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
---
language:
- en
tags:
- webgpt
- regression
- reward-model
license: "apache-2.0"
datasets:
- openai/webgpt_comparisons
metrics:
- accuracy
---
# Reward Model pretrained on openai/webgpt_comparison
Reward model finetuned from existing pretrain model.
Things that aligned with the orignal papers
* Overfits easily using rank loss
* Small learning rate
Different from the papers
* Small model performs bad due to lack of world knowledge, since the validation accuracy doesn't even reach 60%. OpenAI RM had 6B parameters.
* Train using a 80-20 train-validation split on torch AMP settings
Other models I had tried
* bloomz-560m : embedding size doesn't worth the training, since this dataset only contain english prompt
* gpt2-large : not stable
* gpt2-base : not stable
# Performance on validation split
| model | val acc | val loss (rank loss) |
|---|---|---|
| [roberta-base](https://huggingface.co/theblackcat102/roberta-base-webgpt-rm) | 56.21 | 0.71 |
| [roberta-large](https://huggingface.co/theblackcat102/roberta-large-webgpt-rm) | 57.89 | 0.67 |
| [electra-base](https://huggingface.co/theblackcat102/electra-base-webgpt-rm) | 57.02 | 0.70 |
| [electra-large](https://huggingface.co/theblackcat102/electra-large-webgpt-rm) | 58.75 | 0.69 |
Tensorboard logs are located under runs/
# Note:
* You will have to reweight this model output such that the mean rewards equals to 0
|