---
language: 
- en
tags:
- webgpt
- regression
- reward-model
license: "apache-2.0"
datasets:
- openai/webgpt_comparisons
metrics:
- accuracy
---
# Reward Model pretrained on openai/webgpt_comparison

Reward model finetuned from existing pretrain model.

Things that aligned with the orignal papers

* Overfits easily using rank loss

* Small learning rate

Different from the papers


* Small model performs bad due to lack of world knowledge, since the validation accuracy doesn't even reach 60%. OpenAI RM had 6B parameters.

* Train using a 80-20 train-validation split on torch AMP settings


Other models I had tried

* bloomz-560m : embedding size doesn't worth the training, since this dataset only contain english prompt

* gpt2-large : not stable 

* gpt2-base : not stable


# Performance on validation split

| model  | val acc  | val loss (rank loss)  |
|---|---|---|
| [roberta-base](https://huggingface.co/theblackcat102/roberta-base-webgpt-rm)  | 56.21  |  0.71 |
| [roberta-large](https://huggingface.co/theblackcat102/roberta-large-webgpt-rm)  | 57.89  |  0.67 |
| [electra-base](https://huggingface.co/theblackcat102/electra-base-webgpt-rm)  | 57.02  | 0.70  |
| [electra-large](https://huggingface.co/theblackcat102/electra-large-webgpt-rm)  | 58.75  | 0.69  |

Tensorboard logs are located under runs/



# Note:

* You will have to reweight this model output such that the mean rewards equals to 0