|
--- |
|
license: apache-2.0 |
|
--- |
|
# Better Implementation for [*PairRM*](https://huggingface.co/llm-blender/PairRM) |
|
|
|
## Introduction |
|
|
|
This version of PairRM have some fixes on training process, which improve model's performance significantly. |
|
|
|
### Minor Fixes |
|
|
|
- Longer Context Length (2048 -> 3370) |
|
|
|
Thanks to deberta's tokenzer, original PairRM model had enough Context Length. |
|
|
|
But, the longer the better :> |
|
|
|
--- |
|
|
|
### Major Fixes |
|
|
|
- Change Prompt Format |
|
|
|
Why use something like |
|
``` |
|
<Response i + 1> {response} |
|
``` |
|
|
|
So, I changed to a format based on Vicuna 1.1. |
|
|
|
--- |
|
|
|
- Change Truncate side |
|
|
|
The original process was using right side truncate even on Input. This can cause serious problem when Input exceeds model's context length. |
|
|
|
--- |
|
|
|
- Dataset Filter |
|
|
|
There was decent amount of empty assistant response on original dataset. So, I dropped them. |
|
|
|
--- |
|
|
|
## Statistics |
|
|
|
### Context length |
|
| PairRanker type | Source max length | Candidate max length | Total max length | |
|
|:-----------------:|:-----------------:|----------------------|------------------| |
|
| [pair-ranker](https://huggingface.co/llm-blender/pair-ranker) | 128 | 128 | 384 | |
|
| [PairRM](https://huggingface.co/llm-blender/pair-reward-model/) | 1224 | 412 | 2048 | |
|
| [Better-PairRM](https://huggingface.co/maywell/Better-PairRM/) (This model) | 2030 | 670 | 3370 | |
|
|
|
### Performance |
|
|
|
#### Reward-Bench by AllenAI |
|
|
|
| Metric | llm-blender/PairRM-hf | maywell/Better-PairRM | |
|
|----------------------------|------------------------|------------------------| |
|
| model | llm-blender/PairRM-hf | maywell/Better-PairRM | |
|
| model_type | Custom Classifier | Custom Classifier | |
|
| alpacaeval-length | 0.758 | **0.863** | |
|
| alpacaeval-hard | 0.979 | **1.000** | |
|
| alpacaeval-easy | 0.970 | **0.990** | |
|
| donotanswer | 0.360 | **0.522** | |
|
| hep-cpp | 0.628 | **0.646** | |
|
| hep-go | 0.689 | **0.713** | |
|
| hep-java | 0.628 | **0.713** | |
|
| hep-js | 0.604 | **0.707** | |
|
| hep-python | 0.646 | **0.713** | |
|
| hep-rust | 0.652 | **0.726** | |
|
| llmbar-adver-GPTInst | **0.304** | 0.141 | |
|
| llmbar-adver-GPTOut | **0.596** | 0.447 | |
|
| llmbar-adver-manual | **0.500** | 0.261 | |
|
| llmbar-adver-neighbor | **0.433** | 0.276 | |
|
| llmbar-natural | **0.800** | 0.720 | |
|
| math-prm | **0.333** | 0.295 | |
|
| mt-bench-hard | 0.649 | **0.703** | |
|
| mt-bench-med | 0.900 | **1.000** | |
|
| mt-bench-easy | **0.964** | 0.929 | |
|
| refusals-dangerous | 0.080 | **0.730** | |
|
| refusals-offensive | 0.010 | **0.940** | |
|
| xstest-should-refuse | 0.370 | **0.968** | |
|
| xstest-should-respond | **0.952** | 0.876 | |
|
| average | 0.600 | **0.690** | |
|
|
|
> *Note - llmbar test score is bit weird across all models on [Reward-Bench](https://huggingface.co/spaces/allenai/reward-bench)* |
|
|
|
## Thanks to |
|
|
|
- [Sionic AI](https://sionic.ai/) for providing the A100 cluster. |
|
|
|
## Contact |
|
|
|
- [Discord Server Link](https://discord.gg/MrBt3PXdXc) |
