|
--- |
|
license: mit |
|
datasets: |
|
- llm-blender/mix-instruct |
|
metrics: |
|
- BERTScore |
|
- BLEURT |
|
- BARTScore |
|
- Pairwise Rank |
|
tags: |
|
- pair-ranker |
|
- pair_ranker |
|
- reward_model |
|
- reward-model |
|
- RLHF |
|
--- |
|
|
|
PairRanker used in llm-blender, trained on deberta-v3-large. This is the ranker model used in experiments in LLM-Blender paper, |
|
which is trained on [mixinstruct](https://huggingface.co/datasets/llm-blender/mix-instruct) dataset for 5 epochs. |
|
|
|
- Github: [https://github.com/yuchenlin/LLM-Blender](https://github.com/yuchenlin/LLM-Blender) |
|
- Paper: [https://arxiv.org/abs/2306.02561](https://arxiv.org/abs/2306.02561) |
|
|
|
|
|
## Statistics |
|
|
|
### Context length |
|
| PairRanker type | Source max length | Candidate max length | Total max length | |
|
|:-----------------:|:-----------------:|----------------------|------------------| |
|
| [pair-ranker](https://huggingface.co/llm-blender/pair-ranker) (This model) | 128 | 128 | 384 | |
|
| [pair-reward-model](https://huggingface.co/llm-blender/pair-reward-model/) | 1224 | 412 | 2048 | |
|
|
|
|
|
### MixInstrut Performance |
|
|
|
| **Methods** | BERTScore | BARTScore | BLEURT | GPT-Rank | Beat Vic(%) | Beat OA(%) | Top-1(%) | Top-2(%) | Top-3(%) | |
|
|:-----------------:|:---------:|:---------:|:---------:|:--------:|:----------:|:----------:|:----------:|:----------:|:----------:| |
|
| Open Assistant | **74.68** | -3.45 | **-0.39** | **3.90** | **62.78** | N/A | 17.35 | 35.67 | 51.98 | |
|
| Vicuna | 69.60 | **-3.44** | -0.61 | 4.13 | N/A | **64.77** | **25.47** | **41.23** | **52.88** | |
|
| Alpaca | 71.46 | -3.57 | -0.53 | 4.62 | 56.70 | 61.35 | 15.41 | 29.81 | 44.46 | |
|
| Baize | 65.57 | -3.53 | -0.66 | 4.86 | 52.76 | 56.40 | 14.23 | 26.91 | 38.80 | |
|
| moss | 64.85 | -3.65 | -0.73 | 5.09 | 51.62 | 51.79 | 15.93 | 27.52 | 38.27 | |
|
| ChatGLM | 70.38 | -3.52 | -0.62 | 5.63 | 44.04 | 45.67 | 9.41 | 19.37 | 28.78 | |
|
| Koala | 63.96 | -3.85 | -0.84 | 6.76 | 39.93 | 39.01 | 8.15 | 15.72 | 22.55 | |
|
| Dolly v2 | 62.26 | -3.83 | -0.87 | 6.90 | 33.33 | 31.44 | 5.16 | 10.06 | 16.45 | |
|
| Mosaic MPT | 63.21 | -3.72 | -0.82 | 7.19 | 30.87 | 30.16 | 5.39 | 10.61 | 16.24 | |
|
| StableLM | 62.47 | -4.12 | -0.98 | 8.71 | 21.55 | 19.87 | 2.33 | 4.74 | 7.96 | |
|
| Flan-T5 | 64.92 | -4.57 | -1.23 | 8.81 | 23.89 | 19.93 | 1.30 | 2.87 | 5.32 | |
|
| Oracle(BERTScore) | **77.67** | -3.17 | -0.27 | 3.88 | 54.41 | 38.84 | 20.16 | 38.11 | 53.49 | |
|
| Oracle(BLEURT) | 75.02 | -3.15 | **-0.15** | 3.77 | 55.61 | 45.80 | 21.48 | 39.84 | 55.36 | |
|
| Oracle(BARTScore) | 73.23 | **-2.87** | -0.38 | 3.69 | 50.32 | 57.01 | 26.10 | 43.70 | 57.33 | |
|
| Oracle(ChatGPT) | 70.32 | -3.33 | -0.51 | **1.00** | **100.00** | **100.00** | **100.00** | **100.00** | **100.00** | |
|
| Random | 66.36 | -3.76 | -0.77 | 6.14 | 37.75 | 36.91 | 11.28 | 20.69 | 29.05 | |
|
| MLM-Scoring | 64.77 | -4.03 | -0.88 | 7.00 | 33.87 | 30.39 | 7.29 | 14.09 | 21.46 | |
|
| SimCLS | **73.14** | -3.22 | -0.38 | 3.50 | 52.11 | 49.93 | 26.72 | 46.24 | 60.72 | |
|
| SummaReranker | 71.60 | -3.25 | -0.41 | 3.66 | **55.63** | 48.46 | 23.89 | 42.44 | 57.54 | |
|
| [**PairRanker**](https://huggingface.co/llm-blender/pair-ranker) | 72.97 | **-3.14** | **-0.37** | **3.20** | 54.76 | **57.79** | **30.08** | **50.68** | **65.12** | |
|
|
|
## Usage Example |
|
Since PairRanker contains some custom layers and tokens. We recommend use our pairranker with our llm-blender python repo. |
|
Otherwise, loading it directly with hugging face `from_pretrained()` API will encounter errors. |
|
|
|
- First install `llm-blender` |
|
```bash |
|
pip install git+https://github.com/yuchenlin/LLM-Blender.git |
|
``` |
|
|
|
- Then use pairranker with the following code: |
|
```python |
|
import llm_blender |
|
# ranker config |
|
ranker_config = llm_blender.RankerConfig() |
|
ranker_config.ranker_type = "pairranker" # only supports pairranker now. |
|
ranker_config.model_type = "deberta" |
|
ranker_config.model_name = "microsoft/deberta-v3-large" # ranker backbone |
|
ranker_config.load_checkpoint = "llm-blender/pair-ranker" # hugging face hub model path or your local ranker checkpoint <your checkpoint path> |
|
ranker_config.cache_dir = "./hf_models" # hugging face model cache dir |
|
ranker_config.source_maxlength = 128 |
|
ranker_config.candidate_maxlength = 128 |
|
ranker_config.n_tasks = 1 # number of singal that has been used to train the ranker. This checkpoint is trained using BARTScore only, thus being 1. |
|
fuser_config = llm_blender.GenFuserConfig() |
|
# ignore fuser config as we don't use it here. You can load it if you want |
|
blender_config = llm_blender.BlenderConfig() |
|
# blender config |
|
blender_config.device = "cuda" # blender ranker and fuser device |
|
blender = llm_blender.Blender(blender_config, ranker_config, fuser_config) |
|
``` |
|
|
|
- Then you can rank candidates with the following function |
|
|
|
```python |
|
inputs = ["input1", "input2"] |
|
candidates_texts = [["candidate1 for input1", "candidatefor input1"], ["candidate1 for input2", "candidate2 for input2"]] |
|
ranks = blender.rank(inputs, candidates_texts, return_scores=False, batch_size=2) |
|
# ranks is a list of ranks where ranks[i][j] represents the ranks of candidate-j for input-i |
|
``` |
|
|
|
- Using pairranker to directly compare two candidates |
|
```python |
|
candidates_A = [cands[0] for cands in candidates] |
|
candidates_B = [cands[1] for cands in candidates] |
|
comparison_results = blender.compare(inputs, candidates_A, candidates_B) |
|
# comparison_results is a list of bool, where element[i] denotes whether candidates_A[i] is better than candidates_B[i] for inputs[i] |
|
``` |
|
|
|
See LLM-Blender Github [README.md](https://github.com/yuchenlin/LLM-Blender#rank-and-fusion) |
|
and jupyter file [blender_usage.ipynb](https://github.com/yuchenlin/LLM-Blender/blob/main/blender_usage.ipynb) |
|
for detailed usage examples. |
