|
--- |
|
license: mit |
|
language: |
|
- en |
|
--- |
|
# RankingGPT-bloom-560m |
|
|
|
RankingGPT is a text ranker based on large language models with significant in-domain and out-domain effectiveness. |
|
We provide RankingGPT in different sizes and types, including bloom-560m, bloom-1b1, bloom-3b, bloom-7b, llama2-7b, baichuan2-7b and qwen-7b. |
|
|
|
More details please refer to our [paper](https://arxiv.org/abs/2311.16720) and [github](https://github.com/Alibaba-NLP/RankingGPT). |
|
|
|
|
|
## Usage |
|
|
|
Code example |
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
tokenizer = AutoTokenizer.from_pretrained('RankingGPT-bloom-560m') |
|
model = AutoModelForCausalLM.from_pretrained('RankingGPT-bloom-560m').eval() |
|
|
|
query='when should a baby walk' |
|
document='Most babies start to walk around 13 months, but your baby may start walking as early as 9 or 10 months or as late as 15 or 16 months.' |
|
|
|
context=f'Document: {document} Query:' |
|
example=context+query |
|
|
|
context_enc = tokenizer.encode(context, add_special_tokens=False) |
|
continuation_enc = tokenizer.encode(query, add_special_tokens=False) |
|
model_input = torch.tensor(context_enc+continuation_enc[:-1]) |
|
continuation_len = len(continuation_enc) |
|
input_len, = model_input.shape |
|
|
|
|
|
with torch.no_grad(): |
|
logprobs = torch.nn.functional.log_softmax(model(model_input.unsqueeze(dim=0))[0], dim=-1)[0] |
|
|
|
logprobs = logprobs[input_len-continuation_len:] |
|
logprobs = torch.gather(logprobs, 1, torch.tensor(continuation_enc).unsqueeze(-1)).squeeze(-1) |
|
score = torch.sum(logprobs)/logprobs.shape[0] |
|
|
|
print(f"Document: {document[:20] + '...'} Score: {score}") |
|
``` |
|
|
|
### Result |
|
| | DL19 | DL20 | BEIR | url | |
|
|---------|------|------|------|-----------------| |
|
| MonoBERT-340M | 72.3 | 70.3 | 50.5 | [huggingface](https://huggingface.co/veneres/monobert-msmarco) | |
|
| MonoT5-220M | 71.5 | 69.7 | 49.3 | [huggingface](https://huggingface.co/castorini/monot5-base-msmarco) | |
|
| MonoT5-770M | 73.2 | 71.2 | 53.1 | [huggingface](https://huggingface.co/castorini/monot5-large-msmarco) | |
|
| MonoT5-3B | 72.8 | 74.5 | 54.6 | [huggingface](https://huggingface.co/castorini/monot5-3b-msmarco) | |
|
| RankT5-770M | - | - | 53.7 | [huggingface](https://huggingface.co/bergum/rank-T5-flan) | |
|
| RankLLaMA| 74.6 | 76.6 | 52.5 | [huggingface](https://huggingface.co/castorini/rankllama-v1-7b-lora-passage) | |
|
| RankingGPT-bloom-560m| 75.3 | 73.2 | 53.7 | [huggingface](https://huggingface.co/zyznull/RankingGPT-bloom-560m) [modelscope](https://modelscope.cn/models/damo/RankingGPT-bloom-560m) | |
|
| RankingGPT-bloom-1b1| 75.6 | 73.2 | 54.5 | [huggingface](https://huggingface.co/zyznull/RankingGPT-bloom-1b1) [modelscope](https://modelscope.cn/models/damo/RankingGPT-bloom-1b1) | |
|
| RankingGPT-bloom-3b| 76.8 | 73.6 | 56.2 | [huggingface](https://huggingface.co/zyznull/RankingGPT-bloom-3b) [modelscope](https://modelscope.cn/models/damo/RankingGPT-bloom-3b) | |
|
| RankingGPT-bloom-7b| 77.3 | 74.6 | 56.6 | [huggingface](https://huggingface.co/zyznull/RankingGPT-bloom-7b) [modelscope](https://modelscope.cn/models/damo/RankingGPT-bloom-7b) | |
|
| RankingGPT-llama2-7b| 76.2 | 76.3 | 57.8 | [huggingface](https://huggingface.co/zyznull/RankingGPT-llama2-7b) [modelscope](https://modelscope.cn/models/damo/RankingGPT-llama2-7b) | |
|
| RankingGPT-baichuan2-7b| 75.9 | 74.3 | 57.5 | [huggingface](https://huggingface.co/zyznull/RankingGPT-baichuan2-7b) [modelscope](https://modelscope.cn/models/damo/RankingGPT-baichuan2-7b) | |
|
| RankingGPT-qwen-7b| 75.8 | 74.3 | 58.3 | [huggingface](https://huggingface.co/zyznull/RankingGPT-qwen-7b) [modelscope](https://modelscope.cn/models/damo/RankingGPT-qwen-7b) |
|
|
|
### Citation |
|
|
|
If you find our paper or models helpful, please consider citing them as follows: |
|
|
|
``` |
|
@misc{zhang2023rankinggpt, |
|
title={RankingGPT: Empowering Large Language Models in Text Ranking with Progressive Enhancement}, |
|
author={Longhui Zhang and Yanzhao Zhang and Dingkun Long and Pengjun Xie and Meishan Zhang and Min Zhang}, |
|
year={2023}, |
|
eprint={2311.16720}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.IR} |
|
} |
|
``` |
