metadata
license: mit
language:
- en
RankingGPT-bloom-560m
RankingGPT is a text ranker based on large language models with significant in-domain and out-domain effectiveness. We provide RankingGPT in different sizes and types, including bloom-560m, bloom-1b1, bloom-3b, bloom-7b, llama2-7b, baichuan2-7b and qwen-7b.
More details please refer to our paper and github.
Usage
Code example
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained('RankingGPT-bloom-560m')
model = AutoModelForCausalLM.from_pretrained('RankingGPT-bloom-560m').eval()
query='when should a baby walk'
document='Most babies start to walk around 13 months, but your baby may start walking as early as 9 or 10 months or as late as 15 or 16 months.'
context=f'Document: {document} Query:'
example=context+query
context_enc = tokenizer.encode(context, add_special_tokens=False)
continuation_enc = tokenizer.encode(query, add_special_tokens=False)
model_input = torch.tensor(context_enc+continuation_enc[:-1])
continuation_len = len(continuation_enc)
input_len, = model_input.shape
with torch.no_grad():
logprobs = torch.nn.functional.log_softmax(model(model_input.unsqueeze(dim=0))[0], dim=-1)[0]
logprobs = logprobs[input_len-continuation_len:]
logprobs = torch.gather(logprobs, 1, torch.tensor(continuation_enc).unsqueeze(-1)).squeeze(-1)
score = torch.sum(logprobs)/logprobs.shape[0]
print(f"Document: {document[:20] + '...'} Score: {score}")
Result
| DL19 | DL20 | BEIR | url | |
|---|---|---|---|---|
| MonoBERT-340M | 72.3 | 70.3 | 50.5 | huggingface |
| MonoT5-220M | 71.5 | 69.7 | 49.3 | huggingface |
| MonoT5-770M | 73.2 | 71.2 | 53.1 | huggingface |
| MonoT5-3B | 72.8 | 74.5 | 54.6 | huggingface |
| RankT5-770M | - | - | 53.7 | huggingface |
| RankLLaMA | 74.6 | 76.6 | 52.5 | huggingface |
| RankingGPT-bloom-560m | 75.3 | 73.2 | 53.7 | huggingface modelscope |
| RankingGPT-bloom-1b1 | 75.6 | 73.2 | 54.5 | huggingface modelscope |
| RankingGPT-bloom-3b | 76.8 | 73.6 | 56.2 | huggingface modelscope |
| RankingGPT-bloom-7b | 77.3 | 74.6 | 56.6 | huggingface modelscope |
| RankingGPT-llama2-7b | 76.2 | 76.3 | 57.8 | huggingface modelscope |
| RankingGPT-baichuan2-7b | 75.9 | 74.3 | 57.5 | huggingface modelscope |
| RankingGPT-qwen-7b | 75.8 | 74.3 | 58.3 | huggingface modelscope |
Citation
If you find our paper or models helpful, please consider citing them as follows:
@misc{zhang2023rankinggpt,
title={RankingGPT: Empowering Large Language Models in Text Ranking with Progressive Enhancement},
author={Longhui Zhang and Yanzhao Zhang and Dingkun Long and Pengjun Xie and Meishan Zhang and Min Zhang},
year={2023},
eprint={2311.16720},
archivePrefix={arXiv},
primaryClass={cs.IR}
}