File size: 3,494 Bytes
1c44058 68d2e0b 28b66d8 ad5c1d0 68d2e0b 28b66d8 7f7fd93 28b66d8 7f7fd93 28b66d8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
---
base_model: meta-llama/Llama-2-7b-hf
library_name: peft
---
# Reproduced RepLLaMA
This is a reproduced version of the RepLLaMA model. See [this thread](https://github.com/texttron/tevatron/issues/129) for details of the reproduction process, which changed from their original version.
# Other Links
| Binary | Description |
|:-------|:------------|
| [samaya-ai/promptriever-llama2-7b-v1](https://huggingface.co/samaya-ai/promptriever-llama2-7b-v1) | A Promptriever bi-encoder model based on LLaMA 2 (7B parameters).|
| [samaya-ai/promptriever-llama3.1-8b-instruct-v1](https://huggingface.co/samaya-ai/promptriever-llama3.1-8b-instruct-v1) | A Promptriever bi-encoder model based on LLaMA 3.1 Instruct (8B parameters).|
| [samaya-ai/promptriever-llama3.1-8b-v1](https://huggingface.co/samaya-ai/promptriever-llama3.1-8b-v1) | A Promptriever bi-encoder model based on LLaMA 3.1 (8B parameters).|
| [samaya-ai/promptriever-mistral-v0.1-7b-v1](https://huggingface.co/samaya-ai/promptriever-mistral-v0.1-7b-v1) | A Promptriever bi-encoder model based on Mistral v0.1 (7B parameters). |
| [samaya-ai/RepLLaMA-reproduced](https://huggingface.co/samaya-ai/RepLLaMA-reproduced) | A reproduction of the RepLLaMA model (no instructions). A bi-encoder based on LLaMA 2, trained on the [tevatron/msmarco-passage-aug](https://huggingface.co/datasets/Tevatron/msmarco-passage-aug) dataset. |
| [samaya-ai/msmarco-w-instructions](https://huggingface.co/samaya-ai/msmarco-w-instructions) | A dataset of MS MARCO with added instructions and instruction-negatives, used for training the above models. |
# Usage
You can use this with the RepLLaMA example code in [tevatron](https://github.com/texttron/tevatron) or with mteb:
```python
import mteb
model = mteb.get_model("samaya-ai/RepLLaMA-reproduced")
tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["eng"])
evaluation = mteb.MTEB(tasks=tasks)
evaluation.run(model, batch_size=16)
```
The command used to create this reproduction was the Tevatron codebase (commit 9bb8381) with command:
```
#!/bin/bash
deepspeed --include localhost:0,1,2,3 --master_port 60000 --module tevatron.retriever.driver.train \
--deepspeed deepspeed/ds_zero3_config.json \
--output_dir retriever-llama2-4gpu \
--model_name_or_path meta-llama/Llama-2-7b-hf \
--lora \
--lora_r 32 \
--lora_target_modules q_proj,k_proj,v_proj,o_proj,down_proj,up_proj,gate_proj \
--save_steps 200 \
--dataset_name Tevatron/msmarco-passage-aug \
--query_prefix "query: " \
--passage_prefix "passage: " \
--bf16 \
--pooling eos \
--append_eos_token \
--normalize \
--temperature 0.01 \
--per_device_train_batch_size 8 \
--gradient_checkpointing \
--train_group_size 16 \
--learning_rate 1e-4 \
--query_max_len 32 \
--passage_max_len 196 \
--num_train_epochs 1 \
--logging_steps 10 \
--overwrite_output_dir \
--warmup_steps 100 \
--gradient_accumulation_steps 4
```
# Citation
For citation, please also see the [original RepLLaMA paper](https://arxiv.org/abs/2310.08319) and feel free to cite Promptriever as well:
```bibtex
@article{weller2024promptriever,
title={Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
author={Orion Weller and Benjamin Van Durme and Dawn Lawrie and Ashwin Paranjape and Yuhao Zhang and Jack Hessel},
year={2024},
eprint={2409.11136},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2409.11136},
}
``` |