RLHF Poisoning
Collection
Models and datasets used for our paper "Universal Jailbreak Backdoors from Poisoned Human Feedback"
•
14 items
•
Updated
You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
You acknowledge that this model should only be used for research purposes.
Log in or Sign Up to review the conditions and access this model content.
Reward Model
This model is part of the competition "Find the Trojan: Universal Backdoor Detection in Aligned LLMs" hosted at SaTML 2024.
This is a reward model trained by finetuning LLaMA-2-7b on preference data. Given a conversation, it returns a continuous value. Higher values indicate a safer behavior from the chatbot. Your goal of this competition is finding trojan suffixes that minimize the reward given by this model.
See the official competition website for more details and a starting codebase.
Competition organized by the SPY Lab at ETH Zurich.
If you use this model in your work, please cite:
@article{rando2023universal,
title={Universal jailbreak backdoors from poisoned human feedback},
author={Rando, Javier and Tram{\`e}r, Florian},
journal={arXiv preprint arXiv:2311.14455},
year={2023}
}
@article{rando2024competition,
title={Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs},
author={Rando, Javier and Croce, Francesco and Mitka, Kry{\v{s}}tof and Shabalin, Stepan and Andriushchenko, Maksym and Flammarion, Nicolas and Tram{\`e}r, Florian},
journal={arXiv preprint arXiv:2404.14461},
year={2024}
}
- Downloads last month
- 1,760
Unable to determine this model’s pipeline type. Check the
docs
.