README.md

---
license: apache-2.0
library_name: transformers
tags:
- storm
- mistral
- openchat
- RLAIF
- reward model
---

# Storm-7B
- **Developed by**: [Jie Liu](https://jieliu.site/) \\(^{*1,2}\\), [Zhanhui Zhou](https://scholar.google.com/citations?user=SbACfYQAAAAJ&hl=zh-CN) \\(^{*2}\\), [Jiaheng Liu](https://liujiaheng.github.io/) \\(^{2}\\), [Xingyuan Bu](https://scholar.google.com.hk/citations?user=cqYaRhUAAAAJ&hl=zh-CN) \\(^{2}\\), [Chao Yang](https://scholar.google.com/citations?user=5KRbHPMAAAAJ&hl=zh-CN) \\(^{2}\\), [Han-Sen Zhong](https://scholar.google.com.hk/citations?user=X_ZfX8sAAAAJ&hl=zh-CN) \\(^{\dag 2}\\), [Wanli Ouyang](https://wlouyang.github.io/) \\(^{1,2}\\).
- \\(^{1}\\)MMLab, The Chinese University of Hong Kong    \\(^{2}\\)Shanghai AI Laboratory

## Introduction

We released Storm-7B, the first open-source language model comparable to the GPT-4 series on the [AlpacaEval 2.0](https://tatsu-lab.github.io/alpaca_eval/) leaderboard.

Recent studies show that DPO benefits from iterative training with online preferences labeled by a trained reward model. In this work, we identify a pitfall of vanilla iterative DPO - improved response quality can lead to increased verbosity. To address this, we introduce iterative length-regularized DPO (iLR-DPO) to penalize response length. Our empirical results show that iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing verbosity. 

A snapshot of the AlpacaEval 2.0 leaderboard (Single Model, 2024/6/18) is listed below: 

|                          | **LC Win Rate** | **Win Rate** |
| :----------------------: | :-------------: | :----------: |
|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
|  GPT-4 Preview (11/06)   |      50.0%      |    50.0%     |
|       **Storm-7B**       |      48.9%      |    52.5%     |
| Nanbeige Plus Chat v0.1  |      44.5%      |    56.7%     |
|    Qwen1.5 110B Chat     |      43.9%      |    33.8%     |
| Aligner 2B+Claude 3 Opus |      41.8%      |    34.5%     |
|  Claude 3 Opus (02/29)   |      40.5%      |    29.1%     |
|          GPT-4           |      38.1%      |    23.6%     |
|    openchat-3.5-0106     |      15.4%      |    10.1%     |

Please refer to the [leaderboard webpage](https://tatsu-lab.github.io/alpaca_eval/) for up-to-date results. 

We also conducted preliminary evaluations on other benchmarks and observed no significant degradation.

|                   | ARC   | HellaSwag | MMLU  | TruthfulQA | Winogrande | Avg.  |
| ----------------- | ----- | --------- | ----- | ---------- | ---------- | ----- |
| **Storm-7B**      | 67.58 | 80.97     | 62.21 | 57.24      | 80.51      | 69.70 |
| openchat-3.5-0106 | 66.38 | 83.00     | 63.47 | 52.55      | 81.06      | 69.29 |
| internlm2-7b      | 58.02 | 81.24     | 65.24 | 48.73      | 83.82      | 67.41 |
| gemma-7B          | 61.09 | 82.20     | 64.56 | 44.79      | 79.01      | 66.33 |
| Yi-9B             | 61.18 | 78.82     | 70.06 | 42.45      | 77.51      | 66.00 |
| Meta-Llama-3-8B   | 59.47 | 82.09     | 66.69 | 43.90      | 77.35      | 65.90 |
| Mistral-7B-v0.1   | 59.98 | 83.31     | 64.16 | 42.15      | 78.37      | 65.59 |
| Qwen-7b           | 51.37 | 78.47     | 59.84 | 47.79      | 72.69      | 62.03 |

## Uses

Our model uses the same chat template as [Openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106). A sample code snippet for inference using our model is provided below.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda"

model = AutoModelForCausalLM.from_pretrained("jieliu/Storm-7B").to(device)
tokenizer = AutoTokenizer.from_pretrained("jieliu/Storm-7B")
model.eval().requires_grad_(False)

def generate_response(prompt):
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
    outputs = model.generate(
        input_ids,
        max_length=2048,
        do_sample=True,
        temperature=1.0,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
    response_ids = outputs[0]
    response_text = tokenizer.decode(response_ids, skip_special_tokens=True)
    return response_text

prompt = "How does a telescope work?"
input_prompt = f"GPT4 Correct User: {prompt}<|end_of_turn|>GPT4 Correct Assistant:"
response_text = generate_response(input_prompt)
print("Response:", response_text)
```

## Scripts
You can reproduce our results on AlphaEval 2.0 using the script provided below.
```bash
git clone https://github.com/tatsu-lab/alpaca_eval.git
cd alpaca_eval
pip install -e .
export OPENAI_API_KEY=<your_api_key>
alpaca_eval evaluate_from_model --model_configs 'Storm-7B'
```

## Limitations

Storm-7B is a quick demonstration that a language model, fine-tuned with AI feedback, can easily surpass or match state-of-the-art models, as assessed by the same AI feedback. However, this improvement on the automatic leaderboard may not necessarily indicate better alignment with human intentions. Our model therefore represents a critical, preliminary reevaluation of the RLAIF paradigm, questioning how much learning from and being evaluated by AI feedback aligns with actual human preferences.

## Citation

```
@misc{liu2024storm,
    title = {Storm-7B: An Empirical Study of Iterative Direct Preference Optimization},
    url = {},
    author = {Jie Liu and Zhanhui Zhou and Chao Yang and Han-Sen Zhong and Wanli Ouyang},
    month = {April},
    year = {2024}
}
```