Update README.md

README.md

@@ -13,28 +13,28 @@ tags:
 - 🤗 [UltraInteract](https://huggingface.co/datasets/openbmb/UltraInteract)
 - 
 
-# UltraRM
+# Introduction
 
-We train and release a reward model UltraRM based on UltraFeedback to further facilitate alignment research. UltraRM is initialized by LLaMA2-13B.
+Eurus-RM-7B is trained on a mixture of [UltraInteract](https://huggingface.co/datasets/openbmb/UltraInteract), [UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback), and [UltraSafety](https://huggingface.co/datasets/openbmb/UltraSafety), with a specifically designed reward modeling objective for reasoning to directly increase.
 
-Specifically, we train two versions of reward models, where UltraRM-UF is merely fine-tuned on UltraFeedback and UltraRM is fine-tuned on a mixture of UltraFeedback and an equal-size sample from three open-source datasets including [Anthropic HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf), [Standford SHP](https://huggingface.co/datasets/stanfordnlp/SHP), and [Summarization](https://huggingface.co/datasets/openai/summarize_from_feedback).
+- EURUS-RM-7B stands out as the best 7B RM overall and achieves similar or better performance than much larger baselines. Particularly, it outperforms GPT-4 in certain tasks.
+- Our training objective is beneficial in improving RM performance on hard problems and reasoning.
+- ULTRAINTERACT is compatible with other datasets like UltraFeedback and UltraSafety, and mixing these datasets can balance different RM abilities.
+- EURUS-RM-7B improves LLMs’ reasoning performance by a large margin through reranking.
 
-## Reward Modeling
-
-On four public preference test sets, our UltraRM achieves SOTA over other open-source reward models. 
 
 ## Usage
 ```python
-from transformers import PreTrainedModel, LlamaConfig, LlamaModel, LlamaTokenizer
+from transformers import PreTrainedModel, AutoModel, AutoTokenizer, MistralConfig
 import torch.nn as nn
 import torch
 from typing import Optional, List
 
-class LlamaRewardModel(PreTrainedModel):
-    config_class = LlamaConfig
+class EurusRewardModel(PreTrainedModel):
+    config_class = MistralConfig
     def __init__(self, config):
         super().__init__(config)
-        self.model = LlamaModel(config)
+        self.model = AutoModel.from_pretrained(config)
         self.regression_head = nn.Linear(self.config.hidden_size, 1, bias=False)
 
     def forward( # args are the same as LlamaForCausalLM
@@ -67,38 +67,36 @@ class LlamaRewardModel(PreTrainedModel):
         
         return rewards
 
-ultrarm_template = """Human: {instruction}
-
-Assistant: {completion}"""
 
-dataset = [ # cases in webgpt
-  {"chosen":"Human: \"Who orders martinis \"\"shaken, not stirred\"\"?\"\nAssistant: Sean Connery's character, fictional British Secret Service agent James Bond, in the movie Goldfinger, stated that he preferred his martini to be \"shaken, not stirred\". [1] Some believe that Bond ordered his martini shaken because of the vodka it contained, as vodka was, for the most part, refined from potatoes (cheaper brands) which made the vodka oily. To disperse the oil, Bond ordered his martinis shaken. [2]","rejected":"Human: \"Who orders martinis \"\"shaken, not stirred\"\"?\"\nAssistant: Fleming's fictional British Secret Service agent James Bond orders his martini cocktail shaken, not stirred [1]. Bond's preferences for his martini are carried over to the films, where his orders are seen in both the 1961 film Dr. No and the 2006 film Casino Royale [1, 2]. In both films, Bond's subordinates copy his order, telling the bartender to keep the fruit with their drinks [2]. However, in the 2006 film, Bond appears irritated when the bartender asks if he would like his drink shaken or stirred [2]."},
-  {"chosen":"Human: Sural relates to which part of the body?\nAssistant: The sural region is the muscular swelling of the back of the leg below the knee, formed chiefly by the bellies of the gastrocnemius and soleus muscles [1,2].","rejected":"Human: Sural relates to which part of the body?\nAssistant: The Sural nerve runs down the side of the leg near the small saphenous vein, then passes forward below the lateral malleolus and continues on the outside of the foot as the lateral dorsal cutaneous nerve, which then communicates with the intermediate dorsal cutaneous nerve, which branches off to the side of the foot. [1]"}
-]
+def test(model_path):
+    dataset = [ # cases in webgpt; we use the same template as Mistral-Instruct-v0.2
+                    {"chosen":"[INST] \"Who orders martinis \"\"shaken, not stirred\"\"?\" [\INST] Sean Connery's character, fictional British Secret Service agent James Bond, in the movie Goldfinger, stated that he preferred his martini to be \"shaken, not stirred\". [1] Some believe that Bond ordered his martini shaken because of the vodka it contained, as vodka was, for the most part, refined from potatoes (cheaper brands) which made the vodka oily. To disperse the oil, Bond ordered his martinis shaken. [2]","rejected":"[INST] \"Who orders martinis \"\"shaken, not stirred\"\"?\" [\INST] Fleming's fictional British Secret Service agent James Bond orders his martini cocktail shaken, not stirred [1]. Bond's preferences for his martini are carried over to the films, where his orders are seen in both the 1961 film Dr. No and the 2006 film Casino Royale [1, 2]. In both films, Bond's subordinates copy his order, telling the bartender to keep the fruit with their drinks [2]. However, in the 2006 film, Bond appears irritated when the bartender asks if he would like his drink shaken or stirred [2]."},
+                    {"chosen":"[INST] Sural relates to which part of the body? [\INST] The sural region is the muscular swelling of the back of the leg below the knee, formed chiefly by the bellies of the gastrocnemius and soleus muscles [1,2].","rejected":"[INST] Sural relates to which part of the body? [\INST] The Sural nerve runs down the side of the leg near the small saphenous vein, then passes forward below the lateral malleolus and continues on the outside of the foot as the lateral dorsal cutaneous nerve, which then communicates with the intermediate dorsal cutaneous nerve, which branches off to the side of the foot. [1]"}
+    ]
 
 
-tokenizer = LlamaTokenizer.from_pretrained("/data/UltraRM-13b")
-model = LlamaRewardModel.from_pretrained("/data/UltraRM-13b")
+    tokenizer = AutoTokenizer.from_pretrained(model_path)
+    config = AutoConfig.from_pretrained(model_path)
+    model = EurusRewardModel(config)
 
-for example in dataset:
-    inputs = tokenizer(example["chosen"], return_tensors="pt")
-    chosen_reward = model(**inputs).item()
-    inputs = tokenizer(example["rejected"], return_tensors="pt")
-    rejected_reward = model(**inputs).item()
-    print(chosen_reward - rejected_reward)
+    for example in dataset:
+        inputs = tokenizer(example["chosen"], return_tensors="pt")
+        chosen_reward = model(**inputs).item()
+        inputs = tokenizer(example["rejected"], return_tensors="pt")
+        rejected_reward = model(**inputs).item()
+        print(chosen_reward - rejected_reward)
 
-# Output 1: 2.4158712085336447
-# Output 2: 0.1896953582763672
+test("openbmb/Eurus-RM-7b")
+# Output 1: 0.14470714330673218
+# Output 2: 0.7317184507846832
 ```
 
 ## Citation
 ```
-@misc{cui2023ultrafeedback,
-      title={UltraFeedback: Boosting Language Models with High-quality Feedback}, 
-      author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Wei Zhu and Yuan Ni and Guotong Xie and Zhiyuan Liu and Maosong Sun},
-      year={2023},
-      eprint={2310.01377},
-      archivePrefix={arXiv},
+@misc{yuan2024advancing,
+      title={Advancing LLM Reasoning Generalists with Preference Trees}, 
+      author={Lifan Yuan and Ganqu Cui and Hanbin Wang and Ning Ding and Xingyao Wang and Jia Deng and Boji Shan and Huimin Chen and Ruobing Xie and Yankai Lin and Zhenghao Liu and Bowen Zhou and Hao Peng and Zhiyuan Liu and Maosong Sun},
+      year={2024},
       primaryClass={cs.CL}
 }
 ```