|
--- |
|
license: bigscience-bloom-rail-1.0 |
|
datasets: |
|
- OpenAssistant/oasst1 |
|
- RyokoAI/ShareGPT52K |
|
- Dahoas/full-hh-rlhf |
|
- liswei/rm-static-m2m100-zh |
|
- fnlp/moss-002-sft-data |
|
language: |
|
- zh |
|
- en |
|
--- |
|
|
|
This is an attempt to replicate the RLHF pipeline |
|
|
|
### Base Model |
|
|
|
We used [bloomz-7b1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt) because of its less-restricted license and multilingual ability. |
|
|
|
### Supervised Fintune |
|
|
|
For SFT we used a combination of multiple datasets including: |
|
- [RyokoAI/ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) |
|
- [GPTeacher](https://github.com/teknium1/GPTeacher) |
|
- [Alpaca-GPT4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM) en & zh |
|
- Filtered subset of machine-translated ShareGPT dataset into Chinese |
|
|
|
### Reward Model |
|
|
|
For RM we used the code of [reward-modeling](https://github.com/Dahoas/reward-modeling) repo and datasets from |
|
- [oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1) |
|
- [Dahoas/full-hh-rlhf](https://huggingface.co/datasets/Dahoas/full-hh-rlhf) |
|
- [liswei/rm-static-m2m100-zh](https://huggingface.co/datasets/liswei/rm-static-m2m100-zh) |
|
|
|
### Reinforcement Learning |
|
|
|
For RL we used the code of [trlx](https://github.com/CarperAI/trlx) with slight modification. |
|
|
|
Instead of building value network upon the policy network with a single linear layer, we add another hydra head upon the reference network's frozen bottom layers as value network. |
|
|
|
### Example |
|
|
|
We used Vicuna v1.1 template for model training |
|
|
|
``` |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
checkpoint = "keyfan/bloomz-rlhf" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(checkpoint) |
|
model = AutoModelForCausalLM.from_pretrained(checkpoint).cuda() |
|
|
|
template = ("A chat between a curious human and an artificial intelligence assistant. " |
|
"The assistant gives helpful, detailed, and polite answers to the human's questions. " |
|
"USER: {}\nASSISTANT:") |
|
question = template.format("Who was the president of the United States in 1955?") |
|
inputs = tokenizer.encode(question, return_tensors="pt").cuda() |
|
outputs = model.generate(inputs, do_sample=True, top_p=0.8, max_new_tokens=512) |
|
print(tokenizer.decode(outputs[0])) |
|
``` |
|
|
|
### Evalutions |
|
|
|
Result on the Chinese [BELLE eval set](https://github.com/LianjiaTech/BELLE/tree/main/eval) |
|
|
|
| others | rewrite | classification | generation | summarization | extract | open qa | brainstorming | closed qa | macro ave | macro ave w/o others | |
|
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | |
|
| 0.619 | 0.873 | 0.706 | 0.934 | 0.755 | 0.619 | 0.527 | 0.908 | 0.615 | 0.728 | 0.742 | |
|
|
|
* We found in GPT-4 evaluation the order in which the responses were presented has unneglectable affect on the final score even with the very-well designed Vicuna prompt. So we removed the score on the Vicuna eval set. |
|
|
