Text Generation
Safetensors
Chinese
English
wangrongsheng commited on
Commit
856e460
1 Parent(s): 33e1849

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +127 -0
README.md CHANGED
@@ -1,3 +1,130 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ datasets:
4
+ - shareAI/ShareGPT-Chinese-English-90k
5
+ language:
6
+ - zh
7
+ - en
8
+ pipeline_tag: text-generation
9
  ---
10
+ ![](./assets/aurora.png)
11
+
12
+ <div align="center">
13
+ <h2>
14
+ Aurora: Activating chinese chat capability for Mistral-8x7B sparse Mixture-of-Experts through Instruction-Tuning
15
+ </h2>
16
+ </div>
17
+
18
+ 1. <h1>Please follow our Github: <a href="https://github.com/WangRongsheng/Aurora">https://github.com/WangRongsheng/Aurora</a></h1>
19
+ 2. <h1>Please follow our Paper: <a href="https://arxiv.org/abs/2312.14557">https://arxiv.org/abs/2312.14557</a></h1>
20
+
21
+ ## Overview
22
+
23
+ Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture.
24
+
25
+ ![](./training_loss.png)
26
+
27
+ ## Usage
28
+
29
+ ```python
30
+ import gradio as gr
31
+ import torch
32
+ from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer
33
+ from threading import Thread
34
+ from peft import PeftModel
35
+ import time
36
+
37
+ model_name_or_path = "mistralai/Mixtral-8x7B-Instruct-v0.1" # download weights from https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
38
+ lora_weights = "wangrongsheng/Aurora" # download weights from https://huggingface.co/wangrongsheng/Aurora
39
+
40
+ tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
41
+ model0 = AutoModelForCausalLM.from_pretrained(model_name_or_path, load_in_4bit=True, device_map="auto", torch_dtype=torch.bfloat16)
42
+ model = PeftModel.from_pretrained(
43
+ model0,
44
+ lora_weights,
45
+ )
46
+
47
+ class StopOnTokens(StoppingCriteria):
48
+ def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
49
+ stop_ids = [0,]
50
+ for stop_id in stop_ids:
51
+ if input_ids[0][-1] == stop_id:
52
+ return True
53
+ return False
54
+
55
+ def convert_history_to_text(history):
56
+ text = ""
57
+ if len(history) > 1:
58
+ text = "<s> " + "".join(
59
+ [
60
+ "".join(
61
+ [
62
+ f"[INST]{item[0]}[/INST] {item[1]} ",
63
+ ]
64
+ )
65
+ for item in history[:-1]
66
+ ]
67
+ ) + "</s> "
68
+ text += "".join(
69
+ [
70
+ "".join(
71
+ [
72
+ f"[INST]{history[-1][0]}[/INST]",
73
+ ]
74
+ )
75
+ ]
76
+ )
77
+ return text
78
+
79
+ def predict(message, history):
80
+
81
+ history_transformer_format = history + [[message, ""]]
82
+ stop = StopOnTokens()
83
+
84
+ messages = convert_history_to_text(history_transformer_format)
85
+
86
+ model_inputs = tokenizer([messages], return_tensors="pt").to("cuda")
87
+ streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)
88
+ generate_kwargs = dict(
89
+ model_inputs,
90
+ streamer=streamer,
91
+ max_new_tokens=4096,
92
+ do_sample=True,
93
+ top_p=0.95,
94
+ top_k=1000,
95
+ temperature=1.0,
96
+ num_beams=1,
97
+ pad_token_id=tokenizer.eos_token_id,
98
+ stopping_criteria=StoppingCriteriaList([stop])
99
+ )
100
+ t = Thread(target=model.generate, kwargs=generate_kwargs)
101
+ t.start()
102
+
103
+ partial_message = ""
104
+ t1 = time.time()
105
+ count = 0
106
+ for new_token in streamer:
107
+ if new_token != '<':
108
+ partial_message += new_token
109
+ count += 1
110
+ yield partial_message
111
+ t2 = time.time()
112
+ speed = count/(t2-t1)
113
+ print("inference speed: %f tok/s" % speed)
114
+
115
+
116
+ gr.ChatInterface(predict,chatbot=gr.Chatbot(height=600,),title="MoE").queue().launch()
117
+ ```
118
+
119
+ ## Citation
120
+ If you find our work helpful, feel free to give us a cite.
121
+ ```latex
122
+ @misc{wang2023auroraactivating,
123
+ title={Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning},
124
+ author={Rongsheng Wang and Haoming Chen and Ruizhe Zhou and Yaofei Duan and Kunyan Cai and Han Ma and Jiaxi Cui and Jian Li and Patrick Cheong-Iao Pang and Yapeng Wang and Tao Tan},
125
+ year={2023},
126
+ eprint={2312.14557},
127
+ archivePrefix={arXiv},
128
+ primaryClass={cs.CL}
129
+ }
130
+ ```