Upload README.md

README.md

@@ -1,3 +1,130 @@
 ---
 license: apache-2.0
+datasets:
+- shareAI/ShareGPT-Chinese-English-90k
+language:
+- zh
+- en
+pipeline_tag: text-generation
 ---
+![](./assets/aurora.png)
+
+<div align="center">
+<h2>
+  Aurora: Activating chinese chat capability for Mistral-8x7B sparse Mixture-of-Experts through Instruction-Tuning
+</h2>
+</div>
+
+1. <h1>Please follow our Github: <a href="https://github.com/WangRongsheng/Aurora">https://github.com/WangRongsheng/Aurora</a></h1>
+2. <h1>Please follow our Paper: <a href="https://arxiv.org/abs/2312.14557">https://arxiv.org/abs/2312.14557</a></h1>
+
+## Overview
+
+Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture.
+
+![](./training_loss.png)
+
+## Usage
+
+```python
+import gradio as gr
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer
+from threading import Thread
+from peft import PeftModel
+import time
+
+model_name_or_path = "mistralai/Mixtral-8x7B-Instruct-v0.1" # download weights from https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
+lora_weights = "wangrongsheng/Aurora" # download weights from https://huggingface.co/wangrongsheng/Aurora
+
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+model0 = AutoModelForCausalLM.from_pretrained(model_name_or_path, load_in_4bit=True, device_map="auto", torch_dtype=torch.bfloat16)
+model = PeftModel.from_pretrained(
+    model0,
+    lora_weights,
+)
+
+class StopOnTokens(StoppingCriteria):
+    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
+        stop_ids = [0,]
+        for stop_id in stop_ids:
+            if input_ids[0][-1] == stop_id:
+                return True
+        return False
+
+def convert_history_to_text(history):
+    text = ""
+    if len(history) > 1:
+        text = "<s> " + "".join(
+                [
+                    "".join(
+                        [
+                            f"[INST]{item[0]}[/INST] {item[1]} ",
+                        ]
+                    )
+                    for item in history[:-1]
+                ]
+            ) + "</s> "
+    text += "".join(
+        [
+            "".join(
+                [
+                    f"[INST]{history[-1][0]}[/INST]",
+                ]
+            )
+        ]
+    )
+    return text
+
+def predict(message, history):
+
+    history_transformer_format = history + [[message, ""]]
+    stop = StopOnTokens()
+
+    messages = convert_history_to_text(history_transformer_format)
+
+    model_inputs = tokenizer([messages], return_tensors="pt").to("cuda")
+    streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)
+    generate_kwargs = dict(
+        model_inputs,
+        streamer=streamer,
+        max_new_tokens=4096,
+        do_sample=True,
+        top_p=0.95,
+        top_k=1000,
+        temperature=1.0,
+        num_beams=1,
+        pad_token_id=tokenizer.eos_token_id,
+        stopping_criteria=StoppingCriteriaList([stop])
+        )
+    t = Thread(target=model.generate, kwargs=generate_kwargs)
+    t.start()
+
+    partial_message  = ""
+    t1 = time.time()
+    count = 0
+    for new_token in streamer:
+        if new_token != '<':
+            partial_message += new_token
+            count += 1
+            yield partial_message
+    t2 = time.time()
+    speed = count/(t2-t1)
+    print("inference speed: %f tok/s" % speed)
+
+
+gr.ChatInterface(predict,chatbot=gr.Chatbot(height=600,),title="MoE").queue().launch()
+```
+
+## Citation
+If you find our work helpful, feel free to give us a cite.
+```latex
+@misc{wang2023auroraactivating,
+      title={Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning}, 
+      author={Rongsheng Wang and Haoming Chen and Ruizhe Zhou and Yaofei Duan and Kunyan Cai and Han Ma and Jiaxi Cui and Jian Li and Patrick Cheong-Iao Pang and Yapeng Wang and Tao Tan},
+      year={2023},
+      eprint={2312.14557},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```