--- language: - en library_name: transformers pipeline_tag: text-generation datasets: - jondurbin/airoboros-2.2 - Open-Orca/OpenOrca - garage-bAInd/Open-Platypus - WizardLM/WizardLM_evol_instruct_V2_196k - TokenBender/python_eval_instruct_51k - codefuse-ai/Evol-Instruction-66k tags: - llama-2 - code license: llama2 model-index: - name: SpeechlessCoder results: - task: type: text-generation dataset: type: openai_humaneval name: HumanEval metrics: - name: pass@1 type: pass@1 value: verified: false ---

speechless-thoughts-mistral-7b

[code](https://github.com/uukuguy/multi_loras) speechless-thoughts-mistral-7b is fine-tuned as a baseline of the [speechless-sparsetral-16x7b-MoE](https://huggingface.co/uukuguy/speechless-sparsetral-16x7b-MoE). The specific datasets (speechless-thoughts-252k) are as follows: - jondurbin/airoboros-2.2: Filter categories related to coding, reasoning and planning. 23,462 samples. - Open-Orca/OpenOrca: Filter the 'cot' category in 1M GPT4 dataset. 74,440 samples. - garage-bAInd/Open-Platypus: 100%, 24,926 samples. - WizardLM/WizardLM_evol_instruct_V2_196k: Coding coversation part. 30,185 samples - TokenBender/python_eval_instruct_51k: “python” in output .40,309 samples - Spider: 8,659 samples - codefuse-ai/Evol-Instruction-66k: 100%, 66,862 samples ## Alpaca Prompt Format ``` ### Instruction: ### Response: ``` ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name_or_path="uukuguy/speechless-thoughts-mistral-7b" tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto", trust_remote_code=True).eval() system = ""Below is an instruction that describes a task.\nWrite a response that appropriately completes the request.\n\n"" prompt = f"{system}\n\n### Instruction:\n{instruction}\n\n### Response:" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) pred = model.generate(**inputs, max_length=4096, do_sample=True, top_k=50, top_p=0.99, temperature=0.9, num_return_sequences=1) print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)) ``` ## HumanEval | Metric | Value | | --- | --- | | humaneval-python | | ## lm-evaluation-harness ```json {'ARC (acc_norm)': , 'HellaSwag (acc_norm)': , 'MMLU (acc)': , 'TruthfulQA (mc2)': , 'Winoground (acc)': , 'GSM8K (acc)': , 'DROP (f1)': , 'Open LLM Score': } ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_uukuguy__speechless-thoughts-mistral-7b) | Metric | Value | |-----------------------|---------------------------| | Avg. | 59.72 | | ARC (25-shot) | 58.96 | | HellaSwag (10-shot) | 80.71 | | MMLU (5-shot) | 60.11 | | TruthfulQA (0-shot) | 49.91 | | Winogrande (5-shot) | 77.82 | | GSM8K (5-shot) | 30.78 |