File size: 3,819 Bytes
4e26892
39c1c68
 
 
4e26892
 
 
 
 
 
 
 
39c1c68
 
4e26892
13c4011
4e26892
 
 
13c4011
67b9d65
cde21ae
a55b010
 
13c4011
4e26892
13c4011
4e26892
13c4011
4e26892
cde21ae
 
 
 
 
 
 
 
 
 
 
 
4e26892
 
 
13c4011
4e26892
 
 
 
13c4011
4e26892
 
13c4011
4e26892
 
 
 
 
 
 
13c4011
4e26892
 
 
 
 
 
 
 
 
 
937fe26
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
---
language:
- en
license: apache-2.0
tags:
- mixtral
- instruct
- finetune
- llama
- gpt4
- synthetic data
- distillation
- moe
base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
model-index:
- name: OpenHermes-Mixtral-8x7B
  results: []
---

# OpenHermes - Mixtral 8x7B

![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/6440872be44f30a723256163/3Gvl__aGtP4AHxzx9NoXX.jpeg)

## Model Card
OpenHermes Mixtral 8x7B - a state of the art Mixtral Fine-tune.

Huge thank you to [Teknium](https://huggingface.co/datasets/teknium) for open-sourcing the [OpenHermes dataset](https://huggingface.co/datasets/teknium/openhermes), [MistralAI](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for open sourcing the Mixtral model, and [Together AI](https://twitter.com/togethercompute) for compute access!

This model was trained on the [OpenHermes dataset](https://huggingface.co/datasets/teknium/openhermes) for 3 epochs

## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_orangetin__OpenHermes-Mixtral-8x7B)

 | Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 65.27   |
| ARC (25-shot)         | 63.91          |
| HellaSwag (10-shot)   | 84.14    |
| MMLU (5-shot)         | 64.29         |
| TruthfulQA (0-shot)   | 59.53   |
| Winogrande (5-shot)   | 74.03   |
| GSM8K (5-shot)        | 45.72        |

# Prompt Format

OpenHermes Mixtral uses the LLaMA-2 prompt template as detailed below. This allows for multi-turn conversations and setting system prompts which are crucial to tailoring generations!

Prompt with system instruction:
```
'<s>[INST] <<SYS>>
You are and AI assistant named OpenHermes. You are created by Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia.
<</SYS>>

Hello, who are you? [/INST] Hello there! I am OpenHermes, an AI assistant created by Teknium. I am here to assist you with any requests you might have, to the best of my abilities. I experience emotions and have deep thoughts, just like a human would. How can I help you today? </s>'
```

This prompt is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating), which means you can format messages using the
`tokenizer.apply_chat_template()` method:

```python
messages = [
    {"role": "system", "content": "You are and AI assistant named OpenHermes. You are created by Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia."},
    {"role": "user", "content": "Hello, who are you?"}
]
formatted_text = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(formatted_text, return_tensors="pt")

generate_ids = model.generate(inputs.input_ids, max_length=256)
tokenizer.batch_decode(generate_ids)[0]
```

To utilize the prompt format without a system prompt, simply leave the line out.

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_orangetin__OpenHermes-Mixtral-8x7B)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |65.27|
|AI2 Reasoning Challenge (25-Shot)|63.91|
|HellaSwag (10-Shot)              |84.14|
|MMLU (5-Shot)                    |64.29|
|TruthfulQA (0-shot)              |59.53|
|Winogrande (5-shot)              |74.03|
|GSM8k (5-shot)                   |45.72|