---
language:
- en
license: apache-2.0
tags:
- mistral-7b
- instruct
- finetune
- synthetic data
- distillation
base_model: mistralai/Mistral-7B-v0.1
model-index:
- name: Mistral-Syndicate-7B
  results: []
---

## Mistral-Syndicate-7B


<img src="https://cdn-uploads.huggingface.co/production/uploads/64712903631bb7660aaa7183/yMzb5tbgygtlHnoyR37YO.png" width="600px" />

## Model Description:

Mistral Syndicate is in no way a state-of-the-art model, rather it is a fine-tuning experiment to explore
the training dynamics specific to large language models. The dataset used in finetuning was generated via
a "syndicate" of other open language models both of similar parameter size and larger. Each model would generate a
response for a given instruction, and the group would vote on which model's response was best.

The instruction inputs used for the output label synthesis were a curated subset of [VMWare/open-instruct](https://huggingface.co/datasets/VMware/open-instruct)
with additional instructions synthesized from scratch.

## Prompt template

With context
```
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:

### Input:

### Response:
```

Without context
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:

### Response:
```

## Evaluation Results
_12.30.23_
| Benchmark  | Result |
|------------|--------|
| ARC        | 60.84  |
| HellaSwag  | 82.91  |
| MMLU       | 60.83  |
| TruthfulQA | 43.71  |
| Winogrande | 78.61  |
| GSM8K      | 44.50  |


# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_spmurrayzzz__Mistral-Syndicate-7B)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |61.90|
|AI2 Reasoning Challenge (25-Shot)|60.84|
|HellaSwag (10-Shot)              |82.91|
|MMLU (5-Shot)                    |60.83|
|TruthfulQA (0-shot)              |43.71|
|Winogrande (5-shot)              |78.61|
|GSM8k (5-shot)                   |44.50|


# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_spmurrayzzz__Mistral-Syndicate-7B)

|      Metric       |Value|
|-------------------|----:|
|Avg.               |13.85|
|IFEval (0-Shot)    |24.96|
|BBH (3-Shot)       |20.51|
|MATH Lvl 5 (4-Shot)| 2.42|
|GPQA (0-shot)      | 3.47|
|MuSR (0-shot)      |13.62|
|MMLU-PRO (5-shot)  |18.13|