spmurrayzzz/Mistral-Syndicate-7B

Mistral-Syndicate-7B

Model Description:

Mistral Syndicate is in no way a state-of-the-art model, rather it is a fine-tuning experiment to explore the training dynamics specific to large language models. The dataset used in finetuning was generated via a "syndicate" of other open language models both of similar parameter size and larger. Each model would generate a response for a given instruction, and the group would vote on which model's response was best.

The instruction inputs used for the output label synthesis were a curated subset of VMWare/open-instruct with additional instructions synthesized from scratch.

Prompt template

With context

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:

### Input:

### Response:

Without context

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:

### Response:

Evaluation Results

12.30.23

Benchmark	Result
ARC	60.84
HellaSwag	82.91
MMLU	60.83
TruthfulQA	43.71
Winogrande	78.61
GSM8K	44.50

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	61.90
AI2 Reasoning Challenge (25-Shot)	60.84
HellaSwag (10-Shot)	82.91
MMLU (5-Shot)	60.83
TruthfulQA (0-shot)	43.71
Winogrande (5-shot)	78.61
GSM8k (5-shot)	44.50

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	13.85
IFEval (0-Shot)	24.96
BBH (3-Shot)	20.51
MATH Lvl 5 (4-Shot)	2.42
GPQA (0-shot)	3.47
MuSR (0-shot)	13.62
MMLU-PRO (5-shot)	18.13