Adding Evaluation Results (#1)

a6f7706 verified 3 months ago

No virus

4.41 kB

	---
	language:
	- en
	license: apache-2.0
	model-index:
	- name: mistral-7b-v0.1-layla-v4
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 62.29
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=l3utterfly/mistral-7b-v0.1-layla-v4
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 83.36
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=l3utterfly/mistral-7b-v0.1-layla-v4
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 64.32
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=l3utterfly/mistral-7b-v0.1-layla-v4
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 43.14
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=l3utterfly/mistral-7b-v0.1-layla-v4
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 79.56
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=l3utterfly/mistral-7b-v0.1-layla-v4
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 55.5
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=l3utterfly/mistral-7b-v0.1-layla-v4
	name: Open LLM Leaderboard
	---

	# Model Card

	### Model Description

	Mistral 7B fine-tuned by the OpenHermes 2.5 dataset optimised for multi-turn conversation and character impersonation.

	The dataset has been pre-processed by doing the following:
	1. remove all refusals
	2. remove any mention of AI assistant
	3. split any multi-turn dialog generated in the dataset into multi-turn conversations records
	4. added nfsw generated conversations from the Teatime dataset

	- Developed by: l3utterfly
	- Funded by: Layla Network
	- Model type: Mistral
	- Language(s) (NLP): English
	- License: Apache-2.0
	- Finetuned from model: Mistral 7B

	## Uses

	Base model used by Layla - the offline personal assistant: https://www.layla-network.ai

	Help & support: https://discord.gg/x546YJ6nYC

	Prompt:
	```
	USER:
	ASSISTANT:
	```

	[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_l3utterfly__mistral-7b-v0.1-layla-v4)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|64.69\|
	\|AI2 Reasoning Challenge (25-Shot)\|62.29\|
	\|HellaSwag (10-Shot) \|83.36\|
	\|MMLU (5-Shot) \|64.32\|
	\|TruthfulQA (0-shot) \|43.14\|
	\|Winogrande (5-shot) \|79.56\|
	\|GSM8k (5-shot) \|55.50\|