Adding Evaluation Results (#1)

0068887 verified 3 months ago

No virus

4.76 kB

	---
	license: apache-2.0
	datasets:
	- lodrick-the-lafted/Hermes-100K
	model-index:
	- name: Hermes-Instruct-7B-100K
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 61.52
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=lodrick-the-lafted/Hermes-Instruct-7B-100K
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 82.84
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=lodrick-the-lafted/Hermes-Instruct-7B-100K
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 60.95
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=lodrick-the-lafted/Hermes-Instruct-7B-100K
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 63.62
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=lodrick-the-lafted/Hermes-Instruct-7B-100K
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 76.87
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=lodrick-the-lafted/Hermes-Instruct-7B-100K
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 43.97
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=lodrick-the-lafted/Hermes-Instruct-7B-100K
	name: Open LLM Leaderboard
	---

	<img src=https://huggingface.co/lodrick-the-lafted/Hermes-Instruct-7B-100K/resolve/main/hermes-instruct.png>

	# Hermes-Instruct-7B-v0.2

	[Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) trained with 100K rows of [teknium/openhermes](https://huggingface.co/datasets/teknium/openhermes), in Alpaca format.

	<br />
	<br />

	# Prompt Format

	Both the default Mistral-Instruct tags and Alpaca are fine, so either:
	```
	<s>[INST] {sys_prompt} {instruction} [/INST]
	```

	or


	```
	{sys_prompt}

	### Instruction:
	{instruction}

	### Response:

	```
	The tokenizer default is Alpaca this time around.

	<br />
	<br />

	# Usage

	```python
	from transformers import AutoTokenizer
	import transformers
	import torch

	model = "lodrick-the-lafted/Hermes-Instruct-7B-100K"

	tokenizer = AutoTokenizer.from_pretrained(model)
	pipeline = transformers.pipeline(
	"text-generation",
	model=model,
	model_kwargs={"torch_dtype": torch.bfloat16},
	)

	messages = [{"role": "user", "content": "Give me a cooking recipe for an apple pie."}]
	prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.95)
	print(outputs[0]["generated_text"])
	```


	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_lodrick-the-lafted__Hermes-Instruct-7B-100K)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|64.96\|
	\|AI2 Reasoning Challenge (25-Shot)\|61.52\|
	\|HellaSwag (10-Shot) \|82.84\|
	\|MMLU (5-Shot) \|60.95\|
	\|TruthfulQA (0-shot) \|63.62\|
	\|Winogrande (5-shot) \|76.87\|
	\|GSM8k (5-shot) \|43.97\|