macadeliccc
/

SmaugDolphin-60B

Text Generation

Inference Endpoints

text-generation-inference

Model card Files Files and versions Community

SmaugDolphin-60B / README.md

macadeliccc's picture

Adding Evaluation Results (#2)

ff76607 verified 4 months ago

|

raw history blame contribute delete

No virus

4.08 kB

	---
	license: other
	license_name: yi-license
	license_link: https://huggingface.co/cognitivecomputations/dolphin-2.2-yi-34b-200k/blob/main/LICENSE
	model-index:
	- name: SmaugDolphin-60B
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 73.38
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/SmaugDolphin-60B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 86.55
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/SmaugDolphin-60B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 76.78
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/SmaugDolphin-60B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 67.44
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/SmaugDolphin-60B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 83.5
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/SmaugDolphin-60B
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 70.96
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/SmaugDolphin-60B
	name: Open LLM Leaderboard
	---
	# Smaug Dolphin 60B

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/6455cc8d679315e4ef16fbec/S-OC1x8oocC_kU-_-yUgh.png)

	This model is a MoErge of [abacusai/Smaug-34B-v0.1](https://huggingface.co/abacusai/Smaug-34B-v0.1) and [cognitivecomputations/dolphin-2.2-yi-34b-200k](https://huggingface.co/cognitivecomputations/dolphin-2.2-yi-34b-200k)

	This model works as expected. Evaluations are running now.

	## GGUF + iMatrix

	Available [here]()

	## AWQ

	TODO

	## Example output

	![image](example.png)

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/6455cc8d679315e4ef16fbec/5VMamt1iXKnOGZEAG2WWs.png)

	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_macadeliccc__SmaugDolphin-60B)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|76.44\|
	\|AI2 Reasoning Challenge (25-Shot)\|73.38\|
	\|HellaSwag (10-Shot) \|86.55\|
	\|MMLU (5-Shot) \|76.78\|
	\|TruthfulQA (0-shot) \|67.44\|
	\|Winogrande (5-shot) \|83.50\|
	\|GSM8k (5-shot) \|70.96\|