Adding Evaluation Results (#1)

84f5908 verified 3 months ago

No virus

4.3 kB

	---
	language:
	- en
	license: cc-by-nc-4.0
	model-index:
	- name: Dionysus-Mistral-m3-v6
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 63.14
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=decem/Dionysus-Mistral-m3-v6
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 84.51
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=decem/Dionysus-Mistral-m3-v6
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 62.82
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=decem/Dionysus-Mistral-m3-v6
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 49.49
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=decem/Dionysus-Mistral-m3-v6
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 78.45
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=decem/Dionysus-Mistral-m3-v6
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 64.22
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=decem/Dionysus-Mistral-m3-v6
	name: Open LLM Leaderboard
	---
	# Dionysus-Mistral-m3-v6 - A Fine-tuned Language Model


	## Overview

	Dionysus-Mistral-m3-v6 is a language model fine-tuned on the AIDC-ai-business/Marcoroni-7B-v3 architecture. This model is mainly used for roleplay.

	<img src="./dionysus.jpg" alt="Dionysus" style="transform: scale(0.5);">

	## Model Details

	- Developer: DECEM
	- Base Model: [AIDC-ai-business/Marcoroni-7B-v3](https://huggingface.co/AIDC-ai-business/Marcoroni-7B-v3)
	- Fine-tuning Method: SFT
	- Language: English
	# Prompting

	## Prompt Template for alpaca style

	```
	{system prompt} (without the {})

	### Instruction:
	{user_name}: {prompt} (without the {})

	### Response:
	{character_name}: (without the {})
	```
	```
	stop_words: </s>
	```

	## Open LLM Leaderboard Evaluation Details
	[https://huggingface.co/datasets/open-llm-leaderboard/details_decem__Dionysus-Mistral-m3-v6](https://huggingface.co/datasets/open-llm-leaderboard/details_decem__Dionysus-Mistral-m3-v6)
	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_decem__Dionysus-Mistral-m3-v6)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|67.10\|
	\|AI2 Reasoning Challenge (25-Shot)\|63.14\|
	\|HellaSwag (10-Shot) \|84.51\|
	\|MMLU (5-Shot) \|62.82\|
	\|TruthfulQA (0-shot) \|49.49\|
	\|Winogrande (5-shot) \|78.45\|
	\|GSM8k (5-shot) \|64.22\|