Adding Evaluation Results (#2)

7e772af verified 3 months ago

No virus

6.33 kB

	---
	license: apache-2.0
	tags:
	- generated_from_trainer
	- text generation
	- email generation
	- email
	datasets:
	- aeslc
	- postbot/multi-emails-100k
	widget:
	- text: 'Good Morning Professor Beans,

	Hope you are doing well. I just wanted to reach out and ask if differential calculus
	will be on the exam'
	example_title: email to prof
	- text: 'Hey <NAME>,


	Thank you for signing up for my weekly newsletter. Before we get started, you''ll
	have to confirm your email address.'
	example_title: newsletter
	- text: 'Hi <NAME>,


	I hope this email finds you well. I wanted to reach out and ask about office hours'
	example_title: office hours
	- text: 'Greetings <NAME>,


	I hope you had a splendid evening at the Company sausage eating festival. I am
	reaching out because'
	example_title: festival
	- text: 'Good Morning Harold,


	I was wondering when the next'
	example_title: event
	- text: URGENT - I need the TPS reports
	example_title: URGENT
	- text: 'Hi Archibald,


	I hope this email finds you extremely well.'
	example_title: emails that find you
	- text: 'Hello there.


	I just wanted to reach out and check in to'
	example_title: checking in
	- text: 'Hello <NAME>,


	I hope this email finds you well. I wanted to reach out and see if you''ve enjoyed
	your time with us'
	example_title: work well
	- text: 'Hi <NAME>,


	I hope this email finds you well. I wanted to reach out and see if we could catch
	up'
	example_title: catch up
	- text: I'm <NAME> and I just moved into the area and wanted to reach out and get
	some details on where I could get groceries and
	example_title: grocery
	parameters:
	min_length: 32
	max_length: 128
	no_repeat_ngram_size: 2
	do_sample: true
	temperature: 0.4
	top_k: 30
	top_p: 0.9
	repetition_penalty: 3.5
	length_penalty: 0.9
	base_model: EleutherAI/gpt-neo-1.3B
	model-index:
	- name: gpt-neo-1.3B-emailgen
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 29.95
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 47.95
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 24.11
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 42.55
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 56.27
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 0.0
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen
	name: Open LLM Leaderboard
	---


	# gpt-neo-1.3B-emailgen

	This model is a fine-tuned version of [EleutherAI/gpt-neo-1.3B](https://huggingface.co/EleutherAI/gpt-neo-1.3B) on the postbot/multi-emails-100k dataset.
	It achieves the following results on the evaluation set:
	- Loss: 1.6930

	## Model description

	More information needed

	## Intended uses & limitations

	More information needed

	## Training and evaluation data

	More information needed

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 0.0001
	- train_batch_size: 4
	- eval_batch_size: 4
	- seed: 42
	- distributed_type: multi-GPU
	- gradient_accumulation_steps: 32
	- total_train_batch_size: 128
	- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_ratio: 0.02
	- num_epochs: 2

	### Training results

	\| Training Loss \| Epoch \| Step \| Validation Loss \|
	\|:-------------:\|:-----:\|:----:\|:---------------:\|
	\| 1.8669 \| 1.0 \| 789 \| 1.7866 \|
	\| 1.4049 \| 2.0 \| 1578 \| 1.6930 \|


	### Framework versions

	- Transformers 4.22.2
	- Pytorch 1.10.0+cu113
	- Tokenizers 0.12.1

	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_postbot__gpt-neo-1.3B-emailgen)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|33.47\|
	\|AI2 Reasoning Challenge (25-Shot)\|29.95\|
	\|HellaSwag (10-Shot) \|47.95\|
	\|MMLU (5-Shot) \|24.11\|
	\|TruthfulQA (0-shot) \|42.55\|
	\|Winogrande (5-shot) \|56.27\|
	\|GSM8k (5-shot) \| 0.00\|