zephyr-7b-sft-lora-accum8-lr3e_5

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6104

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.0441	0.51	6	1.9197
1.9066	1.53	13	1.7587
1.7223	2.55	20	1.6647
1.651	3.57	27	1.5758
1.5966	4.51	33	1.4932
1.4568	5.53	40	1.3932
1.3729	6.55	47	1.2837
1.3072	7.57	54	1.2049
1.196	8.51	60	1.1644
1.1675	9.53	67	1.1299
1.1349	10.55	74	1.1030
1.1079	11.57	81	1.0822
1.084	12.51	87	1.0677
1.0777	13.53	94	1.0514
1.0443	14.55	101	1.0369
1.0333	15.57	108	1.0226
1.0317	16.51	114	1.0103
1.0009	17.53	121	0.9972
0.9809	18.55	128	0.9830
0.9619	19.57	135	0.9673
0.9501	20.51	141	0.9589
0.9351	21.53	148	0.9411
0.9052	22.55	155	0.9274
0.8942	23.57	162	0.9113
0.8786	24.51	168	0.8971
0.8429	25.53	175	0.8800
0.8323	26.55	182	0.8607
0.8173	27.57	189	0.8427
0.7873	28.51	195	0.8262
0.7702	29.53	202	0.8127
0.7536	30.55	209	0.7957
0.7211	31.57	216	0.7831
0.7139	32.51	222	0.7689
0.6903	33.53	229	0.7508
0.6678	34.55	236	0.7359
0.6574	35.57	243	0.7226
0.6423	36.51	249	0.7091
0.622	37.53	256	0.6963
0.6055	38.55	263	0.6855
0.5969	39.57	270	0.6763
0.5886	40.51	276	0.6705
0.5708	41.53	283	0.6603
0.5591	42.55	290	0.6498
0.5554	43.57	297	0.6427
0.5488	44.51	303	0.6390
0.5416	45.53	310	0.6331
0.5274	46.55	317	0.6260
0.5333	47.57	324	0.6194
0.5168	48.51	330	0.6157
0.5161	49.53	337	0.6108

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.0
• Datasets 2.14.6
• Tokenizers 0.14.1