lmind_nq_train6000_eval6489_v1_docidx_v3_1e-4_lora2

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 5.3535
• Accuracy: 0.4374

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.3892	1.0	341	0.4544	3.4056
1.3499	2.0	683	0.4577	3.4531
1.2427	3.0	1024	0.4584	3.6711
1.1231	4.0	1366	0.4570	3.8000
0.995	5.0	1707	0.4552	3.9532
0.8693	6.0	2049	0.4526	4.0766
0.7302	7.0	2390	0.4501	4.1717
0.6033	8.0	2732	0.448	4.2778
0.4825	9.0	3073	0.4462	4.3415
0.387	10.0	3415	0.4463	4.4131
0.2933	11.0	3756	0.4434	4.4906
0.2344	12.0	4098	0.4425	4.6517
0.1919	13.0	4439	0.4408	4.7515
0.1581	14.0	4781	0.4421	4.7323
0.1429	15.0	5122	0.4407	4.8101
0.1279	16.0	5464	0.4406	4.8482
0.1231	17.0	5805	0.4411	4.9735
0.1145	18.0	6147	0.4415	5.0121
0.1087	19.0	6488	0.4394	4.9836
0.1084	20.0	6830	0.4388	5.1171
0.1069	21.0	7171	0.4405	5.0120
0.1075	22.0	7513	0.44	5.2343
0.1024	23.0	7854	0.4409	5.1501
0.0981	24.0	8196	0.4403	5.0801
0.097	25.0	8537	0.4416	5.1037
0.0963	26.0	8879	0.4398	5.2064
0.0983	27.0	9220	0.4414	5.0664
0.0969	28.0	9562	0.4410	5.2559
0.0966	29.0	9903	0.4404	5.1960
0.0954	30.0	10245	0.4396	5.2238
0.0931	31.0	10586	0.4402	5.2195
0.0923	32.0	10928	0.4407	5.2871
0.0911	33.0	11269	0.4392	5.3201
0.0934	34.0	11611	0.4387	5.3628
0.091	35.0	11952	0.4390	5.3197
0.0902	36.0	12294	0.4391	5.1868
0.0916	37.0	12635	0.4424	5.1227
0.0905	38.0	12977	0.4367	5.2214
0.0907	39.0	13318	0.4412	5.2412
0.0883	40.0	13660	0.4395	5.3015
0.0892	41.0	14001	0.4392	5.2816
0.0881	42.0	14343	0.4351	5.3583
0.0881	43.0	14684	0.4365	5.2678
0.0898	44.0	15026	0.4372	5.3854
0.0874	45.0	15345	5.3568	0.4392
0.088	46.0	15687	5.3908	0.4358
0.0885	47.0	16028	5.2685	0.4366
0.0872	48.0	16370	5.3500	0.44
0.0869	49.0	16711	5.3612	0.4372
0.0864	49.99	17050	5.3535	0.4374

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1