lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx dataset. It achieves the following results on the evaluation set:

• Loss: 0.7825
• Accuracy: 0.7891

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.6067	0.9997	839	0.7197	1.8440
1.5433	1.9994	1678	0.7247	1.7661
1.4167	2.9991	2517	0.7310	1.6455
1.2948	4.0	3357	0.7366	1.5394
1.1715	4.9997	4196	0.7422	1.4463
1.0458	5.9994	5035	0.7484	1.3537
0.9357	6.9991	5874	0.7546	1.2456
0.8269	8.0	6714	0.7598	1.1735
0.7262	8.9997	7553	0.7649	1.0966
0.6381	9.9970	8390	0.7692	1.0623
0.5784	10.9997	9229	1.0101	0.7731
0.5071	11.9994	10068	0.9538	0.7760
0.4734	12.9991	10907	0.9292	0.7791
0.4302	14.0	11747	0.8846	0.7809
0.3917	14.9997	12586	0.8536	0.7833
0.3632	15.9994	13425	0.8468	0.7846
0.3351	16.9991	14264	0.8244	0.7863
0.3186	18.0	15104	0.8096	0.7871
0.2957	18.9997	15943	0.7865	0.7885
0.2858	19.9970	16780	0.7825	0.7891

Framework versions

• PEFT 0.5.0
• Transformers 4.40.2
• Pytorch 2.3.0
• Datasets 2.19.1
• Tokenizers 0.19.1