squad_qa_baseline_v5_full_meta-llama_Llama-2-7b-hf_3e-5_lora

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.4997
• Accuracy: 0.5972

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0.99	74	1.7199	0.6196
2.2232	1.99	149	1.6485	0.6252
1.6589	3.0	224	1.6497	0.6251
1.6589	4.0	299	1.6766	0.6224
1.5556	4.99	373	1.7691	0.6181
1.3483	5.99	448	1.8441	0.6152
1.0767	7.0	523	1.9540	0.6127
1.0767	8.0	598	2.0592	0.6098
0.8642	8.99	672	2.2436	0.6051
0.6478	9.99	747	2.4147	0.6026
0.5323	11.0	822	2.4719	0.6018
0.5323	12.0	897	2.6090	0.5990
0.4539	12.99	971	2.6601	0.5999
0.3998	13.99	1046	2.7315	0.5994
0.3835	15.0	1121	2.8304	0.6006
0.3835	16.0	1196	2.8454	0.5976
0.3694	16.99	1270	2.8839	0.5994
0.3527	17.99	1345	2.9917	0.5971
0.3529	19.0	1420	2.9956	0.5975
0.3529	20.0	1495	2.9528	0.5979
0.3479	20.99	1569	2.9979	0.5989
0.3385	21.99	1644	2.9651	0.5971
0.3405	23.0	1719	3.0648	0.5974
0.3405	24.0	1794	3.0753	0.598
0.3357	24.99	1868	3.1164	0.5969
0.3284	25.99	1943	3.0272	0.5956
0.3349	27.0	2018	3.1836	0.5964
0.3349	28.0	2093	3.1658	0.5965
0.3297	28.99	2167	3.1892	0.596
0.326	29.99	2242	3.1197	0.5973
0.3284	31.0	2317	3.2153	0.5951
0.3284	32.0	2392	3.2405	0.5967
0.3262	32.99	2466	3.2837	0.5941
0.3225	33.99	2541	3.2351	0.5967
0.3219	35.0	2616	3.1901	0.5972
0.3219	36.0	2691	3.2920	0.5972
0.3209	36.99	2765	3.3388	0.5965
0.3183	37.99	2840	3.3174	0.5964
0.321	39.0	2915	3.2232	0.5975
0.321	40.0	2990	3.3365	0.5956
0.3183	40.99	3064	3.3483	0.5967
0.3153	41.99	3139	3.2347	0.5972
0.3196	43.0	3214	3.3181	0.5982
0.3196	44.0	3289	3.3559	0.5965
0.3154	44.99	3363	3.3142	0.5975
0.3137	45.99	3438	3.3402	0.5967
0.3153	47.0	3513	3.3566	0.5966
0.3153	48.0	3588	3.3560	0.5966
0.3149	48.99	3662	3.3837	0.5966
0.3124	49.5	3700	3.4997	0.5972

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1