albert-xxlarge-v2-Adapters

This model is a fine-tuned version of albert/albert-xxlarge-v2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5628

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: inverse_sqrt
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.121	0.1146	50	1.0992
1.1183	0.2292	100	1.0918
1.1155	0.3438	150	1.0899
1.0405	0.4585	200	0.9750
0.9	0.5731	250	0.9008
0.8362	0.6877	300	0.8511
0.7714	0.8023	350	0.8039
0.7385	0.9169	400	0.7617
0.7422	1.0315	450	0.7265
0.6513	1.1461	500	0.7158
0.7349	1.2607	550	0.6831
0.6515	1.3754	600	0.6679
0.6054	1.4900	650	0.6465
0.6069	1.6046	700	0.6364
0.6132	1.7192	750	0.6344
0.6195	1.8338	800	0.6366
0.6026	1.9484	850	0.6313
0.507	2.0630	900	0.5977
0.5555	2.1777	950	0.5871
0.5855	2.2923	1000	0.5835
0.5642	2.4069	1050	0.5956
0.5937	2.5215	1100	0.5791
0.5763	2.6361	1150	0.5756
0.5041	2.7507	1200	0.5729
0.5753	2.8653	1250	0.5720
0.5546	2.9799	1300	0.5628

Framework versions

• PEFT 0.11.1
• Transformers 4.42.4
• Pytorch 2.1.0
• Datasets 2.20.0
• Tokenizers 0.19.1