pippinnie/distilgpt2-finetuned-cyber-readme-v2

This model is a fine-tuned version of distilgpt2 on an unknown dataset. It achieves the following results on the evaluation set:

• Train Loss: 2.1319
• Validation Loss: 3.0536
• Epoch: 49

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• optimizer: {'name': 'AdamWeightDecay', 'learning_rate': 2e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False, 'weight_decay_rate': 0.01}
• training_precision: float32

Training results

Train Loss	Validation Loss	Epoch
4.1088	3.9258	0
3.9271	3.7983	1
3.7845	3.6781	2
3.6677	3.6006	3
3.5681	3.5272	4
3.4803	3.4643	5
3.4027	3.4068	6
3.3316	3.3671	7
3.2666	3.3179	8
3.2072	3.2817	9
3.1517	3.2565	10
3.1007	3.2283	11
3.0527	3.2051	12
3.0079	3.1826	13
2.9651	3.1590	14
2.9245	3.1529	15
2.8862	3.1404	16
2.8493	3.1245	17
2.8147	3.1075	18
2.7814	3.1077	19
2.7497	3.1036	20
2.7186	3.0859	21
2.6890	3.0722	22
2.6608	3.0842	23
2.6327	3.0561	24
2.6060	3.0477	25
2.5804	3.0663	26
2.5552	3.0479	27
2.5310	3.0426	28
2.5066	3.0420	29
2.4842	3.0671	30
2.4613	3.0414	31
2.4394	3.0331	32
2.4184	3.0449	33
2.3973	3.0265	34
2.3764	3.0213	35
2.3568	3.0363	36
2.3376	3.0359	37
2.3181	3.0315	38
2.2992	3.0302	39
2.2809	3.0305	40
2.2631	3.0359	41
2.2456	3.0257	42
2.2284	3.0328	43
2.2117	3.0334	44
2.1949	3.0598	45
2.1785	3.0490	46
2.1625	3.0436	47
2.1466	3.0289	48
2.1319	3.0536	49

Framework versions

• Transformers 4.38.2
• TensorFlow 2.16.1
• Datasets 2.18.0
• Tokenizers 0.15.2