gpt2-finetuned

This model is a fine-tuned version of openai-community/gpt2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.6944
• Bleu: 0.0294
• Bertscore Precision: 0.1536
• Bertscore Recall: 0.1658
• Bertscore F1: 0.1592

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Bleu	Bertscore Precision	Bertscore Recall	Bertscore F1
4.716	1.0	5750	3.4413	0.0112	0.1417	0.1575	0.1489
4.5916	2.0	11500	3.2372	0.0119	0.1424	0.1583	0.1496
4.325	3.0	17250	3.0534	0.0128	0.1430	0.1587	0.1501
4.1626	4.0	23000	2.9061	0.0136	0.1433	0.1592	0.1505
4.0255	5.0	28750	2.7554	0.0148	0.1438	0.1599	0.1511
3.862	6.0	34500	2.6185	0.0346	0.1446	0.1605	0.1518
3.7367	7.0	40250	2.4945	0.0286	0.1456	0.1611	0.1527
3.7907	8.0	46000	2.3799	0.0401	0.1488	0.1617	0.1548
3.5181	9.0	51750	2.2704	0.0607	0.1490	0.1623	0.1551
3.3377	10.0	57500	2.1710	0.0804	0.1498	0.1627	0.1558
3.294	11.0	63250	2.0876	0.0221	0.1512	0.1633	0.1568
3.1612	12.0	69000	2.0004	0.0234	0.1516	0.1637	0.1572
3.1257	13.0	74750	1.9356	0.0244	0.1518	0.1642	0.1575
3.1347	14.0	80500	1.8769	0.0257	0.1525	0.1646	0.1581
2.8094	15.0	86250	1.8210	0.0268	0.1527	0.1649	0.1584
2.8519	16.0	92000	1.7776	0.0275	0.1530	0.1652	0.1587
2.782	17.0	97750	1.7438	0.0282	0.1532	0.1654	0.1589
2.9097	18.0	103500	1.7183	0.0289	0.1535	0.1657	0.1591
2.881	19.0	109250	1.6999	0.0293	0.1536	0.1658	0.1592
2.6302	20.0	115000	1.6944	0.0294	0.1536	0.1658	0.1592

Framework versions

• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1