oop_and_text_gpt2

This model is a fine-tuned version of openai-community/gpt2-medium on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.6509
• Accuracy: 0.1604
• Num Input Tokens Seen: 5873664

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Input Tokens Seen
No log	0	0	8.7649	0.1981	0
1.6819	0.2092	50	3.0818	0.1934	409600
1.6936	0.4184	100	1.8988	0.2358	819200
1.8252	0.6276	150	1.7960	0.2075	1228800
1.7185	0.8368	200	1.6783	0.2170	1638400
1.6083	1.0460	250	1.6509	0.1604	2048000
1.5989	1.2552	300	1.7008	0.1792	2457600
1.7146	1.4644	350	1.7357	0.1792	2867200
1.5851	1.6736	400	1.7148	0.2075	3276800
1.5185	1.8828	450	1.6637	0.2123	3686400
1.5803	2.0921	500	1.6518	0.1840	4096000
1.5545	2.3013	550	1.6630	0.2075	4505600
1.6077	2.5105	600	1.6762	0.1840	4915200
1.6173	2.7197	650	1.6747	0.1934	5324800
1.5865	2.9289	700	1.6730	0.1981	5734400

Framework versions

• Transformers 4.43.2
• Pytorch 2.4.0
• Datasets 2.20.0
• Tokenizers 0.19.1