finetune_starcoder2

This model is a fine-tuned version of ise-uiuc/Magicoder-S-DS-6.7B on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.3220

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 1
• eval_batch_size: 5
• seed: 0
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 2000

Training results

Training Loss	Epoch	Step	Validation Loss
0.8634	0.5340	50	0.6751
0.6619	1.0681	100	0.4653
0.5147	1.6021	150	0.4231
0.4761	2.1362	200	0.3912
0.4348	2.6702	250	0.3663
0.4123	3.2043	300	0.3515
0.3893	3.7383	350	0.3407
0.3769	4.2724	400	0.3329
0.3719	4.8064	450	0.3266
0.3578	5.3405	500	0.3220

Framework versions

• PEFT 0.11.1
• Transformers 4.42.4
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1