imdatta0's picture
End of training
712bd91 verified
|
raw
history blame
4.07 kB
metadata
license: llama2
library_name: peft
tags:
  - generated_from_trainer
base_model: meta-llama/Llama-2-7b-hf
model-index:
  - name: >-
      llama_2_7b_Magiccoder_evol_downNupNgateNqNkNvNo_r8_lr0.0001_bg88_alpha8_0_41_normlayergrad
    results: []

llama_2_7b_Magiccoder_evol_downNupNgateNqNkNvNo_r8_lr0.0001_bg88_alpha8_0_41_normlayergrad

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.1291

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 0.02
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
1.3048 0.0203 31 1.2412
1.1685 0.0405 62 1.2116
1.1585 0.0608 93 1.1957
1.1653 0.0810 124 1.1848
1.1337 0.1013 155 1.1766
1.1308 0.1215 186 1.1674
1.1602 0.1418 217 1.1661
1.1277 0.1620 248 1.1611
1.1375 0.1823 279 1.1615
1.1095 0.2025 310 1.1582
1.1043 0.2228 341 1.1551
1.1269 0.2431 372 1.1511
1.1175 0.2633 403 1.1533
1.1252 0.2836 434 1.1491
1.1192 0.3038 465 1.1513
1.1074 0.3241 496 1.1437
1.117 0.3443 527 1.1459
1.1058 0.3646 558 1.1427
1.1144 0.3848 589 1.1416
1.0913 0.4051 620 1.1435
1.111 0.4254 651 1.1388
1.1239 0.4456 682 1.1397
1.1074 0.4659 713 1.1402
1.1076 0.4861 744 1.1396
1.0887 0.5064 775 1.1374
1.0872 0.5266 806 1.1358
1.0802 0.5469 837 1.1351
1.0812 0.5671 868 1.1338
1.097 0.5874 899 1.1352
1.1052 0.6076 930 1.1352
1.088 0.6279 961 1.1328
1.0977 0.6482 992 1.1315
1.0902 0.6684 1023 1.1312
1.1085 0.6887 1054 1.1309
1.0956 0.7089 1085 1.1294
1.0897 0.7292 1116 1.1310
1.0816 0.7494 1147 1.1304
1.0995 0.7697 1178 1.1296
1.1032 0.7899 1209 1.1292
1.0878 0.8102 1240 1.1297
1.0988 0.8304 1271 1.1294
1.0915 0.8507 1302 1.1294
1.0823 0.8710 1333 1.1295
1.0867 0.8912 1364 1.1289
1.098 0.9115 1395 1.1290
1.1071 0.9317 1426 1.1292
1.061 0.9520 1457 1.1291
1.0708 0.9722 1488 1.1292
1.0977 0.9925 1519 1.1291

Framework versions

  • PEFT 0.7.1
  • Transformers 4.40.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.16.1
  • Tokenizers 0.19.1