Hugging Face's logo

stojchet
/
lr_sft1

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
stojchet commited on
Commit
1d75dac
•
1 Parent(s): 31a8c54

End of training

Browse files
Files changed (1) hide show
  1. README.md +143 -0
README.md ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: deepseek-ai/deepseek-coder-1.3b-base
3
+ datasets:
4
+ - generator
5
+ library_name: peft
6
+ license: other
7
+ tags:
8
+ - trl
9
+ - sft
10
+ - generated_from_trainer
11
+ model-index:
12
+ - name: lr_sft1
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/stojchets/huggingface/runs/lr_sft1)
20
+ # lr_sft1
21
+
22
+ This model is a fine-tuned version of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) on the generator dataset.
23
+ It achieves the following results on the evaluation set:
24
+ - Loss: 1.1672
25
+
26
+ ## Model description
27
+
28
+ More information needed
29
+
30
+ ## Intended uses & limitations
31
+
32
+ More information needed
33
+
34
+ ## Training and evaluation data
35
+
36
+ More information needed
37
+
38
+ ## Training procedure
39
+
40
+ ### Training hyperparameters
41
+
42
+ The following hyperparameters were used during training:
43
+ - learning_rate: 0.00141
44
+ - train_batch_size: 8
45
+ - eval_batch_size: 8
46
+ - seed: 42
47
+ - gradient_accumulation_steps: 16
48
+ - total_train_batch_size: 128
49
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
50
+ - lr_scheduler_type: linear
51
+ - num_epochs: 1
52
+
53
+ ### Training results
54
+
55
+ | Training Loss | Epoch | Step | Validation Loss |
56
+ |:-------------:|:------:|:----:|:---------------:|
57
+ | 1.2662 | 0.0128 | 1 | 1.2175 |
58
+ | 1.2021 | 0.0256 | 2 | 1.1878 |
59
+ | 1.2099 | 0.0384 | 3 | 1.1839 |
60
+ | 1.1084 | 0.0512 | 4 | 1.1874 |
61
+ | 1.1652 | 0.064 | 5 | 1.1817 |
62
+ | 1.1503 | 0.0768 | 6 | 1.1817 |
63
+ | 1.1545 | 0.0896 | 7 | 1.1776 |
64
+ | 1.2043 | 0.1024 | 8 | 1.1785 |
65
+ | 1.1557 | 0.1152 | 9 | 1.1759 |
66
+ | 1.1748 | 0.128 | 10 | 1.1749 |
67
+ | 1.2061 | 0.1408 | 11 | 1.1757 |
68
+ | 1.1357 | 0.1536 | 12 | 1.1757 |
69
+ | 1.1039 | 0.1664 | 13 | 1.1753 |
70
+ | 1.2229 | 0.1792 | 14 | 1.1755 |
71
+ | 1.148 | 0.192 | 15 | 1.1750 |
72
+ | 1.1819 | 0.2048 | 16 | 1.1746 |
73
+ | 1.1758 | 0.2176 | 17 | 1.1745 |
74
+ | 1.1895 | 0.2304 | 18 | 1.1742 |
75
+ | 1.1277 | 0.2432 | 19 | 1.1741 |
76
+ | 1.1258 | 0.256 | 20 | 1.1739 |
77
+ | 1.1493 | 0.2688 | 21 | 1.1733 |
78
+ | 1.1295 | 0.2816 | 22 | 1.1733 |
79
+ | 1.1768 | 0.2944 | 23 | 1.1736 |
80
+ | 1.206 | 0.3072 | 24 | 1.1735 |
81
+ | 1.1397 | 0.32 | 25 | 1.1732 |
82
+ | 1.1736 | 0.3328 | 26 | 1.1734 |
83
+ | 1.1412 | 0.3456 | 27 | 1.1740 |
84
+ | 1.1383 | 0.3584 | 28 | 1.1745 |
85
+ | 1.1216 | 0.3712 | 29 | 1.1742 |
86
+ | 1.1127 | 0.384 | 30 | 1.1731 |
87
+ | 1.1234 | 0.3968 | 31 | 1.1724 |
88
+ | 1.1406 | 0.4096 | 32 | 1.1724 |
89
+ | 1.186 | 0.4224 | 33 | 1.1723 |
90
+ | 1.154 | 0.4352 | 34 | 1.1721 |
91
+ | 1.114 | 0.448 | 35 | 1.1724 |
92
+ | 1.1148 | 0.4608 | 36 | 1.1728 |
93
+ | 1.1422 | 0.4736 | 37 | 1.1726 |
94
+ | 1.1561 | 0.4864 | 38 | 1.1721 |
95
+ | 1.1964 | 0.4992 | 39 | 1.1716 |
96
+ | 1.1288 | 0.512 | 40 | 1.1714 |
97
+ | 1.142 | 0.5248 | 41 | 1.1713 |
98
+ | 1.149 | 0.5376 | 42 | 1.1711 |
99
+ | 1.1104 | 0.5504 | 43 | 1.1710 |
100
+ | 1.12 | 0.5632 | 44 | 1.1709 |
101
+ | 1.1256 | 0.576 | 45 | 1.1710 |
102
+ | 1.162 | 0.5888 | 46 | 1.1710 |
103
+ | 1.0982 | 0.6016 | 47 | 1.1710 |
104
+ | 1.1383 | 0.6144 | 48 | 1.1710 |
105
+ | 1.1394 | 0.6272 | 49 | 1.1708 |
106
+ | 1.1196 | 0.64 | 50 | 1.1707 |
107
+ | 1.156 | 0.6528 | 51 | 1.1705 |
108
+ | 1.105 | 0.6656 | 52 | 1.1703 |
109
+ | 1.1455 | 0.6784 | 53 | 1.1701 |
110
+ | 1.1266 | 0.6912 | 54 | 1.1698 |
111
+ | 1.1063 | 0.704 | 55 | 1.1695 |
112
+ | 1.127 | 0.7168 | 56 | 1.1693 |
113
+ | 1.1501 | 0.7296 | 57 | 1.1690 |
114
+ | 1.1383 | 0.7424 | 58 | 1.1688 |
115
+ | 1.1174 | 0.7552 | 59 | 1.1686 |
116
+ | 1.1413 | 0.768 | 60 | 1.1685 |
117
+ | 1.1871 | 0.7808 | 61 | 1.1684 |
118
+ | 1.1796 | 0.7936 | 62 | 1.1683 |
119
+ | 1.123 | 0.8064 | 63 | 1.1683 |
120
+ | 1.1645 | 0.8192 | 64 | 1.1682 |
121
+ | 1.1165 | 0.832 | 65 | 1.1681 |
122
+ | 1.0805 | 0.8448 | 66 | 1.1680 |
123
+ | 1.2018 | 0.8576 | 67 | 1.1678 |
124
+ | 1.0869 | 0.8704 | 68 | 1.1677 |
125
+ | 1.1286 | 0.8832 | 69 | 1.1676 |
126
+ | 1.0889 | 0.896 | 70 | 1.1676 |
127
+ | 1.1395 | 0.9088 | 71 | 1.1675 |
128
+ | 1.1756 | 0.9216 | 72 | 1.1674 |
129
+ | 1.1575 | 0.9344 | 73 | 1.1674 |
130
+ | 1.1073 | 0.9472 | 74 | 1.1673 |
131
+ | 1.163 | 0.96 | 75 | 1.1673 |
132
+ | 1.1789 | 0.9728 | 76 | 1.1673 |
133
+ | 1.1267 | 0.9856 | 77 | 1.1673 |
134
+ | 1.1416 | 0.9984 | 78 | 1.1672 |
135
+
136
+
137
+ ### Framework versions
138
+
139
+ - PEFT 0.10.0
140
+ - Transformers 4.43.0.dev0
141
+ - Pytorch 2.2.2+cu121
142
+ - Datasets 2.19.2
143
+ - Tokenizers 0.19.1