Hugging Face's logo

thrunlab
/
Mistral_Sparse_refined_web_50p_2024-03-10

Text Generation
Transformers
Safetensors
sparse_mistral
Generated from Trainer
custom_code
Model card Files Community
vxbrandon commited on
Commit
3b68e99
•
1 Parent(s): e6ef6c5

End of training

Browse files
Files changed (8) hide show
  1. README.md +9 -109
  2. adapter_config.json +3 -3
  3. adapter_model.safetensors +1 -1
  4. model-00001-of-00003.safetensors +1 -1
  5. model-00002-of-00003.safetensors +1 -1
  6. model-00003-of-00003.safetensors +1 -1
  7. sparsification_sftt.py +7 -2
  8. training_args.bin +1 -1
README.md CHANGED
@@ -15,7 +15,7 @@ should probably proofread and complete it, then remove this comment. -->
15
 
16
  This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
17
  It achieves the following results on the evaluation set:
18
- - Loss: 2.0919
19
 
20
  ## Model description
21
 
@@ -39,122 +39,22 @@ The following hyperparameters were used during training:
39
  - eval_batch_size: 1
40
  - seed: 0
41
  - distributed_type: multi-GPU
42
- - num_devices: 4
43
  - gradient_accumulation_steps: 4
44
- - total_train_batch_size: 16
45
- - total_eval_batch_size: 4
46
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
47
  - lr_scheduler_type: linear
48
- - training_steps: 2600
49
 
50
  ### Training results
51
 
52
  | Training Loss | Epoch | Step | Validation Loss |
53
  |:-------------:|:-----:|:----:|:---------------:|
54
- | 2.415 | 0.0 | 25 | 2.5680 |
55
- | 2.2287 | 0.01 | 50 | 2.5235 |
56
- | 2.2559 | 0.01 | 75 | 2.4801 |
57
- | 2.2193 | 0.02 | 100 | 2.4598 |
58
- | 2.2245 | 0.02 | 125 | 2.4460 |
59
- | 2.2962 | 0.02 | 150 | 2.4369 |
60
- | 2.2261 | 0.03 | 175 | 2.4264 |
61
- | 2.2229 | 0.03 | 200 | 2.4212 |
62
- | 2.2524 | 0.04 | 225 | 2.4164 |
63
- | 2.2935 | 0.04 | 250 | 2.4106 |
64
- | 2.2075 | 0.04 | 275 | 2.4061 |
65
- | 2.2153 | 0.05 | 300 | 2.4061 |
66
- | 2.229 | 0.05 | 325 | 2.4056 |
67
- | 2.0697 | 0.06 | 350 | 2.4035 |
68
- | 2.2312 | 0.06 | 375 | 2.3997 |
69
- | 2.2712 | 0.06 | 400 | 2.3958 |
70
- | 2.2631 | 0.07 | 425 | 2.3973 |
71
- | 2.2205 | 0.07 | 450 | 2.3978 |
72
- | 2.1442 | 0.08 | 475 | 2.3974 |
73
- | 2.2079 | 0.08 | 500 | 2.3962 |
74
- | 2.2704 | 0.08 | 525 | 2.3899 |
75
- | 2.1625 | 0.09 | 550 | 2.3895 |
76
- | 2.3011 | 0.09 | 575 | 2.3901 |
77
- | 2.086 | 0.1 | 600 | 2.3900 |
78
- | 2.1868 | 0.1 | 625 | 2.3922 |
79
- | 2.2738 | 0.1 | 650 | 2.3903 |
80
- | 2.2612 | 0.11 | 675 | 2.3872 |
81
- | 2.2295 | 0.11 | 700 | 2.3852 |
82
- | 2.2134 | 0.12 | 725 | 2.3819 |
83
- | 2.1193 | 0.12 | 750 | 2.3861 |
84
- | 2.1635 | 0.12 | 775 | 2.3861 |
85
- | 2.2542 | 0.13 | 800 | 2.3881 |
86
- | 2.2632 | 0.13 | 825 | 2.3833 |
87
- | 2.243 | 0.14 | 850 | 2.3839 |
88
- | 2.1096 | 0.14 | 875 | 2.3840 |
89
- | 2.1746 | 0.14 | 900 | 2.3868 |
90
- | 2.1214 | 0.15 | 925 | 2.3827 |
91
- | 2.2598 | 0.15 | 950 | 2.3829 |
92
- | 2.193 | 0.16 | 975 | 2.3859 |
93
- | 2.2361 | 0.16 | 1000 | 2.3822 |
94
- | 2.2412 | 0.16 | 1025 | 2.3807 |
95
- | 2.2359 | 0.17 | 1050 | 2.3820 |
96
- | 2.2822 | 0.17 | 1075 | 2.3794 |
97
- | 2.2076 | 0.18 | 1100 | 2.3812 |
98
- | 2.2422 | 0.18 | 1125 | 2.3811 |
99
- | 2.2015 | 0.18 | 1150 | 2.3806 |
100
- | 2.2625 | 0.19 | 1175 | 2.3791 |
101
- | 2.1099 | 0.19 | 1200 | 2.3781 |
102
- | 2.2019 | 0.2 | 1225 | 2.3777 |
103
- | 2.184 | 0.2 | 1250 | 2.3786 |
104
- | 2.2258 | 0.2 | 1275 | 2.3768 |
105
- | 2.2587 | 0.21 | 1300 | 2.3779 |
106
- | 2.235 | 0.21 | 1325 | 2.3769 |
107
- | 2.1314 | 0.22 | 1350 | 2.3781 |
108
- | 2.2081 | 0.22 | 1375 | 2.3782 |
109
- | 2.2186 | 0.22 | 1400 | 2.3761 |
110
- | 2.1597 | 0.23 | 1425 | 2.3770 |
111
- | 2.1417 | 0.23 | 1450 | 2.3785 |
112
- | 2.1131 | 0.24 | 1475 | 2.3752 |
113
- | 2.1621 | 0.24 | 1500 | 2.3780 |
114
- | 2.0999 | 0.24 | 1525 | 2.3763 |
115
- | 2.1708 | 0.25 | 1550 | 2.3777 |
116
- | 2.2863 | 0.25 | 1575 | 2.3756 |
117
- | 2.2396 | 0.26 | 1600 | 2.3780 |
118
- | 2.2023 | 0.26 | 1625 | 2.3773 |
119
- | 2.2884 | 0.26 | 1650 | 2.3766 |
120
- | 2.1907 | 0.27 | 1675 | 2.3769 |
121
- | 2.2297 | 0.27 | 1700 | 2.3771 |
122
- | 2.2199 | 0.28 | 1725 | 2.3801 |
123
- | 2.1971 | 0.28 | 1750 | 2.3770 |
124
- | 2.1796 | 0.28 | 1775 | 2.3773 |
125
- | 2.1441 | 0.29 | 1800 | 2.3769 |
126
- | 2.224 | 0.29 | 1825 | 2.3786 |
127
- | 2.2156 | 0.3 | 1850 | 2.3801 |
128
- | 2.2023 | 0.3 | 1875 | 2.3758 |
129
- | 2.2222 | 0.3 | 1900 | 2.3768 |
130
- | 2.2751 | 0.31 | 1925 | 2.3757 |
131
- | 2.2201 | 0.31 | 1950 | 2.3742 |
132
- | 2.2462 | 0.32 | 1975 | 2.3753 |
133
- | 2.1603 | 0.32 | 2000 | 2.3756 |
134
- | 2.1849 | 0.32 | 2025 | 2.3755 |
135
- | 2.1675 | 0.33 | 2050 | 2.3764 |
136
- | 2.1657 | 0.33 | 2075 | 2.3755 |
137
- | 2.2616 | 0.34 | 2100 | 2.3740 |
138
- | 2.3012 | 0.34 | 2125 | 2.3738 |
139
- | 2.2098 | 0.34 | 2150 | 2.3754 |
140
- | 2.169 | 0.35 | 2175 | 2.3755 |
141
- | 2.1896 | 0.35 | 2200 | 2.3728 |
142
- | 2.1382 | 0.36 | 2225 | 2.3724 |
143
- | 2.131 | 0.36 | 2250 | 2.3719 |
144
- | 2.2687 | 0.36 | 2275 | 2.3695 |
145
- | 2.2486 | 0.37 | 2300 | 2.3693 |
146
- | 2.1445 | 0.37 | 2325 | 2.3708 |
147
- | 2.1863 | 0.38 | 2350 | 2.3685 |
148
- | 2.2238 | 0.38 | 2375 | 2.3670 |
149
- | 2.2373 | 0.38 | 2400 | 2.3680 |
150
- | 2.0792 | 0.39 | 2425 | 2.3685 |
151
- | 2.0805 | 0.39 | 2450 | 2.3662 |
152
- | 2.166 | 0.4 | 2475 | 2.3678 |
153
- | 2.249 | 0.4 | 2500 | 2.3683 |
154
- | 2.2577 | 0.4 | 2525 | 2.3706 |
155
- | 2.1109 | 0.41 | 2550 | 2.3691 |
156
- | 2.1343 | 0.41 | 2575 | 2.3677 |
157
- | 2.2144 | 0.42 | 2600 | 2.3693 |
158
 
159
 
160
  ### Framework versions
 
15
 
16
  This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
17
  It achieves the following results on the evaluation set:
18
+ - Loss: 2.2038
19
 
20
  ## Model description
21
 
 
39
  - eval_batch_size: 1
40
  - seed: 0
41
  - distributed_type: multi-GPU
42
+ - num_devices: 3
43
  - gradient_accumulation_steps: 4
44
+ - total_train_batch_size: 12
45
+ - total_eval_batch_size: 3
46
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
47
  - lr_scheduler_type: linear
48
+ - training_steps: 100
49
 
50
  ### Training results
51
 
52
  | Training Loss | Epoch | Step | Validation Loss |
53
  |:-------------:|:-----:|:----:|:---------------:|
54
+ | 2.3494 | 0.0 | 25 | 2.5710 |
55
+ | 2.4899 | 0.01 | 50 | 2.5247 |
56
+ | 2.3538 | 0.01 | 75 | 2.4854 |
57
+ | 2.2439 | 0.01 | 100 | 2.4641 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
 
59
 
60
  ### Framework versions
adapter_config.json CHANGED
@@ -19,11 +19,11 @@
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
 
22
  "v_proj",
23
  "down_proj",
24
- "up_proj",
25
- "gate_proj",
26
- "q_proj"
27
  ],
28
  "task_type": "CAUSAL_LM"
29
  }
 
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
+ "up_proj",
23
  "v_proj",
24
  "down_proj",
25
+ "q_proj",
26
+ "gate_proj"
 
27
  ],
28
  "task_type": "CAUSAL_LM"
29
  }
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bd4e8b381905e417363fa265c6a21ec8d5c47b32522723663d53ce9658c023c3
3
  size 281061608
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:94743bd541c5b4b5e9a8027a9bf4fa67a3830b2d41b59a56eb10af673ef436e6
3
  size 281061608
model-00001-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3a5336ceb88d83727181978159b424f8c1b95e8253fcdecc9112d5681756a573
3
  size 4943162336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:498c080dcf48c045521ff16d442ce23a45ec7083b31895c1c40a1136539fb7ed
3
  size 4943162336
model-00002-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2456526ecb117aebc612d16313d67a761eec452e372720f66a0f2c7c201704f6
3
  size 4999819336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5361b93f43dd55633500a4ea89d2b1b78aaa2f0574d04d83efd432c86094e5ff
3
  size 4999819336
model-00003-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ba920a2cec936486b08a59db3a59ca7ec4e079186458a05622b1ff87ece605bd
3
  size 4540516344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a5bb57df83d05596ed401063f7f9b17eb3bd77718b9c05e571fd673c3f0a9b59
3
  size 4540516344
sparsification_sftt.py CHANGED
@@ -320,7 +320,6 @@ class MistralSparseSiluMLP(MistralMLP):
320
  # Sparse activation function
321
  self.sparse_act_fn = SparseSiLU(threshold=self.dead_threshold)
322
 
323
-
324
  def activate_stats(self, is_collect_histogram: bool = True):
325
  self.is_stats = True
326
  self.dead_percentage = 0
@@ -381,6 +380,9 @@ class MistralSparseSiluMLP(MistralMLP):
381
  return self.down_proj(post_act * self.up_proj(x))
382
 
383
  else:
 
 
 
384
  pre_act = self.gate_proj(x)
385
  post_act = self.act_fn(pre_act)
386
  if self.kill_sparse_swish_outputs:
@@ -871,7 +873,10 @@ def set_sparse_threshold(model, sparsity_level: float, use_relu: bool = False):
871
 
872
 
873
  def plot_histogram(
874
- bin_edges, histogram_counts: torch.tensor, title: str = "Activation Distribution", fig_dir: str = "figures"
 
 
 
875
  ):
876
  plt.bar(
877
  bin_edges[:-1], histogram_counts, width=np.diff(bin_edges), edgecolor="black"
 
320
  # Sparse activation function
321
  self.sparse_act_fn = SparseSiLU(threshold=self.dead_threshold)
322
 
 
323
  def activate_stats(self, is_collect_histogram: bool = True):
324
  self.is_stats = True
325
  self.dead_percentage = 0
 
380
  return self.down_proj(post_act * self.up_proj(x))
381
 
382
  else:
383
+ self.count += 1
384
+ if self.count <= 1:
385
+ print("USING SparseSILU!!!!")
386
  pre_act = self.gate_proj(x)
387
  post_act = self.act_fn(pre_act)
388
  if self.kill_sparse_swish_outputs:
 
873
 
874
 
875
  def plot_histogram(
876
+ bin_edges,
877
+ histogram_counts: torch.tensor,
878
+ title: str = "Activation Distribution",
879
+ fig_dir: str = "figures",
880
  ):
881
  plt.bar(
882
  bin_edges[:-1], histogram_counts, width=np.diff(bin_edges), edgecolor="black"
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:05e6f7a32604fa61f389a8051cfbc20ca0269a2ed9c87034e6b49d5a98a9ea80
3
  size 6456
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:990b5362067c4bd760007683596bec970c263d55089fcbdbf8243c0c703bfcf6
3
  size 6456