Vui Seng Chua
commited on
Commit
·
9e57370
1
Parent(s):
9a98eea
Add plot
Browse files- README.md +5 -1
- perplexity_vs_sparsity.png +0 -0
- plot.py +19 -0
README.md
CHANGED
@@ -1,5 +1,8 @@
|
|
1 |
This repo contains sparsity report for each of the pruned model in the table below.
|
2 |
|
|
|
|
|
|
|
3 |
Pruning ```meta-llama/Meta-Llama-3.1-8B``` with Wanda
|
4 |
| Weight Target Sparsity | Perplexity (lower is better) |
|
5 |
|------------------------|-----------------------------|
|
@@ -12,7 +15,8 @@ Pruning ```meta-llama/Meta-Llama-3.1-8B``` with Wanda
|
|
12 |
| 60 | 20.2265 |
|
13 |
| 70 | 103.5209 |
|
14 |
|
15 |
-
|
|
|
16 |
|
17 |
> For a more granular sparsity report within a given tile, pls continue below
|
18 |
|
|
|
1 |
This repo contains sparsity report for each of the pruned model in the table below.
|
2 |
|
3 |
+
The report (csv) shows layer-wise sparsity, sparsity by tile of 128x16, sparsity by col and row global to its layers.
|
4 |
+
|
5 |
+
|
6 |
Pruning ```meta-llama/Meta-Llama-3.1-8B``` with Wanda
|
7 |
| Weight Target Sparsity | Perplexity (lower is better) |
|
8 |
|------------------------|-----------------------------|
|
|
|
15 |
| 60 | 20.2265 |
|
16 |
| 70 | 103.5209 |
|
17 |
|
18 |
+
![Perplexity over Sparsity](./perplexity_vs_sparsity.png)
|
19 |
+
|
20 |
|
21 |
> For a more granular sparsity report within a given tile, pls continue below
|
22 |
|
perplexity_vs_sparsity.png
ADDED
![]() |
plot.py
ADDED
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
import matplotlib.pyplot as plt
|
2 |
+
|
3 |
+
# Data from the table
|
4 |
+
sparsity = [0, 10, 20, 30, 40, 50, 60, 70]
|
5 |
+
perplexity = [5.8393, 5.8781, 6.0102, 6.3076, 7.0094, 9.0642, 20.2265, 103.5209]
|
6 |
+
|
7 |
+
# Final adjustment: plot size to 6x4 inches
|
8 |
+
plt.figure(figsize=(6, 4))
|
9 |
+
plt.plot(sparsity, perplexity, marker='o', linestyle='-', color='b')
|
10 |
+
plt.axhline(y=5.8393, color='g', linestyle='--', label='Perplexity at 0% Sparsity')
|
11 |
+
plt.title("Perplexity vs. Weight Target Sparsity", fontsize=14)
|
12 |
+
plt.xlabel("Weight Target Sparsity (%)", fontsize=12)
|
13 |
+
plt.ylabel("Perplexity (lower is better)", fontsize=12)
|
14 |
+
plt.legend(fontsize=10)
|
15 |
+
plt.grid(True)
|
16 |
+
plt.xticks(fontsize=10)
|
17 |
+
plt.yticks(fontsize=10)
|
18 |
+
plt.show()
|
19 |
+
plt.savefig("perplexity_vs_sparsity.png")
|