Vui Seng Chua commited on
Commit
9e57370
·
1 Parent(s): 9a98eea
Files changed (3) hide show
  1. README.md +5 -1
  2. perplexity_vs_sparsity.png +0 -0
  3. plot.py +19 -0
README.md CHANGED
@@ -1,5 +1,8 @@
1
  This repo contains sparsity report for each of the pruned model in the table below.
2
 
 
 
 
3
  Pruning ```meta-llama/Meta-Llama-3.1-8B``` with Wanda
4
  | Weight Target Sparsity | Perplexity (lower is better) |
5
  |------------------------|-----------------------------|
@@ -12,7 +15,8 @@ Pruning ```meta-llama/Meta-Llama-3.1-8B``` with Wanda
12
  | 60 | 20.2265 |
13
  | 70 | 103.5209 |
14
 
15
- The report (csv) shows layer-wise sparsity, sparsity by tile of 128x16, sparsity by col and row global to its layers.
 
16
 
17
  > For a more granular sparsity report within a given tile, pls continue below
18
 
 
1
  This repo contains sparsity report for each of the pruned model in the table below.
2
 
3
+ The report (csv) shows layer-wise sparsity, sparsity by tile of 128x16, sparsity by col and row global to its layers.
4
+
5
+
6
  Pruning ```meta-llama/Meta-Llama-3.1-8B``` with Wanda
7
  | Weight Target Sparsity | Perplexity (lower is better) |
8
  |------------------------|-----------------------------|
 
15
  | 60 | 20.2265 |
16
  | 70 | 103.5209 |
17
 
18
+ ![Perplexity over Sparsity](./perplexity_vs_sparsity.png)
19
+
20
 
21
  > For a more granular sparsity report within a given tile, pls continue below
22
 
perplexity_vs_sparsity.png ADDED
plot.py ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import matplotlib.pyplot as plt
2
+
3
+ # Data from the table
4
+ sparsity = [0, 10, 20, 30, 40, 50, 60, 70]
5
+ perplexity = [5.8393, 5.8781, 6.0102, 6.3076, 7.0094, 9.0642, 20.2265, 103.5209]
6
+
7
+ # Final adjustment: plot size to 6x4 inches
8
+ plt.figure(figsize=(6, 4))
9
+ plt.plot(sparsity, perplexity, marker='o', linestyle='-', color='b')
10
+ plt.axhline(y=5.8393, color='g', linestyle='--', label='Perplexity at 0% Sparsity')
11
+ plt.title("Perplexity vs. Weight Target Sparsity", fontsize=14)
12
+ plt.xlabel("Weight Target Sparsity (%)", fontsize=12)
13
+ plt.ylabel("Perplexity (lower is better)", fontsize=12)
14
+ plt.legend(fontsize=10)
15
+ plt.grid(True)
16
+ plt.xticks(fontsize=10)
17
+ plt.yticks(fontsize=10)
18
+ plt.show()
19
+ plt.savefig("perplexity_vs_sparsity.png")