Find important layers with iMatrix?
#2
by Downtown-Case - opened
With text LLMs, we can get an "importance" map of layers by dumping iMatrix data. For instance, with DeepseekV4, it looks like:
Layer Tensor Σ(Act²) Min Max μ σ % Active N Entropy E (norm) ZD CosSim
0.01.394.559 I =========================================================================================================================================================================
...
0.01.395.027 I 42 ffn_down_exps 2188310.00 0.0034 2307.5815 4.17 42.45 100.00% 524288 13.8542 72.92% 0.95% 0.0075
0.01.395.029 I 41 ffn_down_exps 1342491.50 0.0014 1482.0859 2.56 30.71 100.00% 524288 13.2723 69.85% 0.84% 0.0061
0.01.395.030 I 39 ffn_down_exps 326504.12 0.0048 237.1173 0.62 1.95 100.00% 524288 17.6960 93.14% 3.59% 0.0121
0.01.395.032 I 40 ffn_down_exps 312423.81 0.0026 411.8223 0.60 5.31 100.00% 524288 15.6911 82.58% 0.84% 0.0248
0.01.395.033 I 37 ffn_down_exps 222362.38 0.0007 489.9371 0.42 6.33 100.00% 524288 14.1953 74.71% 0.24% 0.0012
0.01.395.035 I 36 ffn_down_exps 156658.64 0.0230 3400.4160 0.30 5.36 100.00% 524288 16.2856 85.71% 0.23% 0.0081
0.01.395.036 I 38 ffn_down_exps 120999.50 0.0140 894.2704 0.23 1.63 100.00% 524288 17.5129 92.17% 0.95% 0.0046
0.01.395.038 I 20 ffn_down_exps 118100.43 0.0088 302.3193 0.23 1.26 100.00% 524288 17.8586 93.99% 0.80% 0.0530
0.01.395.039 I 35 ffn_down_exps 106088.88 0.0172 89.1712 0.20 0.60 100.00% 524288 17.9852 94.66% 2.55% 0.1036
0.01.395.041 I 34 ffn_down_exps 105549.59 0.0193 57.0928 0.20 0.74 100.00% 524288 17.7523 93.43% 1.45% 0.0199
0.01.395.042 I 22 ffn_down_exps 96935.09 0.0094 124.3763 0.18 0.48 100.00% 524288 18.1703 95.63% 2.99% 0.0478
0.01.395.044 I 32 ffn_down_exps 86532.07 0.0165 152.3934 0.17 1.09 100.00% 524288 17.4677 91.94% 0.60% 0.0596
0.01.395.045 I 30 ffn_down_exps 83201.19 0.0204 380.0406 0.16 1.08 100.00% 524288 17.3665 91.40% 0.47% 0.0130
0.01.395.047 I 29 ffn_down_exps 77579.06 0.0181 310.9636 0.15 1.10 100.00% 524288 17.1501 90.26% 0.46% 0.0343
0.01.395.048 I 26 ffn_down_exps 75697.02 0.0131 323.1925 0.14 0.63 100.00% 524288 18.0774 95.14% 1.35% 0.0090
0.01.395.050 I 28 ffn_down_exps 70282.80 0.0140 95.2578 0.13 0.30 100.00% 524288 18.3332 96.49% 3.29% 0.0554
0.01.395.051 I 33 ffn_down_exps 69956.70 0.0164 523.0419 0.13 0.87 100.00% 524288 18.1195 95.37% 0.65% 0.0158
0.01.395.053 I 31 ffn_down_exps 69351.28 0.0149 96.7603 0.13 0.44 100.00% 524288 18.0518 95.01% 2.42% 0.0188
0.01.395.054 I 18 ffn_down_exps 67680.16 0.0069 215.6972 0.13 0.50 100.00% 524288 18.2169 95.88% 1.26% 0.0091
0.01.395.056 I 21 ffn_down_exps 62098.25 0.0068 83.9246 0.12 0.29 100.00% 524288 18.3104 96.37% 2.87% 0.0494
0.01.395.057 I 24 ffn_down_exps 56159.25 0.0044 60.2592 0.11 0.31 100.00% 524288 18.1794 95.68% 2.64% 0.0146
0.01.395.059 I 27 ffn_down_exps 47317.23 0.0120 49.1935 0.09 0.32 100.00% 524288 18.2059 95.82% 1.53% 0.0212
0.01.395.060 I 25 ffn_down_exps 45894.51 0.0141 317.9896 0.09 0.57 100.00% 524288 18.2478 96.04% 0.57% 0.0186
0.01.395.061 I 23 ffn_down_exps 43613.12 0.0078 557.6200 0.08 1.00 100.00% 524288 17.8259 93.82% 0.36% 0.0184
0.01.395.063 I 17 ffn_down_exps 38313.35 0.0057 123.1870 0.07 0.23 100.00% 524288 18.4736 97.23% 1.24% 0.0284
0.01.395.064 I 19 ffn_down_exps 34140.08 0.0026 25.3453 0.07 0.15 100.00% 524288 18.0750 95.13% 3.60% 0.0337
0.01.395.066 I 16 ffn_down_exps 26969.89 0.0052 43.9049 0.05 0.13 100.00% 524288 18.4342 97.02% 1.78% 0.0919
0.01.395.067 I 7 ffn_down_exps 21039.93 0.0040 16.9352 0.04 0.09 100.00% 524288 18.2099 95.84% 4.02% 0.0307
0.01.395.069 I 8 ffn_down_exps 19691.43 0.0030 60.6809 0.04 0.21 100.00% 524288 17.9929 94.70% 1.01% 0.0338
0.01.395.070 I 6 ffn_down_exps 19079.35 0.0020 80.1530 0.04 0.19 100.00% 524288 17.9975 94.72% 1.39% 0.0262
0.01.395.072 I 15 ffn_down_exps 18530.90 0.0043 16.3784 0.04 0.05 100.00% 524288 18.5393 97.58% 5.38% 0.1252
0.01.395.073 I 14 ffn_down_exps 18457.52 0.0038 32.4825 0.04 0.07 100.00% 524288 18.4495 97.10% 2.92% 0.1021
0.01.395.075 I 12 ffn_down_exps 17647.14 0.0032 25.4635 0.03 0.08 100.00% 524288 18.3605 96.63% 2.86% 0.0544
0.01.395.076 I 13 ffn_down_exps 17050.14 0.0043 85.4791 0.03 0.14 100.00% 524288 18.3883 96.78% 0.90% 0.0714
0.01.395.077 I 0 ffn_down_exps 14612.26 0.0017 111.8095 0.03 0.39 100.00% 524288 16.1380 84.94% 0.57% 0.0000
0.01.395.079 I 11 ffn_down_exps 13912.36 0.0033 120.1434 0.03 0.24 100.00% 524288 18.2498 96.05% 0.27% 0.0460
0.01.395.080 I 9 ffn_down_exps 12179.78 0.0025 186.1999 0.02 0.44 100.00% 524288 17.7973 93.67% 0.08% 0.0278
0.01.395.082 I 10 ffn_down_exps 11200.07 0.0018 40.7784 0.02 0.09 100.00% 524288 18.2278 95.94% 1.16% 0.0437
0.01.395.083 I 5 ffn_down_exps 10742.23 0.0018 34.6680 0.02 0.09 100.00% 524288 18.1245 95.39% 1.51% 0.0310
0.01.395.085 I 3 ffn_down_exps 8788.01 0.0007 22.1987 0.02 0.06 100.00% 524288 17.9272 94.35% 2.39% 0.0278
0.01.395.086 I 2 ffn_down_exps 8732.57 0.0031 64.2533 0.02 0.12 100.00% 524288 17.8960 94.19% 0.64% 0.0337
0.01.395.088 I 4 ffn_down_exps 6289.40 0.0008 22.9802 0.01 0.05 100.00% 524288 17.9751 94.61% 2.15% 0.0645
0.01.395.089 I 1 ffn_down_exps 6086.41 0.0026 6.2940 0.01 0.03 100.00% 524288 18.4388 97.05% 1.50% 0.0152
0.01.395.091 I 42 ffn_down_shexp 34612.52 0.0160 1341.1948 16.90 93.21 100.00% 2048 7.1877 65.34% 2.54% 0.0310
0.01.395.092 I 41 ffn_down_shexp 2724.08 0.0114 212.8199 1.33 10.03 100.00% 2048 7.2744 66.13% 1.51% 0.0351
0.01.395.094 I 40 ffn_down_shexp 827.90 0.0074 43.3332 0.40 1.79 100.00% 2048 9.0304 82.09% 1.86% 0.0707
0.01.395.095 I 39 ffn_down_shexp 786.74 0.0171 37.1856 0.38 1.40 100.00% 2048 9.5218 86.56% 1.51% 0.0738
0.01.395.097 I 38 ffn_down_shexp 350.24 0.0090 18.1399 0.17 0.58 100.00% 2048 9.6738 87.94% 2.39% 0.0668
0.01.395.098 I 37 ffn_down_shexp 331.55 0.0115 19.0700 0.16 0.69 100.00% 2048 9.0949 82.68% 1.95% 0.0813
0.01.395.099 I 36 ffn_down_shexp 316.18 0.0118 7.7782 0.15 0.35 100.00% 2048 9.9761 90.69% 3.37% 0.2196
0.01.395.101 I 20 ffn_down_shexp 234.34 0.0115 8.4255 0.11 0.33 100.00% 2048 9.4939 86.31% 4.00% 0.1512
0.01.395.102 I 0 ffn_down_shexp 230.03 0.0010 3.6309 0.11 0.17 100.00% 2048 10.2337 93.03% 5.08% 0.0000
0.01.395.103 I 22 ffn_down_shexp 229.66 0.0093 6.3586 0.11 0.26 100.00% 2048 9.7870 88.97% 5.27% 0.1002
0.01.395.105 I 34 ffn_down_shexp 211.43 0.0078 7.5232 0.10 0.34 100.00% 2048 9.3473 84.98% 3.52% 0.0922
0.01.395.106 I 35 ffn_down_shexp 181.20 0.0122 2.7099 0.09 0.13 100.00% 2048 10.3347 93.95% 5.66% 0.1978
0.01.395.107 I 21 ffn_down_shexp 171.94 0.0055 18.8485 0.08 0.62 100.00% 2048 8.0438 73.13% 1.07% 0.0879
0.01.395.109 I 31 ffn_down_shexp 171.21 0.0092 3.8894 0.08 0.20 100.00% 2048 9.8624 89.66% 3.81% 0.1497
0.01.395.110 I 33 ffn_down_shexp 146.38 0.0069 5.2085 0.07 0.22 100.00% 2048 9.5951 87.23% 2.44% 0.1711
0.01.395.111 I 27 ffn_down_shexp 142.28 0.0062 10.0730 0.07 0.35 100.00% 2048 9.0720 82.47% 1.51% 0.0197
0.01.395.113 I 32 ffn_down_shexp 140.41 0.0069 2.9945 0.07 0.13 100.00% 2048 10.1864 92.60% 3.81% 0.2066
0.01.395.114 I 30 ffn_down_shexp 139.77 0.0095 5.1566 0.07 0.19 100.00% 2048 9.9563 90.51% 2.29% 0.1502
0.01.395.115 I 26 ffn_down_shexp 128.48 0.0074 20.1095 0.06 0.50 100.00% 2048 8.5832 78.03% 1.03% 0.1130
0.01.395.117 I 18 ffn_down_shexp 116.00 0.0071 4.6894 0.06 0.16 100.00% 2048 9.6093 87.36% 4.00% 0.1583
0.01.395.118 I 19 ffn_down_shexp 106.18 0.0030 3.0635 0.05 0.11 100.00% 2048 10.0625 91.48% 5.42% 0.1697
0.01.395.120 I 24 ffn_down_shexp 100.60 0.0060 18.3339 0.05 0.41 100.00% 2048 9.0142 81.95% 0.44% 0.0532
0.01.395.121 I 29 ffn_down_shexp 98.30 0.0090 3.2956 0.05 0.12 100.00% 2048 10.0478 91.34% 2.59% 0.2410
0.01.395.122 I 28 ffn_down_shexp 94.77 0.0095 1.9621 0.05 0.07 100.00% 2048 10.4385 94.90% 4.25% 0.1522
0.01.395.124 I 23 ffn_down_shexp 92.10 0.0063 5.0679 0.04 0.13 100.00% 2048 10.1048 91.86% 2.64% 0.1845
0.01.395.125 I 25 ffn_down_shexp 70.68 0.0068 1.6708 0.03 0.07 100.00% 2048 10.2358 93.05% 3.61% 0.0562
0.01.395.126 I 17 ffn_down_shexp 64.33 0.0054 2.6332 0.03 0.09 100.00% 2048 9.6997 88.18% 3.76% 0.1993
0.01.395.128 I 2 ffn_down_shexp 60.10 0.0001 0.9880 0.03 0.05 100.00% 2048 9.7403 88.55% 9.77% 0.3717
0.01.395.129 I 16 ffn_down_shexp 44.27 0.0042 1.2203 0.02 0.05 100.00% 2048 10.0215 91.10% 4.64% 0.1345
0.01.395.130 I 15 ffn_down_shexp 40.14 0.0028 2.0710 0.02 0.08 100.00% 2048 9.5433 86.76% 1.76% 0.0919
0.01.395.132 I 7 ffn_down_shexp 38.57 0.0016 0.9163 0.02 0.03 100.00% 2048 10.3202 93.82% 5.37% 0.3248
0.01.395.133 I 13 ffn_down_shexp 34.90 0.0026 3.3175 0.02 0.09 100.00% 2048 9.2855 84.41% 1.12% 0.0927
0.01.395.134 I 8 ffn_down_shexp 34.03 0.0009 0.6567 0.02 0.03 100.00% 2048 10.3375 93.98% 5.76% 0.3255
0.01.395.136 I 14 ffn_down_shexp 32.17 0.0028 1.2743 0.02 0.04 100.00% 2048 9.9871 90.79% 3.22% 0.0695
0.01.395.137 I 12 ffn_down_shexp 30.67 0.0026 0.7244 0.01 0.03 100.00% 2048 10.1094 91.90% 3.56% 0.2364
0.01.395.138 I 11 ffn_down_shexp 25.76 0.0019 0.4718 0.01 0.02 100.00% 2048 10.1934 92.67% 4.39% 0.3184
0.01.395.139 I 6 ffn_down_shexp 25.16 0.0013 0.6841 0.01 0.02 100.00% 2048 10.3666 94.24% 4.83% 0.2502
0.01.395.141 I 9 ffn_down_shexp 23.50 0.0011 0.5647 0.01 0.03 100.00% 2048 10.0643 91.49% 3.37% 0.2357
0.01.395.142 I 10 ffn_down_shexp 21.54 0.0016 0.2399 0.01 0.01 100.00% 2048 10.3685 94.26% 6.20% 0.2763
0.01.395.143 I 1 ffn_down_shexp 21.02 0.0002 0.2970 0.01 0.01 100.00% 2048 10.3180 93.80% 8.35% 0.3348
0.01.395.145 I 4 ffn_down_shexp 15.74 0.0011 0.1577 0.01 0.01 100.00% 2048 10.4121 94.66% 6.69% 0.4026
0.01.395.146 I 3 ffn_down_shexp 15.25 0.0019 0.1763 0.01 0.01 100.00% 2048 10.5377 95.80% 4.10% 0.3213
0.01.395.148 I 5 ffn_down_shexp 14.10 0.0007 0.5160 0.01 0.01 100.00% 2048 10.2768 93.43% 3.76% 0.2988
...
Σ(Act²): sum of all squared activations (the importance scores)
Min & Max: minimum and maximum squared activations values
μ & σ: Squared activations' mean and standard deviation
% Active: proportion of elements whose average squared activation exceeds a small threshold (1e-5). Helpful to determine how alive/dormant the tensor is during inference
N: number of squared activations
Entropy: entropy of the squared activation distribution, in bits (standard Shannon entropy measurement) S = − ∑ i = 1 N p i log 2 p i
E (norm): Normalized entropy. E ( n o r m ) = − ∑ i = 1 N p i log 2 p i l o g 2 N . These two metrics can be used to determine how well a prompt "exercises" the model's capabilities
ZD Score: z-score distribution as described in 3.1 Layer Importance Scores of Layer-Wise Quantization
CosSim: cosine similarity with respect to the previous layer's tensor. Useful to determine how similar the squared activations of the current layer are to the previous layer's squared activations.
Weighted averages of Σ(Act²), ZD Score and CosSim are also calculated.
When using these statistics, please note that they are computed on the squared activations, not on the actual (raw) activations. Whilst the results are still useful, they're less reliable than using the raw values, and in the case of the cosine similarity, could be misleading if the tensor contains opposite vectors.
(Full output): https://huggingface.co/Downtown-Case/DeepSeek-V4-Flash-0731-128GB-RAM-IK-GGUF/blob/main/KLD_Logs%2BData/iMatrix_layer_activations.txt
And we can see, in this instance, that DeepseekV4 is remarkably "bottom heavy", so it's best to leave the last few FFN down experts unquantized, while the early ones will accept quantization better.
I'd like to do a similar analysis with Minimax H3, but I'm not sure there's a tool to do it? It seems stablediffusion.cpp can generate an importance matrix:
https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/imatrix.md
But doesn't support dumping the layer statistics, yet.
I vibecoded a statistics dump into stable-diffusion.cpp, heh.
Here's an initial one. I might make more iMatrix runs and see if that changes the dump.
=========================================================================================================================================================================================
Computing statistics (614 tensors)
Tensor Σ(Act²) Min Max μ σ % Active N Entropy E (norm) ZD
[INFO ] main.cpp:709 - loading imatrix from '/home/alpha/AI/stable-diffusion-dump.cpp/h3-imatrix.dat'
model.diffusion_model.blocks.0.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.1.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.2.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.3.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.4.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.5.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.6.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.7.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.8.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.9.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.10.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.11.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.12.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.13.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.14.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.15.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.16.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.17.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.18.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.19.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.20.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.21.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.22.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.23.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.24.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.25.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.26.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.27.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.28.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.29.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.30.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.31.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.32.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.33.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.34.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.35.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.36.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.37.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.38.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.39.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.40.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.41.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.42.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.43.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.44.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.45.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.46.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.47.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.48.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.49.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.blocks.0.attn.out_proj.weight 6202.62 0.0083 23.7319 0.87 1.39 100.00% 7168 11.7748 91.94% 8.78%
model.diffusion_model.token_refiner.blocks.0.attn.out_proj.weight 387729.56 0.0011 2565.4746 54.09 116.62 100.00% 7168 11.3557 88.67% 7.14%
model.diffusion_model.blocks.1.attn.out_proj.weight 46959.87 0.0478 90.8520 6.55 9.88 100.00% 7168 11.7239 91.54% 11.45%
model.diffusion_model.token_refiner.blocks.1.attn.out_proj.weight 102254.27 0.0714 339.0525 14.27 23.84 100.00% 7168 11.6727 91.14% 8.86%
model.diffusion_model.blocks.2.attn.out_proj.weight 15045.74 0.0259 30.1027 2.10 3.14 100.00% 7168 11.8200 92.29% 9.50%
model.diffusion_model.blocks.3.attn.out_proj.weight 20624.10 0.0369 38.6121 2.88 4.23 100.00% 7168 11.7696 91.90% 11.58%
model.diffusion_model.blocks.4.attn.out_proj.weight 17481.88 0.0716 40.5009 2.44 2.88 100.00% 7168 12.1245 94.67% 10.92%
model.diffusion_model.blocks.5.attn.out_proj.weight 55775.04 0.1642 134.3691 7.78 14.41 100.00% 7168 11.5477 90.16% 6.64%
model.diffusion_model.blocks.6.attn.out_proj.weight 29734.83 0.0807 78.6473 4.15 5.85 100.00% 7168 11.9323 93.17% 9.88%
model.diffusion_model.blocks.7.attn.out_proj.weight 30285.88 0.0863 123.0555 4.23 6.95 100.00% 7168 11.8313 92.38% 7.44%
model.diffusion_model.blocks.8.attn.out_proj.weight 31849.85 0.1190 81.9460 4.44 6.14 100.00% 7168 11.9445 93.26% 10.03%
model.diffusion_model.blocks.9.attn.out_proj.weight 48854.78 0.1404 338.9897 6.82 14.47 100.00% 7168 11.6182 90.71% 5.82%
model.diffusion_model.blocks.10.attn.out_proj.weight 35505.46 0.1331 96.4819 4.95 6.83 100.00% 7168 11.9917 93.63% 9.05%
model.diffusion_model.blocks.11.attn.out_proj.weight 48252.97 0.1761 102.9977 6.73 7.79 100.00% 7168 12.1311 94.72% 11.54%
model.diffusion_model.blocks.12.attn.out_proj.weight 53961.42 0.1529 142.6175 7.53 11.19 100.00% 7168 11.8749 92.72% 9.95%
model.diffusion_model.blocks.13.attn.out_proj.weight 60442.61 0.3057 182.5312 8.43 11.46 100.00% 7168 12.0179 93.84% 9.56%
model.diffusion_model.blocks.14.attn.out_proj.weight 76698.44 0.3163 461.7632 10.70 18.92 100.00% 7168 11.8546 92.56% 6.64%
model.diffusion_model.blocks.15.attn.out_proj.weight 76113.62 0.4402 126.4336 10.62 11.87 100.00% 7168 12.1827 95.12% 11.19%
model.diffusion_model.blocks.16.attn.out_proj.weight 76839.54 0.4645 235.6785 10.72 12.62 100.00% 7168 12.1892 95.17% 9.24%
model.diffusion_model.blocks.17.attn.out_proj.weight 99362.79 0.6574 224.4041 13.86 14.76 100.00% 7168 12.2369 95.55% 11.01%
model.diffusion_model.blocks.18.attn.out_proj.weight 150093.05 0.8259 272.5168 20.94 22.88 100.00% 7168 12.2312 95.50% 9.78%
model.diffusion_model.blocks.19.attn.out_proj.weight 200740.75 1.0498 543.2661 28.01 36.41 100.00% 7168 12.0522 94.10% 9.86%
model.diffusion_model.blocks.20.attn.out_proj.weight 185256.77 1.0173 441.6166 25.84 28.71 100.00% 7168 12.2209 95.42% 9.97%
model.diffusion_model.blocks.21.attn.out_proj.weight 181773.20 1.2037 510.2438 25.36 27.34 100.00% 7168 12.2572 95.70% 10.52%
model.diffusion_model.blocks.22.attn.out_proj.weight 234816.45 1.1254 758.1772 32.76 40.99 100.00% 7168 12.1652 94.99% 8.58%
model.diffusion_model.blocks.23.attn.out_proj.weight 301210.31 1.1076 810.9752 42.02 47.55 100.00% 7168 12.2117 95.35% 9.67%
model.diffusion_model.blocks.24.attn.out_proj.weight 278978.03 1.9128 593.9188 38.92 39.19 100.00% 7168 12.2835 95.91% 11.72%
model.diffusion_model.blocks.25.attn.out_proj.weight 336258.47 1.4878 693.4986 46.91 44.27 100.00% 7168 12.3487 96.42% 11.02%
model.diffusion_model.blocks.26.attn.out_proj.weight 504076.22 4.4263 606.0920 70.32 60.54 100.00% 7168 12.3913 96.75% 11.86%
model.diffusion_model.blocks.27.attn.out_proj.weight 626798.00 2.4192 1294.5457 87.44 87.12 100.00% 7168 12.2984 96.03% 10.91%
model.diffusion_model.blocks.28.attn.out_proj.weight 1009886.19 1.7355 2674.4634 140.89 153.51 100.00% 7168 12.2104 95.34% 10.53%
model.diffusion_model.blocks.29.attn.out_proj.weight 1467630.12 4.3360 2823.6702 204.75 198.42 100.00% 7168 12.3165 96.17% 10.73%
model.diffusion_model.blocks.30.attn.out_proj.weight 1590480.88 0.9601 4118.7700 221.89 227.47 100.00% 7168 12.2937 95.99% 9.35%
model.diffusion_model.blocks.31.attn.out_proj.weight 2168693.00 9.2232 3632.9077 302.55 318.45 100.00% 7168 12.2450 95.61% 10.60%
model.diffusion_model.blocks.32.attn.out_proj.weight 2558283.25 1.5779 7803.0479 356.90 425.88 100.00% 7168 12.1413 94.80% 9.43%
model.diffusion_model.blocks.33.attn.out_proj.weight 3365109.50 11.0787 6192.7021 469.46 452.57 100.00% 7168 12.2870 95.94% 12.10%
model.diffusion_model.blocks.34.attn.out_proj.weight 3053948.50 11.2750 4811.2090 426.05 426.80 100.00% 7168 12.2610 95.73% 10.73%
model.diffusion_model.blocks.35.attn.out_proj.weight 3893367.75 8.2484 6912.2124 543.16 584.18 100.00% 7168 12.1871 95.16% 11.20%
model.diffusion_model.blocks.36.attn.out_proj.weight 4264811.50 4.1692 10259.6270 594.98 619.44 100.00% 7168 12.2337 95.52% 11.09%
model.diffusion_model.blocks.37.attn.out_proj.weight 2904622.25 2.6665 4261.7778 405.22 412.94 100.00% 7168 12.2048 95.30% 11.57%
model.diffusion_model.blocks.38.attn.out_proj.weight 2157850.50 2.2985 6944.4595 301.04 396.95 100.00% 7168 11.9637 93.41% 11.05%
model.diffusion_model.blocks.39.attn.out_proj.weight 1541379.75 2.8263 2987.7583 215.04 258.41 100.00% 7168 12.1053 94.52% 10.31%
model.diffusion_model.blocks.40.attn.out_proj.weight 1892064.50 2.2907 4378.9624 263.96 254.58 100.00% 7168 12.2522 95.66% 13.24%
model.diffusion_model.blocks.41.attn.out_proj.weight 2023697.12 1.7891 6292.1758 282.32 365.78 100.00% 7168 11.9924 93.64% 10.63%
model.diffusion_model.blocks.42.attn.out_proj.weight 1716527.12 4.0379 2447.4929 239.47 240.23 100.00% 7168 12.2447 95.61% 11.50%
model.diffusion_model.blocks.43.attn.out_proj.weight 3018289.75 5.4196 8978.5547 421.08 564.70 100.00% 7168 12.0512 94.10% 8.78%
model.diffusion_model.blocks.44.attn.out_proj.weight 4260718.50 4.4132 11793.5410 594.41 872.95 100.00% 7168 11.8697 92.68% 9.75%
model.diffusion_model.blocks.45.attn.out_proj.weight 5511802.50 4.2116 19069.2031 768.95 1271.54 100.00% 7168 11.8119 92.23% 6.68%
model.diffusion_model.blocks.46.attn.out_proj.weight 3879571.50 14.1795 10186.3018 541.23 865.25 100.00% 7168 11.8123 92.23% 7.83%
model.diffusion_model.blocks.47.attn.out_proj.weight 4805472.00 8.9552 18217.1719 670.41 1115.15 100.00% 7168 11.6970 91.33% 9.82%
model.diffusion_model.blocks.48.attn.out_proj.weight 9444024.00 41.5092 30417.0000 1317.53 1802.15 100.00% 7168 12.0854 94.36% 8.18%
model.diffusion_model.blocks.49.attn.out_proj.weight 9850817.00 15.2147 78521.4766 1374.28 2588.65 100.00% 7168 11.8982 92.90% 5.23%
model.diffusion_model.blocks.0.attn.qkv_proj.weight 286.07 0.0008 2.2998 0.05 0.12 100.00% 5376 11.1911 90.31% 3.96%
model.diffusion_model.token_refiner.blocks.0.attn.qkv_proj.weight 3252.54 0.0482 6.8970 0.61 0.54 100.00% 5376 11.9982 96.82% 10.83%
model.diffusion_model.blocks.1.attn.qkv_proj.weight 1326.91 0.0004 85.6209 0.25 1.71 100.00% 5376 9.4494 76.25% 1.79%
model.diffusion_model.token_refiner.blocks.1.attn.qkv_proj.weight 2940.08 0.0257 62.3044 0.55 1.11 100.00% 5376 11.6592 94.08% 4.69%
model.diffusion_model.blocks.2.attn.qkv_proj.weight 216.18 0.0002 2.9871 0.04 0.10 100.00% 5376 10.8160 87.28% 5.58%
model.diffusion_model.blocks.3.attn.qkv_proj.weight 234.18 0.0004 2.5943 0.04 0.10 100.00% 5376 11.0209 88.93% 6.29%
model.diffusion_model.blocks.4.attn.qkv_proj.weight 311.64 0.0004 2.3351 0.06 0.12 100.00% 5376 11.0436 89.12% 7.22%
model.diffusion_model.blocks.5.attn.qkv_proj.weight 589.20 0.0017 2.7713 0.11 0.19 100.00% 5376 11.2837 91.05% 8.72%
model.diffusion_model.blocks.6.attn.qkv_proj.weight 400.53 0.0004 1.7937 0.07 0.12 100.00% 5376 11.4204 92.16% 9.10%
model.diffusion_model.blocks.7.attn.qkv_proj.weight 434.70 0.0008 1.8351 0.08 0.12 100.00% 5376 11.4201 92.15% 9.11%
model.diffusion_model.blocks.8.attn.qkv_proj.weight 410.30 0.0010 1.5757 0.08 0.11 100.00% 5376 11.5575 93.26% 9.52%
model.diffusion_model.blocks.9.attn.qkv_proj.weight 559.29 0.0010 4.7429 0.10 0.15 100.00% 5376 11.5636 93.31% 8.24%
model.diffusion_model.blocks.10.attn.qkv_proj.weight 498.32 0.0012 4.1042 0.09 0.15 100.00% 5376 11.4946 92.76% 8.13%
model.diffusion_model.blocks.11.attn.qkv_proj.weight 602.01 0.0021 6.7469 0.11 0.18 100.00% 5376 11.5629 93.31% 7.89%
model.diffusion_model.blocks.12.attn.qkv_proj.weight 613.14 0.0015 2.8188 0.11 0.16 100.00% 5376 11.6443 93.96% 8.63%
model.diffusion_model.blocks.13.attn.qkv_proj.weight 671.53 0.0008 2.5930 0.12 0.17 100.00% 5376 11.5998 93.61% 9.24%
model.diffusion_model.blocks.14.attn.qkv_proj.weight 832.93 0.0021 6.7109 0.15 0.21 100.00% 5376 11.6399 93.93% 9.08%
model.diffusion_model.blocks.15.attn.qkv_proj.weight 838.98 0.0037 2.9675 0.16 0.21 100.00% 5376 11.6228 93.79% 9.21%
model.diffusion_model.blocks.16.attn.qkv_proj.weight 891.95 0.0019 5.4387 0.17 0.29 100.00% 5376 11.3750 91.79% 7.51%
model.diffusion_model.blocks.17.attn.qkv_proj.weight 1022.04 0.0029 8.9663 0.19 0.33 100.00% 5376 11.4562 92.45% 7.07%
model.diffusion_model.blocks.18.attn.qkv_proj.weight 1281.73 0.0042 16.9813 0.24 0.45 100.00% 5376 11.4236 92.18% 6.51%
model.diffusion_model.blocks.19.attn.qkv_proj.weight 1702.52 0.0061 18.2810 0.32 0.53 100.00% 5376 11.5631 93.31% 6.53%
model.diffusion_model.blocks.20.attn.qkv_proj.weight 1645.29 0.0041 13.2556 0.31 0.57 100.00% 5376 11.2942 91.14% 7.14%
model.diffusion_model.blocks.21.attn.qkv_proj.weight 1680.26 0.0045 12.7208 0.31 0.59 100.00% 5376 11.3048 91.22% 6.49%
model.diffusion_model.blocks.22.attn.qkv_proj.weight 2035.36 0.0068 30.8893 0.38 0.75 100.00% 5376 11.3370 91.48% 6.62%
model.diffusion_model.blocks.23.attn.qkv_proj.weight 2848.44 0.0027 37.9520 0.53 1.02 100.00% 5376 11.3874 91.89% 6.55%
model.diffusion_model.blocks.24.attn.qkv_proj.weight 3437.13 0.0041 44.6511 0.64 1.19 100.00% 5376 11.4871 92.70% 5.97%
model.diffusion_model.blocks.25.attn.qkv_proj.weight 4038.83 0.0028 73.2934 0.75 1.53 100.00% 5376 11.4724 92.58% 5.00%
model.diffusion_model.blocks.26.attn.qkv_proj.weight 5487.58 0.0189 33.7092 1.02 1.45 100.00% 5376 11.7123 94.51% 7.03%
model.diffusion_model.blocks.27.attn.qkv_proj.weight 7926.66 0.0235 102.0933 1.47 2.24 100.00% 5376 11.7269 94.63% 7.20%
model.diffusion_model.blocks.28.attn.qkv_proj.weight 10070.42 0.0170 49.0417 1.87 2.27 100.00% 5376 11.8055 95.26% 7.59%
model.diffusion_model.blocks.29.attn.qkv_proj.weight 14702.81 0.0433 50.8014 2.73 3.10 100.00% 5376 11.8126 95.32% 8.89%
model.diffusion_model.blocks.30.attn.qkv_proj.weight 17100.02 0.0353 48.2687 3.18 3.28 100.00% 5376 11.8450 95.58% 10.01%
model.diffusion_model.blocks.31.attn.qkv_proj.weight 23879.76 0.0418 54.8158 4.44 4.50 100.00% 5376 11.8212 95.39% 10.60%
model.diffusion_model.blocks.32.attn.qkv_proj.weight 30758.80 0.0586 66.4006 5.72 5.52 100.00% 5376 11.8628 95.73% 10.40%
model.diffusion_model.blocks.33.attn.qkv_proj.weight 37598.50 0.0696 85.5452 6.99 7.01 100.00% 5376 11.8087 95.29% 10.83%
model.diffusion_model.blocks.34.attn.qkv_proj.weight 39814.91 0.1096 79.8723 7.41 6.81 100.00% 5376 11.8712 95.80% 11.07%
model.diffusion_model.blocks.35.attn.qkv_proj.weight 45450.11 0.1078 87.7067 8.45 8.11 100.00% 5376 11.8594 95.70% 10.06%
model.diffusion_model.blocks.36.attn.qkv_proj.weight 43477.61 0.1467 93.7542 8.09 7.37 100.00% 5376 11.9384 96.34% 10.19%
model.diffusion_model.blocks.37.attn.qkv_proj.weight 38891.90 0.1036 162.5714 7.23 7.09 100.00% 5376 11.9208 96.20% 8.65%
model.diffusion_model.blocks.38.attn.qkv_proj.weight 42470.65 0.1255 179.4583 7.90 7.32 100.00% 5376 12.0024 96.85% 8.95%
model.diffusion_model.blocks.39.attn.qkv_proj.weight 41799.04 0.1393 107.5959 7.78 6.43 100.00% 5376 12.0218 97.01% 9.47%
model.diffusion_model.blocks.40.attn.qkv_proj.weight 44062.07 0.1667 118.1432 8.20 6.93 100.00% 5376 11.9990 96.83% 9.28%
model.diffusion_model.blocks.41.attn.qkv_proj.weight 46795.35 0.2129 137.7783 8.70 7.52 100.00% 5376 12.0145 96.95% 8.98%
model.diffusion_model.blocks.42.attn.qkv_proj.weight 48308.53 0.2800 121.6657 8.99 8.05 100.00% 5376 11.9777 96.65% 9.11%
model.diffusion_model.blocks.43.attn.qkv_proj.weight 57143.88 0.3141 225.4996 10.63 9.94 100.00% 5376 12.0111 96.92% 8.30%
model.diffusion_model.blocks.44.attn.qkv_proj.weight 55940.87 0.3621 162.3853 10.41 9.64 100.00% 5376 12.0122 96.93% 8.11%
model.diffusion_model.blocks.45.attn.qkv_proj.weight 65976.30 0.3680 231.7699 12.27 12.29 100.00% 5376 11.9741 96.63% 8.02%
model.diffusion_model.blocks.46.attn.qkv_proj.weight 53983.43 0.1689 262.3690 10.04 11.06 100.00% 5376 11.9432 96.38% 6.68%
model.diffusion_model.blocks.47.attn.qkv_proj.weight 64306.93 0.2219 345.5754 11.96 13.71 100.00% 5376 11.9210 96.20% 6.53%
model.diffusion_model.blocks.48.attn.qkv_proj.weight 90387.17 0.6217 478.5000 16.81 19.92 100.00% 5376 11.9320 96.29% 5.52%
model.diffusion_model.blocks.49.attn.qkv_proj.weight 51064.44 0.0961 234.6013 9.50 10.87 100.00% 5376 11.9354 96.31% 5.80%
model.diffusion_model.blocks.0.mlp.fc1.weight 0.00 0.0000 0.0007 0.00 0.00 0.41% 5376 8.9639 72.33% 0.43%
model.diffusion_model.token_refiner.blocks.0.mlp.fc1.weight 0.09 0.0000 0.0002 0.00 0.00 44.49% 5376 11.5685 93.35% 11.94%
model.diffusion_model.blocks.1.mlp.fc1.weight 0.00 0.0000 0.0000 0.00 0.00 0.76% 5376 11.0307 89.01% 6.47%
model.diffusion_model.token_refiner.blocks.1.mlp.fc1.weight 0.08 0.0000 0.0005 0.00 0.00 33.05% 5376 11.1114 89.66% 8.85%
model.diffusion_model.blocks.2.mlp.fc1.weight 0.00 0.0000 0.0000 0.00 0.00 0.45% 5376 11.1341 89.85% 5.17%
model.diffusion_model.blocks.3.mlp.fc1.weight 0.00 0.0000 0.0001 0.00 0.00 0.47% 5376 11.3441 91.54% 6.21%
model.diffusion_model.blocks.4.mlp.fc1.weight 0.01 0.0000 0.0001 0.00 0.00 0.71% 5376 11.3887 91.90% 6.72%
model.diffusion_model.blocks.5.mlp.fc1.weight 0.01 0.0000 0.0001 0.00 0.00 0.63% 5376 11.5154 92.92% 6.10%
model.diffusion_model.blocks.6.mlp.fc1.weight 0.01 0.0000 0.0002 0.00 0.00 0.33% 5376 11.5930 93.55% 2.42%
model.diffusion_model.blocks.7.mlp.fc1.weight 0.01 0.0000 0.0004 0.00 0.00 0.95% 5376 11.3082 91.25% 1.86%
model.diffusion_model.blocks.8.mlp.fc1.weight 0.01 0.0000 0.0002 0.00 0.00 0.63% 5376 11.5922 93.54% 4.37%
model.diffusion_model.blocks.9.mlp.fc1.weight 0.01 0.0000 0.0001 0.00 0.00 0.91% 5376 11.5766 93.42% 5.34%
model.diffusion_model.blocks.10.mlp.fc1.weight 0.01 0.0000 0.0003 0.00 0.00 1.30% 5376 11.3515 91.60% 2.62%
model.diffusion_model.blocks.11.mlp.fc1.weight 0.01 0.0000 0.0001 0.00 0.00 1.62% 5376 11.4863 92.69% 5.73%
model.diffusion_model.blocks.12.mlp.fc1.weight 0.01 0.0000 0.0002 0.00 0.00 1.77% 5376 11.5485 93.19% 5.75%
model.diffusion_model.blocks.13.mlp.fc1.weight 0.01 0.0000 0.0002 0.00 0.00 2.40% 5376 11.4956 92.76% 4.58%
model.diffusion_model.blocks.14.mlp.fc1.weight 0.01 0.0000 0.0002 0.00 0.00 2.21% 5376 11.5758 93.41% 5.45%
model.diffusion_model.blocks.15.mlp.fc1.weight 0.01 0.0000 0.0002 0.00 0.00 2.92% 5376 11.5166 92.93% 5.78%
model.diffusion_model.blocks.16.mlp.fc1.weight 0.01 0.0000 0.0001 0.00 0.00 3.79% 5376 11.4926 92.74% 5.84%
model.diffusion_model.blocks.17.mlp.fc1.weight 0.02 0.0000 0.0002 0.00 0.00 4.07% 5376 11.4444 92.35% 5.47%
model.diffusion_model.blocks.18.mlp.fc1.weight 0.02 0.0000 0.0002 0.00 0.00 5.75% 5376 11.4429 92.34% 5.93%
model.diffusion_model.blocks.19.mlp.fc1.weight 0.02 0.0000 0.0002 0.00 0.00 5.80% 5376 11.5169 92.94% 5.92%
model.diffusion_model.blocks.20.mlp.fc1.weight 0.02 0.0000 0.0002 0.00 0.00 6.88% 5376 11.3411 91.52% 5.80%
model.diffusion_model.blocks.21.mlp.fc1.weight 0.02 0.0000 0.0002 0.00 0.00 6.88% 5376 11.3362 91.48% 5.17%
model.diffusion_model.blocks.22.mlp.fc1.weight 0.02 0.0000 0.0004 0.00 0.00 8.72% 5376 11.3691 91.74% 4.99%
model.diffusion_model.blocks.23.mlp.fc1.weight 0.03 0.0000 0.0006 0.00 0.00 13.06% 5376 11.2306 90.63% 4.50%
model.diffusion_model.blocks.24.mlp.fc1.weight 0.03 0.0000 0.0006 0.00 0.00 14.60% 5376 11.3207 91.35% 4.33%
model.diffusion_model.blocks.25.mlp.fc1.weight 0.04 0.0000 0.0013 0.00 0.00 14.96% 5376 11.3041 91.22% 3.03%
model.diffusion_model.blocks.26.mlp.fc1.weight 0.04 0.0000 0.0009 0.00 0.00 13.60% 5376 11.5444 93.16% 3.46%
model.diffusion_model.blocks.27.mlp.fc1.weight 0.04 0.0000 0.0010 0.00 0.00 17.43% 5376 11.7009 94.42% 3.42%
model.diffusion_model.blocks.28.mlp.fc1.weight 0.04 0.0000 0.0006 0.00 0.00 21.13% 5376 11.7918 95.15% 4.19%
model.diffusion_model.blocks.29.mlp.fc1.weight 0.05 0.0000 0.0004 0.00 0.00 24.29% 5376 11.8006 95.22% 6.03%
model.diffusion_model.blocks.30.mlp.fc1.weight 0.05 0.0000 0.0003 0.00 0.00 31.14% 5376 11.8569 95.68% 7.51%
model.diffusion_model.blocks.31.mlp.fc1.weight 0.05 0.0000 0.0003 0.00 0.00 36.74% 5376 11.8876 95.93% 9.23%
model.diffusion_model.blocks.32.mlp.fc1.weight 0.06 0.0000 0.0004 0.00 0.00 45.63% 5376 11.8809 95.87% 9.32%
model.diffusion_model.blocks.33.mlp.fc1.weight 0.07 0.0000 0.0003 0.00 0.00 52.38% 5376 11.9095 96.10% 10.12%
model.diffusion_model.blocks.34.mlp.fc1.weight 0.07 0.0000 0.0007 0.00 0.00 55.41% 5376 11.9268 96.24% 8.20%
model.diffusion_model.blocks.35.mlp.fc1.weight 0.07 0.0000 0.0009 0.00 0.00 54.00% 5376 11.9192 96.18% 6.05%
model.diffusion_model.blocks.36.mlp.fc1.weight 0.06 0.0000 0.0008 0.00 0.00 46.09% 5376 11.9947 96.79% 5.92%
model.diffusion_model.blocks.37.mlp.fc1.weight 0.05 0.0000 0.0004 0.00 0.00 35.40% 5376 12.0564 97.29% 8.54%
model.diffusion_model.blocks.38.mlp.fc1.weight 0.05 0.0000 0.0007 0.00 0.00 35.03% 5376 12.0459 97.20% 5.15%
model.diffusion_model.blocks.39.mlp.fc1.weight 0.06 0.0000 0.0008 0.00 0.00 41.31% 5376 12.0283 97.06% 5.04%
model.diffusion_model.blocks.40.mlp.fc1.weight 0.06 0.0000 0.0008 0.00 0.00 48.85% 5376 12.0419 97.17% 4.84%
model.diffusion_model.blocks.41.mlp.fc1.weight 0.07 0.0000 0.0007 0.00 0.00 54.33% 5376 12.0563 97.29% 6.68%
model.diffusion_model.blocks.42.mlp.fc1.weight 0.08 0.0000 0.0008 0.00 0.00 58.76% 5376 12.0190 96.99% 6.57%
model.diffusion_model.blocks.43.mlp.fc1.weight 0.08 0.0000 0.0002 0.00 0.00 62.05% 5376 12.0579 97.30% 10.19%
model.diffusion_model.blocks.44.mlp.fc1.weight 0.09 0.0000 0.0013 0.00 0.00 67.39% 5376 11.9975 96.81% 5.30%
model.diffusion_model.blocks.45.mlp.fc1.weight 0.09 0.0000 0.0002 0.00 0.00 71.91% 5376 12.0553 97.28% 9.30%
model.diffusion_model.blocks.46.mlp.fc1.weight 0.11 0.0000 0.0004 0.00 0.00 84.02% 5376 12.0534 97.27% 8.22%
model.diffusion_model.blocks.47.mlp.fc1.weight 0.15 0.0000 0.0005 0.00 0.00 93.77% 5376 12.0324 97.10% 8.00%
model.diffusion_model.blocks.48.mlp.fc1.weight 0.22 0.0000 0.0010 0.00 0.00 97.64% 5376 12.0129 96.94% 8.02%
model.diffusion_model.blocks.49.mlp.fc1.weight 0.27 0.0000 0.0023 0.00 0.00 99.76% 5376 12.0143 96.95% 6.88%
model.diffusion_model.blocks.0.mlp.fc2.weight 3.00 0.0000 0.1968 0.00 0.00 81.35% 14336 11.5709 83.80% 1.32%
model.diffusion_model.token_refiner.blocks.0.mlp.fc2.weight 7.29 0.0000 0.1284 0.00 0.00 53.12% 14336 10.8849 78.83% 4.06%
model.diffusion_model.blocks.1.mlp.fc2.weight 20.65 0.0000 0.2931 0.00 0.00 83.55% 14336 11.0241 79.84% 7.62%
model.diffusion_model.token_refiner.blocks.1.mlp.fc2.weight 36.70 0.0000 9.1506 0.00 0.13 24.80% 14336 3.4407 24.92% 0.06%
model.diffusion_model.blocks.2.mlp.fc2.weight 1.86 0.0000 0.0711 0.00 0.00 68.26% 14336 11.3337 82.08% 2.04%
model.diffusion_model.blocks.3.mlp.fc2.weight 1.99 0.0000 0.0507 0.00 0.00 70.65% 14336 11.1181 80.52% 1.86%
model.diffusion_model.blocks.4.mlp.fc2.weight 2.43 0.0000 0.2495 0.00 0.00 68.17% 14336 9.6622 69.98% 0.45%
model.diffusion_model.blocks.5.mlp.fc2.weight 1.85 0.0000 0.1314 0.00 0.00 67.01% 14336 10.2581 74.29% 0.80%
model.diffusion_model.blocks.6.mlp.fc2.weight 2.40 0.0000 0.4231 0.00 0.00 75.99% 14336 10.1143 73.25% 0.28%
model.diffusion_model.blocks.7.mlp.fc2.weight 2.78 0.0000 0.3254 0.00 0.00 69.96% 14336 9.5543 69.20% 0.53%
model.diffusion_model.blocks.8.mlp.fc2.weight 2.98 0.0000 0.3524 0.00 0.00 76.55% 14336 10.3920 75.26% 0.61%
model.diffusion_model.blocks.9.mlp.fc2.weight 2.42 0.0000 0.0394 0.00 0.00 75.26% 14336 11.5016 83.30% 3.68%
model.diffusion_model.blocks.10.mlp.fc2.weight 2.54 0.0000 0.0572 0.00 0.00 72.17% 14336 11.0734 80.20% 2.76%
model.diffusion_model.blocks.11.mlp.fc2.weight 4.27 0.0000 0.0927 0.00 0.00 82.55% 14336 11.3785 82.41% 2.95%
model.diffusion_model.blocks.12.mlp.fc2.weight 4.13 0.0000 0.0654 0.00 0.00 81.09% 14336 11.3725 82.37% 3.25%
model.diffusion_model.blocks.13.mlp.fc2.weight 4.44 0.0000 0.1785 0.00 0.00 82.97% 14336 11.2698 81.62% 2.33%
model.diffusion_model.blocks.14.mlp.fc2.weight 4.88 0.0000 0.0640 0.00 0.00 85.13% 14336 11.4058 82.61% 3.15%
model.diffusion_model.blocks.15.mlp.fc2.weight 6.64 0.0000 0.1982 0.00 0.00 89.17% 14336 11.3924 82.51% 2.65%
model.diffusion_model.blocks.16.mlp.fc2.weight 6.76 0.0000 0.0954 0.00 0.00 90.32% 14336 11.5294 83.50% 3.56%
model.diffusion_model.blocks.17.mlp.fc2.weight 11.36 0.0000 0.2421 0.00 0.00 95.55% 14336 11.2727 81.64% 1.98%
model.diffusion_model.blocks.18.mlp.fc2.weight 24.56 0.0000 2.9967 0.00 0.03 97.37% 14336 9.6005 69.53% 0.33%
model.diffusion_model.blocks.19.mlp.fc2.weight 18.84 0.0000 1.7301 0.00 0.02 97.31% 14336 11.0827 80.27% 0.93%
model.diffusion_model.blocks.20.mlp.fc2.weight 19.20 0.0000 0.8827 0.00 0.01 97.68% 14336 11.2916 81.78% 1.39%
model.diffusion_model.blocks.21.mlp.fc2.weight 21.54 0.0000 0.9157 0.00 0.01 97.79% 14336 11.2451 81.44% 1.26%
model.diffusion_model.blocks.22.mlp.fc2.weight 32.93 0.0000 3.1965 0.00 0.03 98.66% 14336 10.5730 76.57% 0.50%
model.diffusion_model.blocks.23.mlp.fc2.weight 123.01 0.0000 0.9572 0.01 0.02 99.98% 14336 12.6889 91.90% 4.28%
model.diffusion_model.blocks.24.mlp.fc2.weight 148.64 0.0000 0.9917 0.01 0.02 100.00% 14336 12.8901 93.36% 5.86%
model.diffusion_model.blocks.25.mlp.fc2.weight 241.09 0.0000 1.6987 0.02 0.04 100.00% 14336 12.8178 92.83% 4.75%
model.diffusion_model.blocks.26.mlp.fc2.weight 228.81 0.0000 3.0080 0.02 0.04 99.99% 14336 12.7928 92.65% 3.64%
model.diffusion_model.blocks.27.mlp.fc2.weight 264.85 0.0000 3.1997 0.02 0.05 100.00% 14336 12.7515 92.35% 3.19%
model.diffusion_model.blocks.28.mlp.fc2.weight 375.01 0.0001 3.7782 0.03 0.07 100.00% 14336 12.7525 92.36% 3.47%
model.diffusion_model.blocks.29.mlp.fc2.weight 388.76 0.0001 2.4725 0.03 0.07 100.00% 14336 12.6647 91.72% 3.83%
model.diffusion_model.blocks.30.mlp.fc2.weight 481.15 0.0001 2.5943 0.03 0.07 100.00% 14336 12.8256 92.89% 5.12%
model.diffusion_model.blocks.31.mlp.fc2.weight 584.35 0.0000 3.2965 0.04 0.08 100.00% 14336 12.8572 93.12% 5.04%
model.diffusion_model.blocks.32.mlp.fc2.weight 814.95 0.0000 3.2539 0.06 0.08 100.00% 14336 13.0729 94.68% 8.66%
model.diffusion_model.blocks.33.mlp.fc2.weight 1040.49 0.0000 1.4147 0.07 0.08 100.00% 14336 13.1948 95.56% 11.13%
model.diffusion_model.blocks.34.mlp.fc2.weight 1446.80 0.0001 2.4119 0.10 0.10 100.00% 14336 13.2855 96.22% 10.93%
model.diffusion_model.blocks.35.mlp.fc2.weight 1854.77 0.0001 2.0814 0.13 0.12 100.00% 14336 13.2893 96.25% 11.83%
model.diffusion_model.blocks.36.mlp.fc2.weight 1461.90 0.0001 29.0783 0.10 0.27 100.00% 14336 13.1183 95.01% 2.23%
model.diffusion_model.blocks.37.mlp.fc2.weight 1044.24 0.0001 1.1472 0.07 0.08 100.00% 14336 13.1992 95.60% 11.24%
model.diffusion_model.blocks.38.mlp.fc2.weight 839.62 0.0001 5.5590 0.06 0.08 100.00% 14336 13.2490 95.96% 8.00%
model.diffusion_model.blocks.39.mlp.fc2.weight 1233.28 0.0002 216.8184 0.09 1.81 100.00% 14336 11.6166 84.13% 0.02%
model.diffusion_model.blocks.40.mlp.fc2.weight 1266.00 0.0002 2.7289 0.09 0.09 100.00% 14336 13.3008 96.33% 11.27%
model.diffusion_model.blocks.41.mlp.fc2.weight 1531.04 0.0001 7.8280 0.11 0.12 100.00% 14336 13.2899 96.25% 9.74%
model.diffusion_model.blocks.42.mlp.fc2.weight 2453.81 0.0002 13.9164 0.17 0.22 100.00% 14336 13.1794 95.45% 8.35%
model.diffusion_model.blocks.43.mlp.fc2.weight 2673.96 0.0001 18.7274 0.19 0.27 100.00% 14336 13.1282 95.08% 7.74%
model.diffusion_model.blocks.44.mlp.fc2.weight 3178.08 0.0001 49.0114 0.22 0.53 100.00% 14336 13.0334 94.39% 2.91%
model.diffusion_model.blocks.45.mlp.fc2.weight 2925.45 0.0001 253.6084 0.20 2.13 100.00% 14336 12.5030 90.55% 0.07%
model.diffusion_model.blocks.46.mlp.fc2.weight 3621.52 0.0001 14.9919 0.25 0.35 100.00% 14336 13.1587 95.30% 6.26%
model.diffusion_model.blocks.47.mlp.fc2.weight 6046.40 0.0001 22.3388 0.42 0.57 100.00% 14336 13.1360 95.14% 7.25%
model.diffusion_model.blocks.48.mlp.fc2.weight 21895.48 0.0001 951.2448 1.53 8.76 100.00% 14336 12.5303 90.75% 0.86%
model.diffusion_model.blocks.49.mlp.fc2.weight 39558.48 0.0001 324.8035 2.76 7.76 100.00% 14336 12.1881 88.27% 4.39%
model.diffusion_model.audio_patch_proj.weight 36.06 0.9065 1.3968 1.13 0.11 100.00% 32 4.9931 99.86% 15.62%
model.diffusion_model.condition_proj.weight 3321343.25 0.7390 3215828.0000 648.70 44941.20 100.00% 5120 0.4035 3.27% 0.02%
model.diffusion_model.final_layer.adaln_proj.linear.weight 0.14 0.0000 0.1245 0.02 0.04 50.00% 8 0.6728 22.43% 12.50%
model.diffusion_model.final_layer.audio_out.weight 58.63 0.0000 0.9794 0.01 0.04 34.47% 5376 9.3019 75.06% 6.06%
model.diffusion_model.final_layer.video_out.weight 13.19 0.0000 0.1096 0.00 0.01 33.35% 5376 9.8022 79.10% 9.26%
model.diffusion_model.video_patch_proj.weight 81.73 0.7328 1.2404 0.85 0.11 100.00% 96 6.5747 99.84% 13.54%
This is actually very interesting though! What stands out to me:
adaln_proj.linear.weight must be full precision.
fc2 mlp layers 39, 45, 47, 48, and 49 have crazy statistics. These should be high-precision too.
The mlp.fc1 weights are way less active than fc2, and have fewer outliers. It seems like they could stand ~4 bit quantization better than the others?
H3 seems to be "bottom heavy" too, where most tensors get more sensitive in later layers, especially layers past ~33. Layers 0 and 1 seem to be more sensitive, too.
And there are a few outliers that looks like they could use extra precision, if your quantization tool will allow such a thing. For Example, mlp.fc2 layers: 49-42, 39, 36 have crazy high max values, but 0, 1, 18 and 22 are higher than surrounding layers. 27.attn.qkv_proj is an outlier, too.