File size: 30,551 Bytes
41d4286
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
|     | layer_id                                                     | layer_type    | param_type   | shape        |   nparam |      nnz |   sparsity |
|----:|:-------------------------------------------------------------|:--------------|:-------------|:-------------|---------:|---------:|-----------:|
|   0 | nncf_module.bert.embeddings.word_embeddings                  | NNCFEmbedding | weight       | [30522, 768] | 23440896 | 23440896 |   0        |
|   1 | nncf_module.bert.embeddings.position_embeddings              | NNCFEmbedding | weight       | [512, 768]   |   393216 |   393216 |   0        |
|   2 | nncf_module.bert.embeddings.token_type_embeddings            | NNCFEmbedding | weight       | [2, 768]     |     1536 |     1536 |   0        |
|   3 | nncf_module.bert.embeddings.LayerNorm                        | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|   4 | nncf_module.bert.embeddings.LayerNorm                        | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|   5 | nncf_module.bert.encoder.layer.0.attention.self.query        | NNCFLinear    | weight       | [320, 768]   |   245760 |    93583 |   0.61921  |
|   6 | nncf_module.bert.encoder.layer.0.attention.self.query        | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
|   7 | nncf_module.bert.encoder.layer.0.attention.self.key          | NNCFLinear    | weight       | [320, 768]   |   245760 |    98270 |   0.600138 |
|   8 | nncf_module.bert.encoder.layer.0.attention.self.key          | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
|   9 | nncf_module.bert.encoder.layer.0.attention.self.value        | NNCFLinear    | weight       | [320, 768]   |   245760 |   113605 |   0.53774  |
|  10 | nncf_module.bert.encoder.layer.0.attention.self.value        | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
|  11 | nncf_module.bert.encoder.layer.0.attention.output.dense      | NNCFLinear    | weight       | [768, 320]   |   245760 |   117208 |   0.523079 |
|  12 | nncf_module.bert.encoder.layer.0.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  13 | nncf_module.bert.encoder.layer.0.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  14 | nncf_module.bert.encoder.layer.0.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  15 | nncf_module.bert.encoder.layer.0.intermediate.dense          | NNCFLinear    | weight       | [185, 768]   |   142080 |    97073 |   0.316772 |
|  16 | nncf_module.bert.encoder.layer.0.intermediate.dense          | NNCFLinear    | bias         | [185]        |      185 |      185 |   0        |
|  17 | nncf_module.bert.encoder.layer.0.output.dense                | NNCFLinear    | weight       | [768, 185]   |   142080 |    94692 |   0.33353  |
|  18 | nncf_module.bert.encoder.layer.0.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  19 | nncf_module.bert.encoder.layer.0.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  20 | nncf_module.bert.encoder.layer.0.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  21 | nncf_module.bert.encoder.layer.1.attention.self.query        | NNCFLinear    | weight       | [320, 768]   |   245760 |   118436 |   0.518083 |
|  22 | nncf_module.bert.encoder.layer.1.attention.self.query        | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
|  23 | nncf_module.bert.encoder.layer.1.attention.self.key          | NNCFLinear    | weight       | [320, 768]   |   245760 |   118116 |   0.519385 |
|  24 | nncf_module.bert.encoder.layer.1.attention.self.key          | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
|  25 | nncf_module.bert.encoder.layer.1.attention.self.value        | NNCFLinear    | weight       | [320, 768]   |   245760 |   107511 |   0.562537 |
|  26 | nncf_module.bert.encoder.layer.1.attention.self.value        | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
|  27 | nncf_module.bert.encoder.layer.1.attention.output.dense      | NNCFLinear    | weight       | [768, 320]   |   245760 |   111189 |   0.547571 |
|  28 | nncf_module.bert.encoder.layer.1.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  29 | nncf_module.bert.encoder.layer.1.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  30 | nncf_module.bert.encoder.layer.1.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  31 | nncf_module.bert.encoder.layer.1.intermediate.dense          | NNCFLinear    | weight       | [315, 768]   |   241920 |   148783 |   0.384991 |
|  32 | nncf_module.bert.encoder.layer.1.intermediate.dense          | NNCFLinear    | bias         | [315]        |      315 |      315 |   0        |
|  33 | nncf_module.bert.encoder.layer.1.output.dense                | NNCFLinear    | weight       | [768, 315]   |   241920 |   143166 |   0.408209 |
|  34 | nncf_module.bert.encoder.layer.1.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  35 | nncf_module.bert.encoder.layer.1.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  36 | nncf_module.bert.encoder.layer.1.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  37 | nncf_module.bert.encoder.layer.2.attention.self.query        | NNCFLinear    | weight       | [576, 768]   |   442368 |   162735 |   0.632128 |
|  38 | nncf_module.bert.encoder.layer.2.attention.self.query        | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  39 | nncf_module.bert.encoder.layer.2.attention.self.key          | NNCFLinear    | weight       | [576, 768]   |   442368 |   164795 |   0.627471 |
|  40 | nncf_module.bert.encoder.layer.2.attention.self.key          | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  41 | nncf_module.bert.encoder.layer.2.attention.self.value        | NNCFLinear    | weight       | [576, 768]   |   442368 |   135670 |   0.69331  |
|  42 | nncf_module.bert.encoder.layer.2.attention.self.value        | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  43 | nncf_module.bert.encoder.layer.2.attention.output.dense      | NNCFLinear    | weight       | [768, 576]   |   442368 |   138445 |   0.687037 |
|  44 | nncf_module.bert.encoder.layer.2.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  45 | nncf_module.bert.encoder.layer.2.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  46 | nncf_module.bert.encoder.layer.2.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  47 | nncf_module.bert.encoder.layer.2.intermediate.dense          | NNCFLinear    | weight       | [339, 768]   |   260352 |   154035 |   0.408359 |
|  48 | nncf_module.bert.encoder.layer.2.intermediate.dense          | NNCFLinear    | bias         | [339]        |      339 |      339 |   0        |
|  49 | nncf_module.bert.encoder.layer.2.output.dense                | NNCFLinear    | weight       | [768, 339]   |   260352 |   150816 |   0.420723 |
|  50 | nncf_module.bert.encoder.layer.2.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  51 | nncf_module.bert.encoder.layer.2.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  52 | nncf_module.bert.encoder.layer.2.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  53 | nncf_module.bert.encoder.layer.3.attention.self.query        | NNCFLinear    | weight       | [576, 768]   |   442368 |   170623 |   0.614296 |
|  54 | nncf_module.bert.encoder.layer.3.attention.self.query        | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  55 | nncf_module.bert.encoder.layer.3.attention.self.key          | NNCFLinear    | weight       | [576, 768]   |   442368 |   178401 |   0.596714 |
|  56 | nncf_module.bert.encoder.layer.3.attention.self.key          | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  57 | nncf_module.bert.encoder.layer.3.attention.self.value        | NNCFLinear    | weight       | [576, 768]   |   442368 |   171905 |   0.611398 |
|  58 | nncf_module.bert.encoder.layer.3.attention.self.value        | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  59 | nncf_module.bert.encoder.layer.3.attention.output.dense      | NNCFLinear    | weight       | [768, 576]   |   442368 |   169172 |   0.617576 |
|  60 | nncf_module.bert.encoder.layer.3.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  61 | nncf_module.bert.encoder.layer.3.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  62 | nncf_module.bert.encoder.layer.3.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  63 | nncf_module.bert.encoder.layer.3.intermediate.dense          | NNCFLinear    | weight       | [368, 768]   |   282624 |   163163 |   0.422685 |
|  64 | nncf_module.bert.encoder.layer.3.intermediate.dense          | NNCFLinear    | bias         | [368]        |      368 |      368 |   0        |
|  65 | nncf_module.bert.encoder.layer.3.output.dense                | NNCFLinear    | weight       | [768, 368]   |   282624 |   157506 |   0.442701 |
|  66 | nncf_module.bert.encoder.layer.3.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  67 | nncf_module.bert.encoder.layer.3.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  68 | nncf_module.bert.encoder.layer.3.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  69 | nncf_module.bert.encoder.layer.4.attention.self.query        | NNCFLinear    | weight       | [576, 768]   |   442368 |   175772 |   0.602657 |
|  70 | nncf_module.bert.encoder.layer.4.attention.self.query        | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  71 | nncf_module.bert.encoder.layer.4.attention.self.key          | NNCFLinear    | weight       | [576, 768]   |   442368 |   177087 |   0.599684 |
|  72 | nncf_module.bert.encoder.layer.4.attention.self.key          | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  73 | nncf_module.bert.encoder.layer.4.attention.self.value        | NNCFLinear    | weight       | [576, 768]   |   442368 |   163996 |   0.629277 |
|  74 | nncf_module.bert.encoder.layer.4.attention.self.value        | NNCFLinear    | bias         | [576]        |      576 |      576 |   0        |
|  75 | nncf_module.bert.encoder.layer.4.attention.output.dense      | NNCFLinear    | weight       | [768, 576]   |   442368 |   159335 |   0.639813 |
|  76 | nncf_module.bert.encoder.layer.4.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  77 | nncf_module.bert.encoder.layer.4.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  78 | nncf_module.bert.encoder.layer.4.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  79 | nncf_module.bert.encoder.layer.4.intermediate.dense          | NNCFLinear    | weight       | [386, 768]   |   296448 |   167726 |   0.434214 |
|  80 | nncf_module.bert.encoder.layer.4.intermediate.dense          | NNCFLinear    | bias         | [386]        |      386 |      386 |   0        |
|  81 | nncf_module.bert.encoder.layer.4.output.dense                | NNCFLinear    | weight       | [768, 386]   |   296448 |   159865 |   0.460732 |
|  82 | nncf_module.bert.encoder.layer.4.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  83 | nncf_module.bert.encoder.layer.4.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  84 | nncf_module.bert.encoder.layer.4.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  85 | nncf_module.bert.encoder.layer.5.attention.self.query        | NNCFLinear    | weight       | [384, 768]   |   294912 |   114186 |   0.612813 |
|  86 | nncf_module.bert.encoder.layer.5.attention.self.query        | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
|  87 | nncf_module.bert.encoder.layer.5.attention.self.key          | NNCFLinear    | weight       | [384, 768]   |   294912 |   132782 |   0.549757 |
|  88 | nncf_module.bert.encoder.layer.5.attention.self.key          | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
|  89 | nncf_module.bert.encoder.layer.5.attention.self.value        | NNCFLinear    | weight       | [384, 768]   |   294912 |   134830 |   0.542813 |
|  90 | nncf_module.bert.encoder.layer.5.attention.self.value        | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
|  91 | nncf_module.bert.encoder.layer.5.attention.output.dense      | NNCFLinear    | weight       | [768, 384]   |   294912 |   131941 |   0.552609 |
|  92 | nncf_module.bert.encoder.layer.5.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  93 | nncf_module.bert.encoder.layer.5.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
|  94 | nncf_module.bert.encoder.layer.5.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
|  95 | nncf_module.bert.encoder.layer.5.intermediate.dense          | NNCFLinear    | weight       | [336, 768]   |   258048 |   153916 |   0.403537 |
|  96 | nncf_module.bert.encoder.layer.5.intermediate.dense          | NNCFLinear    | bias         | [336]        |      336 |      336 |   0        |
|  97 | nncf_module.bert.encoder.layer.5.output.dense                | NNCFLinear    | weight       | [768, 336]   |   258048 |   145794 |   0.435012 |
|  98 | nncf_module.bert.encoder.layer.5.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
|  99 | nncf_module.bert.encoder.layer.5.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 100 | nncf_module.bert.encoder.layer.5.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 101 | nncf_module.bert.encoder.layer.6.attention.self.query        | NNCFLinear    | weight       | [448, 768]   |   344064 |   131878 |   0.616705 |
| 102 | nncf_module.bert.encoder.layer.6.attention.self.query        | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 103 | nncf_module.bert.encoder.layer.6.attention.self.key          | NNCFLinear    | weight       | [448, 768]   |   344064 |   144502 |   0.580014 |
| 104 | nncf_module.bert.encoder.layer.6.attention.self.key          | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 105 | nncf_module.bert.encoder.layer.6.attention.self.value        | NNCFLinear    | weight       | [448, 768]   |   344064 |   130911 |   0.619516 |
| 106 | nncf_module.bert.encoder.layer.6.attention.self.value        | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 107 | nncf_module.bert.encoder.layer.6.attention.output.dense      | NNCFLinear    | weight       | [768, 448]   |   344064 |   125928 |   0.633998 |
| 108 | nncf_module.bert.encoder.layer.6.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 109 | nncf_module.bert.encoder.layer.6.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 110 | nncf_module.bert.encoder.layer.6.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 111 | nncf_module.bert.encoder.layer.6.intermediate.dense          | NNCFLinear    | weight       | [280, 768]   |   215040 |   135283 |   0.370894 |
| 112 | nncf_module.bert.encoder.layer.6.intermediate.dense          | NNCFLinear    | bias         | [280]        |      280 |      280 |   0        |
| 113 | nncf_module.bert.encoder.layer.6.output.dense                | NNCFLinear    | weight       | [768, 280]   |   215040 |   131619 |   0.387932 |
| 114 | nncf_module.bert.encoder.layer.6.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 115 | nncf_module.bert.encoder.layer.6.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 116 | nncf_module.bert.encoder.layer.6.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 117 | nncf_module.bert.encoder.layer.7.attention.self.query        | NNCFLinear    | weight       | [448, 768]   |   344064 |   132120 |   0.616002 |
| 118 | nncf_module.bert.encoder.layer.7.attention.self.query        | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 119 | nncf_module.bert.encoder.layer.7.attention.self.key          | NNCFLinear    | weight       | [448, 768]   |   344064 |   152223 |   0.557574 |
| 120 | nncf_module.bert.encoder.layer.7.attention.self.key          | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 121 | nncf_module.bert.encoder.layer.7.attention.self.value        | NNCFLinear    | weight       | [448, 768]   |   344064 |   141066 |   0.590001 |
| 122 | nncf_module.bert.encoder.layer.7.attention.self.value        | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 123 | nncf_module.bert.encoder.layer.7.attention.output.dense      | NNCFLinear    | weight       | [768, 448]   |   344064 |   135662 |   0.605707 |
| 124 | nncf_module.bert.encoder.layer.7.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 125 | nncf_module.bert.encoder.layer.7.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 126 | nncf_module.bert.encoder.layer.7.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 127 | nncf_module.bert.encoder.layer.7.intermediate.dense          | NNCFLinear    | weight       | [211, 768]   |   162048 |   109590 |   0.323719 |
| 128 | nncf_module.bert.encoder.layer.7.intermediate.dense          | NNCFLinear    | bias         | [211]        |      211 |      211 |   0        |
| 129 | nncf_module.bert.encoder.layer.7.output.dense                | NNCFLinear    | weight       | [768, 211]   |   162048 |   107335 |   0.337635 |
| 130 | nncf_module.bert.encoder.layer.7.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 131 | nncf_module.bert.encoder.layer.7.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 132 | nncf_module.bert.encoder.layer.7.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 133 | nncf_module.bert.encoder.layer.8.attention.self.query        | NNCFLinear    | weight       | [448, 768]   |   344064 |   129148 |   0.62464  |
| 134 | nncf_module.bert.encoder.layer.8.attention.self.query        | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 135 | nncf_module.bert.encoder.layer.8.attention.self.key          | NNCFLinear    | weight       | [448, 768]   |   344064 |   130060 |   0.621989 |
| 136 | nncf_module.bert.encoder.layer.8.attention.self.key          | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 137 | nncf_module.bert.encoder.layer.8.attention.self.value        | NNCFLinear    | weight       | [448, 768]   |   344064 |   108162 |   0.685634 |
| 138 | nncf_module.bert.encoder.layer.8.attention.self.value        | NNCFLinear    | bias         | [448]        |      448 |      448 |   0        |
| 139 | nncf_module.bert.encoder.layer.8.attention.output.dense      | NNCFLinear    | weight       | [768, 448]   |   344064 |   103447 |   0.699338 |
| 140 | nncf_module.bert.encoder.layer.8.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 141 | nncf_module.bert.encoder.layer.8.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 142 | nncf_module.bert.encoder.layer.8.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 143 | nncf_module.bert.encoder.layer.8.intermediate.dense          | NNCFLinear    | weight       | [108, 768]   |    82944 |    63275 |   0.237136 |
| 144 | nncf_module.bert.encoder.layer.8.intermediate.dense          | NNCFLinear    | bias         | [108]        |      108 |      108 |   0        |
| 145 | nncf_module.bert.encoder.layer.8.output.dense                | NNCFLinear    | weight       | [768, 108]   |    82944 |    62725 |   0.243767 |
| 146 | nncf_module.bert.encoder.layer.8.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 147 | nncf_module.bert.encoder.layer.8.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 148 | nncf_module.bert.encoder.layer.8.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 149 | nncf_module.bert.encoder.layer.9.attention.self.query        | NNCFLinear    | weight       | [320, 768]   |   245760 |   107145 |   0.564026 |
| 150 | nncf_module.bert.encoder.layer.9.attention.self.query        | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
| 151 | nncf_module.bert.encoder.layer.9.attention.self.key          | NNCFLinear    | weight       | [320, 768]   |   245760 |   101811 |   0.58573  |
| 152 | nncf_module.bert.encoder.layer.9.attention.self.key          | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
| 153 | nncf_module.bert.encoder.layer.9.attention.self.value        | NNCFLinear    | weight       | [320, 768]   |   245760 |    52182 |   0.787671 |
| 154 | nncf_module.bert.encoder.layer.9.attention.self.value        | NNCFLinear    | bias         | [320]        |      320 |      320 |   0        |
| 155 | nncf_module.bert.encoder.layer.9.attention.output.dense      | NNCFLinear    | weight       | [768, 320]   |   245760 |    53210 |   0.783488 |
| 156 | nncf_module.bert.encoder.layer.9.attention.output.dense      | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 157 | nncf_module.bert.encoder.layer.9.attention.output.LayerNorm  | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 158 | nncf_module.bert.encoder.layer.9.attention.output.LayerNorm  | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 159 | nncf_module.bert.encoder.layer.9.intermediate.dense          | NNCFLinear    | weight       | [53, 768]    |    40704 |    33461 |   0.177943 |
| 160 | nncf_module.bert.encoder.layer.9.intermediate.dense          | NNCFLinear    | bias         | [53]         |       53 |       53 |   0        |
| 161 | nncf_module.bert.encoder.layer.9.output.dense                | NNCFLinear    | weight       | [768, 53]    |    40704 |    32551 |   0.2003   |
| 162 | nncf_module.bert.encoder.layer.9.output.dense                | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 163 | nncf_module.bert.encoder.layer.9.output.LayerNorm            | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 164 | nncf_module.bert.encoder.layer.9.output.LayerNorm            | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 165 | nncf_module.bert.encoder.layer.10.attention.self.query       | NNCFLinear    | weight       | [384, 768]   |   294912 |   112430 |   0.618768 |
| 166 | nncf_module.bert.encoder.layer.10.attention.self.query       | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
| 167 | nncf_module.bert.encoder.layer.10.attention.self.key         | NNCFLinear    | weight       | [384, 768]   |   294912 |   109594 |   0.628384 |
| 168 | nncf_module.bert.encoder.layer.10.attention.self.key         | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
| 169 | nncf_module.bert.encoder.layer.10.attention.self.value       | NNCFLinear    | weight       | [384, 768]   |   294912 |    61774 |   0.790534 |
| 170 | nncf_module.bert.encoder.layer.10.attention.self.value       | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
| 171 | nncf_module.bert.encoder.layer.10.attention.output.dense     | NNCFLinear    | weight       | [768, 384]   |   294912 |    64183 |   0.782366 |
| 172 | nncf_module.bert.encoder.layer.10.attention.output.dense     | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 173 | nncf_module.bert.encoder.layer.10.attention.output.LayerNorm | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 174 | nncf_module.bert.encoder.layer.10.attention.output.LayerNorm | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 175 | nncf_module.bert.encoder.layer.10.intermediate.dense         | NNCFLinear    | weight       | [86, 768]    |    66048 |    50455 |   0.236086 |
| 176 | nncf_module.bert.encoder.layer.10.intermediate.dense         | NNCFLinear    | bias         | [86]         |       86 |       86 |   0        |
| 177 | nncf_module.bert.encoder.layer.10.output.dense               | NNCFLinear    | weight       | [768, 86]    |    66048 |    49741 |   0.246896 |
| 178 | nncf_module.bert.encoder.layer.10.output.dense               | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 179 | nncf_module.bert.encoder.layer.10.output.LayerNorm           | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 180 | nncf_module.bert.encoder.layer.10.output.LayerNorm           | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 181 | nncf_module.bert.encoder.layer.11.attention.self.query       | NNCFLinear    | weight       | [384, 768]   |   294912 |    88129 |   0.701168 |
| 182 | nncf_module.bert.encoder.layer.11.attention.self.query       | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
| 183 | nncf_module.bert.encoder.layer.11.attention.self.key         | NNCFLinear    | weight       | [384, 768]   |   294912 |    85288 |   0.710802 |
| 184 | nncf_module.bert.encoder.layer.11.attention.self.key         | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
| 185 | nncf_module.bert.encoder.layer.11.attention.self.value       | NNCFLinear    | weight       | [384, 768]   |   294912 |    47258 |   0.839756 |
| 186 | nncf_module.bert.encoder.layer.11.attention.self.value       | NNCFLinear    | bias         | [384]        |      384 |      384 |   0        |
| 187 | nncf_module.bert.encoder.layer.11.attention.output.dense     | NNCFLinear    | weight       | [768, 384]   |   294912 |    49311 |   0.832794 |
| 188 | nncf_module.bert.encoder.layer.11.attention.output.dense     | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 189 | nncf_module.bert.encoder.layer.11.attention.output.LayerNorm | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 190 | nncf_module.bert.encoder.layer.11.attention.output.LayerNorm | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 191 | nncf_module.bert.encoder.layer.11.intermediate.dense         | NNCFLinear    | weight       | [105, 768]   |    80640 |    62254 |   0.228001 |
| 192 | nncf_module.bert.encoder.layer.11.intermediate.dense         | NNCFLinear    | bias         | [105]        |      105 |      105 |   0        |
| 193 | nncf_module.bert.encoder.layer.11.output.dense               | NNCFLinear    | weight       | [768, 105]   |    80640 |    61669 |   0.235255 |
| 194 | nncf_module.bert.encoder.layer.11.output.dense               | NNCFLinear    | bias         | [768]        |      768 |      768 |   0        |
| 195 | nncf_module.bert.encoder.layer.11.output.LayerNorm           | LayerNorm     | weight       | [768]        |      768 |      768 |   0        |
| 196 | nncf_module.bert.encoder.layer.11.output.LayerNorm           | LayerNorm     | bias         | [768]        |      768 |      768 |   0        |
| 197 | nncf_module.qa_outputs                                       | NNCFLinear    | weight       | [2, 768]     |     1536 |     1536 |   0        |
| 198 | nncf_module.qa_outputs                                       | NNCFLinear    | bias         | [2]          |        2 |        2 |   0        |