Hugging Face's logo

legraphista
/
Yuan2-M32-hf-IMat-GGUF

Text Generation
GGUF
quantized
GGUF
imatrix
quantization
imat
static
16bit
8bit
6bit
5bit
4bit
3bit
2bit
1bit
Model card Files Community
1
legraphista commited on
Commit
2107810
•
1 Parent(s): 4c9edb3

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +190 -0
imatrix.log ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ warning: not compiled with GPU offload support, --n-gpu-layers option will be ignored
2
+ warning: see main README.md for information on enabling GPU BLAS support
3
+ main: build = 8 (589f21d)
4
+ main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
5
+ main: seed = 1717069408
6
+ llama_model_loader: loaded meta data with 24 key-value pairs and 1875 tensors from Yuan2-M32-hf-IMat-GGUF/Yuan2-M32-hf.gguf (version GGUF V3 (latest))
7
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
8
+ llama_model_loader: - kv 0: general.architecture str = yuan2_moe
9
+ llama_model_loader: - kv 1: general.name str = yuan2_moe
10
+ llama_model_loader: - kv 2: yuan2_moe.context_length u32 = 4096
11
+ llama_model_loader: - kv 3: yuan2_moe.embedding_length u32 = 2048
12
+ llama_model_loader: - kv 4: yuan2_moe.block_count u32 = 24
13
+ llama_model_loader: - kv 5: yuan2_moe.feed_forward_length u32 = 8192
14
+ llama_model_loader: - kv 6: yuan2_moe.rope.dimension_count u32 = 128
15
+ llama_model_loader: - kv 7: yuan2_moe.attention.head_count u32 = 16
16
+ llama_model_loader: - kv 8: yuan2_moe.attention.head_count_kv u32 = 16
17
+ llama_model_loader: - kv 9: yuan2_moe.expert_count u32 = 32
18
+ llama_model_loader: - kv 10: yuan2_moe.expert_used_count u32 = 2
19
+ llama_model_loader: - kv 11: yuan2_moe.attention.layer_norm_rms_epsilon f32 = 0.000001
20
+ llama_model_loader: - kv 12: general.file_type u32 = 1
21
+ llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
22
+ llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,135040] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
23
+ llama_model_loader: - kv 15: tokenizer.ggml.scores arr[f32,135040] = [0.000000, 0.000000, 0.000000, 0.0000...
24
+ llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,135040] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
25
+ llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 77185
26
+ llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 77185
27
+ llama_model_loader: - kv 19: tokenizer.ggml.seperator_token_id u32 = 77185
28
+ llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 77185
29
+ llama_model_loader: - kv 21: tokenizer.ggml.mask_token_id u32 = 77185
30
+ llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = false
31
+ llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
32
+ llama_model_loader: - type f32: 121 tensors
33
+ llama_model_loader: - type f16: 1754 tensors
34
+ llm_load_vocab: mismatch in special tokens definition ( 2729/135040 vs 346/135040 ).
35
+ llm_load_print_meta: format = GGUF V3 (latest)
36
+ llm_load_print_meta: arch = yuan2_moe
37
+ llm_load_print_meta: vocab type = SPM
38
+ llm_load_print_meta: n_vocab = 135040
39
+ llm_load_print_meta: n_merges = 0
40
+ llm_load_print_meta: n_ctx_train = 4096
41
+ llm_load_print_meta: n_embd = 2048
42
+ llm_load_print_meta: n_head = 16
43
+ llm_load_print_meta: n_head_kv = 16
44
+ llm_load_print_meta: n_layer = 24
45
+ llm_load_print_meta: n_rot = 128
46
+ llm_load_print_meta: n_embd_head_k = 256
47
+ llm_load_print_meta: n_embd_head_v = 256
48
+ llm_load_print_meta: n_gqa = 1
49
+ llm_load_print_meta: n_embd_k_gqa = 4096
50
+ llm_load_print_meta: n_embd_v_gqa = 4096
51
+ llm_load_print_meta: f_norm_eps = 0.0e+00
52
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-06
53
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
54
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
55
+ llm_load_print_meta: n_ff = 8192
56
+ llm_load_print_meta: n_expert = 32
57
+ llm_load_print_meta: n_expert_used = 2
58
+ llm_load_print_meta: rope scaling = linear
59
+ llm_load_print_meta: freq_base_train = 10000.0
60
+ llm_load_print_meta: freq_scale_train = 1
61
+ llm_load_print_meta: n_yarn_orig_ctx = 4096
62
+ llm_load_print_meta: rope_finetuned = unknown
63
+ llm_load_print_meta: model type = 40B
64
+ llm_load_print_meta: model ftype = F16
65
+ llm_load_print_meta: model params = 40.22 B
66
+ llm_load_print_meta: model size = 74.91 GiB (16.00 BPW)
67
+ llm_load_print_meta: general.name = yuan2_moe
68
+ llm_load_print_meta: BOS token = 77185 '<eod>'
69
+ llm_load_print_meta: EOS token = 77185 '<eod>'
70
+ llm_load_print_meta: UNK token = 0 '<unk>'
71
+ llm_load_print_meta: SEP token = 77185 '<eod>'
72
+ llm_load_print_meta: PAD token = 77185 '<eod>'
73
+ llm_load_print_meta: LF token = 13 '<0x0A>'
74
+ llm_load_tensors: ggml ctx size = 1.06 MiB
75
+ llm_load_tensors: CPU buffer size = 76712.85 MiB
76
+ ....................................................................................................
77
+ llama_new_context_with_model: n_ctx = 512
78
+ llama_new_context_with_model: freq_base = 10000.0
79
+ llama_new_context_with_model: freq_scale = 1
80
+ llama_kv_cache_init: CPU KV buffer size = 192.38 MiB
81
+ llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB
82
+ llama_new_context_with_model: CPU input buffer size = 6.01 MiB
83
+ llama_new_context_with_model: CPU compute buffer size = 275.88 MiB
84
+ llama_new_context_with_model: graph splits (measure): 1
85
+
86
+ system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
87
+ compute_imatrix: tokenizing the input ..
88
+ compute_imatrix: tokenization took 145.195 ms
89
+ compute_imatrix: computing over 227 chunks with batch_size 512
90
+ compute_imatrix: 6.78 seconds per pass - ETA 25.65 minutes
91
+ [1]6.5558,[2]5.0024,[3]5.8688,[4]6.8477,
92
+ save_imatrix: stored collected data after 10 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
93
+ [5]7.1188,[6]7.3938,[7]6.3059,[8]6.7000,[9]6.7433,
94
+ save_imatrix: stored collected data after 20 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
95
+ [10]7.2226,[11]7.5399,[12]7.9695,[13]8.3739,[14]7.8757,
96
+ save_imatrix: stored collected data after 30 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
97
+ [15]8.2691,[16]8.3867,[17]8.6503,[18]8.8671,[19]9.0091,
98
+ save_imatrix: stored collected data after 40 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
99
+ [20]9.1840,[21]9.2799,[22]9.0400,[23]8.6569,[24]8.7209,
100
+ save_imatrix: stored collected data after 50 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
101
+ [25]8.1930,[26]7.7850,[27]7.5167,[28]7.2632,[29]6.9825,
102
+ save_imatrix: stored collected data after 60 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
103
+ [30]7.3951,[31]7.3448,[32]7.5309,[33]7.3902,[34]7.3415,
104
+ save_imatrix: stored collected data after 70 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
105
+ [35]7.0182,[36]6.7552,[37]6.4881,[38]6.4493,[39]6.4314,
106
+ save_imatrix: stored collected data after 80 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
107
+ [40]6.4113,[41]6.4790,[42]6.5106,[43]6.6147,[44]6.7387,
108
+ save_imatrix: stored collected data after 90 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
109
+ [45]6.8810,[46]6.9200,[47]7.0145,[48]7.1681,[49]7.3595,
110
+ save_imatrix: stored collected data after 100 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
111
+ [50]7.5659,[51]7.6763,[52]7.6824,[53]7.6250,[54]7.5348,
112
+ save_imatrix: stored collected data after 110 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
113
+ [55]7.4143,[56]7.5141,[57]7.5952,[58]7.6657,[59]7.7468,
114
+ save_imatrix: stored collected data after 120 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
115
+ [60]7.7793,[61]7.8927,[62]7.9396,[63]8.0284,[64]8.0642,
116
+ save_imatrix: stored collected data after 130 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
117
+ [65]8.1467,[66]8.2391,[67]8.2560,[68]8.2841,[69]8.3739,
118
+ save_imatrix: stored collected data after 140 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
119
+ [70]8.5129,[71]8.6025,[72]8.6186,[73]8.5808,[74]8.5795,
120
+ save_imatrix: stored collected data after 150 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
121
+ [75]8.6234,[76]8.6382,[77]8.6283,[78]8.5947,[79]8.5555,
122
+ save_imatrix: stored collected data after 160 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
123
+ [80]8.5826,[81]8.5578,[82]8.5244,[83]8.5821,[84]8.6244,
124
+ save_imatrix: stored collected data after 170 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
125
+ [85]8.6236,[86]8.6298,[87]8.6283,[88]8.6120,[89]8.5763,
126
+ save_imatrix: stored collected data after 180 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
127
+ [90]8.5623,[91]8.5481,[92]8.5802,[93]8.5909,[94]8.6161,
128
+ save_imatrix: stored collected data after 190 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
129
+ [95]8.6344,[96]8.6459,[97]8.6175,[98]8.5774,[99]8.5987,
130
+ save_imatrix: stored collected data after 200 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
131
+ [100]8.6399,[101]8.6517,[102]8.6304,[103]8.5887,[104]8.5743,
132
+ save_imatrix: stored collected data after 210 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
133
+ [105]8.5806,[106]8.6410,[107]8.6477,[108]8.6413,[109]8.6781,
134
+ save_imatrix: stored collected data after 220 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
135
+ [110]8.7056,[111]8.7298,[112]8.7006,[113]8.7188,[114]8.7299,
136
+ save_imatrix: stored collected data after 230 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
137
+ [115]8.7238,[116]8.7355,[117]8.7230,[118]8.7087,[119]8.6851,
138
+ save_imatrix: stored collected data after 240 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
139
+ [120]8.6678,[121]8.7011,[122]8.7138,[123]8.7270,[124]8.7333,
140
+ save_imatrix: stored collected data after 250 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
141
+ [125]8.7264,[126]8.7703,[127]8.6855,[128]8.6916,[129]8.6864,
142
+ save_imatrix: stored collected data after 260 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
143
+ [130]8.6621,[131]8.6297,[132]8.5725,[133]8.5187,[134]8.4626,
144
+ save_imatrix: stored collected data after 270 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
145
+ [135]8.4089,[136]8.3466,[137]8.2958,[138]8.2468,[139]8.1934,
146
+ save_imatrix: stored collected data after 280 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
147
+ [140]8.1758,[141]8.1481,[142]8.1078,[143]8.0591,[144]8.0292,
148
+ save_imatrix: stored collected data after 290 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
149
+ [145]8.0041,[146]7.9806,[147]7.9489,[148]7.9131,[149]7.8880,
150
+ save_imatrix: stored collected data after 300 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
151
+ [150]7.8550,[151]7.8215,[152]7.8359,[153]7.7834,[154]7.7633,
152
+ save_imatrix: stored collected data after 310 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
153
+ [155]7.8523,[156]7.9496,[157]8.0368,[158]8.1009,[159]8.1471,
154
+ save_imatrix: stored collected data after 320 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
155
+ [160]8.1647,[161]8.1946,[162]8.2248,[163]8.2249,[164]8.2250,
156
+ save_imatrix: stored collected data after 330 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
157
+ [165]8.2689,[166]8.2907,[167]8.3209,[168]8.3421,[169]8.3452,
158
+ save_imatrix: stored collected data after 340 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
159
+ [170]8.3679,[171]8.3964,[172]8.4240,[173]8.4850,[174]8.5304,
160
+ save_imatrix: stored collected data after 350 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
161
+ [175]8.5662,[176]8.5695,[177]8.5835,[178]8.5995,[179]8.6060,
162
+ save_imatrix: stored collected data after 360 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
163
+ [180]8.5969,[181]8.6079,[182]8.6110,[183]8.6246,[184]8.6350,
164
+ save_imatrix: stored collected data after 370 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
165
+ [185]8.6868,[186]8.6822,[187]8.6447,[188]8.6949,[189]8.7372,
166
+ save_imatrix: stored collected data after 380 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
167
+ [190]8.7771,[191]8.8391,[192]8.9060,[193]8.8979,[194]8.8961,
168
+ save_imatrix: stored collected data after 390 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
169
+ [195]8.8519,[196]8.8544,[197]8.8626,[198]8.8997,[199]8.8840,
170
+ save_imatrix: stored collected data after 400 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
171
+ [200]8.8698,[201]8.8631,[202]8.8561,[203]8.8675,[204]8.8596,
172
+ save_imatrix: stored collected data after 410 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
173
+ [205]8.8652,[206]8.8837,[207]8.8930,[208]8.9143,[209]8.9504,
174
+ save_imatrix: stored collected data after 420 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
175
+ [210]8.9338,[211]8.9362,[212]8.9262,[213]8.9121,[214]8.9058,
176
+ save_imatrix: stored collected data after 430 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
177
+ [215]8.8761,[216]8.8852,[217]8.8815,[218]8.8830,[219]8.8819,
178
+ save_imatrix: stored collected data after 440 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
179
+ [220]8.9316,[221]8.9905,[222]9.0298,[223]9.0636,[224]9.1087,
180
+ save_imatrix: stored collected data after 450 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
181
+ [225]9.0895,[226]9.0782,[227]9.0585,
182
+ save_imatrix: stored collected data after 454 chunks in Yuan2-M32-hf-IMat-GGUF/imatrix.dat
183
+
184
+ llama_print_timings: load time = 51125.10 ms
185
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
186
+ llama_print_timings: prompt eval time = 1527383.56 ms / 116224 tokens ( 13.14 ms per token, 76.09 tokens per second)
187
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
188
+ llama_print_timings: total time = 1572971.60 ms / 116225 tokens
189
+
190
+ Final estimate: PPL = 9.0585 +/- 0.10484