llama_model_loader: loaded meta data with 46 key-value pairs and 959 tensors from DeepSeek-V2.5-IMat-GGUF/DeepSeek-V2.5.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = DeepSeek V2.5 llama_model_loader: - kv 3: general.version str = V2.5 llama_model_loader: - kv 4: general.basename str = DeepSeek llama_model_loader: - kv 5: general.size_label str = 160x14B llama_model_loader: - kv 6: general.license str = other llama_model_loader: - kv 7: general.license.name str = deepseek llama_model_loader: - kv 8: general.license.link str = https://github.com/deepseek-ai/DeepSe... llama_model_loader: - kv 9: deepseek2.block_count u32 = 60 llama_model_loader: - kv 10: deepseek2.context_length u32 = 163840 llama_model_loader: - kv 11: deepseek2.embedding_length u32 = 5120 llama_model_loader: - kv 12: deepseek2.feed_forward_length u32 = 12288 llama_model_loader: - kv 13: deepseek2.attention.head_count u32 = 128 llama_model_loader: - kv 14: deepseek2.attention.head_count_kv u32 = 128 llama_model_loader: - kv 15: deepseek2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 16: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 17: deepseek2.expert_used_count u32 = 6 llama_model_loader: - kv 18: general.file_type u32 = 7 llama_model_loader: - kv 19: deepseek2.leading_dense_block_count u32 = 1 llama_model_loader: - kv 20: deepseek2.vocab_size u32 = 102400 llama_model_loader: - kv 21: deepseek2.attention.q_lora_rank u32 = 1536 llama_model_loader: - kv 22: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 23: deepseek2.attention.key_length u32 = 192 llama_model_loader: - kv 24: deepseek2.attention.value_length u32 = 128 llama_model_loader: - kv 25: deepseek2.expert_feed_forward_length u32 = 1536 llama_model_loader: - kv 26: deepseek2.expert_count u32 = 160 llama_model_loader: - kv 27: deepseek2.expert_shared_count u32 = 2 llama_model_loader: - kv 28: deepseek2.expert_weights_scale f32 = 16.000000 llama_model_loader: - kv 29: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 30: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 31: deepseek2.rope.scaling.factor f32 = 40.000000 llama_model_loader: - kv 32: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 33: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 35: tokenizer.ggml.pre str = deepseek-llm llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,102400] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,102400] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,99757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e... llama_model_loader: - kv 39: tokenizer.ggml.bos_token_id u32 = 100000 llama_model_loader: - kv 40: tokenizer.ggml.eos_token_id u32 = 100001 llama_model_loader: - kv 41: tokenizer.ggml.padding_token_id u32 = 100001 llama_model_loader: - kv 42: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 43: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 44: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 45: general.quantization_version u32 = 2 llama_model_loader: - type f32: 300 tensors llama_model_loader: - type q8_0: 659 tensors llm_load_vocab: special tokens cache size = 18 llm_load_vocab: token to piece cache size = 0.6411 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = deepseek2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 102400 llm_load_print_meta: n_merges = 99757 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 163840 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_layer = 60 llm_load_print_meta: n_head = 128 llm_load_print_meta: n_head_kv = 128 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 192 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 24576 llm_load_print_meta: n_embd_v_gqa = 16384 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 12288 llm_load_print_meta: n_expert = 160 llm_load_print_meta: n_expert_used = 6 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = yarn llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 0.025 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 236B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 235.74 B llm_load_print_meta: model size = 233.41 GiB (8.50 BPW) llm_load_print_meta: general.name = DeepSeek V2.5 llm_load_print_meta: BOS token = 100000 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 100001 '<｜end▁of▁sentence｜>' llm_load_print_meta: PAD token = 100001 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 126 'Ä' llm_load_print_meta: max token length = 256 llm_load_print_meta: n_layer_dense_lead = 1 llm_load_print_meta: n_lora_q = 1536 llm_load_print_meta: n_lora_kv = 512 llm_load_print_meta: n_ff_exp = 1536 llm_load_print_meta: n_expert_shared = 2 llm_load_print_meta: expert_weights_scale = 16.0 llm_load_print_meta: rope_yarn_log_mul = 0.1000 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.80 MiB llm_load_tensors: offloading 5 repeating layers to GPU llm_load_tensors: offloaded 5/61 layers to GPU llm_load_tensors: CPU buffer size = 218873.36 MiB llm_load_tensors: CUDA0 buffer size = 20135.96 MiB .................................................................................................... llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 0.025 llama_kv_cache_init: CUDA_Host KV buffer size = 2200.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 200.00 MiB llama_new_context_with_model: KV self size = 2400.00 MiB, K (f16): 1440.00 MiB, V (f16): 960.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.39 MiB llama_new_context_with_model: CUDA0 compute buffer size = 1422.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 81.01 MiB llama_new_context_with_model: graph nodes = 4480 llama_new_context_with_model: graph splits = 990 system_info: n_threads = 25 (n_threads_batch = 25) / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | compute_imatrix: tokenizing the input .. compute_imatrix: tokenization took 227.203 ms compute_imatrix: computing over 139 chunks with batch_size 512 compute_imatrix: 75.87 seconds per pass - ETA 2 hours 55.77 minutes [1]5.5015,[2]3.7649,[3]3.6729,[4]4.0519,[5]4.0107,[6]3.8092,[7]4.0137,[8]3.8730,[9]4.3245, save_imatrix: stored collected data after 10 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [10]4.5113,[11]4.2002,[12]4.4440,[13]4.7697,[14]5.0262,[15]5.1554,[16]5.3737,[17]5.5523,[18]5.6777,[19]5.7885, save_imatrix: stored collected data after 20 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [20]5.5213,[21]5.5350,[22]5.4706,[23]5.5090,[24]5.4301,[25]5.5961,[26]5.5111,[27]5.6447,[28]5.4996,[29]5.2319, save_imatrix: stored collected data after 30 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [30]5.0140,[31]4.9498,[32]4.8967,[33]4.7701,[34]4.5632,[35]4.3906,[36]4.3358,[37]4.2842,[38]4.2968,[39]4.2397, save_imatrix: stored collected data after 40 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [40]4.1962,[41]4.1293,[42]4.0756,[43]4.1196,[44]4.1897,[45]4.2780,[46]4.2952,[47]4.1724,[48]4.0553,[49]3.9527, save_imatrix: stored collected data after 50 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [50]3.8533,[51]3.8519,[52]3.8249,[53]3.9112,[54]3.9904,[55]4.0774,[56]4.0375,[57]4.0570,[58]4.0796,[59]4.1426, save_imatrix: stored collected data after 60 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [60]4.2123,[61]4.2756,[62]4.3190,[63]4.3468,[64]4.3812,[65]4.4107,[66]4.4078,[67]4.4155,[68]4.4210,[69]4.4617, save_imatrix: stored collected data after 70 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [70]4.4805,[71]4.4887,[72]4.5195,[73]4.5275,[74]4.5322,[75]4.5374,[76]4.5511,[77]4.5618,[78]4.5849,[79]4.5736, save_imatrix: stored collected data after 80 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [80]4.5966,[81]4.5972,[82]4.6062,[83]4.6034,[84]4.6068,[85]4.6044,[86]4.6032,[87]4.5966,[88]4.6224,[89]4.6473, save_imatrix: stored collected data after 90 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [90]4.6531,[91]4.6850,[92]4.7205,[93]4.6958,[94]4.7014,[95]4.6854,[96]4.7096,[97]4.7276,[98]4.7255,[99]4.6876, save_imatrix: stored collected data after 100 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [100]4.6500,[101]4.6125,[102]4.5737,[103]4.5371,[104]4.5050,[105]4.4702,[106]4.4360,[107]4.4042,[108]4.3785,[109]4.3936, save_imatrix: stored collected data after 110 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [110]4.4266,[111]4.4703,[112]4.5140,[113]4.5552,[114]4.6251,[115]4.6639,[116]4.6870,[117]4.7021,[118]4.7271,[119]4.7273, save_imatrix: stored collected data after 120 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [120]4.7050,[121]4.6528,[122]4.6047,[123]4.6342,[124]4.6665,[125]4.6711,[126]4.6768,[127]4.6893,[128]4.7165,[129]4.7215, save_imatrix: stored collected data after 130 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat [130]4.7351,[131]4.7572,[132]4.7526,[133]4.7442,[134]4.6946,[135]4.6445,[136]4.6437,[137]4.5964,[138]4.5544,[139]4.5095, save_imatrix: stored collected data after 139 chunks in DeepSeek-V2.5-IMat-GGUF/imatrix.dat llama_print_timings: load time = 166881.68 ms llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: prompt eval time = 15552712.87 ms / 71168 tokens ( 218.54 ms per token, 4.58 tokens per second) llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: total time = 15670205.49 ms / 71169 tokens Final estimate: PPL = 4.5095 +/- 0.05621