mradermacher

/

Llama-3.1-8B-Instruct-Fei-v1-Uncensored-GGUF

like 1

{
"memory": {
"ram_capacity": "31.88 GB",
"ram_unused": "24.63 GB"
},
"gpu": {
"gpu_names": [
"NVIDIA GeForce RTX 3070"
],
"vram_recommended_capacity": "8.00 GB",
"vram_unused": "6.93 GB"
},
"os": {
"platform": "win32",
"version": "10.0.19045"
},
"app": {
"version": "0.2.27",
"downloadsDir": "C:\mytools\gguf"
},
"model": {}
}

C:/mytools/llama.cpp $ ./llama-cli.exe -m ../gguf/Publisher/Repository/Llama-3.1-8B-Instruct-Fei-v1-Uncensored.Q4_K_M.gguf -p "hi" -n 128
Log start
main: build = 3464 (49ce0ab6)
main: built with cc (GCC) 14.1.0 for x86_64-w64-mingw32
main: seed = 1722132883
llama_model_loader: loaded meta data with 38 key-value pairs and 292 tensors from ../gguf/Publisher/Repository/Llama-3.1-8B-Instruct-Fei-v1-Uncensored.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct
llama_model_loader: - kv 3: general.version str = v1
llama_model_loader: - kv 4: general.finetune str = Instruct
llama_model_loader: - kv 5: general.basename str = Meta-Llama-3.1
llama_model_loader: - kv 6: general.size_label str = 8B
llama_model_loader: - kv 7: general.license str = llama3.1
llama_model_loader: - kv 8: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv 9: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv 10: llama.block_count u32 = 32
llama_model_loader: - kv 11: llama.context_length u32 = 131072
llama_model_loader: - kv 12: llama.embedding_length u32 = 4096
llama_model_loader: - kv 13: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 14: llama.attention.head_count u32 = 32
llama_model_loader: - kv 15: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 16: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 17: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 18: general.file_type u32 = 15
llama_model_loader: - kv 19: llama.vocab_size u32 = 128256
llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 22: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 128009
llama_model_loader: - kv 28: tokenizer.ggml.padding_token_id u32 = 128009
llama_model_loader: - kv 29: tokenizer.chat_template str = {{ '<|begin_of_text|>' }}{% if messag...
llama_model_loader: - kv 30: general.quantization_version u32 = 2
llama_model_loader: - kv 31: general.url str = https://huggingface.co/mradermacher/L...
llama_model_loader: - kv 32: mradermacher.quantize_version str = 2
llama_model_loader: - kv 33: mradermacher.quantized_by str = mradermacher
llama_model_loader: - kv 34: mradermacher.quantized_at str = 2024-07-28T00:15:55+02:00
llama_model_loader: - kv 35: mradermacher.quantized_on str = db1
llama_model_loader: - kv 36: general.source.url str = https://huggingface.co/aifeifei799/Ll...
llama_model_loader: - kv 37: mradermacher.convert_type str = hf
llama_model_loader: - type f32: 66 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 128256
llm_load_print_meta: n_merges = 280147
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 8.03 B
llm_load_print_meta: model size = 4.58 GiB (4.89 BPW)
llm_load_print_meta: general.name = Meta Llama 3.1 8B Instruct
llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
llm_load_print_meta: PAD token = 128009 '<|eot_id|>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size = 0.14 MiB
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 292, got 291
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '../gguf/Publisher/Repository/Llama-3.1-8B-Instruct-Fei-v1-Uncensored.Q4_K_M.gguf'
main: error: unable to load model

You need to update your llama.cpp.

C:\mytools\llama.cpp>llama-cli.exe --verbose
Log start
main: build = 3464 (49ce0ab6)
main: built with cc (GCC) 14.1.0 for x86_64-w64-mingw32
main: seed = 1722133068

update llama.cpp, :)
C:/mytools/llama.cpp $ ./llama-cli.exe -m ../gguf/Publisher/Repository/Llama-3.1-8B-Instruct-Fei-v1-Uncensored.Q4_K_M.gguf -p "hi" -n 128
Log start
main: build = 3482 (e54c35e4)
main: built with cc (GCC) 14.1.0 for x86_64-w64-mingw32
main: seed = 1722133272
llama_model_loader: loaded meta data with 38 key-value pairs and 292 tensors from ../gguf/Publisher/Repository/Llama-3.1-8B-Instruct-Fei-v1-Uncensored.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct
llama_model_loader: - kv 3: general.version str = v1
llama_model_loader: - kv 4: general.finetune str = Instruct
llama_model_loader: - kv 5: general.basename str = Meta-Llama-3.1
llama_model_loader: - kv 6: general.size_label str = 8B
llama_model_loader: - kv 7: general.license str = llama3.1
llama_model_loader: - kv 8: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv 9: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv 10: llama.block_count u32 = 32
llama_model_loader: - kv 11: llama.context_length u32 = 131072
llama_model_loader: - kv 12: llama.embedding_length u32 = 4096
llama_model_loader: - kv 13: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 14: llama.attention.head_count u32 = 32
llama_model_loader: - kv 15: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 16: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 17: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 18: general.file_type u32 = 15
llama_model_loader: - kv 19: llama.vocab_size u32 = 128256
llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 22: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 128009
llama_model_loader: - kv 28: tokenizer.ggml.padding_token_id u32 = 128009
llama_model_loader: - kv 29: tokenizer.chat_template str = {{ '<|begin_of_text|>' }}{% if messag...
llama_model_loader: - kv 30: general.quantization_version u32 = 2
llama_model_loader: - kv 31: general.url str = https://huggingface.co/mradermacher/L...
llama_model_loader: - kv 32: mradermacher.quantize_version str = 2
llama_model_loader: - kv 33: mradermacher.quantized_by str = mradermacher
llama_model_loader: - kv 34: mradermacher.quantized_at str = 2024-07-28T00:15:55+02:00
llama_model_loader: - kv 35: mradermacher.quantized_on str = db1
llama_model_loader: - kv 36: general.source.url str = https://huggingface.co/aifeifei799/Ll...
llama_model_loader: - kv 37: mradermacher.convert_type str = hf
llama_model_loader: - type f32: 66 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 128256
llm_load_print_meta: n_merges = 280147
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 8.03 B
llm_load_print_meta: model size = 4.58 GiB (4.89 BPW)
llm_load_print_meta: general.name = Meta Llama 3.1 8B Instruct
llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
llm_load_print_meta: PAD token = 128009 '<|eot_id|>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size = 0.14 MiB
llm_load_tensors: CPU buffer size = 4685.30 MiB
........................................................................................
llama_new_context_with_model: n_ctx = 131072
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 500000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 16384.00 MiB
llama_new_context_with_model: KV self size = 16384.00 MiB, K (f16): 8192.00 MiB, V (f16): 8192.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.49 MiB
llama_new_context_with_model: CPU compute buffer size = 8480.01 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 1

system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
sampling:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 131072, n_batch = 2048, n_predict = 128, n_keep = 1

hi, I have a question about my account.
How do I pay for my account?
If you are asking about your subscription, you can view your subscription status and payment information in your account settings. Please see our guide on how to manage your subscription for more details.
If you are having trouble with your subscription, please contact our support team and we'll be happy to help.
If you have a question about a one-time payment, please let me know the specific details and I'll do my best to assist you. hi, I have a question about my account.
Hi, I have a question about my account.
I was wondering if you
llama_print_timings: load time = 6316.03 ms
llama_print_timings: sample time = 11.53 ms / 128 runs ( 0.09 ms per token, 11104.36 tokens per second)
llama_print_timings: prompt eval time = 220.37 ms / 2 tokens ( 110.19 ms per token, 9.08 tokens per second)
llama_print_timings: eval time = 19633.13 ms / 127 runs ( 154.59 ms per token, 6.47 tokens per second)
llama_print_timings: total time = 19962.67 ms / 129 tokens
Log end

"llama.cpp error: 'done_getting_tensors: wrong number of tensors; expected 292, got 291'"
up llama.cpp >= 3482 (e54c35e4)
This issue has been resolved.

C:/mytools/llama.cpp $ ./llama-cli.exe -v
Log start
main: build = 3482 (e54c35e4)
main: built with cc (GCC) 14.1.0 for x86_64-w64-mingw32
main: seed = 1722133370

newer than the version I use... :)