Hugging Face's logo

legraphista
/
Hermes-3-Llama-3.1-8B-IMat-GGUF

Text Generation
GGUF
English
Llama-3
instruct
finetune
chatml
gpt4
synthetic data
distillation
function calling
json mode
axolotl
roleplaying
chat
quantized
GGUF
quantization
imat
imatrix
static
16bit
8bit
6bit
5bit
4bit
3bit
2bit
1bit
conversational
Model card Files Community
legraphista commited on
Commit
3711eaf
•
1 Parent(s): 75477f0

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +153 -0
imatrix.log ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ llama_model_loader: loaded meta data with 34 key-value pairs and 292 tensors from Hermes-3-Llama-3.1-8B-IMat-GGUF/Hermes-3-Llama-3.1-8B.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
2
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
3
+ llama_model_loader: - kv 0: general.architecture str = llama
4
+ llama_model_loader: - kv 1: general.type str = model
5
+ llama_model_loader: - kv 2: general.name str = Hermes 3 Llama 3.1 8B
6
+ llama_model_loader: - kv 3: general.organization str = NousResearch
7
+ llama_model_loader: - kv 4: general.basename str = Hermes-3-Llama-3.1
8
+ llama_model_loader: - kv 5: general.size_label str = 8B
9
+ llama_model_loader: - kv 6: general.license str = llama3
10
+ llama_model_loader: - kv 7: general.base_model.count u32 = 1
11
+ llama_model_loader: - kv 8: general.base_model.0.name str = Meta Llama 3.1 8B
12
+ llama_model_loader: - kv 9: general.base_model.0.organization str = Meta Llama
13
+ llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Met...
14
+ llama_model_loader: - kv 11: general.tags arr[str,12] = ["Llama-3", "instruct", "finetune", "...
15
+ llama_model_loader: - kv 12: general.languages arr[str,1] = ["en"]
16
+ llama_model_loader: - kv 13: llama.block_count u32 = 32
17
+ llama_model_loader: - kv 14: llama.context_length u32 = 131072
18
+ llama_model_loader: - kv 15: llama.embedding_length u32 = 4096
19
+ llama_model_loader: - kv 16: llama.feed_forward_length u32 = 14336
20
+ llama_model_loader: - kv 17: llama.attention.head_count u32 = 32
21
+ llama_model_loader: - kv 18: llama.attention.head_count_kv u32 = 8
22
+ llama_model_loader: - kv 19: llama.rope.freq_base f32 = 500000.000000
23
+ llama_model_loader: - kv 20: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
24
+ llama_model_loader: - kv 21: general.file_type u32 = 7
25
+ llama_model_loader: - kv 22: llama.vocab_size u32 = 128256
26
+ llama_model_loader: - kv 23: llama.rope.dimension_count u32 = 128
27
+ llama_model_loader: - kv 24: tokenizer.ggml.model str = gpt2
28
+ llama_model_loader: - kv 25: tokenizer.ggml.pre str = llama-bpe
29
+ llama_model_loader: - kv 26: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
30
+ llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
31
+ llama_model_loader: - kv 28: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
32
+ llama_model_loader: - kv 29: tokenizer.ggml.bos_token_id u32 = 128000
33
+ llama_model_loader: - kv 30: tokenizer.ggml.eos_token_id u32 = 128040
34
+ llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 128040
35
+ llama_model_loader: - kv 32: tokenizer.chat_template str = {{bos_token}}{% for message in messag...
36
+ llama_model_loader: - kv 33: general.quantization_version u32 = 2
37
+ llama_model_loader: - type f32: 66 tensors
38
+ llama_model_loader: - type q8_0: 226 tensors
39
+ llm_load_vocab: special tokens cache size = 256
40
+ llm_load_vocab: token to piece cache size = 0.7994 MB
41
+ llm_load_print_meta: format = GGUF V3 (latest)
42
+ llm_load_print_meta: arch = llama
43
+ llm_load_print_meta: vocab type = BPE
44
+ llm_load_print_meta: n_vocab = 128256
45
+ llm_load_print_meta: n_merges = 280147
46
+ llm_load_print_meta: vocab_only = 0
47
+ llm_load_print_meta: n_ctx_train = 131072
48
+ llm_load_print_meta: n_embd = 4096
49
+ llm_load_print_meta: n_layer = 32
50
+ llm_load_print_meta: n_head = 32
51
+ llm_load_print_meta: n_head_kv = 8
52
+ llm_load_print_meta: n_rot = 128
53
+ llm_load_print_meta: n_swa = 0
54
+ llm_load_print_meta: n_embd_head_k = 128
55
+ llm_load_print_meta: n_embd_head_v = 128
56
+ llm_load_print_meta: n_gqa = 4
57
+ llm_load_print_meta: n_embd_k_gqa = 1024
58
+ llm_load_print_meta: n_embd_v_gqa = 1024
59
+ llm_load_print_meta: f_norm_eps = 0.0e+00
60
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-05
61
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
62
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
63
+ llm_load_print_meta: f_logit_scale = 0.0e+00
64
+ llm_load_print_meta: n_ff = 14336
65
+ llm_load_print_meta: n_expert = 0
66
+ llm_load_print_meta: n_expert_used = 0
67
+ llm_load_print_meta: causal attn = 1
68
+ llm_load_print_meta: pooling type = 0
69
+ llm_load_print_meta: rope type = 0
70
+ llm_load_print_meta: rope scaling = linear
71
+ llm_load_print_meta: freq_base_train = 500000.0
72
+ llm_load_print_meta: freq_scale_train = 1
73
+ llm_load_print_meta: n_ctx_orig_yarn = 131072
74
+ llm_load_print_meta: rope_finetuned = unknown
75
+ llm_load_print_meta: ssm_d_conv = 0
76
+ llm_load_print_meta: ssm_d_inner = 0
77
+ llm_load_print_meta: ssm_d_state = 0
78
+ llm_load_print_meta: ssm_dt_rank = 0
79
+ llm_load_print_meta: model type = 8B
80
+ llm_load_print_meta: model ftype = Q8_0
81
+ llm_load_print_meta: model params = 8.03 B
82
+ llm_load_print_meta: model size = 7.95 GiB (8.50 BPW)
83
+ llm_load_print_meta: general.name = Hermes 3 Llama 3.1 8B
84
+ llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
85
+ llm_load_print_meta: EOS token = 128040 '<|im_end|>'
86
+ llm_load_print_meta: PAD token = 128040 '<|im_end|>'
87
+ llm_load_print_meta: LF token = 128 'Ä'
88
+ llm_load_print_meta: EOT token = 128040 '<|im_end|>'
89
+ llm_load_print_meta: max token length = 256
90
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
91
+ ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
92
+ ggml_cuda_init: found 1 CUDA devices:
93
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
94
+ llm_load_tensors: ggml ctx size = 0.27 MiB
95
+ llm_load_tensors: offloading 32 repeating layers to GPU
96
+ llm_load_tensors: offloading non-repeating layers to GPU
97
+ llm_load_tensors: offloaded 33/33 layers to GPU
98
+ llm_load_tensors: CPU buffer size = 532.31 MiB
99
+ llm_load_tensors: CUDA0 buffer size = 7605.34 MiB
100
+ .........................................................................................
101
+ llama_new_context_with_model: n_ctx = 512
102
+ llama_new_context_with_model: n_batch = 512
103
+ llama_new_context_with_model: n_ubatch = 512
104
+ llama_new_context_with_model: flash_attn = 0
105
+ llama_new_context_with_model: freq_base = 500000.0
106
+ llama_new_context_with_model: freq_scale = 1
107
+ llama_kv_cache_init: CUDA0 KV buffer size = 64.00 MiB
108
+ llama_new_context_with_model: KV self size = 64.00 MiB, K (f16): 32.00 MiB, V (f16): 32.00 MiB
109
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.49 MiB
110
+ llama_new_context_with_model: CUDA0 compute buffer size = 258.50 MiB
111
+ llama_new_context_with_model: CUDA_Host compute buffer size = 9.01 MiB
112
+ llama_new_context_with_model: graph nodes = 1030
113
+ llama_new_context_with_model: graph splits = 2
114
+
115
+ system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
116
+ compute_imatrix: tokenizing the input ..
117
+ compute_imatrix: tokenization took 42.013 ms
118
+ compute_imatrix: computing over 125 chunks with batch_size 512
119
+ compute_imatrix: 0.75 seconds per pass - ETA 1.55 minutes
120
+ [1]6.5294,[2]5.2572,[3]4.6884,[4]5.8580,[5]5.9745,[6]4.9979,[7]5.2920,[8]5.8251,[9]5.9930,
121
+ save_imatrix: stored collected data after 10 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
122
+ [10]5.3755,[11]5.8342,[12]6.3525,[13]6.8022,[14]7.2438,[15]7.5480,[16]7.8856,[17]8.0749,[18]7.7882,[19]7.4444,
123
+ save_imatrix: stored collected data after 20 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
124
+ [20]7.4155,[21]7.6047,[22]7.5125,[23]7.8480,[24]7.8675,[25]8.2301,[26]8.2276,[27]8.0914,[28]8.0742,[29]8.0705,
125
+ save_imatrix: stored collected data after 30 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
126
+ [30]8.0540,[31]7.6576,[32]7.2706,[33]7.0903,[34]6.9371,[35]7.0092,[36]7.0515,[37]7.0219,[38]7.1002,[39]7.2472,
127
+ save_imatrix: stored collected data after 40 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
128
+ [40]7.3291,[41]7.2846,[42]7.1899,[43]7.2619,[44]7.2908,[45]7.4095,[46]7.3172,[47]7.4408,[48]7.5346,[49]7.6345,
129
+ save_imatrix: stored collected data after 50 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
130
+ [50]7.5249,[51]7.6517,[52]7.7858,[53]7.8870,[54]7.9717,[55]7.9944,[56]7.9542,[57]7.9644,[58]7.9124,[59]7.9322,
131
+ save_imatrix: stored collected data after 60 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
132
+ [60]7.9002,[61]7.8969,[62]7.9549,[63]8.0105,[64]7.9395,[65]7.9150,[66]7.9417,[67]7.9349,[68]7.9387,[69]7.9429,
133
+ save_imatrix: stored collected data after 70 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
134
+ [70]7.9583,[71]7.9599,[72]7.9700,[73]7.9487,[74]7.9034,[75]7.9109,[76]7.9282,[77]7.8986,[78]7.9046,[79]7.9482,
135
+ save_imatrix: stored collected data after 80 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
136
+ [80]7.9852,[81]7.9814,[82]7.9836,[83]8.0226,[84]7.9148,[85]7.9197,[86]7.9340,[87]7.9525,[88]7.9872,[89]8.0079,
137
+ save_imatrix: stored collected data after 90 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
138
+ [90]7.9379,[91]7.8548,[92]7.7770,[93]7.7098,[94]7.6401,[95]7.5784,[96]7.5469,[97]7.5560,[98]7.6133,[99]7.6978,
139
+ save_imatrix: stored collected data after 100 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
140
+ [100]7.7826,[101]7.8411,[102]7.9709,[103]7.9998,[104]8.0367,[105]7.9608,[106]7.9741,[107]7.9274,[108]7.8736,[109]7.8201,
141
+ save_imatrix: stored collected data after 110 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
142
+ [110]7.8671,[111]7.9228,[112]7.9391,[113]7.9329,[114]7.9656,[115]8.0049,[116]8.0187,[117]8.0360,[118]8.0694,[119]8.0208,
143
+ save_imatrix: stored collected data after 120 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
144
+ [120]8.0207,[121]8.0026,[122]8.0207,[123]8.0415,[124]8.0669,[125]8.0627,
145
+ save_imatrix: stored collected data after 125 chunks in Hermes-3-Llama-3.1-8B-IMat-GGUF/imatrix.dat
146
+
147
+ llama_print_timings: load time = 3984.97 ms
148
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
149
+ llama_print_timings: prompt eval time = 74660.71 ms / 64000 tokens ( 1.17 ms per token, 857.21 tokens per second)
150
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
151
+ llama_print_timings: total time = 78713.95 ms / 64001 tokens
152
+
153
+ Final estimate: PPL = 8.0627 +/- 0.12730