lightblue
/

Karasu-Mixtral-8x22B-v0.1-gguf

Inference Endpoints

Model card Files Files and versions Community

ptrdvn commited on Apr 12

Commit

18dc5ba

•

1 Parent(s): 1149b1d

Update README.md

Files changed (1) hide show

README.md +3 -3

README.md CHANGED Viewed

@@ -22,10 +22,10 @@ If you would like a nice easy GUI and have >64GB of RAM, then you could also run
 ```bash
 cd llama.cpp
-./convert.py --outfile Karasu-Mixtral-8x22B-v0.1-q3_k_m --outtype f16 /workspace/llm_training/axolotl/mixtral_8x22B_training/merged_model_multiling
-./quantize /workspace/Karasu-Mixtral-8x22B-v0.1.gguf /workspace/Karasu-Mixtral-8x22B-v0.1_q3_k_m.gguf Q3_K_M
-./gguf-split --split --split-max-size 5G /workspace/Karasu-Mixtral-8x22B-v0.1_q3_k_m.gguf /workspace/somewhere-sensible
 ```

 ```bash
 cd llama.cpp
+./convert-hf-to-gguf.py --outfile  /workspace/Karasu-Mixtral-8x22B-v0.1.gguf --outtype f16 /workspace/llm_training/axolotl/mixtral_8x22B_training/merged_model_multiling
+./quantize /workspace/Karasu-Mixtral-8x22B-v0.1.gguf /workspace/Karasu-Mixtral-8x22B-v0.1-Q3_K_M.gguf Q3_K_M
+./gguf-split  --split --split-max-tensors 128  /workspace/Karasu-Mixtral-8x22B-v0.1-Q3_K_M.gguf /workspace/split_gguf_q3km/Karasu-Mixtral-8x22B-v0.1-Q3_K_M
 ```