leafspark's picture
readme: add extra params
130ddce verified
|
raw
history blame
3.81 kB
metadata
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
  - code
  - deepseek
  - gguf
  - bf16
  - chinese
  - english
metrics:
  - accuracy

Deepseek-V2-Chat-GGUF

Quantizised from https://huggingface.co/deepseek-ai/DeepSeek-V2-Chat

Using llama.cpp fork: https://github.com/fairydreaming/llama.cpp/tree/deepseek-v2

TODO: Make llamafile for Q2_K and Q4_K_M

Warning: This will not work unless you compile llama.cpp from the repo provided (and set metadata KV overrides)!

How to use:

Downloading the bf16:

  • Find the relevant directory
  • Download all files
  • Run merge.py
  • Merged GGUF should appear

Downloading the quantizations:

  • Find the relevant directory
  • Download all files
  • Point to the first split (most programs should load all the splits automatically now)

Running in llama.cpp:

To start in command line interactive mode (text completion):

main -m DeepSeek-V2-Chat.{quant}.gguf -c {context length} --color -i

To use llama.cpp OpenAI compatible server:

server \
  -m DeepSeek-V2-Chat.{quant}.gguf \
  -c {context_length} \
  (--color [recommended: colored output in supported terminals]) \
  (-i [note: interactive mode]) \
  (--mlock [note: avoid using swap]) \
  (--verbose) \
  (--log-disable [note: disable logging to file, may be useful for prod]) \
  (--metrics [note: prometheus compatible monitoring endpoint]) \
  (--api-key [string]) \
  (--port [int]) \
  (--flash-attn [note: must be fully offloaded to supported GPU])

Making an importance matrix:

imatrix \
  -m DeepSeek-V2-Chat.{quant}.gguf \
  -f groups_merged.txt \
  --verbosity [0, 1, 2] \
  -ngl {GPU offloading; must build with CUDA} \
 --ofreq {recommended: 1}

Making a quant:

quantize \
  DeepSeek-V2-Chat.bf16.gguf \
  DeepSeek-V2-Chat.{quant}.gguf \
  {quant} \
  (--imatrix [file])

Quants:

- bf16 [size: 439gb]
- q8_0 (later, please use q4_k_m for now) [estimated size: 233.27gb]
- q4_k_m [size: 132gb]
- q2_k [size: 80gb]
- iq2_xxs [size: 61.5gb]
- iq3_xs (uploading) [size: 89.6gb]
- iq1_m [size: 27.3gb]

Note: Use iMatrix quants only if you can fully offload to GPU, otherwise speed will be affected a lot.

Planned Quants (using importance matrix):

- q5_k_m
- q5_k_s
- q3_k_m
- q6_k
- iq4_nl
- iq4_xs
- iq2_xs
- iq2_s
- iq2_m
- iq1_s (note: for fun only, this quant is likely useless)

Note: the model files do not have some DeepSeek v2 specific parameters, will look into adding them

Please use commit 039896407afd40e54321d47c5063c46a52da3e01, otherwise use these metadata KV overrides:

deepseek2.attention.q_lora_rank=int:1536
deepseek2.attention.kv_lora_rank=int:512
deepseek2.expert_shared_count=int:2
deepseek2.expert_feed_forward_length=int:1536
deepseek2.experts_weight_scale=int:16
deepseek2.leading_dense_block_count=int:1
rope.scaling.yarn_log_multiplier=float:0.0707

A precompiled AVX2 version is avaliable at llama.cpp-039896407afd40e54321d47c5063c46a52da3e01.zip in the root of this repo.

License:

  • DeepSeek license for model weights
  • MIT license for any repo code

Performance:

~1.5t/s with Ryzen 3 3700x (96gb 3200mhz) [Q2_K]

iMatrix:

Find imatrix.dat in the root of this repo, made with a Q2_K quant (see here for info: https://github.com/ggerganov/llama.cpp/issues/5153#issuecomment-1913185693)

Using groups_merged.txt, find it here: https://github.com/ggerganov/llama.cpp/discussions/5263#discussioncomment-8395384

Censorship:

This model is quite censored, finetuning on toxic DPO might help.