Edit model card

Yi-1.5-6B-Chat-GGUF

Original Model

01-ai/Yi-1.5-6B-Chat

Run with LlamaEdge

  • LlamaEdge version: coming soon

  • Prompt template

    • Prompt type: chatml

    • Prompt string

      <|im_start|>system
      {system_message}<|im_end|>
      <|im_start|>user
      {prompt}<|im_end|>
      <|im_start|>assistant
      
    • Reverse prompt: <|im_end|>

  • Context size: 4096

  • Run as LlamaEdge service

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-1.5-6B-Chat-Q5_K_M.gguf \
      llama-api-server.wasm \
      --prompt-template chatml \
      --reverse-prompt "<|im_end|>" \
      --ctx-size 4096 \
      --model-name Yi-1.5-6B-Chat
    
  • Run as LlamaEdge command app

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-1.5-6B-Chat-Q5_K_M.gguf \
      llama-chat.wasm \
      --prompt-template chatml \
      --reverse-prompt "<|im_end|>" \
      --ctx-size 4096
    

Quantized GGUF Models

Name Quant method Bits Size Use case
Yi-1.5-6B-Chat-Q2_K.gguf Q2_K 2 2.34 GB smallest, significant quality loss - not recommended for most purposes
Yi-1.5-6B-Chat-Q3_K_L.gguf Q3_K_L 3 3.24 GB small, substantial quality loss
Yi-1.5-6B-Chat-Q3_K_M.gguf Q3_K_M 3 2.99 GB very small, high quality loss
Yi-1.5-6B-Chat-Q3_K_S.gguf Q3_K_S 3 2.71 GB very small, high quality loss
Yi-1.5-6B-Chat-Q4_0.gguf Q4_0 4 3.48 GB legacy; small, very high quality loss - prefer using Q3_K_M
Yi-1.5-6B-Chat-Q4_K_M.gguf Q4_K_M 4 3.67 GB medium, balanced quality - recommended
Yi-1.5-6B-Chat-Q4_K_S.gguf Q4_K_S 4 3.5 GB small, greater quality loss
Yi-1.5-6B-Chat-Q5_0.gguf Q5_0 5 4.2 GB legacy; medium, balanced quality - prefer using Q4_K_M
Yi-1.5-6B-Chat-Q5_K_M.gguf Q5_K_M 5 4.3 GB large, very low quality loss - recommended
Yi-1.5-6B-Chat-Q5_K_S.gguf Q5_K_S 5 4.2 GB large, low quality loss - recommended
Yi-1.5-6B-Chat-Q6_K.gguf Q6_K 6 4.97 GB very large, extremely low quality loss
Yi-1.5-6B-Chat-Q8_0.gguf Q8_0 8 6.44 GB very large, extremely low quality loss - not recommended
Yi-1.5-6B-Chat-f16.gguf f16 16 12.1 GB

Quantized with llama.cpp b3135

Downloads last month
2,983
GGUF
Model size
6.06B params
Architecture
llama
Inference Examples
Inference API (serverless) has been turned off for this model.

Quantized from