ToRoLaMa-7B-GGUF

• Author: All By AI
• Original model: ToRoLaMa 7b v1.0

Description

GGUF format model files for ToRoLaMa 7b v1.0.

Prompt template: (Chat Format)

Cuộc hội thoại giữa người dùng và một trí thông minh nhân tạo. Đưa ra câu trả lời chính xác, giúp ích cho người dùng.

USER: chào
ASSISTANT: Xin chào! Tôi có thể giúp gì cho bạn hôm nay?</s>
USER:...

Quantisation methods (using llama.cpp b1796)

• GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
• GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
• GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
• GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
• GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw

How to use GGUF files

Note: Don't clone the entire repo! Only pick and download a single file.

Automatically download models using:

• LM Studio
• LoLLMS Web UI
• Faraday.dev