Hugging Face's logo

tensorblock
/
typhoon-7b-GGUF

Text Generation
Transformers
GGUF
Thai
pretrained
TensorBlock
GGUF
Inference Endpoints
Model card Files Community
typhoon-7b-GGUF / README.md
morriszms's picture
morriszms
Upload folder using huggingface_hub
7a834d6 verified
preview code
|
raw
history blame contribute delete
4.42 kB
metadata
license: apache-2.0
language:
  - th
library_name: transformers
pipeline_tag: text-generation
tags:
  - pretrained
  - TensorBlock
  - GGUF
base_model: scb10x/typhoon-7b
TensorBlock

Feedback and support: TensorBlock's Twitter/X, Telegram Group and Discord server

scb10x/typhoon-7b - GGUF

This repo contains GGUF format model files for scb10x/typhoon-7b.

The files were quantized using machines provided by TensorBlock, and they are compatible with llama.cpp as of commit b4011.

Prompt template

Model file specification

Filename Quant type File Size Description
typhoon-7b-Q2_K.gguf Q2_K 2.547 GB smallest, significant quality loss - not recommended for most purposes
typhoon-7b-Q3_K_S.gguf Q3_K_S 2.963 GB very small, high quality loss
typhoon-7b-Q3_K_M.gguf Q3_K_M 3.293 GB very small, high quality loss
typhoon-7b-Q3_K_L.gguf Q3_K_L 3.575 GB small, substantial quality loss
typhoon-7b-Q4_0.gguf Q4_0 3.844 GB legacy; small, very high quality loss - prefer using Q3_K_M
typhoon-7b-Q4_K_S.gguf Q4_K_S 3.873 GB small, greater quality loss
typhoon-7b-Q4_K_M.gguf Q4_K_M 4.085 GB medium, balanced quality - recommended
typhoon-7b-Q5_0.gguf Q5_0 4.673 GB legacy; medium, balanced quality - prefer using Q4_K_M
typhoon-7b-Q5_K_S.gguf Q5_K_S 4.673 GB large, low quality loss - recommended
typhoon-7b-Q5_K_M.gguf Q5_K_M 4.798 GB large, very low quality loss - recommended
typhoon-7b-Q6_K.gguf Q6_K 5.554 GB very large, extremely low quality loss
typhoon-7b-Q8_0.gguf Q8_0 7.194 GB very large, extremely low quality loss - not recommended

Downloading instruction

Command line

Firstly, install Huggingface Client

pip install -U "huggingface_hub[cli]"

Then, downoad the individual model file the a local directory

huggingface-cli download tensorblock/typhoon-7b-GGUF --include "typhoon-7b-Q2_K.gguf" --local-dir MY_LOCAL_DIR

If you wanna download multiple model files with a pattern (e.g., *Q4_K*gguf), you can try:

huggingface-cli download tensorblock/typhoon-7b-GGUF --local-dir MY_LOCAL_DIR --local-dir-use-symlinks False --include='*Q4_K*gguf'