yothinS/Typhoon_OCR_7B.gguf

Typhoon-OCR-7B - GGUF (Multi-Quantization)

This repository contains the GGUF weights for typhoon-ai/typhoon-ocr-7b, an advanced Vision-Language Model optimized for Optical Character Recognition (OCR) in Thai and English, developed by SCB 10X.

The models have been quantified using llama.cpp with CUDA acceleration, providing various sizes suitable for local deployment on hardware with limited resources (e.g., 16GB RAM).

---

📁 Available Files & Quantization Details

To run this Vision-Language Model (VLM), you need two components downloaded into the same directory:

1. The Main Text Model: (.gguf files listed below)
2. The Vision Projector Model (The Eyes): You MUST download typhoon-ocr-mmproj-f16.gguf to process images.

Main Models Table

File Name	Quantization Type	Size	Recommended Use Case
typhoon-ocr-Q2_K.gguf	2-bit	~2.5 GB	Extreme resource constraint / Testing purposes
typhoon-ocr-Q4_K_M.gguf	4-bit	~4.5 GB	Recommended Balance for 16GB RAM systems
typhoon-ocr-Q6_K.gguf	6-bit	~6.1 GB	High quality, minimal degradation of Thai vocabulary
typhoon-ocr-Q8_0.gguf	8-bit	~7.7 GB	Near-lossless precision, requires higher VRAM/RAM

Multi-Modal Component

• typhoon-ocr-mmproj-f16.gguf (~1.3 GB): The required vision adapter component. Works universally with all quant variants above.

---

🚀 How to Run Locally

1. Using llama.cpp (CLI)

You can run the model directly via the llama-cli using the multi-modal argument (--mmproj):

llama-cli -m typhoon-ocr-Q4_K_M.gguf \
          --mmproj typhoon-ocr-mmproj-f16.gguf \
          --image /path/to/your/document.png \
          -p "<|im_start|><|role|>user<|im_end|><|image|>Extract text from this image.<|im_start|><|role|>assistant<|im_end|>" \
          -n 2048

2. Using Ollama (Custom Modelfile)

To run this model natively in Ollama, create a file named Modelfile in your directory and add the following configuration:

# Path to the main model and vision projector
FROM ./typhoon-ocr-Q4_K_M.gguf
MMPROJ ./typhoon-ocr-mmproj-f16.gguf

# Set Parameters
PARAMETER temperature 0.1
PARAMETER stop <|im_end|>
PARAMETER stop <|endoftext|>

# Template for Qwen2.5-VL / Typhoon OCR
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""

Then compile and run it via terminal:

ollama create typhoon-ocr -f ./Modelfile
ollama run typhoon-ocr

---

📝 Usage & Prompt Notes

As stated by the original developers (Typhoon AI), this model is highly specialized for Document Parsing and OCR. Standard conversational chit-chat prompts might yield suboptimal results. Always structuralize your queries towards data extraction or text transcribing.

🤝 Acknowledgements

Credits go to Typhoon AI (SCB 10X) for developing and releasing the original typhoon-ocr-7b weights. This GGUF version is optimized purely for the open-source community to leverage efficient local edge inference.