nanash66/typhoon-ocr1.5-2b-ROCMFP4-GGUF

Typhoon-OCR 1.5 2B — ROCmFP4 GGUF (Thai iMatrix Calibrated)

This repository provides ROCmFP4 quantized GGUF weights for typhoon-ai/typhoon-ocr1.5-2b, calibrated specifically for high-fidelity Thai document OCR using Importance Matrix (iMatrix).

The text backbone is quantized to Q4_0_ROCMFP4 (UE4M3-scale experimental with Q6_K token embeddings), while the Vision Projector (mmproj-f16.gguf) remains unquantized in full FP16 to ensure zero loss in visual document resolution.

---

🌟 Key Highlights

• 41% Smaller than Q8_0 (67% Smaller than FP16): Text backbone compressed down to 1.08 GB (from 3.28 GB FP16).
• High-Precision Thai Preservation: Calibrated on a custom Thai Calibration Mix (70% general Thai corpus + 30% complex Thai official & legal documents), preventing degradation of Thai vowels, tone marks (วรรณยุกต์), and specialized vocabulary.
• 99.61% Empirical Character Match: Verified against the Q8_0 baseline on multi-page official public sector and legal documents with zero missing tone marks.
• Optimized for AMD RDNA3 / RDNA3.5: High inference speed (approx. 28.6 tokens/sec decode) and ultra-low VRAM footprint (approx. 2.1 GB at 16K context) on AMD Radeon 890M / 880M / 780M iGPUs (Strix Point / Strix Halo / Phoenix).

---

📊 Benchmark & Quality Verification

Empirical test on multi-page dense Thai technical documents:

Metric	Full FP16	Q8_0 Baseline	Q4_0_ROCMFP4 (This Model)
Model Size (LLM Backbone)	3.28 GB	1.83 GB	1.08 GB (📉 -41% vs Q8)
Vision Projector (mmproj)	782 MB	782 MB	782 MB (FP16 unquantized)
Total VRAM (16K Context, 2 Slots)	~5.2 GB	~3.2 GB	~2.1 GB (⚡ -34% VRAM)
Decode Speed (Radeon 890M)	~18 tok/s	~21–25 tok/s	~28.6 tok/s (🚀 +20–30%)
Avg Time per Full PDF Page	~65s	~45s	~29.1s
Thai OCR Fidelity vs Q8_0	100%	100%	99.61% match*

*Note: Thai characters and tone marks are 100% identical; the 0.39% difference is due to minor markdown bold formatting variations.

📁 Repository Files

File	Size	Description
typhoon-ocr1.5-2b-Q4_0_ROCMFP4-imatrix.gguf	1.08 GB	Quantized text model backbone (Q4_0_ROCMFP4 with Q6_K token embeddings)
typhoon-ocr1.5-2b.mmproj-f16.gguf	781 MB	Full FP16 multimodal vision projector (Required for image/PDF input)

---

🚀 Quick Start Guide

1. Requirements

• llama.cpp with ROCmFP4 support (or standard Vulkan / HIP-enabled llama.cpp builds).
• AMD Ryzen AI 300 series (Radeon 890M / 880M), Ryzen 7000/8000 (Radeon 780M), Strix Halo, or discrete AMD Radeon GPUs.

2. Launching llama-server

Run llama-server exposing an OpenAI-compatible vision endpoint:

llama-server \
  -m typhoon-ocr1.5-2b-Q4_0_ROCMFP4-imatrix.gguf \
  --mmproj typhoon-ocr1.5-2b.mmproj-f16.gguf \
  -ngl 99 \
  -c 16384 \
  -np 2 \
  --port 8080 \
  --host 0.0.0.0 \
  -fa on

Note on Context Window: Set context (-c) to at least 8192 per slot (or 16384 for -np 2) because each high-resolution document image generates ~2,000–2,500 visual tokens from the vision encoder.

3. Recommended Prompt & Best Practices (From Typhoon AI)

Typhoon-OCR 1.5 is a task-specific model fine-tuned to achieve optimal OCR and structural extraction when invoked with the official prompt structure and image preprocessing rules specified in the official model card.

A. The Official Standard Prompt

For high-fidelity markdown formatting, clean HTML tables, and figure descriptions, use the exact official prompt:

prompt = """Extract all text from the image.
Instructions:
- Only return the clean Markdown.
- Do not include any explanation or extra text.
- You must include all information on the page.
Formatting Rules:
- Tables: Render tables using <table>...</table> in clean HTML format.
- Equations: Render equations using LaTeX syntax with inline ($...$) and block ($$...$$).
- Images/Charts/Diagrams: Wrap any clearly defined visual areas (e.g. charts, diagrams, pictures) in:
<figure>
Describe the image's main elements (people, objects, text), note any contextual clues (place, event, culture), mention visible text and its meaning, provide deeper analysis when relevant (especially for financial charts, graphs, or documents), comment on style or architecture if relevant, then give a concise overall summary. Describe in Thai.
</figure>
- Page Numbers: Wrap page numbers in <page_number>...</page_number> (e.g., <page_number>14</page_number>).
- Checkboxes: Use ☐ for unchecked and ☑ for checked boxes."""

B. Fixed Image Dimension & Resizing Rule

Image Scaling: The model was trained with a target dimension of 1800 px. Resizing input images so that their longest dimension does not exceed 1800 px ensures optimal visual token allocation, prevents hallucination, and preserves OCR accuracy.

from PIL import Image

def resize_if_needed(img: Image.Image, max_size: int = 1800) -> Image.Image:
    width, height = img.size
    if width > max_size or height > max_size:
        if width >= height:
            scale = max_size / float(width)
            new_size = (max_size, int(height * scale))
        else:
            scale = max_size / float(height)
            new_size = (int(width * scale), max_size)
        return img.resize(new_size, Image.Resampling.LANCZOS)
    return img

C. Option 1: Using the Official typhoon-ocr Library

Because llama-server provides an OpenAI-compatible endpoint, you can use the official typhoon-ocr library directly with your local server:

pip install typhoon-ocr -U

from typhoon_ocr import ocr_document

# Connect directly to your local llama-server
markdown = ocr_document(
    "document_page.png",
    model="typhoon-ocr",
    figure_language="Thai",
    task_type="v1.5",
    base_url="http://localhost:8080/v1",
    api_key="no-key"
)
print(markdown)

D. Option 2: Direct Python Client (Requests / OpenAI API)

import base64
import io
import requests
from PIL import Image

def ocr_image(image_path: str, server_url: str = "http://localhost:8080/v1/chat/completions"):
    # 1. Resize image according to official 1800px guideline
    img = Image.open(image_path)
    img = resize_if_needed(img, 1800)
    
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    img_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")

    # 2. Official prompt
    prompt = """Extract all text from the image.
Instructions:
- Only return the clean Markdown.
- Do not include any explanation or extra text.
- You must include all information on the page.
Formatting Rules:
- Tables: Render tables using <table>...</table> in clean HTML format.
- Equations: Render equations using LaTeX syntax with inline ($...$) and block ($$...$$).
- Images/Charts/Diagrams: Wrap any clearly defined visual areas (e.g. charts, diagrams, pictures) in:
<figure>
Describe the image's main elements (people, objects, text), note any contextual clues (place, event, culture), mention visible text and its meaning, provide deeper analysis when relevant (especially for financial charts, graphs, or documents), comment on style or architecture if relevant, then give a concise overall summary. Describe in Thai.
</figure>
- Page Numbers: Wrap page numbers in <page_number>...</page_number> (e.g., <page_number>14</page_number>).
- Checkboxes: Use ☐ for unchecked and ☑ for checked boxes."""

    payload = {
        "model": "typhoon-ocr1.5-2b",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}}
                ]
            }
        ],
        "temperature": 0.0,
        "max_tokens": 8192
    }

    response = requests.post(server_url, json=payload, timeout=300)
    return response.json()["choices"][0]["message"]["content"]

# Example usage:
# print(ocr_image("sample_page.png"))

4. Intended Uses & Limitations

• Task-Specific: This model is designed specifically for document OCR using the prescribed prompt format. It does not include conversational chat guardrails or general Visual Question Answering (VQA) capabilities.
• Hallucination Risk: Due to the nature of vision-language models, slight hallucinations may occur on degraded, low-resolution, or heavily blurred images. Always review critical extractions.

---

💖 Credits & Acknowledgements

We express our deep appreciation and credit to the original creators and open-source contributors:

1. Original Model Creators:
   • SCB 10X / Typhoon AI Team for creating the state-of-the-art Thai Vision-Language OCR model:
     • Model: typhoon-ai/typhoon-ocr1.5-2b
     • Organization: SCB 10X
2. GGUF Conversion:
   • mradermacher (Nico) for the initial GGUF conversion and hosting the base FP16 GGUF weights:
     • Model Repository: mradermacher/typhoon-ocr1.5-2b-GGUF
3. ROCmFP4 Runtime & Engine:
   • charlie12345 for the ROCmFP4 experimental implementation in charlie12345/rocmfp4-llama enabling native 4-bit float acceleration on AMD iGPUs and Vulkan/HIP.
   • Georgi Gerganov & the llama.cpp community for the foundational inference framework.
4. Calibration Dataset:
   • eaddario for the eaddario/imatrix-calibration dataset used in the Thai token importance matrix calculation.

---

📜 License

This model inherits the Apache 2.0 license from the base scb10x/typhoon-ocr1.5-2b model.