iamkushagratomar/glm-ocr-gguf

GLM-OCR GGUF

GGUF conversions of GLM-OCR for use with llama.cpp and compatible runtimes. The repo contains an F16 conversion plus several quantized variants so you can pick the size/quality trade-off that suits your hardware.

Note: These are unofficial community conversions. All credit for the model goes to the original authors. Please refer to the original model card for the architecture, training details, intended use, and license terms.

Available files

File	Quant	Approx. size	Notes
glm-ocr-f16.gguf	F16	~1.9 GB	Full 16-bit precision, reference quality
glm-ocr-Q8_0.gguf	Q8_0	~1.0 GB	Near-lossless, recommended for OCR
glm-ocr-Q5_K_M.gguf	Q5_K_M	~0.7 GB	Very good quality
glm-ocr-Q5_K_S.gguf	Q5_K_S	~0.66 GB	Slightly smaller than Q5_K_M
glm-ocr-Q4_K_M.gguf	Q4_K_M	~0.6 GB	Good balance of size and quality
glm-ocr-Q4_K_S.gguf	Q4_K_S	~0.57 GB	Slightly smaller than Q4_K_M
glm-ocr-Q3_K_M.gguf	Q3_K_M	~0.5 GB	Noticeable quality loss
glm-ocr-Q3_K_S.gguf	Q3_K_S	~0.47 GB	Smallest, lowest quality

Sizes are approximate. See the Files and versions tab for exact sizes.

Which file should I use?

OCR is unusually sensitive to quantization. A single wrong character counts as an error, and small models lose accuracy faster than large ones at low bit widths. Since this model is under 1B parameters, the size savings from aggressive quantization are small in absolute terms.

• Best accuracy: Q8_0 (or f16 if you have the memory).
• Good compromise: Q5_K_M or Q4_K_M.
• Q3 variants: only if you are very tight on memory. Expect more character errors, repeated tokens, and garbled output on dense or small text.

If OCR quality looks off with a low-bit quant, try Q8_0 on the same image before drawing conclusions.

Usage

Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # omit the flag for CPU-only
cmake --build build --config Release -j$(nproc)

Use a recent llama.cpp build. Support for this architecture is new, so older builds may fail to load the model.

Download

hf download YOUR_USERNAME/glm-ocr-gguf glm-ocr-Q8_0.gguf --local-dir .

Text-only sanity check

./build/bin/llama-cli -m glm-ocr-Q8_0.gguf -ngl 99 -c 4096 -p "Hello"

This only exercises the text decoder. It does not process images.

Image OCR

GLM-OCR is a vision-language model. To process images you need the vision projector (mmproj) file in addition to the language model GGUF, and a llama.cpp build that supports this architecture's vision tower:

./build/bin/llama-mtmd-cli \
  -m glm-ocr-Q8_0.gguf \
  --mmproj mmproj-glm-ocr-F16.gguf \
  --image page.png \
  -p "Extract all text from this image." \
  -ngl 99

Or serve it over an OpenAI-compatible API:

./build/bin/llama-server -m glm-ocr-Q8_0.gguf --mmproj mmproj-glm-ocr-F16.gguf -ngl 99 -c 8192

Status of vision support: the mmproj filename above is a placeholder. If an mmproj file is not listed in this repo's files, the vision encoder has not been converted here, and the GGUFs in this repo can only be used as text decoders. Check the llama.cpp release notes for current GLM-OCR multimodal support.

Memory notes

• Image inputs produce many tokens, so the KV cache grows quickly. If you hit out-of-memory errors, lower the context (-c 4096) or offload fewer layers (-ngl).
• A 4 GB GPU is enough for Q8_0 with a modest context length.

Conversion details

• Converted from the original Hugging Face checkpoint with llama.cpp's convert_hf_to_gguf.py (--outtype f16).
• Quantized from the F16 file with llama-quantize, no importance matrix.
• No fine-tuning or other modification of the weights was performed.

To reproduce:

python convert_hf_to_gguf.py /path/to/GLM-OCR --outfile glm-ocr-f16.gguf --outtype f16
./build/bin/llama-quantize glm-ocr-f16.gguf glm-ocr-Q8_0.gguf Q8_0
./build/bin/llama-quantize glm-ocr-f16.gguf glm-ocr-Q4_K_M.gguf Q4_K_M

Limitations

• Quantization introduces small numerical differences from the original model. Outputs will not be identical to the full-precision checkpoint, and lower-bit files are more likely to make character-level mistakes.
• Accuracy on handwriting, low-resolution scans, dense tables, and complex layouts depends heavily on the original model and the prompt you use.
• Always verify OCR output before using it for anything where errors matter (financial, legal, medical, etc.).

License

These files are derived from the original model and are distributed under the same license as the base model. Please check the original repository for the authoritative license text, and make sure the license field in this card's metadata matches it.

Acknowledgements

• The original GLM-OCR authors for the model.
• llama.cpp and the GGUF format by Georgi Gerganov and contributors.