Add note on running the GGUF with llama-cpp-python (text embeddings, Linux x86_64, T4)
#4
by vinamrag-code - opened
Users who load this GGUF with the PyPI build of llama-cpp-python (0.3.36) get:
llama_model_load: error loading model: unknown model architecture: 'gemma-embedding2'
This PR adds a short subsection after "Quick Start" pointing to a community-built prebuilt CUDA wheel that can load the model.
Details:
- Works on: Linux x86_64, Python 3.13, CUDA 13 runtime, compute capability 7.5 GPUs (Colab T4).
- Does not work on: Windows, macOS, ARM, or other GPU generations.
- Tested with embeddinggemma-2-BF16.gguf for text embeddings only (not images, video or audio). All layers offloaded to the GPU, 768-dimensional output.
- Community-built, not affiliated with Unsloth. Build details and the exact llama.cpp commit are listed in the wheel repo's README: https://huggingface.co/vinamrag-code/llama-cpp-python-cuda-wheels
Related discussion: #
Happy to adjust the wording, placement, or remove the example if you prefer a link only.