peonist-ai/halogen-npu-embeddinggemma-2

embeddinggemma-2 for the Ryzen AI NPU

EmbeddingGemma 2, converted to run on the Ryzen AI NPU of AMD Strix Halo inside halogen-flash-server, beside the Flash model on the GPU and behind the same port. Questions go to the Discord. Bugs go to the server repository's issues.

These files run only inside the halogen-flash-server container, on the NPU. They will not load in transformers or any other runtime.

What is here

devices/                    the NPU program (text, pictures and audio)
embeddinggemma-2.hnpw       the weights, converted for the NPU
tokenizer/                  the tokenizer
LICENSE                     the Apache License 2.0

Use it

Start halogen-flash-server 0.18.3 or later with HALOGEN_DOWNLOAD on and HALOGEN_NPU_MODELS=embeddinggemma-2. The server fetches these files into /models/npu/embeddinggemma-2/ and checks each one's size and checksum before it starts.

The files at the head of this repository need halogen-flash-server 0.18.12 or later; each server version downloads the revision it was released with, so an older server keeps working unchanged.

It puts texts, pictures and audio clips in one 768-dimension space on /v1/embeddings. A text is up to 8,192 tokens. A picture is an image_url item, an audio clip an input_audio item (WAV, up to 30 seconds). prompt_name picks the model's query or document prompt.

The host needs the NPU driver, XRT with its NPU plugin, and the GPU's fabric clock held while the NPU works beside the GPU. The setup and every request shape are in docs/NPU.md.

Source and license

Converted from google/embeddinggemma-2 at revision 914f7f89142e33e77833254d9c9b90c3cef7303b. The weights here are changed from the original: converted to the NPU's number formats and layout. Apache-2.0, as the source. The license text is in LICENSE. The source's model card asks that deployments follow the Gemma Prohibited Use Policy. The name EmbeddingGemma is used here only to say which model these files come from.