dtsj89/puki-ears

Puki's ears: kol-stream for phones (ONNX, int8)

kol-stream by MaorB (Hebrew and English speech recognition on Google's EmbeddingGemma 2 audio encoder, trained on ivrit.ai's data), converted to ONNX for Puki, a toddler game that listens to the child on the phone.

• encoder.onnx: audio frames (1, M, 320) float32 -> the joiner's encoder side (1, T, 512). The log-mel features are inside the graph; the frames are the raw audio cut as Gemma's feature extractor cuts it (160 zeros in front, 320 samples every 160, 16 kHz).
• step.onnx: encoder rows (N, 512) and the last two pieces (1, 2) int64 -> piece scores (N, 2048); piece 0 is blank. Greedy search: see KolEars.kt in the Puki repository.
• tokens.txt: the 2,048 sentencepiece pieces, one per line.

The matrix multiplies are int8 (dynamic quantization), and the graphs are already optimized (ONNX Runtime's "extended" level), so they load as they are. Made by train/kolstream_export.py. On the test clips the output matches the original model; the int8 build differs only on a rare name. Measured on a Galaxy Tab A7 Lite (4 threads): a word in about 0.55 s, a 2 s sentence in 1.4 s.

License

Apache 2.0, as the original model and EmbeddingGemma 2. The Hebrew training data is ivrit.ai's (CC BY 4.0 with an AI-training purpose clause); see the original model card for details and limitations (no punctuation, less accurate than ivrit.ai's Whisper turbo).