PsarAI-Omni v0.1

Speech in, speech out, with one model thinking in between, for Khmer, English and French. There is no ASR, translation or text-to-speech model in the loop: Gemma-4-E2B-it hears the question through its own audio encoder, writes a short spoken reply, and a speech decoder turns Gemma's hidden states straight into speech units for a fast HiFi-GAN vocoder (Jarvis voice, or any voice from a few seconds of reference audio).

question audio ─► Gemma-4-E2B-it audio encoder + LoRA adapter (Khmer / English / French speech)
               ─► Gemma writes the reply (voice-assistant system prompt), sentence by sentence
each sentence  ─► base Gemma final hidden states + token embeddings ─► speech decoder ─► mHuBERT units
               ─► durations + pitch ─► unit HiFi-GAN ─► 24 kHz speech (Jarvis or cloned voice)

GGUF version (llama.cpp, one server, voice cloning): nphearum/PsarAI-Omni-GGUF.

Usage

python psarai_omni.py question.wav --out reply.wav            # --voice ref.wav to clone, --gemma PATH for a local Gemma

from psarai_omni import Omni
omni = Omni()                                   # downloads google/gemma-4-E2B-it unless Omni(gemma=PATH)
text, wav = omni.chat("question.wav")           # float32 at omni.sample_rate (24000)
for sentence, wav in omni.stream("question.wav", question="What is the speaker talking about? Answer in English."):
    play(wav)                                   # first sentence plays while Gemma is still writing
voice = omni.voice("reference.wav")
wav = omni.say("សួស្តី លោក។ Good evening, sir.", voice=voice)

Needs torch, transformers with Gemma 4, peft, torchvision (Gemma 4 processor), safetensors, numpy, soundfile, soxr (torchaudio for cloning). About 10 GB of GPU memory at peak with the per-layer embedding table kept in CPU RAM (the default).

Results (v0.1)

	value
Khmer speech transcription by the adapted Gemma (FLEURS, CER)	12.8% (base Gemma 30.7%)
Khmer speech → English translation (FLEURS, chrF)	34.9 (base Gemma 24.9)
English / French transcription (FLEURS, CER)	4.0% / 5.0%
spoken reply heard back by an ASR vs Gemma's reply text, Khmer (CER, n=4)	7.3%
spoken reply heard back by an ASR vs Gemma's reply text, English (CER, n=10)	11.7%
spoken reply heard back by an ASR vs Gemma's reply text, French (CER, n=2)	7.7%
first audio after the question ends (median, RTX 3090)	0.63 s

The speech decoder alone, on held-out sentences: Jarvis-style 4.2% CER, Gemma-style replies 12.0% (PsarAI-Speech T2U reading the same text: 4.0% / 10.1%).

Files

File	What
adapter/	LoRA on Gemma's audio encoder (r32) and LM attention (r8), audio projections trained in full
speech_decoder.safetensors	Gemma hidden states + token embeddings ─► mHuBERT units (CTC), durations, pitch (66M)
vocoder.safetensors, voices.safetensors	speaker-conditioned unit HiFi-GAN and the Jarvis embedding (from PsarAI-Speech)
campplus.safetensors, campplus.py	CAM++ speaker encoder for voice cloning
psarai_omni.py, psarai_speech.py	inference code (psarai_speech.py provides the T2U and vocoder classes)

Training

• Audio adapter: stage 1, transcription of Khmer (DDD, FLEURS, Jarvis, designed voices), English and French speech (6000 steps); stage 2, 45% transcription + 55% instructions (reply, describe, summarize, translate) whose targets were written by text-only Gemma from the transcript (self-distillation, 3000 steps).
• Speech decoder: initialized from the PsarAI-Speech T2U unit decoder; trained on 34k Jarvis clips (Gemma teacher-forced on their text) plus 31k of Gemma's own reply sentences with units from the PsarAI-Speech T2U.

Limitations

• Research release. Gemma's Khmer replies are weak: they often restate what was said instead of answering.
• Audio input is limited to 30 s (Gemma 4). Replies are short by design (system prompt).
• Base model terms: Gemma Terms of Use apply to the adapter and to outputs of the combined model.