iamdhruvhere/Omnivoice-FP16-CPU

OmniVoice Dynamic FP16 (CPU Optimized)

This repository provides a dynamic FP16 quantized LLM backbone for k2-fsa/OmniVoice. By dynamically casting FP16 weights during forward passes, the model size is reduced down to 1.40 GB for seamless deployment on CPU runtimes.

---

Technical Overview

• Base Model: k2-fsa/OmniVoice
• Target Layer: model.llm (Linear projections quantized to FP16)
• Weights Size: ~1.40 GB
• Precision: Dynamic FP16 Storage / Float32 Compute
• Inference Platform: CPU / Google Colab

---

Audio Sample

An example Hindi generation test is included:

• Audio File: test_fp16_success.wav
• Prompt: "ज़िंदगी की भागदौड़ में कभी-कभी बस ठहरकर गहरी सांस लेना ही सबसे बड़ा सुकून होता है।"

---

Installation

pip install -q huggingface_hub soundfile onnxruntime
pip install -q git+https://github.com/k2-fsa/OmniVoice.git

---

How to Run Inference

Because the LLM uses custom dynamic FP16 weight buffers, use the loader wrapper below:

import torch
import torch.nn as nn
import torch.nn.functional as F
from huggingface_hub import hf_hub_download
import soundfile as sf
from omnivoice import OmniVoice, OmniVoiceGenerationConfig

class DynamicFP16Linear(nn.Module):
    def __init__(self, in_features, out_features, bias=True):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.register_buffer("weight_fp16", torch.empty((out_features, in_features), dtype=torch.float16))
        if bias:
            self.register_buffer("bias", torch.empty(out_features, dtype=torch.float32))
        else:
            self.bias = None

    def forward(self, x):
        w = self.weight_fp16.to(x.dtype)
        return F.linear(x, w, self.bias)

def patch_fp16_linear(module):
    for name, child in module.named_children():
        if isinstance(child, nn.Linear):
            has_bias = child.bias is not None
            patched = DynamicFP16Linear(child.in_features, child.out_features, bias=has_bias)
            setattr(module, name, patched)
        else:
            patch_fp16_linear(child)

model = OmniVoice.from_pretrained(
    "k2-fsa/OmniVoice",
    device_map="cpu",
    dtype=torch.float32,
    load_asr=True
)
patch_fp16_linear(model.llm)

weights_path = hf_hub_download(
    repo_id="iamdhruvhere/Omnivoice-INT16-CPU",
    filename="omnivoice_llm_fp16.pt"
)
model.llm.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

gen_config = OmniVoiceGenerationConfig(
    num_step=64,
    guidance_scale=2.0,
    denoise=True
)

prompt = "नमस्ते, यह ओम्नीवॉइस का नया मॉडल है।"
with torch.no_grad():
    audio = model.generate(
        text=prompt,
        language="hi",
        generation_config=gen_config,
        speed=1.0
    )

sf.write("output.wav", audio[0], model.sampling_rate)
print("Audio saved successfully as output.wav!")

---

License

Subject to the original Apache 2.0 License.