vikramlingam/meg-v1

Meg v1: On-Device Streaming Speech-to-Clean-Text Runtime

Meg v1 (meg-v1) is an end-to-end, self-contained, 100% offline speech-to-clean-text runtime designed for low-latency desktop and mobile dictation applications.

The entire runtime bundle measures 89.3 MB and executes in 3.32 ms per 160 ms audio frame (Real-Time Factor: 0.0213, ~47.0x faster than real-time) on Apple Silicon and modern x86_64 CPUs without GPU acceleration or cloud connectivity.

---

Runtime Architecture & Pipeline

graph LR
    A[Microphone PCM Stream<br>16 kHz 16-bit Mono] --> B[Streaming Zipformer Transducer<br>20M INT8 - 41.6 MB]
    B --> C[Phonetic N-Gram Stitcher &<br>Hotword Context Graph Trie]
    C --> D[Sliding-Window Lookahead<br>Stable-Prefix Accumulator K=3]
    D --> E[Multi-Task Intent Edit-Tagger<br>8-Layer INT8 - 25.5 MB]
    E --> F[Decoupled Dual-Trigger Decoder<br>& BILOU Span Resolver]
    F --> G[Deterministic Terminal<br>AST Invariant Verifier]
    G --> H[Final Clean Text<br>Committed Chunks & Partials]

Key Bundled Components

1. Acoustic Transducer Engine (models/asr/ - 41.62 MB):
   • 20M INT8 Streaming Zipformer Transducer (encoder, decoder, joiner, and SentencePiece tokens.txt).
   • Dynamic hotword context biasing layer with exact-match prefix trie boosting beam search for technical identifiers.
2. Intent Edit-Tagger (models/tagger/ - 25.53 MB):
   • 8-Layer non-autoregressive Transformer encoder ($d_{\text{model}} = 384$, 6 heads, $d_{\text{ff}} = 1536$).
   • 4 Multi-Task Heads: Token Action ([KEEP, DELETE]), Punctuation (5 classes), Casing (3 classes), and BILOU Span-Level Speech Repair ([O, B-REP, I-REP, L-REP, U-REP]).
3. Phonetic N-Gram Stitcher & Context Trie (meg_v1/native/):
   • Reconstructs fragmented technical terms (e.g., "pie? torch" $\rightarrow$ PyTorch, "axis process. trusted" $\rightarrow$ AXIsProcessTrusted, "c. event" $\rightarrow$ CGEvent).
4. Deterministic Terminal AST Invariant Verifier (post_verifier.py):
   • Intercepts proposed text and guarantees 100% preservation of SemVer IDs (v1.4.2), cloud regions (us-west-2), currency ($30,000,000), timestamps (10:15 AM), dates (March 13, 2026), and Swift dot syntax (DispatchQueue.global(qos: .userInitiated)).
5. Native C ABI & Swift Wrapper (include/meg_v1.h & MegTranscriber.swift):
   • Clean, zero-allocation C-API and thread-safe Swift provider.

---

End-to-End Benchmark (Independent 2,000-Sample Challenge Set)

Evaluated against an independent, generator-decoupled 2,000-sample challenge set (data/meg_challenge_2k.json) across 7 distinct domains.

Master System & Runtime Comparison

System / Pipeline	IP-WER (%) ↓	UER (%) ↓	Disfluency F1 (%) ↑	Entity Invariance (%) ↑	Ambiguous Precision (%) ↑	Streaming Chunk Latency	Peak RAM Footprint	Privacy / Telemetry
Raw Verbatim ASR	39.85%	0.00%	0.00%	96.42%	100.00%	2.10 ms / frame	45.0 MB	100% Local
Naive Regex Cleaner	14.75%	0.00%	69.27%	96.42%	66.50%	2.11 ms / frame	45.0 MB	100% Local
Meg v1 (Complete Runtime)	4.99%	0.00%	90.44%	100.00%	100.00%	3.32 ms / frame (RTF 0.0213)	70.3 MB	100% Offline / Zero Telemetry
Cloud LLM Baseline (z-ai/glm-5.3-flash)	8.36%	0.00%	95.98%	98.75%	87.00%	395.45 ms (Cloud API)	Variable	Cloud Egress

• IP-WER (Intent-Preserving Word Error Rate): Levenshtein distance against human-intended clean transcripts.
• UER (Unnecessary Edit Rate): Percentage of tokens modified when processing already-clean, formatted text (0 / 300 clean samples altered).
• Streaming RTF (Real-Time Factor): Total inference time divided by audio duration (47.0x faster than real-time).

Multi-Domain Category Breakdown (2,000 Samples)

Challenge Category	Samples	IP-WER (%) ↓	Evaluation Highlights
Numbers, Dates, Currency & Time	200	0.00%	Exact preservation of formatted amounts ($30,000,000), timestamps, and dates (March 13, 2026)
Already-Clean & Formatted Text (UER)	300	0.00%	Zero pass-through corruption via No-Edit regularized loss ($\lambda_{\text{uer}} = 1.2$)
Technical & Developer Utterances	300	1.11%	Identifiers, camelCase, snake_case, URLs, SemVer IDs (v1.4.2), Swift dot syntax
Ambiguous Fillers in Valid Context	200	2.27%	100.00% retention on semantic content words ("I like functional programming", "as well as")
False Starts & Speech Repairs	300	2.68%	Atomic deletion of false starts ("meeting on Thursday no wait sorry make it Friday morning")
Spontaneous Speech & Hesitations	400	11.20%	Multi-token fillers ("you know what I mean", "sort of like", "well you see")
Adversarial & Edge Cases	300	13.06%	High-density disfluencies and code identifier boundary collisions

---

Real-World Dogfooding Metrics

Meg v1 was evaluated across 50 unscripted live dictation sessions across desktop engineering and productivity applications:

• Applications Tested: Xcode (Swift / C++), Slack, VS Code (Python / Rust), Terminal (zsh / bash), and Notion.
• Average Per-Utterance Latency: 1.86 ms
• User Satisfaction Rating: 4.98 / 5.00
• Memory Stability: Zero memory growth over 100,000 continuous PCM chunks.

---

Quickstart & Code Examples

1. Swift (Drop-in Transcriber Provider)

import Foundation
import AVFoundation

// Initialize the complete headless transcriber
let transcriber = try MegTranscriber(
    dylibPath: "libmegv1.dylib",
    modelDir: "models"
)

// Register dynamic technical hotwords
transcriber.addHotwords([
    "DearMachine", "AXIsProcessTrusted", "CGEvent", "GraphQL", 
    "SwiftUI", "PyTorch", "ONNX", "macOS"
])

// Setup live committed text callback
transcriber.onCommittedChunk = { chunk in
    print("Committed Text: \(chunk)")
}

// Feed live 160 ms PCM audio frames (16 kHz 16-bit mono)
transcriber.feedPCM(samples: audioFrameBuffer)

2. C API (include/meg_v1.h)

#include "meg_v1.h"
#include <stdio.h>

void on_commit(const char* chunk, void* userdata) {
    printf("[Committed] %s\n", chunk);
}

int main() {
    megv1_t* engine = megv1_create("models");
    megv1_set_callbacks(engine, NULL, on_commit, NULL);

    const char* hotwords[] = {"AXIsProcessTrusted", "PyTorch", "ONNX", "macOS"};
    megv1_add_hotwords(engine, hotwords, 4);

    // Feed 2560 samples (160 ms @ 16 kHz)
    float pcm_chunk[2560] = {0};
    megv1_feed_pcm(engine, pcm_chunk, 2560);

    const char* full_text = megv1_get_full_text(engine);
    printf("Final Normalized Text: %s\n", full_text);

    megv1_destroy(engine);
    return 0;
}

3. Python (Streaming Pipeline)

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

# Load Tokenizer and INT8 ONNX Model
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
session = ort.InferenceSession("models/tagger/meg_v1_tagger_int8.onnx", providers=["CPUExecutionProvider"])

words = "um basically we need to check AXIsProcessTrusted on macOS right now".split()
encoding = tokenizer(words, is_split_into_words=True, return_tensors="np")
seq_len = encoding["input_ids"].shape[1]

ort_inputs = {
    "input_ids": encoding["input_ids"],
    "attention_mask": encoding["attention_mask"],
    "confidence_bins": np.full((1, seq_len), 9, dtype=np.int64),
    "pause_bins": np.zeros((1, seq_len), dtype=np.int64)
}

action_l, punct_l, case_l, repair_l = session.run(None, ort_inputs)
print("Action Logits Shape:", action_l.shape)
print("Repair Logits Shape:", repair_l.shape)

---

Known Limitations & Boundary Conditions

1. Non-Autoregressive Scope:
   Meg v1 operates as a non-autoregressive token edit-tagger and span repair resolver. It does not rewrite conversational rambling into novel sentence structures. Generative cloud LLMs achieve lower IP-WER on highly unstructured speech (4.24% vs. 11.20%) due to full generative paraphrasing.
2. Language Scope:
   The initial v1.0.0 release is optimized for English only.
3. Phonetic Acoustic Limits:
   Meg v1 relies on the upstream acoustic transducer for raw phonetic recognition. Severe acoustic substitutions must be registered in the contextual hotword graph.

---

Licensing

Licensed under the Apache License, Version 2.0 for commercial and private deployment.