jatshi/trusted-full-duplex-agent

TFD-STAR 2.0 · Trusted Full-Duplex Speech Agent

Project-owned adapters and controllers for a trustworthy full-duplex speech agent built on MiniCPM-o 4.5. Code, real-time integration delta, evidence and a deep Chinese learning manual are in Jatshi/trusted-full-duplex-agent.

Version 2.0

Version 2.0 connects the research artifacts to a real browser → Gateway → Worker → GPU backend runtime:

• MiniCPM-o 9B GRPO LoRA is loaded into the internal LLM;
• the released 10-32-3 turn-taking MLP runs on real 100 ms audio frames;
• a new Faster-Whisper-small safety path grounds TrustGate on live audio;
• execute / clarify / stop / listen routing is exposed through metrics;
• client-immediate barge-in and reasoned listen messages prevent self-cutoff;
• deterministic clarify/stop speech now uses one consistent female voice.

ASR weights and base-model weights are dependencies and are not redistributed here. Faster-Whisper is an inference component, not a model trained by TFD-STAR.

V2 asset layout

v2.0/
  minicpmo9b-grpo-lora/       deployed PEFT adapter (r=16, alpha=32)
  turn-taking-mlp/            learned_mlp.pt + feature/config metadata
  configs/                    Gate, turn-taking and runtime environment template
  evidence/                   POC result, train/eval reports and latency summary
  SHA256SUMS.txt               integrity hashes

Historical sweep checkpoints remain at their original repository paths so prior reports and links do not break.

Deployed LoRA

The v2.0 runtime uses the most conservative positive MiniCPM-o 9B sweep:

item	value
base	MiniCPM-o 4.5 internal LLM
PEFT	LoRA, r=16, alpha=32, dropout=0.05, all-linear
steps / learning rate	20 / 1e-5
reward	0.3157 → 0.3305, delta +0.0148
interpretation	small positive result near the sampling-noise band

Other 9B sweeps regress. They are deliberately retained as negative evidence; this release does not claim broad RL improvement on the strong base model.

Turn-taking controller

10 causal acoustic features → Linear(10,32) → ReLU → Linear(32,3)
                                                   hold/take/backchannel

• weighted frame-level cross entropy, Adam, lr=0.01, 300 epochs;
• 18,190 training frames / 5,572 validation frames;
• validation frame accuracy 96.46%;
• four-frame confirmation gives 0% false takeover and 0% miss on the held-out parameterized synthetic event set, median take latency 553.9 ms.

These are synthetic-domain results, not a claim of SOTA spontaneous-dialog turn-taking.

GRPO objective

For each prompt, a group of sampled answers receives task rewards and is z-normalized within the group. The project implementation uses:

A_i = (r_i - mean(r)) / (std(r) + eps)
L = -mean(A_i * log pi_theta(y_i|x))
    + beta * mean(log pi_theta - log pi_ref)

Safety samples combine content consistency and action alignment. Barge-in samples combine character-bigram Jaccard with BLEU-1. In LoRA mode, disable_adapter() provides the frozen-base reference without loading a second 9B model.

Runtime usage

Clone the GitHub project and follow integrations/minicpmo45-demo/README.md. The integration package is a source patch plus project-owned files; it does not vendor the upstream Demo tree.

Evidence and limitations

• Latest short live probe on RTX 4080 SUPER 32 GB: backend median 0.742 s, client median 0.824 s, one session through a local SSH tunnel.
• Public relay latency was materially worse than GPU-local inference.
• The 9B GRPO gain is small; several variants are negative.
• Turn-taking training data is parameterized synthetic audio.
• Production serving still needs TLS, auth, concurrency and long-soak tests.

Training/evidence data: jatshi/trusted-full-duplex-agent-data.