RelayS2S

Checkpoints for RelayS2S: Dual-Path Speculative Generation for Real-Time Dialogue.

A duplex speech-to-speech model that drafts a short response prefix speculatively and hands off to a cascaded ASR → LLM pipeline for the continuation. A small learned verifier decides whether the drafted prefix is safe to say out loud.

Code:

License note. Tagged apache-2.0 because the backbone is Qwen2.5-0.5B, which is Apache-2.0. Confirm this is the licence you intend for the trained weights and the speech encoder before relying on it.

Contents

Path	What it is
s2s/s2s.ckpt	Fast-path duplex S2S model (PyTorch Lightning checkpoint, 6.0 GB)
tokenizer/	Qwen2.5-0.5B tokenizer used by the S2S model
verifier/best_model.pt	Prefix verifier (~170K params)
verifier/tokenizer/	Tokenizer the verifier uses to count prefix tokens
pretrained/speech_encoder.pt	Conformer speech encoder, stage-0 initialisation
pretrained/adapter.pt	CNN adapter, stage-0 initialisation

s2s.ckpt is a training checkpoint: about 58% of it is Adam optimizer state that inference does not use. The weights themselves are 2.79 GB.

Two things you need to know before using this

1. The model only detects end-of-turn over exact digital zero.

It was trained on synthesised dialogues whose inter-turn gaps are literally 0.0 — 100% zero samples. It learned that as the end-of-turn cue. Real microphone audio never produces it: background noise at RMS 1e-5, about −100 dB and completely inaudible, is enough to stop [BOS] firing entirely and silently. There is no error; the model simply never responds.

Live microphone input therefore needs a noise gate that zeroes quiet frames before the model sees them. A per-frame RMS threshold of 0.01 with a two-frame hangover works. Dataset replay needs no gate.

2. The verifier threshold is 0.405, not 0.5.

That is the tuned value in verifier/inference_config.json, giving 65.6% recall on bad prefixes at 96.0% acceptance of good ones — roughly a 9.6% discard rate. Using 0.5 will discard more than intended.

Special tokens

The S2S tokenizer adds five control tokens to the Qwen vocabulary. They must be registered before loading, or the embedding matrix will not match: [SIL] (stay silent), [BOS] (start speaking), [EOS], [BOC] (backchannel), [STP] (stop — user barge-in), [HOD].

The model consumes a 160 ms frame clock and emits exactly one token per frame.

Citation

@article{relays2s2025,
  title  = {RelayS2S: Dual-Path Speculative Generation for Real-Time Dialogue},
  year   = {2025},
  eprint = {XXXX.XXXXX},
  archivePrefix = {arXiv}
}