Streaming Turn

Streaming, audio-only turn-taking model for Pipecat voice agents. On every 40 ms of user audio it gives the probability that the user's turn is over, and whether speech over the bot is a backchannel or a floor-take. v0.6 adds a cue, while the user is still speaking, that their turn is about to end, and v0.7 a cue for the bot to backchannel while the user is mid-turn. It never sees a transcript.

  • 4.0M parameters (4.2M for v0.6 and v0.7): log-mel, causal conv encoder, GRU. 16 kHz mono in, 25 frames a second out.
  • 0.37 ms per 40 ms chunk on one Apple M-series core (ONNX Runtime).
  • English only. Experimental: little live testing, v0.5 has not yet been compared with v0.4 live, and nobody has yet heard a bot act on the outputs v0.6 and v0.7 add.

Versions

Folder End the turn when
streaming-turn-v0.7 (latest, experimental) as v0.5: eot ≥ 0.6, or eot_late ≥ 0.4
streaming-turn-v0.6 (experimental) as v0.5: eot ≥ 0.6, or eot_late ≥ 0.4
streaming-turn-v0.5 eot ≥ 0.6, or eot_late ≥ 0.4
streaming-turn-v0.4 eot ≥ 0.5, or eot_late ≥ 0.4
streaming-turn-v0.2 eot ≥ 0.5, or 0.8 s after the last sound if eot reached 0.3

v0.4 adds eot_late, a second end-of-turn output for a user who has gone quiet. v0.5 is v0.4 trained twice as long, with synthetic sessions of a person talking to an agent (pauses while dictating, one-word answers, "mm-hmm" over the bot) in a tenth of each batch.

v0.6 is v0.5 with one more output, eot_soon: while the user is still speaking, the probability that their turn ends within 240 ms and within 480 ms. An app can use it to start a lead-in ("ok, so...") or to get a reply ready before the stop. It comes from a small head (a GRU of 96 units) trained on v0.5's frozen network, so every other output is v0.5's, bit for bit, and an agent ends turns exactly as it does on v0.5. It has one more state tensor to feed back, cue_state.

v0.7 is v0.6 with one more output, bc_cue: while the user is speaking, the probability that a listener would start a backchannel ("mm-hmm", "right") within half a second. Read through a check that the user is mid-turn and a budget of a few a minute, it says when a bot could backchannel. It comes from the same head as eot_soon, with no new training, so every other output is v0.6's, bit for bit, and an agent that ignores it behaves exactly as it does on v0.6. The model gives the moment: it does not make the audio or choose the words.

Each folder holds streaming-turn-fp32.onnx (the streaming model), model.json (tensor names, chunk size, class lists) and a .pt file (PyTorch config and weights). v0.7's .pt file is the same file as v0.6's, which already held the new output's weights.

Use with Pipecat

agent.turn_analyzers is in the code repository, jptaylor/streaming-turn.

from pathlib import Path

from huggingface_hub import snapshot_download
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import BotStartedSpeakingFrame, BotStoppedSpeakingFrame
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContext,
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

from agent.turn_analyzers import StreamingTurnAnalyzer, StreamingTurnParams

VERSION = "streaming-turn-v0.5"  # "-v0.6" and "-v0.7" take the same settings and add the cue outputs
repo = snapshot_download("jptaylor/streaming-turn", allow_patterns=f"{VERSION}/*")
analyzer = StreamingTurnAnalyzer(
    Path(repo) / VERSION,
    params=StreamingTurnParams(threshold=0.6, late_threshold=0.4),
    # v0.4: StreamingTurnParams(threshold=0.5, late_threshold=0.4)
    # v0.2: StreamingTurnParams(threshold=0.5, likely_threshold=0.3)
)


class StreamingTurnStop(TurnAnalyzerUserTurnStopStrategy):
    """Tells the model when the bot is speaking: it was trained with that input."""

    async def process_frame(self, frame):
        if isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
            analyzer.set_bot_speaking(isinstance(frame, BotStartedSpeakingFrame))
        return await super().process_frame(frame)


user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    LLMContext(),
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(),
        user_turn_strategies=UserTurnStrategies(stop=[StreamingTurnStop(turn_analyzer=analyzer)]),
    ),
)

Outputs

One frame per 40 ms chunk.

Output
eot probability that the user's turn is over
eot_late the same, for a user who has gone quiet (v0.4 and later)
intent continuer, assessment, negative, floor_take, non_speech. Backchannel = continuer + assessment; floor-take = negative + floor_take
leadin none, acknowledgement, hesitation, marker, reaction (barely beats guessing)
bc_cue v0.7 only: probability that a listener would start a backchannel within half a second. Read it while the user holds the floor and the bot is silent, and with the mid-turn check below
eot_soon v0.6 and v0.7: probability that the user's turn ends within 240 ms and within 480 ms. Read it while the user is speaking and the bot is silent
vad, vap auxiliary: voice activity now, and both speakers' activity over the next 2 s. The third value of vap, the user speaking 0.6 to 1.2 s from now, is v0.7's mid-turn check

Inputs: chunk (640 samples, float32) and bot_state (is the bot speaking, and for how long), plus the streaming state tensors named in model.json. v0.6 and v0.7 have one more, cue_state. In v0.7 bc_cue comes before eot_soon, so eot_soon and the state outputs sit one place later than in v0.6: read outputs by name.

Results

Held-out Open Yap 1K conversations (1,259 turn ends, 8,191 pauses), replayed through Pipecat's Silero VAD with nobody answering, at the settings above:

Cuts in on a pause Answers within 500 ms Within 1 s Waits for the 3 s timeout
v0.5 4.1% 58.0% 79.9% 1.2%
v0.4 4.9% 56.4% 79.7% 2.9%
v0.2 4.6% 51.4% 82.2% 15.5%

On held-out otoSpeech conversations v0.5 answers 44.1% of turn ends within 1 s (v0.4: 44.0%) and leaves 2.8% to the timeout (v0.4: 11.0%), cutting in on 7.8% of pauses (7.7%).

On Smart Turn v3.2's English test clips (7,820, read 200 ms after the speech ends): v0.5 93.6%, v0.4 93.3%, v0.2 93.8%, Smart Turn v3.2 93.5%.

On held-out synthetic sessions of a person dictating to an agent, v0.5 cuts in on 4% of mid-turn pauses and v0.4 on half. Those sessions use new voices but the same scripts and speech vendors as v0.5's training ones; no real recordings of that case exist yet.

v0.6's cue that the turn is about to end

Experimental. The cue is given when the 240 ms output of eot_soon reaches 0.30 for two frames (sturn.cues.CueTracker in the code repository), a setting chosen on half of the held-out conversations and read once on the other half:

Held-out conversations Open Yap otoSpeech
Turn ends the cue comes before, while the user is still speaking 10.0% 15.0%
Median time before the stop 0.06 s 0.08 s
False starts per pause the user goes on from 4.3% 4.3%
Cues given: before a turn end / false starts 192 / 262 163 / 202
eot at 0.3, the earlier way to act ahead of the stop: before a turn end / false starts 194 / 747 132 / 224
  • It is a cleaner early signal than eot at 0.3, not an early one: about a quarter of a second ahead of a VAD's report of the stop.
  • When two people talk it is wrong more often than right. In six recorded sessions of a person talking to a bot, 13 of its 14 cues came before a turn end, covering 13 of 28 turn ends. That is the only evidence on bot sessions, and it is small.
  • It adds no accuracy over an output v0.5 already has: turn end against pause before the stop, AUC 0.81, where the best vap bin scores 0.82. What it adds is one calibrated number.
  • A second cue, for when the bot should backchannel, was trained and left out of v0.6: it scored no better than vap. v0.7 has it, with the check that makes it usable.

The model card in the code repository has the full results: models/streaming-turn-v0.6.

v0.7's cue for the bot to backchannel

Experimental, and nobody has heard it: nothing here plays a backchannel. The cue is given when bc_cue reaches 0.45 for two frames and the model is at least 92% sure the user is still speaking a second from now (the third value of vap), no faster than four a minute and never more than two close together. That is sturn.cues.CueTracker in the code repository, with CueParams(bc_threshold=0.45, bc_mid_turn=0.92, bc_max_per_minute=4). The setting was chosen on half of the held-out conversations. The other half was read three times, because two faults were found in the per-minute budget after a read; the thresholds did not change. The last read:

Held-out conversations Open Yap otoSpeech
What real listeners did: backchannels a minute of the user's floor / share at a turn end 2.58 / 0.9% 1.84 / 2.5%
Cues a minute of the user's floor 2.45 1.99
Cues that land as the turn ends or after it 0.6% (16 of 2,745) 0.9% (5 of 587)
A real listener's backchannel within half a second of the cue / times random placement 12.4% / 9.7 9.7% / 7.7
bc_cue at 0.5 with no check: cues a minute / share at a turn end / a listener within half a second 3.52 / 5.8% / 11.6% 2.44 / 9.3% / 8.5%
  • By itself the cue is not usable: 6% to 9% of its cues land as the user's turn ends, where a reply is due and not an "mm-hmm". The mid-turn check removes them, at about the rate real listeners backchannel.
  • It adds no accuracy over vap: 1.02 times its average precision. What v0.7 adds is the output and the rule for reading it.
  • The check keeps cues away from phrase ends too, where a backchannel sounds most natural. Most cues fall while the user is speaking: 72% on Open Yap and 89% on otoSpeech.
  • It needs a long turn. In six recorded sessions of a person talking to a bot it never fires: no stretch of the user's speech there is longer than 4.2 s.
  • A backchannel the bot plays is not bot speech to the model: do not report it through bot_state.

The model card in the code repository has the full results: models/streaming-turn-v0.7.

Limitations

  • The held-out conversations are from a corpus the model trained on (different conversations).
  • Keep bot_state up to date.
  • v0.5 is one of three runs, chosen on held-out data. A second seed of the same recipe scored about two points lower on the measure that chose it.
  • A finished sentence it takes for an unfinished one waits for the 3 s timeout.
  • v0.6's eot_soon is experimental: nobody has heard a bot act on it, it gives more false starts than hits when two people talk, and on Open Yap it gives a false start on more pauses (4.3%) than v0.5 cuts in on in the same replay (at most 3.7%).
  • v0.7's bc_cue is experimental: nobody has heard a bot backchannel on it, a real listener's backchannel is near about one cue in nine, and in short question-and-answer turns it does not fire. The bot's own "mm-hmm" in a pause can move the model (planted at -20 dB, it turned 2 of 53 pauses into cut-ins on v0.5): play it through echo cancellation, or only while the user is speaking.
  • Half of the end-of-turn targets on Smart Turn clips are Smart Turn v3.2's own predictions, so it is not independent of v3.2.
  • Trained on human-to-human conversation and single-speaker clips, with simulated echo, and for v0.5 on synthetic speech. Phone codecs are untested.

Training data

Open Yap 1K v1.0, otoSpeech-full-duplex-280h (CC BY 4.0), the English part of the Smart Turn v3.2 training data (CC BY 4.0), and 600 synthetic hard negatives. v0.5 adds 1,355 synthetic agent sessions, with the user's speech from ElevenLabs v4 and the bot's from Deepgram Aura-2. A frozen Whisper base.en encoder was the distillation target, and Smart Turn v3.2 gave soft end-of-turn labels. v0.6's head was trained on the same data with v0.5's weights frozen: no new audio. v0.7 has no new training: bc_cue is an output of v0.6's head, learned from the moments the other speaker in a real conversation backchannelled. No training data is included here.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support