Pipecat Tone

Pipecat Tone tells a voice agent how each turn of a call was meant. It reads one turn of text, and the turn before it when you have it, and returns the speaker's emotion, the move the turn makes, whether it is sarcastic, and whether it is only a listening sound such as "mm-hm". Each comes with a calibrated confidence. It needs no names or speaker labels, and it runs on a CPU in a few milliseconds, so your language model spends no tokens on it.

Release	rc1 · 2026-10-16
Size	24.2M parameters. 96.3 MB ONNX in full precision, 48.2 MB with the weights in 16 bits (model_fp16.onnx)
Speed	p95 per turn on 4 CPU threads (Apple M1 Max): 5.4 ms with the turn before, 3.6 ms without. On 1 thread: 12.1 ms with the turn before. The 16-bit file takes the same time
Runtime	ONNX Runtime 1.24 or later and tokenizers, on CPU. A Python reference runtime ships with it
Licence	Apache-2.0

Quick start

pip install -U huggingface_hub
hf download pipecat-ai/tone --revision rc1 --local-dir pipecat-tone
cd pipecat-tone
pip install -r requirements.txt
python example.py

example.py reads a support call, each turn with the one before it:

"hi um yeah my internet's been down since this morning"
    emotion neutral (low) · move answers (0.92)
"I'm sorry to hear that. Which plan are you on?"
    emotion neutral (low) · move asks (0.96)
"I think my husband set it up, I'm not sure"
    emotion neutral (low) · move hedges (0.99)
"No problem. Can I take the account holder's date of birth?"
    emotion neutral (low) · move asks (0.95)
'Why do you need that?'
    emotion neutral (low) · move withholds (0.98)
"It's only to confirm the account. The next engineer visit is in about forty minutes, I'm afraid."
    emotion neutral (low) · move answers (0.90)
'Oh great, a forty minute wait. Love that.'
    emotion angry (low) · move other (0.92) · sarcastic (0.99)
'Let me try a reset from my side first. The light on your router should turn green.'
    emotion neutral (low) · move answers (0.93)
'mm-hm'
    emotion neutral (low) · move other (0.92) · backchannel
'Do you know which router model you have?'
    emotion neutral (low) · move asks (0.96)
'No idea, sorry.'
    emotion neutral (low) · move doesnt_know (1.00)
"That's fine. Can you try loading a page now?"
    emotion neutral (low) · move answers (0.73)
"That's brilliant, it's working, thank you so much!"
    emotion happy (medium) · move answers (0.86)
'Is there anything else I can help with?'
    emotion neutral (low) · move asks (0.96)
"thanks, that's all i needed"
    emotion happy (low) · move closes (0.98)

Then it prints one turn's full output as JSON: each label with its confidence and the probability of every class.

The 16-bit file is half the download and gives the same tags:

hf download pipecat-ai/tone --revision rc1 --local-dir pipecat-tone-small --exclude model.onnx --exclude model.opt.onnx
cd pipecat-tone-small
pip install -r requirements.txt
python example.py --fp16

In your own code, from the downloaded folder:

import sys
sys.path.insert(0, "runtime")

from tonetext.runtime.classifier import ToneTextClassifier

tone = ToneTextClassifier(".")  # or ToneTextClassifier(".", precision="fp16") for the 16-bit file
tags = tone.tag("oh great, a forty minute wait", previous="The next engineer visit is in about forty minutes, I'm afraid.")
print(tags.emotion.label, tags.move.label, tags.sarcasm.sarcastic, tags.sarcasm.probability)

Input and output

• Input: text, the turn, typed or from speech to text (at most 400 characters), and previous, the turn before it (optional, at most 480). Who said either is not needed. The model reads at most 160 tokens, so a long input loses the end of the turn before first.
• emotion: neutral, happy, amused, angry, sad, afraid, surprised, disgusted or contemptuous, with an intensity (low, medium or high).
• move: asks (wants a reply or an action), answers (gives what was asked for, or offers information), hedges (unsure or second-hand), doesnt_know, withholds (declines or dodges), closes (winds the exchange down), or other.
• sarcasm: sarcastic, the model's decision, set so that at most 1.5% of sincere everyday lines in development data are called sarcastic, and probability, so you can set your own threshold.
• backchannel: whether the turn is only a listening sound ("mm-hm", "yeah", "go on"). The other outputs are still returned.
• Confidence is the chance the label is right, calibrated on development data that includes voice-agent calls.
• Send the turn before when you have it. The move and sarcasm lean on it.

How well it reads calls

Labels are the majority vote of a committee of three large language models. None of these turns were trained on.

Real calls. 1,500 turns, half caller and half agent, from held-out conversations of nine openly licensed support and assistant datasets (credited below). Real calls are calm: 84% of turns are neutral and 14% happy. So emotion is scored over those two, and the move over asks, answers, closes and other, the moves with at least 30 turns.

	With the turn before	Without it
Emotion macro-F1 (neutral, happy)	0.940 (0.923 to 0.956)	0.940
Emotion right, all nine	95.7%	95.6%
Move macro-F1 (asks, answers, closes, other)	0.843 (0.803 to 0.876)	0.798
Asks or not	96.1%	95.0%
Sincere turns called sarcastic	1 of 1,496	3 of 1,496
Calibration error: emotion, move, sarcasm	0.079, 0.027, 0.016	0.078, 0.015, 0.017

Callers and agents read alike: emotion 0.947 and 0.926, the move 0.932 and 0.945.

Calls with more feeling. 897 turns from 299 calls in 13 kinds of service, written for this test by Qwen3.8-27B: 100 sarcastic caller turns, and 104 sincere turns written to look sarcastic.

	With the turn before	Without it
Emotion macro-F1, all nine (885 turns)	0.598 (0.535 to 0.698)	0.584
Move macro-F1 (868 turns)	0.800 (0.757 to 0.833)	0.771
Sarcasm F1 (100 sarcastic)	0.887 (0.839 to 0.928)	0.868
Sincere turns called sarcastic (797)	1.0%	2.0%
Calibration error: emotion, move, sarcasm	0.036, 0.039, 0.014	0.040, 0.050, 0.017

For scale, two of the labellers agree with each other on these calls at emotion 0.623, move 0.836 and sarcasm 0.950.

The 16-bit file gives the same tags as full precision on every test line but 2 emotions of 8,207 with the turn before, and 1 of 8,117 without. No measure above moves.

How it was made

• Model: MiniLM-L6 (sentence-transformers/all-MiniLM-L6-v2 at 1110a24, Apache-2.0), fine-tuned end to end with five small heads. The move and sarcasm heads read four learned queries over the tokens, so they can weigh a turn against the one before.
• Text: about 1.08 million turns from the thirteen openly licensed datasets credited below, dialogue and spoken-style lines written for this project by Qwen3.8-27B (Apache-2.0), Project Gutenberg dialogue, GoEmotions (Google Research, published in a repository under Apache-2.0) and Civil Comments (CC0). Names were removed from the turn before.
• Speech-to-text errors: turns spoken by Kokoro-82M (hexgrad/Kokoro-82M, Apache-2.0), half of them through an 8 kHz phone line, and transcribed again by Whisper large-v3-turbo (openai/whisper-large-v3-turbo, MIT). The 31,939 transcripts that came back with errors train on their clean turn's labels.
• Labels: the committee's votes on 25,303 turns for emotion, 9,097 for the move and 10,000 for sarcasm, 6,000 of them from the open datasets. Five teacher models (jhu-clsp/ettin-encoder-1b, MIT), trained on those votes, label the rest.
• Training: the turn before is dropped from a fifth of the turns each epoch, so the model also works without it. The best of three seeds was chosen on development data that includes real calls. Calibration: one temperature each for emotion and the move, and a curve for sarcasm, fitted on the same data.

Limits

• Use it to give a voice agent a reading of each turn. Do not use it for moderation, safety decisions or judging real people.
• English only, and one turn of context.
• Rare feelings. Real calls hold few angry, sad or afraid turns, so the model's sense of those comes mostly from other text. On the generated calls the agent's own turns score 0.398 on emotion over all nine, set by a handful of sad and afraid turns.
• Sarcasm and emotion. A sarcastic turn sometimes gets the emotion its words say: "oh great, thanks a lot" can read happy while flagged sarcastic. Read emotion together with sarcasm.
• Emotion confidence on real calls is less well calibrated than the rest (error 0.079).
• These are not your users. The real calls come from datasets in which people played both the caller and the agent, and the labels come from language models.

Data credits

• NatCS (Gung et al., 2023; Amazon), CDLA-Permissive-2.0
• Taskmaster-1 (Byrne et al., 2019; Google), Taskmaster-2 and Taskmaster-3 (Byrne et al., 2020; Google), CC BY 4.0
• ABCD, the Action-Based Conversations Dataset (Chen et al., 2021; ASAPP), MIT
• MultiWOZ 2.2 (Zang et al., 2020; Budzianowski et al., 2018), MIT
• STAR (Mosig et al., 2020; Rasa), MIT
• CCPE-M (Radlinski et al., 2019; Google), CC BY 4.0
• Harper Valley Bank (Wu et al., 2020; Gridspace and Stanford), CC BY 4.0
• AMI Meeting Corpus manual annotations (Carletta et al., 2005; the AMI Consortium), CC BY 4.0
• DiPCo, the Dinner Party Corpus (Van Segbroeck et al., 2020; Amazon), CDLA-Permissive-1.0
• DynaSent round 2 (Potts et al., 2021), CC BY 4.0
• iSarcasmEval, SemEval-2022 Task 6 (Abu Farha et al., 2022), MIT

Their turns were cleaned of transcription marks and paired with the turn before. None of their text ships here.

The numbers come from the training repository's report for this release, 2026-10-15-tonetext-r2.md (at commit bc5a253). The model files were packaged from git tag pipecat-tone-rc1-draft (commit f0a0012). That repository is private.