npr1138/soulsync-emotion-wav2vec2

Model card: SoulSync speech emotion classifier (v2)

Generated by python -m eval.make_docs from eval/results/. Numbers are not typed by hand.

Summary

A facebook/wav2vec2-base encoder fine-tuned to classify a 3-second speech clip as happy, sad, angry, neutral or anxious. Used in SoulSync to give the language model a hint about the speaker's tone. It is a demo component, not a clinical or diagnostic tool.

Architecture	Wav2Vec2ForSequenceClassification, CNN feature encoder frozen, 90,369,413 trainable parameters
Input	16 kHz mono, 3 s, zero-mean unit-variance per clip, zero-padded. Longer audio: 3 s windows with 50% overlap, probabilities averaged
Output	5-way softmax
Formats	PyTorch (378.3 MB), ONNX FP32 (378.5 MB), ONNX INT8 (122.1 MB)

Files

File	Use
config.json, model.safetensors	PyTorch / transformers (Wav2Vec2ForSequenceClassification)
emotion_w2v2_int8.onnx	onnxruntime, CPU, no PyTorch needed

Input must be preprocessed exactly as in the app: 16 kHz mono, 3 s windows, zero-mean unit-variance per window, zero-padded (see soulsync/emotion.py in the code repository).

python scripts/hub.py download --repo <this repo id>

Data

Four acted, English-language corpora via the Kaggle collection uldisvalainis/audio-emotions: RAVDESS (2076 files, 24 speakers), CREMA-D (7442 files, 91 speakers), TESS (2800 files, 2 speakers), SAVEE (480 files, 4 speakers). Details and problems found in the packaging: DATASET.md.

• "Fearful" is used as "anxious". "Disgusted" and "surprised" are dropped ({'disgusted': 1923, 'surprised': 652}).
• RAVDESS song clips are excluded.
• Labels are read from filenames. The folder labels are wrong for 405 SAVEE clips.

Split

Speaker-independent, seed 42. Speaker overlap between splits: {'train_val': 0, 'train_test': 0, 'val_test': 0}.

Split	Clips	Speakers	By corpus
train	7635	93	{'CREMA-D': 4808, 'RAVDESS': 647, 'SAVEE': 180, 'TESS': 2000}
val	878	14	{'CREMA-D': 680, 'RAVDESS': 108, 'SAVEE': 90}
test	877	14	{'CREMA-D': 679, 'RAVDESS': 108, 'SAVEE': 90}

TESS has two speakers with 1,000 clips each, so both are in train and the held-out sets contain no TESS.

Training

Optimiser	AdamW, lr 5e-05, weight decay 0.01, 10% warm-up then linear decay
Batch size	16, fp32 on Apple MPS
Loss	cross-entropy, class weights [0.977, 0.9757, 0.9763, 1.1073, 0.9763]
Augmentation	random 3 s crop for longer clips; Wav2Vec2's built-in time masking
Selection	best validation macro-F1, early stopping with patience 3
Result	12 epochs run, best at epoch 9 (validation macro-F1 81.8%), 96.33 minutes

Total training wall-clock across all runs in this project: 160.0 minutes.

Metrics

Test split: 877 clips, 14 speakers never seen in training. Scored once.

Metric	Value	95% CI (clip bootstrap)
Accuracy	71.2%	68.3% to 74.2%
Macro-F1	71.1%	68.2% to 74.1%

Class	Precision	Recall	F1	Support
happy	79.0%	63.1%	70.2%	179
sad	61.2%	74.9%	67.3%	179
angry	74.3%	85.5%	79.5%	179
neutral	76.9%	74.5%	75.7%	161
anxious	68.0%	58.1%	62.6%	179

By corpus: CREMA-D 72.5% (n=679), RAVDESS 69.4% (n=108), SAVEE 63.3% (n=90). Per-speaker accuracy ranges from 48.5% to 92.7%.

How much to trust these numbers

• The confidence intervals resample clips. They do not cover seed-to-seed variation (one training run) or the fact that there are only 14 test speakers.
• Validation macro-F1 was 81.8% and test macro-F1 is 71.1%. The gap shows how much results move between two sets of 14 speakers.
• Trained without TESS and tested on TESS (an unseen corpus, 2 unseen speakers): 60.6% accuracy, 53.4% macro-F1 on 2000 clips.

Baseline

The v1 notebook's recipe (RAVDESS only, split by file, wav2vec2-base-960h) was reproduced because the original weights were unavailable. It scores 61.9% on its own file-level validation split and 32.4% accuracy (28.8% macro-F1) on this test set. The original notebook reported 82.9% on its leaky validation split, so the reproduction is weaker than the original model was and the "before" figure is likely pessimistic.

Emotion2Vec and the ensemble

v1 discarded every Emotion2Vec score through a label mismatch. With the mapping fixed, on the same test set: Emotion2Vec alone 89.3% accuracy; ensemble at Wav2Vec2 weight 0.4 89.5%. The weight was chosen on validation (macro-F1 gain +9.2 points, 95% CI +6.9 to +11.7).

Read this number with care. Emotion2Vec is a public model fine-tuned on EmoBox, a collection of academic emotion corpora. If that includes RAVDESS, CREMA-D or SAVEE, it has heard our test speakers, and its score here is not a speaker-independent result. Nearly all of the ensemble's accuracy comes from Emotion2Vec; the fine-tuned Wav2Vec2 adds little on top of it.

Quantisation

Version	Size	Accuracy	Macro-F1	CPU p50 / p95 per 3 s clip
pytorch_fp32	378.3 MB	71.2%	71.1%	70.2 / 72.2 ms
onnx_fp32	378.5 MB	71.2%	71.1%	101.6 / 106.3 ms
onnx_int8	122.1 MB	71.6%	71.4%	65.5 / 66.7 ms

Limitations

• Acted speech. Actors reading fixed sentences exaggerate emotion. Performance on spontaneous speech is unmeasured and expected to be lower.
• English only. Mostly North American and British speakers.
• Five coarse classes. "Anxious" is a relabelled "fearful". Real anxiety rarely sounds like acted fear.
• Short clips. Trained on 3-second windows of single sentences.
• Not a clinical tool. It must not be used to assess mental state, risk or diagnosis. In SoulSync it only adjusts the tone of a reply, and the UI says the estimate is often wrong.
• One seed, one run.

Licences

RAVDESS: CC BY-NC-SA 4.0. CREMA-D: ODbL 1.0 (contents DbCL 1.0). TESS: CC BY-NC 4.0. SAVEE: registration licence, no redistribution. Base model facebook/wav2vec2-base: Apache-2.0. Checked against the source pages on 2026-10-14; details in DATASET.md. Because two corpora are non-commercial, treat these weights as non-commercial research output.