haimgoldfisher/senti-whisper-full-model

Senti-Whisper — Joint Transcription + Sentiment

A single model that transcribes speech and classifies its sentiment (negative / neutral / positive) in one pass, using the acoustic signal (prosody, tone, energy) that a text-only sentiment pipeline throws away.

It is a frozen ivrit-ai/whisper-large-v3 with a lightweight attention-pooling sentiment head tapped off the encoder. The transcription path is the original Whisper, unchanged.

Architecture

log-mel ─► Whisper encoder ──┬──► Whisper ASR decoder   (UNTOUCHED, frozen)
                             │         └─► transcript
                             │
                             └──► [tapped edge: encoder hidden states]
                                        └─► SentimentHead   (the only trained part)
                                               └─► softmax(neg / neutral / pos)

• Base (frozen): ivrit-ai/whisper-large-v3 (encoder + ASR decoder, untouched).
• Trained head: attention-pooling over the encoder frames of the utterance → 2-layer MLP → 3 logits. Padding frames are masked so short clips don't pool silence. Sentiment is a classification, not a sequence to generate, so the branch is a pooling classifier rather than a second autoregressive decoder.

Intended use & scope

• Use for: utterance-level sentiment from speech where tone matters (call-center analytics, sarcasm/frustration that text misses), plus a transcript.
• Out of scope / caveats: this checkpoint is stage 1 — trained on acted English emotion (RAVDESS). It learns sentiment mostly from prosody, not lexical content, and is not validated on Hebrew or on real (non-acted), telephony-band audio. Treat it as a proof-of-pipeline, not a production model.

How to use

The repo stores a combined state dict (full_senti_whisper.pt: Whisper + head). The head is a custom class, so include it when loading:

import torch, torch.nn as nn, librosa
from transformers import WhisperForConditionalGeneration, WhisperProcessor

LABELS = ["negative", "neutral", "positive"]
BASE = "ivrit-ai/whisper-large-v3"
ENC_FPS, SR = 50, 16_000

class SentimentHead(nn.Module):
    def __init__(self, d_model, num_classes=3, hidden_dim=None, dropout=0.1):
        super().__init__()
        hidden_dim = hidden_dim or d_model // 2
        self.attn = nn.Linear(d_model, 1)
        self.mlp = nn.Sequential(
            nn.LayerNorm(d_model), nn.Dropout(dropout),
            nn.Linear(d_model, hidden_dim), nn.GELU(), nn.Dropout(dropout),
            nn.Linear(hidden_dim, num_classes))
    def forward(self, x, mask):
        s = self.attn(x).masked_fill(mask.unsqueeze(-1) == 0, float("-inf"))
        return self.mlp((torch.softmax(s, 1) * x).sum(1))

# --- load base + head from the combined checkpoint ---
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = WhisperProcessor.from_pretrained(BASE)
whisper = WhisperForConditionalGeneration.from_pretrained(BASE).to(device).eval()
head = SentimentHead(whisper.config.d_model).to(device).eval()

state = torch.load("full_senti_whisper.pt", map_location=device)
# keys are prefixed "whisper." / "sentiment_head." (adjust if you saved differently)
whisper.load_state_dict({k[len("whisper."):]: v for k, v in state.items()
                         if k.startswith("whisper.")}, strict=False)
head.load_state_dict({k[len("sentiment_head."):]: v for k, v in state.items()
                      if k.startswith("sentiment_head.")})

@torch.no_grad()
def predict(path):
    audio, _ = librosa.load(path, sr=SR)
    feats = processor.feature_extractor(audio, sampling_rate=SR,
                                        return_tensors="pt").input_features.to(device)
    hs = whisper.model.encoder(feats).last_hidden_state          # [1, 1500, D]
    n = max(1, min(hs.shape[1], int(len(audio) / SR * ENC_FPS)))
    mask = torch.zeros(1, hs.shape[1], dtype=torch.long, device=device); mask[:, :n] = 1
    sentiment = LABELS[head(hs, mask).argmax(-1).item()]
    transcript = processor.batch_decode(whisper.generate(feats),
                                        skip_special_tokens=True)[0]
    return {"transcript": transcript, "sentiment": sentiment}

print(predict("example.wav"))

Training data

RAVDESS — 1,440 acted clips, 24 actors, 8 emotions. The 8 emotions are collapsed to 3-class sentiment; cross-valence ambiguous emotions are dropped to avoid label noise:

→ negative	→ neutral	→ positive	dropped (ambiguous)
angry, sad, fearful, disgust	neutral, calm	happy	surprised

Training procedure

• Frozen base, head-only training. The Whisper encoder runs once per clip (under no_grad); its hidden states are cached, then the small head is trained on the cache — so training is fast and never touches ASR weights (no WER regression).
• Pooling: attention pooling over the real (non-padded) encoder frames.
• Loss: class-weighted cross-entropy (RAVDESS is non-neutral-heavy).
• Hyperparameters: AdamW, lr 2e-4, weight decay 1e-2, cosine schedule, ~15 epochs, batch 32, fp16. Best checkpoint by validation macro-F1.

Evaluation

⚠️ Fill these in from your run — placeholders, not measured values.

split	macro-F1	accuracy
validation	TBD	TBD
test (held-out)	TBD	TBD

Per-class precision/recall and the confusion matrix come from the classification_report / confusion_matrix cells in the training notebook. For KS-2959 the key comparison is macro-F1 vs. a text-only cascade on a sarcasm/ambiguous-tone subset — the test that justifies using acoustics at all.

Limitations & bias

• Acted, English, clean audio. RAVDESS is studio-acted with two fixed sentences — strong prosody, almost no lexical variety, no telephony noise. Expect a domain gap to real Hebrew call-center audio.
• Prosody-only (stage 1). It learns tone, not words. Stage 2 (adding neutral-tone spoken content / a sentiment corpus like CMU-MOSEI) is needed to also learn sentiment from lexical content.
• No Hebrew sentiment data. There is no public Hebrew speech-sentiment set; a Hebrew production model requires labeling in-house audio (with PII handling).
• License: trained on RAVDESS (CC BY-NC-SA 4.0), so this checkpoint inherits a non-commercial, share-alike restriction. The base Whisper is Apache-2.0; the combined artifact follows the more restrictive RAVDESS terms.

Citation

• RAVDESS: Livingstone & Russo, 2018, PLoS ONE — DOI 10.5281/zenodo.1188976.
• Whisper: Radford et al., 2022, Robust Speech Recognition via Large-Scale Weak Supervision.
• Base model: ivrit-ai/whisper-large-v3.