hum2song contour encoder

A melody-contour encoder for query by melody: humming, whistling, and singing with wrong or nonsense lyrics. It maps a key-normalized pitch track (F0) to one L2-normalized 256-d vector, so a query and a song's vocal melody can be compared by cosine similarity. Lyrics and timbre never reach the model.

Paper: "Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody", Luigi Rivolta, 2026 (arXiv, cs.SD). The arXiv link will be added here once the paper is announced.

Code: https://github.com/luiginotmario/hum2song

Files

File	What it is
e2b/model.safetensors, e2b/best.pt, e2b/config.json	E2b, the main model in the paper and the live model for open retrieval
whistle_closedset_ft/model.safetensors, whistle_closedset_ft/best.pt, whistle_closedset_ft/config.json	Closed-set fine-tune of E2b on real MLEnd whistles (paper Section "closed set", experiment D-044, all train performers, seed 0). Experiment weights only, for the 8 MLEnd songs

The .pt files are what the repo's load_checkpoint reads (model state_dict, config, step; no optimizer state). The .safetensors files hold the same weights.

Architecture

Input is key-normalized pitch plus a voicing flag at 20 ms frames. A convolutional front end halves the frame rate; a 6-layer pre-norm transformer (d=256, 4 heads) and a masked attention pool give one L2-normalized 256-d vector. About 5.2M parameters, one tower for queries and songs.

Training

• E2b: symmetric InfoNCE on HumTrans hum-to-MIDI pairs, then fine-tuned with the CLEWS loss (Serrà et al.) on cover-to-original windows from CHAD. Query F0 comes from RMVPE; whistles use a spectral-peak tracker.
• Whistle closed-set fine-tune: starts from E2b; InfoNCE with same-song masking on MLEnd Hums and Whistles whistle-to-hum and hum-to-hum pairs from the 136 train performers only; 300 steps, LR 3e-5; checkpoint selected on validation performers. Train, validation and test performers are disjoint, but all splits use the same 8 songs.

Training-data provenance: HumTrans (hums with MIDI labels), CHAD (cover and original song fragments), MLEnd Hums and Whistles (hums and whistles of 8 songs). No song audio or datasets are distributed here; check each dataset's own terms before reuse.

Results (from the paper)

MLEnd test performers, 339 real whistles from 26 performers, whistle-to-hums query-by-example over the 8 songs (chance top-1 0.125). 95% CIs are bootstrap over clips.

Model	Top-1	Top-3
E2b (zero-shot)	0.631 [0.581, 0.681] (214/339)	0.770
Whistle closed-set fine-tune	0.673 [0.622, 0.723] (228/339)	0.820

Performer-bootstrap CIs are wide (E2b [0.500, 0.753], fine-tune [0.536, 0.801]). The fine-tune is closed-set on known songs; it is not a claim about new songs.

E2b on hums: HumTrans test top-1 / top-10 0.996 / 1.000; MIR-QBSH (4,431 queries, 48 MIDI songs) 0.949 / 0.990.

Limitations

• Synthetic data did not transfer. Three generations of synthetic whistles and wrong-lyrics singing (physics-based, a commercial neural music model, and open-source neural singing plus a DDSP whistle with a learned human-error model) raised synthetic-query retrieval but never beat E2b on real whistles. The best open-source run reached 0.619 (paired difference -0.012, 95% CI [-0.038, +0.015]). Real queries are the missing step.
• The real-whistle benchmark has only 8 songs and amateur performers; one seed per run.
• The whistle fine-tune is tuned to MLEnd's 8 songs. For open retrieval use E2b.
• Melody-only: it cannot use lyrics, and songs without a clear vocal melody are hard.

Usage

Clone the code and install the training package (training/), then:

from pathlib import Path
import numpy as np, torch
from huggingface_hub import hf_hub_download
from hum2song.contour.train import load_checkpoint
from hum2song.contour.features import rmvpe_contour
from hum2song.contour.evaluate import embed_contours, query_contour

ckpt = hf_hub_download("luigir5/hum2song", "e2b/best.pt")
model, config, step = load_checkpoint(Path(ckpt), torch.device("cpu"))

# track: (2, frames) F0 in Hz and confidence at 10 ms, e.g. from RMVPE
# (hum2song.contour.trackers) or the whistle spectral-peak tracker.
t = np.arange(400) * 0.01
track = np.stack([440 * 2 ** (np.floor(t) / 12), np.ones_like(t)])
contour = rmvpe_contour(track)
emb = embed_contours(model, [query_contour(contour)], torch.device("cpu"))[0]
print(emb.shape)  # (256,)

Licence

Weights: CC BY 4.0. Code: see the GitHub repository.

Citation

@misc{rivolta2026whistle,
  title  = {Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody},
  author = {Rivolta, Luigi},
  year   = {2026},
  note   = {arXiv preprint, cs.SD}
}