hewig/lilt-signs

Lilt's music-sign detector

Finds the signs on a page of printed sheet music (115 kinds: noteheads, stems, beams, flags, rests, clefs, key and time signatures, accidentals, dots, ties and slurs, articulations, dynamics, hairpins, ornaments, fermatas, segno and coda, pedal marks, bowings, tuplets, fingerings and ottava brackets). It runs in the browser in Lilt, a music lab for children in Zoey's Lab: the photo is read on the device and is never uploaded.

The model

FCOS with a ResNet-18 feature pyramid from stride 4 to 64 (torchvision), 13.2 M parameters, trained from ImageNet weights for 30 epochs on 640 px crops of pages scaled to 1024 px wide, roughened like phone photos.

signs.onnx is the bare network: a page in, raw per-level outputs out.

• Input page: float32 [1, 3, H, 1024], grey levels mapped to [-1, 1] and copied into three channels, H padded with white to a multiple of 64.
• Outputs for levels p2–p6 (strides 4, 8, 16, 32, 64; box sizes 8 to 128): cls_* class logits [1, 116, h, w] (index 0 is background, index i + 1 is class i of classes.json), reg_* box distances [1, 4, h, w] in units of the level's size, ctr_* centre-ness [1, 1, h, w].
• Decoding is torchvision's FCOS post-processing: score sqrt(sigmoid(cls) * sigmoid(ctr)) above 0.25, at most 4000 candidates per level, per-class NMS at IoU 0.5. Lilt's TypeScript version is tested against torchvision's own output.

Convolution weights are stored as int8 with one scale per output channel and dequantised as the model loads; the arithmetic is float32. That costs nothing measurable against float16 weights.

How well it reads

On 20 whole held-back DeepScoresV2 pages: recall 0.81 and precision 0.89 at IoU 0.5, each detection matched to at most one true sign. Accidentals, key signatures, clefs, fermatas, ornaments, dynamics, hairpins, tuplets, segno and coda are found 95–100 % of the time in the test crops. Thin and tiny signs are the weak spot: stems (55 %), ledger lines (56 %), augmentation dots (66 %), staccato dots, and ottava brackets (1 in 9). It has only seen clean engraving, roughened; real phone photos are untested.

Credit and licence

Trained on the dense subset of DeepScoresV2, Tuggener, Satyawan, Pacha, Schmidhuber and Stadelmann, The DeepScoresV2 Dataset and Benchmark for Music Object Detection, ICPR 2020, released under CC BY 4.0 (Zenodo). The model is released under the same licence.