keybed-seg

A small semantic segmentation model that finds the keybed of a piano or electronic keyboard in a photo or video frame. Given an image it returns a map of how likely each patch is to be part of the strip of white and black keys.

It is meant to run live in a web browser through onnxruntime-web, so it is deliberately tiny and fast rather than maximally accurate. On a laptop CPU it takes about 130 ms per frame.

The model on its own gives you a rough blob. The project it comes from turns that blob into real 3D geometry by fitting a rectangle of known proportions to it, which recovers the full pose of the keyboard rather than four loose corners. That code lives at https://github.com/matheusfillipe/keybed

What goes in and what comes out

input	image, float32, shape [1, 3, 288, 288], RGB
normalisation	ImageNet, mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]
output	mask, float32, shape [1, 1, 144, 144], probability from 0 to 1
size	6.3 MB, opset 20

The frame is squashed to a square rather than letterboxed. Padding to 4:3 measured clearly worse, because the network has never seen a black bar.

Using it in a browser

import * as ort from "onnxruntime-web/wasm";

const session = await ort.InferenceSession.create("keybed_seg2.onnx");
// rgb is a Float32Array of 3 * 288 * 288, planar, ImageNet normalised
const out = await session.run({
  image: new ort.Tensor("float32", rgb, [1, 3, 288, 288]),
});
const mask = out.mask.data; // 144 * 144 probabilities

Using it in python

import numpy as np, onnxruntime

session = onnxruntime.InferenceSession("keybed_seg2.onnx")
image = np.zeros((1, 3, 288, 288), dtype=np.float32)  # your ImageNet normalised frame
mask = session.run(None, {"image": image})[0][0, 0]   # 144 x 144 probabilities

How it was built

The backbone is a MobileNetV3-Small pretrained on ImageNet, with a small U-Net style decoder on top. Using a pretrained backbone rather than training from scratch was the single biggest accuracy win, roughly a factor of three.

Training data is mostly synthetic. A three.js scene renders a 3D keyboard through thousands of camera angles, lighting setups and HDRI environments, with exact corner labels, and those renders are composited onto real photographs as backgrounds. Adding realistic environments and reflective materials to that generator cut held out error from 165 px to 24 px, the largest single improvement of the project.

The published weights are then fine tuned on about a hundred hand labelled frames of one real instrument in one room.

Limitations, honestly

• It is fine tuned on one instrument in one room. It transfers to other setups, but not as well. If you care about your own piano, retrain it. The repository has a full guide and the browser tool that generates the synthetic data.
• Side and overhead views work best. Straight on front views, where the whole control panel of a synth is visible, are the weakest case, because almost all the training data is angled. It will sometimes mark the whole top of the instrument as keybed.
• It only finds the keys. It does not detect hands, read notes, or identify the instrument.
• Synthetic validation loss stops predicting real accuracy after the first few epochs. If you retrain, always choose your checkpoint against real photographs.

Licence

Apache 2.0. The keyboard mesh used by the synthetic data generator is "Piano keys" from Sketchfab, used under CC BY, and it lives in the code repository rather than here.