royshilkrot/scoreboard-layout

Scoreboard layout

Two small ONNX models that find a sports scoreboard in a photo or video frame:

1. fields_board.onnx: a D-FINE-S detector with six classes: the home score, the away score, the clock, the period, other fields, and the board itself.
2. corners.onnx: given a board box from the first model, the board's four corners: top left, top right, bottom right, bottom left. A tilted or perspective view can then be warped flat.

They were made for ScoreSight, which reads scoreboards with OCR, to suggest the boxes and the four-corner correction when a user sets up a new source. Both models run on a CPU with onnxruntime.

In ScoreSight 0.4 they are the ✨ Auto crop and ✨ Auto layout buttons: watch the 2-minute video.

File	Size	Time per picture*
fields_board.onnx	41.7 MB	about 200 ms
fields_board_int8.onnx (dynamic int8 weights)	12.2 MB	about 230–270 ms; slightly less accurate (below)
corners.onnx	14.1 MB	about 7 ms per board

* onnxruntime 1.31, 2 threads, an Intel Core i9-14900K desktop. Expect a laptop to be slower.

Use

example.py runs both models on a picture: python example.py photo.jpg. It needs onnxruntime, opencv-python and numpy. config.json describes every input and output.

• Fields and board: resize the whole picture to 640 × 640 RGB, 0–1, without keeping the aspect ratio. Pass the picture's own width and height as orig_target_sizes. Boxes come back in the picture's pixels, up to 300 per picture with a class and a score. For each role (home, away, clock, period), take its highest-scoring box. A score of at least 0.3 is a reasonable cut.
• Corners: grow the board box by 15% of its width and height on each side and warp it to a 256 × 256 crop. Normalize the crop with ImageNet's mean and standard deviation. The model gives each corner as a share (0–1) of the crop; the crop's inverse warp maps it back to the picture.

Evaluation

Measured on the test split of Scoreboard Fields (72 photos) plus 7 held-out video frames: 79 photos with 78 boards. Validation is 30 photos. Photos of the same board were kept in one split.

Fields: how often the single best box of a role overlaps the labelled box (IoU ≥ 0.5).

	home	away	clock	period	all of home, away and clock right
test	65% (40/62)	66% (39/59)	74% (35/47)	43% (6/14)	55% (22/40 photos)
test, int8	61%	61%	72%	57%	50% (20/40)
validation	77%	73%	68%	50%	50% (8/16)

• The numbers move between training runs: four runs of this setup, two with the board class and two without, scored 55–65% on "all three right".
• Close-ups are much easier than wide shots: on the video frames all three were right on 5 of 7. On the Wikimedia Commons photos, often wide shots of a whole arena, 13 of 28.
• Home and away get swapped sometimes: on the test photos, a home or away best box landed on the other team's score 5 times.

Board: a board box (score ≥ 0.3) overlapped 91% of the test boards (71/78, IoU ≥ 0.5).

Corners, both models together: the photo's best board box → corners, compared with the photo's largest labelled board. A board passes when all four corners are within the given share of its diagonal.

	within 2%	within 5%	within 10%
test (76 photos)	28%	53%	71%
validation (27 photos)	44%	67%	74%

The test corners were labelled starting from an earlier model's suggestions, which may favour that model. A blind relabelling of them is under way. Most corner misses are about which rectangle counts as "the board": a sign above it, an outer frame against the lit panel, or a video screen beside it.

Training

• Data: the train split of Scoreboard Fields (189 photos) plus 7 frames of videos that ScoreSight users sent. The frames aren't published.
• fields_board: fine-tuned from D-FINE-S's COCO checkpoint for 60 epochs at 640 × 640, using D-FINE's augmentations without horizontal flips, because a mirrored picture swaps home and away. The board box is the box around each labelled board quad. This is the checkpoint with the best validation AP (epoch 55).
• corners: MobileNetV3-Large (ImageNet weights) upsampled to four 64 × 64 heatmaps, each read with a soft-argmax, trained with an L1 loss on the corner coordinates. It trains on random-perspective crops of the labelled boards, made from boxes jittered the way a detector's would be. 2,500 steps of 64 crops.

Limits

• The data is small: fewer than 200 training photos, mostly Commons photos of North American and European sports, plus a few broadcast and LED scoreboards.
• Expect to correct the suggestions: they're a starting point for a person to check, not a finished layout.
• One board per photo is the case it was built for: the corner model sees one board box at a time.

License

The models are released under the Apache License 2.0 (LICENSE), the license of D-FINE. The training labels are CC BY 4.0. The photos keep their own licenses; see the dataset.