dialbench-compact: a 13 M-parameter pointer-meter reader for DialBench RPM-10K

Results card for a compact gauge reader: a 13.08 M-parameter ResNet-18 (224 px input; 13.2 MB as static int8 ONNX) trained from scratch on the official training split of DialBench RPM-10K and scored with the official scorer. Self-evaluated, not an official ranking.

No weights are in this repository. The RPM-10K dataset licence is "TBD" in the README of the DialBench repository, so models trained on it stay with the authors' terms until they state them (asked by e-mail on 2026-10-07 and 2026-10-17 and as Event-AHU/DialBench#5 on 2026-10-19; no answer known on 2026-10-20). The repository holds the result, the per-seed predictions and the links needed to recompute every number from the authors' own release with the public code kit. Whether weights can be released depends on the authors' licence terms.

Result (scoped to four metrics)

• Setting: DialBench RPM-10K official split (labels/train.json 8,730 rows, labels/test.json 2,000 rows), official scorer (benchmark/benchmark.py of Event-AHU/DialBench, ported and equivalence-tested against a verbatim copy of its evaluation logic), 5 seeds, self-evaluated. Runs of 2026-10-17; the table was recomputed from the saved predictions on 2026-10-19.
• Wording: best published Acc_eps and Acc_theta in Table 4 of the DialBench paper (arXiv 2511.21982), and behind on Ref and Rel. It is never "best published result" without that scope.

metric (paper Table 4)	ours, 5-seed mean	MRLM	best published	rank
Acc_eps, % (higher is better)	68.56	62.4	62.4 (MRLM)	first: above all 18 published models
Acc_theta, % (higher is better)	74.88	70.9	70.9 (MRLM)	first: above all 18 published models
Ref (lower is better)	0.0759	0.063	0.038 (LLaVA-v1.6-Vicuna-7B)	behind: 10 of the 18 published models are lower
Rel (lower is better)	0.392	0.535	0.215 (LLaVA-v1.6-Vicuna-7B)	behind: 5 of the 18 published models are lower

Per seed (official scorer, n = 2,000):

seed	Acc_eps	Acc_theta	Ref	Rel
0	68.45 %	74.66 %	0.0814	0.3780
1	69.10 %	74.40 %	0.0712	0.3447
2	68.80 %	75.18 %	0.0816	0.4294
3	68.10 %	74.87 %	0.0654	0.3700
4	68.35 %	75.29 %	0.0798	0.4392

• Acc_eps in detail: Wilson 95 % interval of the seed mean [66.49, 70.56]; frame-block bootstrap (71 blocks of 160 consecutive image ids, 5,000 draws) [64.90, 71.90]. Both lower bounds are above MRLM's 62.4 %. The Wilson interval treats images as independent; frames of the same video are not, which is why the block interval is the one that fixes the wording. MRLM has 8.7 B parameters; this model has 0.15 % of that. Acc_theta, Ref and Rel have no intervals; Acc_eps is the pre-registered statistic.
• Edge numbers (seed 0 weights): ONNX fp32 52.31 MB; static int8 13.17 MB with test Acc_eps 68.05 % (Wilson [65.97, 70.06]) against 68.45 % for fp32; one CPU thread, batch 1: 41.1 ms fp32 and 21.5 ms int8 on a Kaggle Xeon 2.2 GHz, 9.5 ms int8 on one Apple M4 thread.

Claim sentence, in the program's wording: best published Acc_eps (and Acc_theta) on DialBench RPM-10K, official split, official scorer, self-evaluated, 5 seeds, as of 2026-10-20 (literature re-checked that day; runs of 2026-10-17). Not claimed: "#1" or "SOTA"; any claim on Ref or Rel; any claim on the re-split below; any claim on MC1296.

What the result does and does not show

• Not reproduced. The published 62.4 % was not reproduced: the MRLM weights are on Baidu Netdisk, and the paper's split naming (Section 3.1 describes an 81 % / 19 % train/test split, Section 3.2 gives 8,730 images for training and 2,000 for "validation"), checkpoint selection and training recipe are not public. "Best published" rests on Table 4 of the paper (re-read on 2026-10-19). Literature checks on 2026-10-17 and 2026-10-20 (arXiv searches, papers citing the DialBench paper, the DialBench repository, web search) found no official-split RPM-10K result above 62.4 % Acc_eps or 70.9 % Acc_theta; the 77.0 % of arXiv 2609.31998 is on its own 1,382-image set, not on RPM-10K.
• The official split probably overlaps. By thumbnail similarity, test images are as close to train images as train images are to each other (docs/DIALBENCH-AUDIT.md in the kit), which is what a random split of video frames looks like; the authors have not published video or instance ids. The number then measures the official protocol, with near-duplicate frames on both sides, not generalisation to unseen videos. On split v3, a twin-purged id-block re-split of ours (stricter, not proven leak-free, no published number, so no "best published" wording attaches), the same recipe scores 54.40 % (3 seeds: 54.61, 54.07, 54.51; Wilson [52.12, 56.65]; 14-block bootstrap [46.72, 61.57]) against a no-learning nearest-neighbour label-copy floor of 34.11 %.
• One frozen configuration. Chosen on validation only (Stage A: ResNet-18 at 224 px, validation 68.09 %; the two other candidates that ran scored 66.36 and 64.74 %; four more were skipped by the pre-registered time guard), trained from scratch once per seed, each model scored once on the official test (EMA weights at the last epoch, window decoder, no test-time augmentation, no checkpoint selection). Every test look is in the kit's TEST-LOG.md; no run was dropped or replaced. GPU runs are not bit-reproducible: seeds fix initialisation, data order and augmentation draws, not cuDNN.
• Two ingredients, not separated. An earlier MobileNetV3-small (30 CPU epochs, one seed) scored 58.25 %. The same configuration trained for 100 GPU epochs validated at 64.74 % against 61.28 %, and the ResNet-18 reached 68.09 %: the gain comes from training length and backbone together, with no ablation between them.
• The predictions are model outputs, not labels, but they carry information about the labels: they fall within the scorer's tolerance of the label on 68.56 % of the official test rows. They contain no label, range or hit mask.

Protocol

• Target, fixed before any GPU run and never lowered: Acc_eps above MRLM's 62.4 % on the official split.
• Claim rule, fixed before the runs (FROZEN-4.md section 0 in the kit): the statistic is the seed mean H of the pre-registered primary configuration; "clears the target" needs the Wilson 95 % lower bound above 62.4 % (H of at least 64.523 %), and the frame-block bootstrap is printed beside it and fixes the wording. Result: H = 68.56 %, both lower bounds above 62.4 %.
• Selection on validation only: a stratified validation part of about 1,000 train.json images is held out; test.json is scored once per frozen configuration and every look is logged.
• Pre-registration: FROZEN-1.md to FROZEN-4.md in the kit, each committed before the thing it governs.
• Scorer: src/dialbench_metrics.py in the kit, numpy only, equivalence-tested against the authors' logic.

What is in this repository

file	what
README.md	this card
results/four_metric_table.md	the table above with its provenance
predictions/official_seed{0..4}.json, predictions/v3_seed{0..2}.json, predictions/README.md	per-seed predictions (file name, predicted number) on the official test rows and on the v3 test part, copied from the kit; no labels, ranges or hit masks; script paths in the README refer to the kit
CHANGELOG.md	dated changes to this card

Code, pre-registrations, test log, scorer tests and the Kaggle kernel sources: github.com/sankalpsthakur/dialbench-compact (Apache-2.0; CI runs the scorer tests and the manifest check).

Reproduce

Everything that touches RPM-10K pixels or labels ran inside private Kaggle kernels; the kit's README.md ("Reproduce") gives the kernel sources, script hashes and the exact push commands. With the authors' release and the kit you can regenerate the split manifests, rerun training on 2 x Tesla T4 (Stage B trained the five official-split seeds and the three v3 seeds in one kernel: 2 GPUs x 6,657 s = 3.70 billed GPU-hours for all eight runs) and re-score the saved predictions without any model.

Licences

• This repository (card, tables, predictions): Apache-2.0, like the kit.
• RPM-10K: licence "TBD" in the DialBench repository; nothing from the dataset is here except image file names and aggregate statistics.
• Weights: none released. They stay private until the authors state the terms for models trained on RPM-10K.
• The scorer port keeps the notice of its source (the DialBench repository's LICENSE file is MIT; its README says BSD 3-Clause).

Citation

DialBench and RPM-10K: Wang et al., DialBench (arXiv 2511.21982). This work: Sankalp Thakur, dialbench-compact, 2026, https://github.com/sankalpsthakur/dialbench-compact.

Card written 2026-10-20 (UTC); literature last checked 2026-10-20. Repeat the check before relying on the "best published" wording.