udwm-identifiability-diagnostic

Diagnostic artifact for a falsifiable claim about distillation loss design, not a trained agent:

Matching teacher–student value disagreement at a single shared latent does not identify epistemic uncertainty. The matched statistic is one number, w* + (g* − Σ̄), that mixes two unknowns: ensemble disagreement (epistemic, w) and per-member aleatoric noise (g). A one-step student can keep that number exactly while trading the two against each other — and along that trade the decision statistic u = w − g, the object model- based RL algorithms gate on, is not identifiable (its sign is not even identifiable).

This checkpoint set reproduces the failure mode itself so that researchers working on calibrated model-based RL (MBPO-style loops that decide how much to trust imagined rollouts from an uncertainty signal) can load, probe, and extend it. Full research record: github.com/nisaral/uncertainty-diffusion-world-models.

What the artifact contains

• checkpoints/ — DelayedBimodal seed-0 checkpoints (teacher + distilled one-step students), one per arm:
  • ordinary_seed0.pt — baseline member matching, no decision loss.
  • hybrid_seed0.pt — the M=1 single-latent decision loss (the object of the identifiability claim).
  • identified_hybrid_seed0.pt — the broken EMA-reweighted identified arm (aleatoric channel annihilated; u-rank collapses to ~noise).
  • identified_eq_seed0.pt — the corrected equal-weight M>=2 arm (recovery, with the known w hole).
  • identified_eq_norm_seed0.pt — equal-weight M>=2 + value-target standardization (the 2026-09-07/08 re-read: standardization is the empirically active knob; u-rank 0.95 on the live critic).
  • lagged_hybrid_seed0.pt, lagged_identified_eq_seed0.pt, lagged_identified_eq_nonorm_seed0.pt — the slow target-critic ("lagged") arms.
  • Each trainer checkpoint stores teacher + student + agent + u_net (trainer.save format).
• configs/delayed_bimodal_distill.yaml — the registered protocol config.
• arms_seed0.json — the eval-time endpoint rows for those arms.
• theory/identifiability_frontier.py + research/proofs/ identifiability-frontier.md — verified theorems (T1–T7) + Proposition 8 (level-set geometry; the M=1 objective's decision-statistic uncertainty does not shrink as population risk → 0; M≥2 identifies with rate √ε).
• research/ — the registered results docs: normalization control (G7), re-adjudication with in-file baselines (G9), DMC budget probe + Amendment 2 (2026-09-08), combined-fix, DMC sanity, corrected-weight, H2/H4.

What the numbers say (as of 2026-09-08)

• Mechanism (adjudicated, fixed configs): M=1 hybrid u-rank collapses in policy; EMA-both annihilates g (collapse control reproduces cross- environment); equal-weight M>=2 recovers the decision-statistic rank.
• Attribution (G7/G9, N=10): with normalize_values held fixed the lagged-critic axis adds ~0 to u-rank on DelayedBimodal; standardization alone lifts the corrected arm to the measured ceiling on the live critic (identified_eq_norm u-rank 0.948, 10/10 ≥ 0.70).
• DMC budget probe (2026-09-08): the 3,600-step operating point was a budget confound; baseline u-rank climbs to 0.93/0.86 at 15k steps. Diagnostic at n=2: the combined lagged arm does not top the DMC table — the lag-axis verdict is open and is being adjudicated at 30 seeds × 15k.

Use

The intended use is diagnostic: load a teacher/student pair, evaluate the local UBE decision statistic u = w − g on real-buffer states under your own value map, and compare student-vs-teacher rank (not RMSE) — rank is the scale-free endpoint (percentile gating is invariant to monotone score transforms; absolute thresholds silently disable the gate on a scale- collapsed student).

Notes

• Research artifact. No SOTA or production claims; all studies are self-gated MBPO/SAC on a small benchmark (DelayedBimodal-v0; DMC hopper-hop-v0 for the out-of-sample transfer thread).
• Gate-payoff usage in model-based RL is a crowded 2026 area (MACURA, AAAI'26 dynamic-uncertainty-filter work, ELVIS, GIRL — see the repo's bib). This artifact is positioned at the mechanism that sits underneath all of them: whether distillation of a diffusion world model silently destroys the "I'm not sure" signal the gate depends on.