promotion/nbpo-policy-diagnostics

NBPO Section-7 policy checkpoints (diagnostic; the gate did not pass)

None of these is a recommended model. Every arm here failed the pre-registered neural-realization gate. They are published because each one is the artifact behind a reported number, and a negative result should be as checkable as a positive one.

The gate, and what happened

A policy is asked to realize the finite-pool Eq. (26) target: the pairwise log-ratio change

h = [log pi(y|x) - log pi(z|x)] - [log pi_t(y|x) - log pi_t(z|x)]

should track eta * z. The gate required held-out normalized MSE < 0.90, sign agreement > 0.65, and positive Pearson and Spearman. No arm cleared it; the best held-out normalized MSE was slightly above 1.0, which is what predicting nothing scores.

arm	target estimator	eta	val nMSE	test nMSE	test sign	test Pearson	test Spearman
eta0p1	sampled	0.1	2.319	2.362	0.485	-0.011	-0.018
eta0p3	sampled	0.3	1.121	1.182	0.490	-0.030	-0.008
eta1p0	sampled	1.0	1.020	1.026	0.505	-0.026	+0.001
lr2em7	sampled	1.0	1.006	1.016	0.497	-0.008	-0.004
lr1em6	sampled	1.0	1.020	1.028	0.512	+0.003	+0.026
DIAGlr1em5	sampled	1.0	22.308	18.623	0.517	-0.008	+0.020
DIAGrb	rao_blackwell	1.0	1.070	1.126	0.535	+0.025	+0.055
DIAGrb_clip100	rao_blackwell	1.0	1.105	1.093	0.542	+0.064	+0.089
DIAGrb_steps1200	rao_blackwell	1.0	1.074	1.077	0.560	+0.089	+0.114

nMSE is MSE / Var(target) on held-out prompts, each arm scored at its own training eta. An arm that never moved would score 1.0.

Target estimators

estimator	what it integrates	max attainable $r^2$
sampled	nothing -- one comparator draw and two Bernoulli flips	0.142
rao_blackwell	the Bernoulli flips only	0.983
canonical	comparator and flips: the exact finite-pool expectation	1.000

The three are unbiased for the same quantity and differ only in variance.

What the diagnostics established

The trainer is not broken: on 8 prompts and 224 pairs the same code reduces the loss 87% below its h = 0 baseline. Longer training helps monotonically (held-out Pearson +0.025 at 300 steps, +0.089 at 1200). Relaxing gradient clipping 100x barely moves the result. One real defect was found and is recorded rather than fixed away: at learning_rate = 0 the online log-probabilities do not reproduce the cached ones, so h carries a spurious component of RMS ~0.35 in the TRAINING signal. Evaluation is unaffected -- both of its terms come from the same cached path.

Contents

Each arms/<label>/ holds the checkpoint, its run_config.yaml, and gate.json with the full metric record including both exact MSE identities.

Licence

CC BY-NC 4.0, inherited from the PKU-SafeRLHF supervision. Base model meta-llama/Llama-3.1-8B-Instruct; preference supervision from promotion/nbpo-saferlhf-preference-models.