zotowata/pc-sho-dlm-code

PC-SHO-DLM

Predictive-Coding Diffusion Language Models with Precision-Conditioned Second-Order Settling

Overview

PC-SHO-DLM is a research architecture that unifies three individually mature research threads:

1. Masked discrete diffusion for iterative text denoising
2. Predictive coding for local error-driven inference and learning
3. Second-order dynamical systems for accelerated latent settling

The core insight: diffusion denoising and predictive coding are both iterative residual-correction procedures, making diffusion a structurally natural host for predictive-coding-based language modeling.

Key Contributions

• Bidirectional predictive transformer with both bottom-up and top-down pathways
• Precision-conditioned second-order settling (damped inertial dynamics with learned uncertainty)
• State continuation across diffusion steps (warm-start from previous denoising step)
• Tokenwise adaptive settling (selective compute allocation to uncertain tokens)
• Globally backpropagation-free training via local prediction-error-driven updates

Theoretical Results

Theorem	Result
A: Reduced-objective exactness	Local post-settling updates = exact gradients via envelope theorem
B: Accelerated settling	O(sqrt(kappa)) vs O(kappa) microsteps over first-order PC
C: Warm-start tracking	Settling cost depends on inter-step drift, not full re-solve
D: Precision optimality	GLS-optimal under heteroscedastic prediction errors

Project Structure

pc-sho-dlm/
├── paper/
│   ├── main.tex          # Full LaTeX manuscript
│   └── references.bib    # Bibliography
├── src/
│   ├── model.py          # Core PC-SHO-DLM architecture
│   └── train.py          # Training loop (local + backprop modes)
├── experiments/
│   ├── ablation_runner.py     # Ablation study runner
│   └── settling_analysis.py   # Settling dynamics analysis
└── README.md

Quick Start

Sanity Check

cd src
python model.py

Train (Local PC mode)

python src/train.py --mode local --device cpu --max_steps 5000

Train (Backprop baseline)

python src/train.py --mode backprop --device cpu --max_steps 5000

Run Ablations

python experiments/ablation_runner.py --device cpu

Analyze Settling Dynamics

python experiments/settling_analysis.py --device cpu

Experimental Program

Stage 1: Proof of Mechanism (20M-100M params)

• Character/subword corpora (text8, enwik8, WikiText-103)
• Establish second-order PC is more compute-efficient than first-order PC

Stage 2: Medium Scale (100M-1B params)

• Open web corpus subset
• Compare against backprop-trained masked diffusion

Stage 3: Structured Generation

• Infilling, editing, code completion
• Test selective settling and state continuation

Falsification Criteria

This proposal should be judged a failure if:

1. Second-order settling does not reduce required microsteps
2. Local learning creates an irrecoverable quality gap vs backprop
3. Precision heads fail to calibrate
4. Dynamics frequently enter limit cycles
5. State continuation does not materially reduce compute

Citation

@article{pcshodlm2026,
  title={PC-SHO-DLM: Predictive-Coding Diffusion Language Models
         with Precision-Conditioned Second-Order Settling},
  year={2026}
}