vit-interp-e3sm: a close-to-vanilla ViT emulator of E3SMv3, built to be taken apart
A plain Vision Transformer that steps the E3SMv3 atmosphere (1Β°, AMIP) forward 6 hours. It was built for an interpretability hackathon: attention heads, edge circuits (Vi-CD) and physical-law tests (LAWFUL) with exege.
Results, the architecture explorer and executed notebooks: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.ngmahfouz/vit-interp/
Model
- 34 prognostic fields (PS, TS; T, U, V, total water on 8 levels) + 5 forcings in, 6-hour tendency out.
- 4Β°Γ4Β° patches β 4050 tokens, learned position embedding, 8 pre-LayerNorm blocks Γ 6 heads (dim 384), global attention, linear decoder; 16.2 M parameters. The state is carried forward and conservation corrections are applied outside the network.
| file | version | recipe | 5-day ACC (2010): PS / T_7 / U_3 / V_3 |
|---|---|---|---|
v1/v1.pt |
v1 | 2000β2009, MSE, one-step then 4- and 8-step rollouts | 0.87 / 0.84 / 0.83 / 0.78 |
v2/v2.pt |
v2 | 1980β2009, + 10 diagnostic outputs, ACE-style correctors in the loss, 3Γ3 output conv, EMA, input noise | 0.92 / 0.89 / 0.87 / 0.84 |
v3/v3.pt |
v3 | v2 made stochastic: noise-conditioned LayerNorms (32-dim per-token noise, zero-init), CRPS on 2-member ensembles, 4- then 8-step rollouts; best 10-day skill, spread/skill β 0.95 | 0.92 / 0.89 / 0.86 / 0.83 |
v4/v4.pt |
v4 | v3 trained on random-length rollouts of 4β16 steps (1β4 days) with the gradient through the last step only; the year-long free run stays stable | 0.89 / 0.86 / 0.84 / 0.80 |
v*/eval_2010.md has each version's full evaluation (year-long free run on 2010 and 1β10-day skill).
v1 uses data-v1/; v2, v3 and v4 use data-v2/. data-v1/, data-v2/ hold the normalization statistics and grid metadata (stats.npz, meta.json; v2 also has
the hybrid coefficients for the correctors). The training data are E3SMv3 output and are not included.
Use
The code is in the vit-interp repository (vit_interp.emulator.Emulator.load, vit_interp.playground, and the
vit-interp adapter for exege). Free runs drift after several months (moist and cold for v2): few-day questions
give the cleanest answers. v3 and v4 are stochastic: Emulator.step draws fresh noise on each call
(pass generator= or noise= to fix it); Emulator.ensemble(x, n, seed) gives ensembles. Load any version by name with
vit_interp.playground.load("v3").
Research artifact, work in progress.