vit-interp-e3sm: a close-to-vanilla ViT emulator of E3SMv3, built to be taken apart

A plain Vision Transformer that steps the E3SMv3 atmosphere (1Β°, AMIP) forward 6 hours. It was built for an interpretability hackathon: attention heads, edge circuits (Vi-CD) and physical-law tests (LAWFUL) with exege.

Results, the architecture explorer and executed notebooks: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.ngmahfouz/vit-interp/

Model

  • 34 prognostic fields (PS, TS; T, U, V, total water on 8 levels) + 5 forcings in, 6-hour tendency out.
  • 4°×4Β° patches β†’ 4050 tokens, learned position embedding, 8 pre-LayerNorm blocks Γ— 6 heads (dim 384), global attention, linear decoder; 16.2 M parameters. The state is carried forward and conservation corrections are applied outside the network.
file version recipe 5-day ACC (2010): PS / T_7 / U_3 / V_3
v1/v1.pt v1 2000–2009, MSE, one-step then 4- and 8-step rollouts 0.87 / 0.84 / 0.83 / 0.78
v2/v2.pt v2 1980–2009, + 10 diagnostic outputs, ACE-style correctors in the loss, 3Γ—3 output conv, EMA, input noise 0.92 / 0.89 / 0.87 / 0.84
v3/v3.pt v3 v2 made stochastic: noise-conditioned LayerNorms (32-dim per-token noise, zero-init), CRPS on 2-member ensembles, 4- then 8-step rollouts; best 10-day skill, spread/skill β‰ˆ 0.95 0.92 / 0.89 / 0.86 / 0.83
v4/v4.pt v4 v3 trained on random-length rollouts of 4–16 steps (1–4 days) with the gradient through the last step only; the year-long free run stays stable 0.89 / 0.86 / 0.84 / 0.80

v*/eval_2010.md has each version's full evaluation (year-long free run on 2010 and 1–10-day skill). v1 uses data-v1/; v2, v3 and v4 use data-v2/. data-v1/, data-v2/ hold the normalization statistics and grid metadata (stats.npz, meta.json; v2 also has the hybrid coefficients for the correctors). The training data are E3SMv3 output and are not included.

Use

The code is in the vit-interp repository (vit_interp.emulator.Emulator.load, vit_interp.playground, and the vit-interp adapter for exege). Free runs drift after several months (moist and cold for v2): few-day questions give the cleanest answers. v3 and v4 are stochastic: Emulator.step draws fresh noise on each call (pass generator= or noise= to fix it); Emulator.ensemble(x, n, seed) gives ensembles. Load any version by name with vit_interp.playground.load("v3").

Research artifact, work in progress.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support