A JEPA Recipe for Tabular Foundation Models — released checkpoints
The two trained arms reported in A JEPA Recipe for Tabular Foundation Models (Mingyu Jeon, Suwan Cho, Jae Young Suh; Modulabs, 2026). Code, configs and the evaluation results are at https://github.com/fbdeme/jepa-pfn. arXiv identifier: to be added once announced.
Tabular foundation models learn to predict cell values in context, whereas world-model self-supervision asks for prediction in representation space. On a tabular foundation-model prior, the latent term of a joint-embedding predictive architecture (JEPA) collapsed in our earlier runs and took the encoder with it to a constant map. We report a recipe under which the latent term survives to convergence beside the value objective: the value head reads the encoder field rather than the predictor, and the target is an exponential moving average (EMA) difference. To bound its cost against the value-only arm, both arms train until a plateau rule stops them, with no fixed step budget. A fixed horizon had confounded a slowdown with a ceiling, since the value-only arm was still improving well past the usual budget. At convergence, in one run per arm, the JEPA arm trails the value-only arm across 147 real datasets, 32:70 wins to losses on classification (29:63 with one entry per dataset name) and 8:24 on regression, the margin small on classification and wider on regression, and the count leans the same way in each stratum and each benchmark. The JEPA arm (jepa) needs 1.42 times as many steps as the value-only arm (ds), and 1.66 times its wall-clock, to reach its plateau.
Checkpoints
Each folder holds the final ckpt.pt ({"model": state_dict, "cfg": training config}), the
config.yaml it was trained from, and the full metrics.jsonl training log. Both arms use the
same 6-layer, 256-wide cell-level in-context transformer (6.62M parameters; the JEPA checkpoint holds 17.52M because it also stores the EMA target copy and the predictor, its online encoder being 6.62M) and were
trained on the TabICL graph-SCM prior until a plateau rule stopped them (no fixed step budget:
stop when the best validation value MSE has not improved by 0.002 for 80 consecutive validations).
| Arm | What | Folder | Stop step | Wall-clock | Final val MSE | ckpt.pt |
|---|---|---|---|---|---|---|
ds |
value-only PFN arm | PFN_tabicl2_ds_conv_s0/ |
178,500 | 24.8 h | 0.4179 | 27 MB |
jepa |
JEPA arm (latent term beside the value objective) | TabularJEPA_v3_tabicl2_headenc_conv_s0/ |
253,750 | 41.3 h | 0.4360 | 70 MB |
Loading
git clone https://github.com/fbdeme/jepa-pfn && cd jepa-pfn && uv sync
uv run hf download fbdeme/jepa-pfn --local-dir runs/
from eval.realdata_bench import load_pfn # builds the right module from ckpt["cfg"]
ds = load_pfn("PFN_tabicl2_ds_conv_s0")
jepa = load_pfn("TabularJEPA_v3_tabicl2_headenc_conv_s0")
The real-data suite the paper reports (147 OpenML-CC18 / Grinsztajn-2022 / TabArena datasets) runs with
python -m eval.suite_bench in that repository; see its README.
Training hardware
One RTX PRO 6000 Blackwell (96 GB); the JEPA arm peaked at about 79 GB of GPU memory.
License
Apache-2.0 (weights and code). The architecture derives from nanoTabPFN (Apache-2.0); the training
prior is the vendored tabicl/prior generator (BSD-3-Clause, Soda team @ Inria).