Tabby-Pretrain

The pretrained PatchTST backbone behind Tabby. It is a encoder only patch transformer trained with a masked-patch objective and a 99-quantile pinball loss, so a single forward pass returns a full predictive distribution.

This repository holds the frozen backbone only — use it for zero-shot forecasting, or as a base to adapt.

Looking for the full model? The post-trained model, backbone and prompt merged into one checkpoint that runs on its own, is published as Tabby. The prompt module on its own is Tabby-Prompt.

Architecture

Parameters	145.84 M (253 tensors)
Layers	20
d_model	768
Attention heads	12 (head dim 64)
MLP hidden	3072 (4× d_model)
Patch size	16
Context window	8192 steps (512 patches)
Quantile levels	99 (0.01 … 0.99)
Dropout	0.1
Pretraining step	175,000
Precision	released in fp32; trained in bf16

The quantile head is monotone by construction: it predicts a base level plus cumulative softplus increments, so the 99 levels can never cross.

Files

File	What it is
model.safetensors	backbone weights, 583 MB, fp32
config.json	architecture config
training_args.json	the recorded pretraining hyperparameters (dataset paths redacted)

The published weights are bit-for-bit the model state dict of the original training checkpoint. Only the optimizer state was dropped, which is what takes the file from 1750 MB down to 583 MB. If you need to resume pretraining rather than run inference, the optimizer state is not in this repository.

Usage

The loader in the Tabby code recognises a config.json + model.safetensors directory and loads it with no key renaming:

from huggingface_hub import snapshot_download
from tabby_prompt.model import load_patchtstfm_backbone

path = snapshot_download("<namespace>/Tabby-Pretrain")
backbone, cfg, step = load_patchtstfm_backbone(path, device="cuda", code_dir=PRETRAIN_CODE)

step comes back as -1 for a snapshot directory — the format carries no step field. The real value, 175000, is in training_args.json and in the pretrain_step metadata key of model.safetensors.

To use it zero-shot through the same predict path Tabby uses, wrap it with prompt_len=0:

from tabby_prompt.model import PatchTSTFMPromptCFG, PromptedPatchTSTFM

cfg = PatchTSTFMPromptCFG(prompt_len=0, context_aware=False,
                          prediction_length=96, context_length=8096)
model = PromptedPatchTSTFM(backbone, cfg).to("cuda").eval()
out = model(context=x, prediction_length=96)   # out["quantile_preds"]: (B, 99, H)

Zero-shot reference

Measured through the Tabby evaluation harness at context_length=8096, each variate forecast independently:

Benchmark	MASE	CRPS
GIFT-Eval, 97 configs	0.7192	0.4922
TIME, 98 out-of-pool tasks	0.6915	0.5715

Both are Seasonal-Naive normalised geometric means. Because each variate is forecast independently, the TIME values are not aligned with the public TIME leaderboard and should only be compared against runs under the same protocol.

Training

Trained for 165,000 steps in bf16 on a mixture of real and synthetic series:

• real series from BLAST (ratio 0.3) and a GIFT-Eval pretraining pool (ratio 0.3), both domain-balanced;
• synthetic series from a Chronos KernelSynth corpus (ratio 0.3), plus CauKer-V2 (ratio 0.1).

Objective: masked patch modelling with mask_ratio=0.4 in contiguous blocks of 8 patches, plus terminal masking of 0–2 patches, scored with a 99-quantile pinball loss. Intermediate supervision is applied at layer exits [0, 5, 10, 15, 20] (lambda_ds=0.5, lambda_fm=0.1).

Optimizer: AdamW, β=(0.9, 0.95), weight decay 0.1, gradient clip 1.0, WSD schedule with 10,000 warmup and 20,000 decay steps, peak and floor LR both 1e-5, effective batch 1200 (200 × 6 accumulation steps).

Full values are in training_args.json.

License

CC BY-NC 4.0 — non-commercial use only.