tiny-random/timestral

This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from mistralai/Timestral-209M.

Structural comparison with mistralai/Timestral-209M:

Feature	Source	Tiny
Forecasting interface	univariate, multivariate, past/known covariates	same
Stored precision	F32	F32
Patch size / block size	32 / 32	same
Transformer layers	24	8; two 3-time + 1-time/variate cycles
Width / attention heads / head dimension	768 / 12 / 64	64 / 1 / 64
SwiGLU intermediate width	2,048	128
Quantile knots	21	same
DHF two-stream decoder and monotone quantile head	present	same

This checkpoint uses Mistral's public timestral inference package (release commit a534bdbd244d64f3d9d743f46c7bf6fc330ed4ff): install it with pip install git+https://github.com/mistralai/timestral.git before running the example. It is a forecasting model, not a Transformers causal-LM or SGLang/vLLM serving model.

File path	Size
model.safetensors	2.7MB

Example usage:

• Transformers (✅ verified)

import torch
from timestral import TimestralForecaster

model_id = "tiny-random/timestral"
device = "cuda" if torch.cuda.is_available() else "cpu"
forecaster = TimestralForecaster.from_pretrained(model_id, device=device, per_shot_horizon_patches=32)
context = torch.sin(torch.arange(96, dtype=torch.float32) / 6).reshape(1, 1, -1)
past = torch.cos(torch.arange(96, dtype=torch.float32) / 5).reshape(1, 1, -1)
known = torch.linspace(0, 1, 128, dtype=torch.float32).reshape(1, 1, -1)
forecast = forecaster.forecast(context, prediction_length=32, past_covariates=past, known_covariates=known)
print(forecast.quantiles.shape)

Codes to create this repo (✅ verified):

Click to expand

import json
from pathlib import Path
from shutil import copyfile

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import save_file
from timestral import TimestralConfig, TimestralModel

source_model_id = "mistralai/Timestral-209M"
save_folder = "/tmp/tiny-random/timestral"
source_revision = "9874d252ab0aae16f7f7c999f98f98d6ad7d8365"
torch.manual_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
# Keep two source 3-time + 1-time/variate cycles.  A 64-wide model with one
# head preserves Timestral's 64-wide attention head exactly.
raw.update(
    d_model=64, num_heads=1, mlp_hidden_dim=128, context_length=1024,
    block_schedule=["time", "time", "time", "time_variate"] * 2,
)
(folder / "config.json").write_text(json.dumps(raw, indent=2) + "\n", encoding="utf-8")
config = TimestralConfig.load(folder / "config.json")
model = TimestralModel(config).float()
with torch.no_grad():
    for name, parameter in sorted(model.named_parameters()):
        if name.endswith("scale"):
            parameter.fill_(1)
        else:
            torch.nn.init.normal_(parameter, mean=0, std=0.02)
save_file(model.state_dict(), folder / "model.safetensors", metadata={"format": "pt"})
for filename in ("LICENSE", "timestral-hourglass.png"):
    try:
        copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename)
    except Exception:
        # The executable checkpoint consists solely of config.json and model.safetensors.
        # Preserve optional presentation assets when their source revision supplies them.
        pass

Printing the model:

Click to expand

TimestralModel(
  (representation): PatchRepresentation(
    (patch_proj): ResidualBlock(
      (hidden_layer): Linear(in_features=64, out_features=256, bias=True)
      (output_layer): Linear(in_features=256, out_features=64, bias=True)
      (residual_layer): Linear(in_features=64, out_features=64, bias=True)
      (activation): SiLU()
    )
  )
  (transformer): Transformer(
    (layers): ModuleList(
      (0-2): 3 x TransformerBlock(
        (attn_norms): ModuleList(
          (0): RMSNorm()
        )
        (attns): ModuleList(
          (0): Attention(
            (wQKV): Linear(in_features=64, out_features=192, bias=False)
            (wO): Linear(in_features=64, out_features=64, bias=False)
            (_gate): Linear(in_features=64, out_features=64, bias=False)
          )
        )
        (mlp_norm): RMSNorm()
        (mlp): FusedSwiGLU(
          (w12): Linear(in_features=64, out_features=256, bias=False)
          (w3): Linear(in_features=128, out_features=64, bias=False)
        )
      )
      (3): TransformerBlock(
        (attn_norms): ModuleList(
          (0-1): 2 x RMSNorm()
        )
        (attns): ModuleList(
          (0-1): 2 x Attention(
            (wQKV): Linear(in_features=64, out_features=192, bias=False)
            (wO): Linear(in_features=64, out_features=64, bias=False)
            (_gate): Linear(in_features=64, out_features=64, bias=False)
          )
        )
        (mlp_norm): RMSNorm()
        (mlp): FusedSwiGLU(
          (w12): Linear(in_features=64, out_features=256, bias=False)
          (w3): Linear(in_features=128, out_features=64, bias=False)
        )
      )
      (4-6): 3 x TransformerBlock(
        (attn_norms): ModuleList(
          (0): RMSNorm()
        )
        (attns): ModuleList(
          (0): Attention(
            (wQKV): Linear(in_features=64, out_features=192, bias=False)
            (wO): Linear(in_features=64, out_features=64, bias=False)
            (_gate): Linear(in_features=64, out_features=64, bias=False)
          )
        )
        (mlp_norm): RMSNorm()
        (mlp): FusedSwiGLU(
          (w12): Linear(in_features=64, out_features=256, bias=False)
          (w3): Linear(in_features=128, out_features=64, bias=False)
        )
      )
      (7): TransformerBlock(
        (attn_norms): ModuleList(
          (0-1): 2 x RMSNorm()
        )
        (attns): ModuleList(
          (0-1): 2 x Attention(
            (wQKV): Linear(in_features=64, out_features=192, bias=False)
            (wO): Linear(in_features=64, out_features=64, bias=False)
            (_gate): Linear(in_features=64, out_features=64, bias=False)
          )
        )
        (mlp_norm): RMSNorm()
        (mlp): FusedSwiGLU(
          (w12): Linear(in_features=64, out_features=256, bias=False)
          (w3): Linear(in_features=128, out_features=64, bias=False)
        )
      )
    )
    (rope): SinCosRoPE()
    (out_norm): RMSNorm()
  )
  (head): QuantileHead(
    (projection): ResidualBlock(
      (hidden_layer): Linear(in_features=64, out_features=256, bias=False)
      (output_layer): Linear(in_features=256, out_features=672, bias=False)
      (residual_layer): Linear(in_features=64, out_features=672, bias=False)
      (activation): SiLU()
    )
  )
)

Test environment:

• huggingface_hub: 1.27.0
• numpy: 2.5.2
• safetensors: 0.8.0
• timestral: 1.0.0
• tokenizers: 0.23.0rc0
• torch: 2.13.0+cu126
• transformers: 5.15.0.dev0

Change log:

• 2026-10-23: Initial version.

This repo is generated by GPT-6.1-Sol