oxfrug/chronos-2-int8-torchao

Smaller Chronos-2 (INT8)

Chronos-2 is Amazon's 120M time-series model: you give it history, it forecasts the next hours/days, no extra training.

This repo is the same model with smaller files. Weights are stored as 8-bit integers (INT8). It is not a fine-tune and not a new architecture.

	Amazon original	this repo
download	478 MB	131 MB
GPU memory while running (3090, one series)	~0.56 GB	~0.25 GB
Python API	Chronos2Pipeline.from_pretrained	load.py (this repo)
license	Apache-2.0	Apache-2.0

Pick this if you want Chronos-2 in Python, but a lighter download and less VRAM.

Pick amazon/chronos-2 if you want the one-liner load and do not care about 350 MB.

Pick an ONNX / TensorRT Chronos-2 if you only care about production latency. Those are a different runtime, not this Python path.

---

Install

pip install "chronos-forecasting>=2.0" torchao safetensors huggingface_hub pandas pyarrow

GPU: a recent PyTorch with CUDA. CPU works; it will be slower.

---

Load (required)

Hugging Face's usual from_pretrained cannot read these INT8 files. Use load.py from this repo:

from pathlib import Path
import importlib.util
from huggingface_hub import snapshot_download

repo = snapshot_download("oxfrug/chronos-2-int8-torchao")

spec = importlib.util.spec_from_file_location("c2int8", Path(repo) / "load.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)

pipe = mod.load(repo, device="cuda")  # "cpu" if you have no GPU

If you already cloned the folder:

from load import load
pipe = load("/path/to/chronos-2-int8-torchao", device="cuda")

You get a normal Chronos2Pipeline. After this, Amazon's docs apply.

---

Forecast

One numpy series:

import numpy as np

history = np.array([12.0, 12.4, 11.9, ...], dtype=np.float32)  # oldest → newest
quantiles, mean = pipe.predict_quantiles(
    inputs=[history],
    prediction_length=24,
    quantile_levels=[0.1, 0.5, 0.9],
)
# median forecast: quantiles[0][:, 1]  (shape: 1 × horizon × 3)

Many series / covariates — same as upstream, with pipe.predict_df(...).

---

Make it faster

INT8 here is smaller, not faster, on an RTX 3090. The model is small; most of the time is starting GPU kernels, not moving weights.

Two switches help both this INT8 and the original FP32 model:

1. TF32 — tell PyTorch to use the GPU's fast float path (it is off by default).
2. torch.compile — fuse those kernels. First call is slow (compile); later calls drop a lot.

import torch
from fast_infer import speedup   # file in this repo

torch.set_float32_matmul_precision("high")
pipe = speedup(pipe)  # TF32 + torch.compile

Or from a terminal, after downloading this folder:

python fast_infer.py                         # this INT8
python fast_infer.py --fp32 amazon/chronos-2 # original model, same knobs

Timed on this machine (RTX 3090, 512 past points, forecast 24, one series):

	time per call	notes
original FP32	~7 ms	no extra knobs
this INT8	~9 ms	smaller, slightly slower
FP32 + compile + TF32	~3 ms	best speed here
INT8 + compile + TF32	~3.6 ms	still a bit behind compiled FP32

Forecasting many series in one call (predict_df with several ids) is the other real win. Amazon's “hundreds of series per second” numbers are batched, not one sine wave.

Half-precision (FP16 / BF16) did not help this 120M model on a 3090.

---

Did INT8 change the forecasts?

This pack only compresses weights.

What i did: take a few public series, hide the last 12–168 points, forecast them with the original model and with this INT8, compare.

• German electricity (hourly) — INT8 stays close. 24h median error vs original +1.8%; 168h actually −3%. Correlation of the two forecasts ≈ 0.99.
• M4 hourly / daily / monthly — same story on shape (high correlation), median error a bit worse (about +9% to +15% MASE on those short holds).
• M4 weekly (13 steps) — the two forecasts diverged (correlation near zero). Do not read that row as “INT8 is better.” Short weekly holds are noisy.

P10–P90 intervals: on 12–24 step holds, both models often cover ~50–60% of points instead of 80%. Compare INT8 to the original on your series if you use the bands, not to the textbook 80%.

Raw dumps: eval/series.json, eval/coverage.json, eval/speed.json.

---

How the file was made

• Start from amazon/chronos-2 (full float weights).
• torchao weight-only INT8: each big linear layer stores integers + a scale. Activations stay float.
• The small quantile head (the part that turns hidden states into P10/P50/P90) is left in float, on purpose.
• No calibration data, no extra training.

That is why from_pretrained fails: Hugging Face does not know this packing. load.py rebuilds the layers and fills them from model.safetensors.

---

Limits

• Smaller files, less GPU RAM — not a speed-up by itself on a 3090.
• Not a domain fine-tune (no Nordic energy LoRA, etc.).
• Not GIFT-Eval / fev-bench.
• If you need intervals, check coverage on your data.

---

Cite the base model

Ansari et al., Chronos-2: From Univariate to Universal Forecasting, 2025.
https://arxiv.org/abs/2510.15821

Quant pack by oxfrug. """