analysis3054 0.5.2
Advanced time-series analytics and forecasting toolkit for commodity and power trading
Requires Python: >=3.10
Keywords: forecasting, time-series, energy, commodities, statistics, pandas

# Analysis3054

Analysis3054 is a Python toolkit for time-series forecasting, market calendar logic, statistical diagnostics, and API-backed data ingestion.

This README is intentionally package-function focused. It documents what is implemented in the Python modules in this repository, with practical usage examples.

## Install

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

Python requirement: `>=3.10`.

## Module Guide

### Forecasting (`analysis3054/forecasting.py`)

This module contains broad forecasting coverage, including:

- Classical: `arima_forecast`, `sarimax_forecast`, `ets_forecast`, `var_forecast`, `vecm_forecast`, `theta_forecast`
- Volatility/regime: `garch_forecast`, `markov_switching_forecast`, `dynamic_factor_forecast`, `unobserved_components_forecast`
- ML/DL methods: `xgboost_forecast`, `lightgbm_forecast`, `catboost_forecast`, `svr_forecast`, `lstm_forecast`, `tcn_forecast`, `transformer_forecast`, `neuralprophet_forecast`
- Chronos/TimeFM family: `chronos2_forecast`, `chronos2_univariate_forecast`, `chronos2_multivariate_forecast`, `chronos2_covariate_forecast`, `chronos_bolt_forecast`, `timesfm_forecast`
- Quantile/anomaly/imputation utilities for Chronos models
- Intraday and demand-focused helpers (burn-day and covariate workflows)

Most functions return typed result objects (`*ForecastResult`) with forecast outputs and model metadata.

### Auto-ML Forecasting (`analysis3054/auto_ml_forecasting.py`)

Feature generation and model-oriented forecast utilities, including:

- `auto_generate_features`
- `bayesian_ridge_covariate_forecast`
- `elastic_net_covariate_forecast`
- `lightgbm_covariate_forecast`
- `xgboost_covariate_forecast`
- `stacked_meta_ensemble_forecast`
- `chronos2_auto_covariate_forecast`

### Forecast Engine (`analysis3054/forecast_engine.py`)

Composable engine API:

- `ForecastEngine`
- `build_default_engine`
- `EngineForecastResult`

Useful when standardizing forecast pipelines across datasets.

Current engine capabilities include:

- model registry introspection (`available_models`, `model_registry`)
- custom backend registration (`register_model`)
- sklearn-style estimator registration (`register_estimator`)
- covariate horizon propagation with automatic future-covariate synthesis
- standardized result fields (`training_time_seconds`, `hyperparameters`, `diagnostics`)

## Capability Map

```mermaid
flowchart LR
    A[analysis3054] --> B[Forecasting APIs]
    A --> C[Forecast Engine]
    A --> D[Stats + Diagnostics]
    A --> E[Holiday Calendars]
    A --> F[Utilities]
    A --> G[DTN API + CSV Updaters]

    B --> B1[Classical + ML + DL + Chronos/TimeFM]
    C --> C1[Model Registry + Covariate Horizon + Diagnostics]
    D --> D1[ADF + Ljung-Box + ACF/PACF + PCA]
    G --> G1[PADD Daily + Rack Daily]
    G1 --> G2[Full Backfill]
    G1 --> G3[Rolling 14-Day Repull]
```

## Forecasting Capability Chart

| Capability Area | Coverage | Representative APIs |
|---|---|---|
| Classical statistical forecasting | `█████` | `arima_forecast`, `sarimax_forecast`, `ets_forecast`, `var_forecast`, `vecm_forecast`, `theta_forecast` |
| Machine learning forecasting | `█████` | `xgboost_forecast`, `lightgbm_forecast`, `catboost_forecast`, `svr_forecast`, `knn_forecast` |
| Deep learning forecasting | `████` | `lstm_forecast`, `tcn_forecast`, `transformer_forecast`, `neuralprophet_forecast` |
| Foundation model forecasting | `█████` | `chronos2_forecast`, `chronos_bolt_forecast`, `timesfm_forecast` |
| Quantile / probabilistic outputs | `█████` | `chronos2_quantile_forecast`, `chronos_bolt_quantile_forecast`, interval-ready model results |
| Covariate-driven forecasting | `█████` | `chronos2_covariate_forecast`, `*_covariate_forecast` in `auto_ml_forecasting.py` |
| Feature-engineering automation | `████` | `auto_generate_features`, `chronos2_feature_generator` |
| Intraday / demand workflows | `████` | `intraday_*` and burn-focused forecast helpers in `forecasting.py` |

Quick interpretation:
- `█████` = mature and broad API coverage in this package.
- `████` = strong, but narrower than top-tier categories.

### Calendar & Holiday APIs

- `analysis3054/holiday_calendars.py`
  - `available_holiday_calendars`
  - `get_holidays`
  - `get_holidays_between`
- `analysis3054/holiday_lookup.py`
  - `is_holiday`
  - `is_financial_holiday`
  - `is_platts_holiday`
  - `resolve_iso_code`
  - `get_market_code`

### Time-Series Utilities (`analysis3054/utils.py`)

Common data engineering helpers:

- Merge/coalesce: `conditional_column_merge`, `conditional_row_merge`, `nearest_key_merge`, `coalesce_merge`
- Time ops: `add_time_features`, `resample_time_series`, `rolling_fill`, `rolling_window_features`, `add_lag_features`
- Quality/transforms: `winsorize_columns`, `scale_columns`, `data_quality_report`
- Domain helper: `get_padd`

### Statistics, Regression, Estimation, Finance, Visualization

- `analysis3054/stats.py`: PCA, cross-correlation, Granger causality, ADF/Ljung-Box diagnostics, ACF/PACF with confidence intervals
- `analysis3054/regression.py`: OLS regression, rolling correlation, CUSUM tests
- `analysis3054/estimators.py`: Bayesian linear, Gaussian process, load-based estimators
- `analysis3054/finance.py`: rolling beta, liquidity-adjusted volatility
- `analysis3054/visualization.py` and `analysis3054/plot.py`: forecast plots, drawdown, ACF/PACF, EIA-style five-year plotting

## Quick Usage

### 1) Forecast Example

```python
import pandas as pd
from analysis3054 import arima_forecast

series = pd.Series([100, 103, 105, 104, 108], index=pd.date_range("2025-01-01", periods=5, freq="D"))
result = arima_forecast(series, horizon=7)
print(result.forecast)
```

### 2) Holiday Lookup Example

```python
from analysis3054 import is_financial_holiday

print(is_financial_holiday("2026-12-25", market="US"))
```

### 3) Utility Example

```python
import pandas as pd
from analysis3054 import add_time_features

df = pd.DataFrame({"date": pd.date_range("2026-01-01", periods=3), "value": [1, 2, 3]})
out = add_time_features(df, date_col="date")
print(out.columns)
```

### 4) ForecastEngine Example (Covariates + Diagnostics)

```python
import pandas as pd
import numpy as np
from analysis3054 import build_default_engine

df = pd.DataFrame(
    {
        "date": pd.date_range("2026-01-01", periods=40, freq="D"),
        "demand": np.linspace(100, 140, 40) + np.sin(np.linspace(0, 6, 40)),
        "temp": np.linspace(30, 45, 40),
    }
)

engine = build_default_engine()
res = engine.forecast(
    df,
    date_col="date",
    target_cols=["demand"],
    model="harmonic",
    horizon=7,
    covariate_cols=["temp"],
    validation_size=7,
)

print(res.forecasts.head())
print(res.diagnostics)
```

## DTN Refined Fuels Demand Puller (PADD + Rack)

Implementation lives in:

- `analysis3054/refined_fuels_api.py`
- `analysis3054/data/kayross/run_dtn_daily.py`

### Supported behavior

- Full backfill (`full history`)
- Rolling repull (default `last 14 days`, inclusive window)
- CSV upsert/merge by endpoint-specific keys
- Same behavior for `padd-daily` and `rack-daily`

### DTN auth options

Use one of:

- `DTN_API_KEY`
- `DTN_ACCESS_TOKEN`
- OAuth client credentials: `DTN_CLIENT_ID` + `DTN_CLIENT_SECRET` (optional `DTN_AUDIENCE`)

### Programmatic API

```python
from analysis3054 import RefinedFuelsUSMDClient, update_padd_daily_csv, update_rack_daily_csv

client = RefinedFuelsUSMDClient(api_key="<your_api_key>")

# Rolling update (default 14 days)
padd_df = update_padd_daily_csv(client, "analysis3054/data/kayross/dtn_refined_fuels.csv")
rack_df = update_rack_daily_csv(client, "analysis3054/data/kayross/dtn_refined_fuels_rack.csv")

# One-time full backfill
update_padd_daily_csv(client, "analysis3054/data/kayross/dtn_refined_fuels.csv", full_history=True)
update_rack_daily_csv(client, "analysis3054/data/kayross/dtn_refined_fuels_rack.csv", full_history=True)
```

### CLI: CSV-only daily updater

```bash
python analysis3054/data/kayross/run_dtn_daily.py
```

Full backfill:

```bash
python analysis3054/data/kayross/run_dtn_daily.py --full-history
```

Custom rolling window:

```bash
python analysis3054/data/kayross/run_dtn_daily.py --lookback-days 21
```

Optional filters:

```bash
python analysis3054/data/kayross/run_dtn_daily.py \
  --dtn-regions 1,2,3 \
  --dtn-products Distillates \
  --dtn-grades "#2 Diesel" \
  --dtn-states TX,LA \
  --rack-average N
```

## Windows Task Scheduler (Daily Run)

Use this for automated daily DTN CSV refresh.

1. Open **Task Scheduler**.
2. Create a new task (not basic task).
3. Trigger: Daily, choose time.
4. Action: **Start a program**.
5. Program/script: path to `python.exe` in your environment, for example:
   - `%USERPROFILE%\\Analysis3054-Codex\\.venv\\Scripts\\python.exe`
6. Add arguments:
   - `%USERPROFILE%\\Analysis3054-Codex\\analysis3054\\data\\kayross\\run_dtn_daily.py`
7. Start in:
   - `%USERPROFILE%\\Analysis3054-Codex`
8. Save task and run once manually to validate output CSV updates.

Optional one-time historical seed task:

- Same command with `--full-history`, then switch back to default daily command.

## Validation

```bash
pytest -q
python3 -m compileall -q analysis3054
```

## DTN Documentation Baseline

DTN integration here is aligned to Refined Fuels Demand OpenAPI `v1.3.0` (as of February 15, 2026):

- Documentation: https://devportal.dtn.com/catalog/Refined%20Fuels/dtn-refined-fuels-demand/documentation#get-/padd-daily
- Release notes: https://devportal.dtn.com/catalog/Refined%20Fuels/dtn-refined-fuels-demand/release-notes
- Local OpenAPI source used for implementation checks:
  - `/Users/alexhoffmann/Downloads/DTN Refined Fuels Demand-1.3.0-OpenAPI.json`