thealper2/decision-transformer-halfcheetah-expert

Decision Transformer — HalfCheetah-v5 (halfcheetah-expert-v2)

Decision Transformer (Chen et al., 2021) trained offline on edbeeching/decision_transformer_gym_replay / halfcheetah-expert-v2 (D4RL) and evaluated autoregressively in Gymnasium HalfCheetah-v5.

Model

Architecture	causal GPT-style Transformer, pre-LayerNorm, token order (R̂ₜ, sₜ, aₜ)
Parameters	727,558
Hidden size / layers / heads	128 / 3 / 1
Context length K	20 timesteps (60 tokens)
Positional information	learned episode-timestep embedding (max 1000)
Dropout	0.1
Input	state 17-d (normalized), action 6-d, return-to-go / 1000
Output	action 6-d, tanh → [-1.0, 1.0]
Loss	MSE on actions at non-padding timesteps

Training data

• Dataset: edbeeching/decision_transformer_gym_replay, config halfcheetah-expert-v2, revision 4441c97718b1f7e03d05f430226b57f658cc156d
• 1000 trajectories × 1000 steps (1,000,000 transitions), no terminal states (time-limit truncation at 1000)
• Trajectory-level split: 900 train / 100 validation (seed 42)
• Train-split returns: min 2045.8, q05 10328.3, median 10699.4, q95 10962.2, max 11252.0

Preprocessing

• Return-to-go: RTGₜ = Σ_{t'≥t} r_{t'}, divided by 1000
• Observation normalization: (s − μ) / (σ + 1e-06), μ/σ computed on training trajectories only (normalization.json)
• Samples: every timestep of every training trajectory is the end of one K-step segment; segments shorter than K (episode start) are left-padded and masked

Training

Optimizer	AdamW, lr 0.0001, weight decay 0.0001
Schedule	10000 linear warmup steps, then cosine
Batch size	64 × 1 grad. accumulation
Epochs / optimizer steps	8 / 112,504
Gradient clipping	0.25
Precision	torch.bfloat16 autocast
Hardware	NVIDIA GeForce RTX 5060 Ti, PyTorch 2.11.0+cu128, CUDA 12.8
Training time	18.0 min
Final train / validation MSE	0.02672 / 0.02626
Checkpoint	best validation MSE 0.02626 (epoch 8, step 112,504)
Seed	42

Evaluation

Gymnasium HalfCheetah-v5 (MuJoCo 3.8.1), 10 episodes per row, env seeds 10000–10009, 1000 steps. Actions are clipped to the action bounds. BC = MLP state → action baseline trained on the same split (no return conditioning). D4RL normalized score uses the halfcheetah reference returns (-280.178953, 12135.0).

Model	Target	Target RTG	Mean Return	Std	D4RL Norm.	Episode Length	Success Rate
DT	low	2046	11058.2	38.1	91.3	1000	N/A
DT	medium	5626	11133.0	110.8	91.9	1000	N/A
DT	dataset_median	10699	11200.9	193.3	92.5	1000	N/A
DT	high	11252	11249.6	90.9	92.9	1000	N/A
DT	above_max	12377	11316.7	115.3	93.4	1000	N/A
BC	N/A	N/A	11105.5	91.7	91.7	1000	N/A

Usage

import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file

path = snapshot_download("thealper2/decision-transformer-halfcheetah-expert")
sys.path.insert(0, path)
from modeling_decision_transformer import DecisionTransformer

cfg = json.load(open(f"{path}/config.json"))
model = DecisionTransformer(**cfg["model_kwargs"]).eval()
model.load_state_dict(load_file(f"{path}/model.safetensors"))
mean, std = torch.tensor(cfg["normalization"]["mean"]), torch.tensor(cfg["normalization"]["std"])
eps, rtg_scale = cfg["normalization"]["eps"], cfg["rtg_scale"]

# history tensors, shape (1, t+1, ·): normalized states, past actions (zeros for the current step),
# RTG / rtg_scale (RTG_0 = target return, RTG_{t+1} = RTG_t - r_t), timesteps 0..t
# action = model.get_action(states, actions, rtg, timesteps)  # -> (1, act_dim)

Files

File	Content
model.safetensors	DT weights (best validation checkpoint)
config.json	model kwargs, normalization, RTG scale, env/dataset info, training config
normalization.json	observation mean / std / eps (training split)
modeling_decision_transformer.py	model definition (PyTorch only)
eval_results.json	per-episode evaluation returns and lengths
training_summary.json, training_metrics.jsonl	training summary and loss curves

Limitations

• Trained on expert data only: returns concentrate in [10328, 10962] (5–95%); targets outside this band are extrapolation.
• Data were collected with mujoco-py (MuJoCo 2.x); evaluation uses Gymnasium on MuJoCo 3.x.
• Single training seed.