ut-amrl/foresight-waypoint-policy

Foresight — Waypoint Grounding Policy

The spatial grounding policy used by Foresight, a navigation policy that iteratively discovers instruction-relevant visual clues and refines its motion plans for open-world navigation.

• Project page: https://amrl.cs.utexas.edu/foresight/
• Paper: https://amrl.cs.utexas.edu/foresight/static/pdfs/ForesightPreprint.pdf
• Code: https://github.com/ut-amrl/foresight_public

What this checkpoint does

The Foresight VLM proposes a trajectory in image space. This policy lifts that image-space plan into metric BEV waypoints for motion control: it consumes an RGB observation plus the rendered plan mask and predicts 10 waypoints of dimension 3.

This is the 48m distance variant, referred to as the 4.8m planning policy in the deployment configs.

Usage

Download into the location the deployment configs expect:

mkdir -p checkpoints/waypoint_policy
hf download ut-amrl/foresight-waypoint-policy \
  gtpassthrough_xformer_kp384_48m.ckpt --local-dir checkpoints/waypoint_policy

legged_deployment/config/waypoint_planner.yaml already points at this path and sets the matching architecture overrides:

grounding_policy.config_name: "model/waypoint/gtpassthrough_xformer"
grounding_policy.checkpoint_path: "checkpoints/waypoint_policy/gtpassthrough_xformer_kp384_48m.ckpt"
grounding_policy.action_head.num_kp: 384
grounding_policy.action_head.dim_feedforward: 384
grounding_policy.action_head.nhead: 8
grounding_policy.action_head.num_layers: 8

These action_head values must match the checkpoint or the weights will not load.

Architecture

Component	Setting
Observation encoder	Depth-Anything-V2 ViT-S, layers 2/5/8/11, out dim 384
Path encoder	EfficientNet-B0, 1 input channel, out dim 64
Spatial encoder	ConvNetSpatial over EfficientNet-B0, spatial softmax with 384 keypoints, learnable temperature
Action head	TransformerActionHead, d_model 128, 8 heads, 8 layers, FFN 384
Output	10 waypoints x 3 dims

The observation encoder expects Depth-Anything-V2 ViT-S weights, which install_all.sh fetches separately.

Training

Trained for 50 epochs with AdamW (lr 3e-5, weight decay 0.01), cosine schedule with 10% warmup, batch size 32 across 3 GPUs (DDP), Huber loss on waypoint regression. Checkpoint selected on best val/loss.

training_config.yaml in this repo is the full Hydra config from the training run.

This is a complete PyTorch Lightning checkpoint, so it still carries optimizer state alongside state_dict. The deployment loader reads the state_dict key and ignores the rest.

Citation

@article{zhang2026foresight,
  title={Foresight: Iterative Reasoning About Clues that Matter for Navigation},
  author={Zhang, Arthur and Qi, Carl and Su, Donne and Meng, Xiangyun and Zhang, Amy and Biswas, Joydeep},
  journal={arXiv preprint arXiv:2606.12550},
  year={2026}
}