ut-amrl/foresight-qwen3vl-2b-sft

Foresight — Qwen3-VL-2B SFT

The supervised fine-tuned vision-language model behind Foresight, a navigation policy that iteratively discovers instruction-relevant visual clues and refines its motion plans for open-world navigation.

• Project page: https://amrl.cs.utexas.edu/foresight/
• Paper: https://amrl.cs.utexas.edu/foresight/static/pdfs/ForesightPreprint.pdf

What this checkpoint does

Given a natural-language goal and a short history of RGB observations, the model proposes an image-space trajectory and critiques its own proposal. Motion planning and critique were co-trained into this single checkpoint, so both roles are served by one vLLM engine at deployment time.

Converting the accepted plan into metric BEV waypoints is handled by a separate grounding policy, which is not part of this repository.

Usage

The deployment stack loads this model through vLLM by repo ID. In legged_deployment/config/vllm_server_sft.yaml:

model_overrides:
  vlm:
    provider_kwargs:
      model: "ut-amrl/foresight-qwen3vl-2b-sft"

It also loads directly with transformers:

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "ut-amrl/foresight-qwen3vl-2b-sft", dtype="bfloat16", device_map="auto"
)
processor = AutoProcessor.from_pretrained("ut-amrl/foresight-qwen3vl-2b-sft")

Prompts are constructed by the Foresight codebase; see https://github.com/ut-amrl/foresight_public for the runnable deployment stack.

Training

Fine-tuned from Qwen/Qwen3-VL-2B-Instruct with LoRA (rank 64, alpha 64) applied to all linear projections in both the language and vision towers. The adapter has been merged into the base weights, so no PEFT dependency is needed at inference. Weights are bfloat16.

Citation

@article{zhang2026foresight,
  title={Foresight: Iterative Reasoning About Clues that Matter for Navigation},
  author={Zhang, Arthur and Qi, Carl and Su, Donne and Meng, Xiangyun and Zhang, Amy and Biswas, Joydeep},
  journal={arXiv preprint arXiv:2606.12550},
  year={2026}
}