Molmo2Fish

Molmo2Fish is Molmo2-8B fine-tuned to track fish in sonar video, and to edit those tracks in response to natural-language feedback.

Track correction is treated as a conversation: the model is shown a set of existing tracks — its own earlier predictions, another tracker's output, or corrupted ground truth — is told in words what is wrong with them, and returns a repaired set of tracks.

Code	github.com/tidalove/molmo2fish
Data	tidalove/cfc-track-instruction
Source video	Caltech Fish Counting

Files

file	what it is
*.safetensors, config.json, …	HuggingFace-format weights at the repo root — what vLLM and launch_scripts/hf_eval.py consume
Molmo2Fish-step420-raw.tar	the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning

Training

Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all three components — LLM, ViT, and connector — with the base weights frozen:

torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \
  --lora_llm --lora_vit --lora_connector --lora_rank 64 \
  --save_folder=/path/to/save/folder

The cfc_correction mixture combines pure tracking, targeted correction, synthetically corrupted correction, correction of real model predictions at two quality levels (molmo_high / molmo_low), and text-only correction. See the dataset card for what each config contains.

Citation

@article{molmo2fish,
    title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance},
    author={Kai van Brunt and Justin Kay and Sara Beery},
    year={2026},
    url={https://arxiv.org/abs/2608.18602}
}