ModalSearcher-8B-SFT

ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search

Code & Documentation | SFT + RL Model | Training Data | Benchmark | arXiv (placeholder)

ModalSearcher-8B-SFT is the supervised fine-tuned checkpoint of ModalSearcher, based on Qwen3-VL-8B-Instruct. It is trained for tool-assisted multimodal deep search: deciding what to search, which modality to use, and when to switch modalities while gathering visual and textual evidence for multi-hop questions.

This repository contains the SFT-only model weights, tokenizer, and processors. Use this checkpoint for inference or as the initialization for further reinforcement learning. For the SFT + RL version, see hansQAQ/ModalSearcher-8B.

Model details

Field	Value
Base model	Qwen3-VL-8B-Instruct
Post-training	Supervised fine-tuning (SFT)
Architecture	Qwen3VLForConditionalGeneration
Weight format	Safetensors, BF16
Intended use	Multimodal search, multi-hop question answering, and RL initialization

Tools

The ModalSearcher inference runner executes the model's tool calls and feeds the resulting evidence back into the conversation.

Tool	Purpose
t2t_search	Retrieve textual evidence and candidate webpages
t2i_search	Search for images from a text query
i2i_search	Reverse-image search, including cropped image regions
read_url	Read webpages and PDFs or inspect retrieved images

For complete search trajectories, run these weights with the ModalSearcher inference runner, configured tool services, and the auxiliary model used for webpage evidence extraction.

Quick start

Clone the project:

git clone https://github.com/Hans-M-Yin/ModalSearcher.git
cd ModalSearcher

Complete the project's Setup, including the inference environment and tool services, then download the checkpoint:

hf download hansQAQ/ModalSearcher-8B-SFT --local-dir ./models/ModalSearcher-8B-SFT

Serve the SFT model with SGLang. Select the GPUs for this service with CUDA_VISIBLE_DEVICES:

nohup python -m sglang.launch_server \
  --model-path ./models/ModalSearcher-8B-SFT \
  --served-model-name ModalSearcher-8B-SFT \
  --tp 2 --port 6657 --context-length 50000 \
  > /tmp/sglang.log 2>&1 &

Set the agent endpoint in the project's .env:

export AGENT_BASE_URL="http://127.0.0.1:6657/v1"
export AGENT_MODEL_NAME="ModalSearcher-8B-SFT"
export AGENT_API_KEY="EMPTY"

Run inference and LLM-as-judge evaluation using the commands in the project's Evaluation section. The inference runner saves tool-use trajectories and final answers; the judge produces per-sample judgments and aggregate accuracy reports.

Training and evaluation resources

• ModalSearch-21K-Full: training data.
• ModalSearch-Bench: evaluation benchmark.
• SFT and RL training code: training frameworks and launch commands.

Limitations

Answers depend on the retrieved evidence and the availability of external tool services. The model can return incorrect or unsupported answers; verify factual claims against the cited sources.

The arXiv link above is a placeholder and will be replaced when the paper is available.