xiaoyu1104/InstanceControl_depth

InstanceControl: Sa2va-Instance-4B (Stage 1)

This repository contains the Sa2va-Instance-4B checkpoint, which serves as Stage 1 (Instance Parsing Model) for InstanceControl, presented in the paper InstanceControl: Controllable Complex Image Generation without Instance Labeling.

• Project Page: InstanceControl Homepage
• GitHub Repository: InstanceControl GitHub
• Paper: arXiv:2606.31924

Model Description

InstanceControl is a multi-instance controllable generation method that eliminates the need for manual instance labeling. It uses a Vision-Language Model (VLM)—specifically this Sa2va-Instance-4B model—to automatically parse instance descriptions from text prompts and predict instance masks based on visual conditions (such as Canny edges, depth, or HED).

Usage

For detailed instructions on setup, environment installation, and running the inference pipeline, please refer to the official GitHub repository.

Predict Instance Masks (Stage 1)

You can run the model to predict instance masks using the following command:

python stage1_Sa2VA/projects/llava_sam2/evaluation/gcg_eval_our_folders.py \
  --model_path /path/to/Sa2va-Instance-4B \
  --image_dir ./example/canny \
  --json_dir ./example/json \
  --save_dir ./results/json_pred_canny

Citation

If you find this project useful, please cite the authors' work:

@article{liu2026instancecontrol,
  title={InstanceControl: Controllable Complex Image Generation without Instance Labeling},
  author={Xiaoyu Liu and Huan Wang and Fan Li and Zhixin Wang and Jiaqi Xu and Ming Liu and Wangmeng Zuo},
  journal={arXiv preprint arXiv:2606.31924},
  year={2026}
}