refer/README.md

Referring Image Segmentation

Getting Started

1. Install the required packages.

pip install -r requirements.txt

2. Prepare RefCOCO datasets following LAVT.

• Download COCO 2014 Train Images [83K/13GB] from COCO, and extract train2014.zip to ./refer/data/images/mscoco/images

• Follow the instructions in ./refer to download and extract refclef.zip, refcoco.zip, refcoco+.zip, refcocog.zip to ./refer/data

Your dataset directory should be:

refer/
├──data/
│  ├── images/mscoco/images/
│  ├── refclef
│  ├── refcoco
│  ├── refcoco+
│  ├── refcocog
├──evaluation/
├──...

Results and Fine-tuned Models of EVP

EVP achieves 76.35 overall IoU and 77.61 mean IoU on the validation set of RefCOCO.

Training

We count the max length of referring sentences and set the token length of lenguage model accrodingly. The checkpoint of the best epoch would be saved at ./checkpoints/.

• Train on RefCOCO

bash train.sh refcoco /path/to/logdir <NUM_GPUS> --token_length 40

• Train on RefCOCO+

bash train.sh refcoco+ /path/to/logdir <NUM_GPUS> --token_length 40

• Train on RefCOCOg

bash train.sh refcocog /path/to/logdir <NUM_GPUS> --token_length 77 --splitBy umd

Evaluation

• Evaluate on RefCOCO

bash test.sh refcoco /path/to/evp_ris_refcoco.pth --token_length 40

• Evaluate on RefCOCO+

bash test.sh refcoco+ /path/to/evp_ris_refcoco+.pth --token_length 40

• Evaluate on RefCOCOg

bash test.sh refcocog /path/to/evp_ris_gref.pth --token_length 77 --splitBy umd

Custom inference

PYTHONPATH="../":$PYTHONPATH python inference.py --img_path test_img.jpg --resume refcoco.pth --token_length 40 --prompt 'green plant'