|
--- |
|
license: apache-2.0 |
|
--- |
|
# PolyFormer: Referring Image Segmentation as Sequential Polygon Generation |
|
|
|
[Project](https://polyformer.github.io/) | [GitHub](https://github.com/amazon-science/polygon-transformer) | [Demo](https://huggingface.co/spaces/koajoel/PolyFormer) |
|
|
|
## Model description |
|
|
|
PolyFormer is a unified framework for referring image segmentation (RIS) and referring expression comprehension (REC) by formulating them as a sequence-to-sequence (seq2seq) prediction problem. For more details, please refer to our paper: |
|
|
|
[PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://arxiv.org/abs/2302.07387) |
|
Jiang Liu*, Hui Ding*, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, R. Manmatha, [CVPR 2023](https://cvpr2023.thecvf.com/Conferences/2023/AcceptedPapers) |
|
|
|
## Training data |
|
|
|
We pre-train PolyFormer on the REC task using Visual Genome, RefCOCO, RefCOCO+, RefCOCOg, and Flickr30k-entities, and the finetune on REC + RIS task using RefCOCO, RefCOCO+, |
|
and RefCOCOg. |
|
|
|
* PolyFormer-B: Swin-B as the visual encoder, BERT-base as the text encoder, 6 transformer encoder layers and 6 decoder layers. |
|
* PolyFormer-L: Swin-L as the visual encoder, BERT-base as the text encoder, 12 transformer encoder layers and 12 decoder layers. |
|
|
|
## Citation |
|
|
|
If you find PolyFormer useful in your research, please cite the following paper: |
|
|
|
``` latex |
|
@article{liu2023polyformer, |
|
title={PolyFormer: Referring Image Segmentation as Sequential Polygon Generation}, |
|
author={Liu, Jiang and Ding, Hui and Cai, Zhaowei and Zhang, Yuting and Satzoda, Ravi Kumar and Mahadevan, Vijay and Manmatha, R}, |
|
journal={arXiv preprint arXiv:2302.07387}, |
|
year={2023} |
|
} |
|
``` |