Paper: ZegOT: Zero-shot Segmentation Through Optimal Transport of Pixels to Text Prompts (ICLR-2024)

Reviewer comment (Experiments):
The experiment setting is not clear. For example, if the proposed method can effectively solve the problem of overfitting the network parameters to the seed class distribution after training, why not conduct a set of experiments under the setting of Inductive (unseen class names and images are not accessible during training). In Table 2, the mIoU (U) and mIoU (S) of ZegCLIP are 87.3 and 92.3 respectively, but hiou should not be 91.1 and should be 89.7. The setting used in the experiment in Table 2 is Conductive, but Table 3 seems to be using Inductive settings with no explanation provided. Would using Transductive settings be better than the previous method? In addition, the author claims to have obtained a SOTA, but it is 2.2 hiou lower on COCO-Stuff164K. For Table 4, I would like to know how much improvement can be achieved by using only MPOT compared to the baseline.

Author response:
>**W2-1**. The experiment setting is not clear to me. For example, If the proposed method can effectively solve the problem of overfitting the network parameters to the seed class distribution after training, why not conduct a set of experiments under the setting of Inductive (unseen class names and images are not accessible during training.).

->  In general, zero-shot semantic segmentation performance in the inductive setting is quite limited to beat the supervised methods and be directly utilized in practice. In this paper, we focus on improving zero-shot segmentation performance which can be comparable to that of the supervised learning methods, thus we conduct experiments under the transductive setting. By efficiently utilizing only class names for unseen classes, our methodology under the transductive setting boosts overall segmentation performance. Nevertheless, per the reviewer’s suggestion, we have implemented the proposed framework in the inductive setting and proved the comparable results to SOTA methods as in the following table.

***Table 1.*** Performance comparison on PASCAL VOC 2012 dataset for the Inductive setting.
| Methods   | IoU(U) | IoU(S) | hIoU |
|-----------|:------:|:------:|:----:|
| ZegFormer |  63.6  |  86.4  | 73.3 |
| Zsseg     |  72.5  |  83.5  | 77.6 |
| ZegCLIP   |  77.8  |  91.9  | 84.3 |
| Ours      |  71.6  |  86.5  | 78.4 |
>**W2-2**.In Table 2, the mIoU (U) and mIoU (S) of ZegCLIP are 87.3 and 92.3 respectively, but hIoU should not be 91.1 and should be 89.7. 

-> Thanks for your careful comments. We have updated the results of ZegCLIP in the revised paper. 

>**W2-3**. The setting used in the experiment in table2 is Conductive. But does Table 3 seem to be using Inductive settings? No explanation was provided, and would using Transductive settings be better than the previous method.

-> Thanks for your careful observation and valuable recommendation. As the reviewer pointed out, we trained our model using transductive settings on the source dataset and solely performed inference on the target test set. To ensure a fair comparison with other methods, we employ the weight of ZegCLIP, which is trained using transductive settings. 

>**W2-4**. In addition, the author claims to have obtained a sota, but it is 2.2 hIoU lower on COCO-Stuff164K. 

-> Thanks for your valuable comments. Per your suggestion, we have narrowed the claim down and revised it. 

>**W2-5**. For table4, I would like to know how much improvement can be achieved by using only MPOT compared to baseline.

->The reviewer is kindly reminded that the results you referred to are already indicated in Table 4 (b) and marked with an 'X'. These results confirm that our proposed MPOT significantly enhances performance, demonstrating a substantial margin of improvement with 2.9 and 2.4 mIoU on the PASCAL VOC and Pascal Context datasets, respectively.

Reviewer comment (Experiments):
The authors include ZegCLIP to "trainable image encoder-based approaches". However, they fix the image encoder and train a new decoder. Such a statement is not accurate.

Author response:
Thanks for your careful comment. This was because ZegCLIP employs a vision prompt tuning approach for tuning the image encoder, which we consider as a trainable encoder.  That being said, we agree with the reviewer, so we now categorize ZegCLIP as a frozen image encoder-based approach.

Reviewer comment (Novelty):
In related work, the authors do not introduce open vocabulary semantic segmentation, which is highly related to this topic in this paper.

Author response:
Thanks for your concrete and helpful comment. We have expanded the scope of zero-shot semantic segmentation to include open-vocabulary semantic segmentation topic in the related work section 2.2 in the revised paper.

Reviewer comment (Evaluation):
The performance of COCO-Stuff does not outperform ZegCLIP. It seems that the proposed method is more useful on simple images such as PASCAL VOC.

Author response:
As the reviewer pointed out, we admit that relying on a few learnable text prompts and lightweight decoder with limited trainable parameters limits the room for performance improvement on a large-scale dataset, compared to the retraining/fine-tuning of the image backbone. We expect this may be solved in a future study, by introducing an advanced MPOT, which balances the trainable parameters and performance for large-scale datasets. We have updated this limitation in the revised paper.

Reviewer comment (Novelty):
The optimal transport plan is just like a spatial attention map for each class and each prompt. Adding a self-attention layer or learnable spatial attention maps might also be effective.

Author response:
The reviewer is kindly reminded that a comparison with the results of the self-attention layer is already presented in Table 4 (b) of the main paper, as indicated  'Self-attention'. Additionally, we compare the results with other matching methods using a one-to-one mapping, referred to as the Bipartite matching method, within the same table. As you correctly commented, both the Self-attention and the Bipartite matching methods yield meaningful results; however, our proposed ZegOT outperforms these methods on both the Pascal VOC and Pascal Context datasets.

Reviewer comment (Experiments):
It seems that the proposed Optimal Transport-based method can be plugged into other methods, such as ZegCLIP. Is it possible to use Optimal Transport in ZegCLIP?

Author response:
Thanks for your concrete suggestion. Per your suggestion, we conducted the experiment to incorporate MPOT into ZegCLIP. Due to the instinctual nature of the framework including network architecture, we integrate MPOT as an additional pipeline for computing loss. To follow the original scheme of ZegCLIP, we use the multiple hand-crafted prompts and use the same number of multiple prompts, as 4. The comparison results are presented in the following table, and we demonstrate that our proposed MPOT acts as a plug-in module for further boosting the performance of the SOTA model. We have added the related contents in the revised manuscript.  

***Table 5.*** Performance of ZegCLIP + MPOT on PASCAL datasets.
| Methods         | Multiple Prompts (N=4) | IoU(U) | IoU(U) | IoU(S) | hIoU |
|-----------------|:------:|:------:|:----:|:----:|:----:|
| ZegCLIP           |  X |  X  |89.9| 92.3 | 91.1 |
| ZegCLIP(reproduced) |  X |  X | 89.7 | 92.2 | 90.9 |
| ZegCLIP (a)        |  O  |  X  | 92.3 | 93.5 | 92.9 |
| ZegCLIP (b) |  O |  O  | 93.0 | 92.8 | 92.9 |

Reviewer comment (Evaluation):
What is the reason behind the similarity of the multiple learned prompts?

Author response:
-> Thanks for your valuable comments. If we do not use our proposed optimal transport, the approach of using multiple learnable text prompts becomes aligning the average features of text prompts with the visual features, yielding all the text prompts cohered. Unfortunately, this cohered text prompts are not well-aligned with diverse features of semantic visual embeddings. Similar negative observations were made in the PLOT, where they conducted image classification with multiple prompts [1]. This is our main motivation of using optimal transport for multi-prompt alignment.

[1] Chen et.al. PLOT: Prompt Learning with Optimal Transport for Vision-Language Models, ICLR2023

Reviewer comment (Reproducibility):
The performance of ZegCLIP is different from that in the original paper.

Author response:
-> Thanks for your careful observation. We have updated the latest results of ZegCLIP in the revised paper.

Reviewer comment (Experiments):
It seems the method is trained in a transductive setting; What is the performance in an inductive setting?

Author response:
-> In general, zero-shot semantic segmentation performance in the inductive setting is quite limited to beat the supervised methods, so that it cannot be directly utilized in practice. Accordingly, our methodology has focused on improving zero-shot segmentation performance, which can be comparable to that of the supervised learning methods. Consequently, we conduct experiments under the transductive setting, by efficiently utilizing only class names for unseen classes to boost overall segmentation performance. Nevertheless, per the reviewer’s suggestion, we have implemented the proposed framework in the inductive setting and proved the comparable results to SOTA methods as in the following table.



***Table 1.*** Performance comparison on PASCAL VOC 2012 dataset for the Inductive setting.
| Methods   | IoU(U) | IoU(S) | hIoU |
|-----------|:------:|:------:|:----:|
| ZegFormer |  63.6  |  86.4  | 73.3 |
| Zsseg     |  72.5  |  83.5  | 77.6 |
| ZegCLIP   |  77.8  |  91.9  | 84.3 |
| Ours      |  71.6  |  86.5  | 78.4 |

Reviewer comment (Experiments):
How does this work under different backbones? Previous works sometimes use Resnet for ZS3. Please consider a comparison.

Author response:
-> Thanks for the constructive comment. We investigated different backbones used in various baseline models for ZS3  as shown in Table 2 below, and the ResNet backbones are mostly utilized as feature backbone. That being said, we don’t need the feature backbone, since our architecture is VLM and decoder-only architecture. Accordingly, we believe that  the suggested experiments with different backbone are not necessary to verifying the performance of our algorithm.


***Table 2.*** Different backbones for ZS3.
| Model     | VLM(CLIP) | Feautre Backbone | Decoder |
|-----------|:------:|:------:|:----:|
| zsseg |  ViT |  ResNet  | - |
| ZegFormer     |  ViT  |  ResNet  | FPN |
| FreeSeg   |  ViT  |  ResNet  | - |
| MVP-SEG      |  ViT  |  DeepLab  | - |
| ZegCLIP      |  ViT  |  -  | Light-weight ViT  |
| Ours      |  ViT  |  -  | FPN / ASPP |

Reviewer comment (Experiments):
The number of classes seems small. What are the results on datasets with larger numbers of classes, such as Pascal Context 459 and ADE-847?

Author response:
-> Thanks for the great comment. To demonstrate our proposed model’s generalization performance on large-scale datasets with larger numbers of classes, we conduct open vocabulary segmentation by training the model with full COCO-171. We are running the experiments now, however, due to the limited computational resources, the results are not prepared. We will try to update the results before the end of the rebuttal period.