Paper: Adaptive Self-training Framework for Fine-grained Scene Graph Generation (ICLR-2024)

Reviewer comment (Experiments):
There are some SOTA SGG methods which might get higher performance than this paper. These methods should also be included for comparisons and analyses, such as VETO[1], GCL[2], PE-Net-Reweigh[3], and so on.  
[1] Vision Relation Transformer for Unbiased Scene Graph Generation, ICCV2023.  
[2] Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation, CVPR 2022.  
[3] Prototype-based Embedding Network for Scene Graph Generation, CVPR 2023.

Author response:
**(W2) Experiments with more state-of-the-art baselines**

We express our appreciation to the reviewer for mentioning studies [1, 2, 3] that we had missed. 

- **Comparison with [2]:** We compared with [2], which utilizes the group collaborative learning (GCL). We updated Table 1 of the main paper in **the modified PDF** to include a comparison with GCL. We observed that GCL significantly improves mR@K, whereas it sacrifices a considerable amount of R@K. Hence, our proposed ST-SGG still surpasses GCL in terms of F@K metrics, which implies a great balanced performance on the overall predicate and tail predicates of our method. 

- **Comparison with [3]:** We also compared with PE-Net [3], which is a new architecture of SGG that utilizes prototype embedding methods for entities and their relationships. However, a direct comparison with PE-Net would be unfair as we used a very simple backbone model, e.g., Motif and VCTree, to apply the self-training framework. Hence, for fair comparisons, we implemented self-training to PE-Net and compared PE-Net+ST-SGG with PE-Net. Please note that we used vanilla PE-Net since PE-Net-Reweight was missing in the official code (https://github.com/VL-Group/PENET). Moreover, we trained PE-Net and PE-Net+ST-SGG with the same hyperparameters, and we fixed the hyperparameters for ST-SGG as: EMA $\alpha^{inc}=0.4$, EMA $\alpha^{dec}=0.4$, the loss coef. $\beta=1.0$, without searching them.


|   **Task**  |                 | **PredCls**|              |                  |**SGCls**|                  |
|:-------------:|:-------------:|:-------------|-------------:|:-------------:|:-------------:|:-------------:|
| **Method** |  **R@50 / 100**  |  **mR@50 / 100** | **F@50 / 100** | **R@50 / 100**  |  **mR@50 / 100** | **F@50 / 100** |
| PE-Net                             |    65.0 / 67.1  |  28.7 / 30.6   |   39.8 / 42.1   |    38.3 / 39.3    |   18.8 / 19.9  |    25.2 / 26.4  |
|   **PE-Net + ST-SGG**    |  62.2 / 64.6    |  31.6 / 33.7   |   **41.9 / 44.3**   |    36.2 / 37.3    |    20.9 / 22.0 |    **26.5 / 27.7**  |


In the above table, we observed that when ST-SGG is applied to PE-Net, the performance is increased in terms of F@K. This implies that the proposed ST-SGG appropriately assigns pseudo-labels to enhance the generalization of PE-Net. This result further demonstrates that our method is indeed model-agnostic to any SGG models.

We have added [2] and [3] into our revised paper. Please see **BLUE** rows of Table 1 in **the modified PDF file.**

Lastly, please understand that regarding the paper [1], as the proceeding of ICCV 2023 was released after the submission date of our paper, it was impossible to compare it with ours. 


[1] Vision Relation Transformer for Unbiased Scene Graph Generation, ICCV2023. 

[2] Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation, CVPR 2022. 

[3] Prototype-based Embedding Network for Scene Graph Generation, CVPR 2023.

Reviewer comment (Experiments):
Comparison with other debiasing methods missing. The paper demonstrated that their method can improve the diversity of the baseline models. However, such improvement of baseline models is prevalent in SGG literature now. Therefore, a direct comparison with other debiasing methods would shed light more on their performances. For example, VCTree+ST-SGG should be compared with VCTree+Resample. With that comparison, the readers would have better understanding of their method’s performance.

Author response:
**(W1) Lack of comparison with debiasing methods. e.g., VCTree + Resamp**

Please note that as for the specific example that the reviewer mentioned (i.e., comparison between VCTree+ST-SGG and VCTree+Resample [1]), we have indeed compared them in rows 18 and 19 of Table 1. 

Upon the reviewer’s request, we provide analysis on the comparisons with debiasing methods presented in Table 1. We observe that when ST-SGG is applied to Motif/VCTree, it shows a lower F@K compared with the case when Resampling is applied to Motif/VCTree. This is primarily attributed to the severe long-tailed distribution in the SGG nature, and as a result, a straightforward resampling approach appears to be more effective in addressing the imbalance of relation classes. However, we observe that applying both Resaming and ST-SGG to Motif/VCTree significantly outperforms Motif/VCTree+Resam. This implies that utilizing the pseudo-label can further enhance the model’s generalization even when resampling is applied.

Based on this result, we argue that ST-SGG stands out as an effective method for improving the generalization capabilities of SGG models. Its ease of implementation and compatibility with other debiasing techniques are notable advantages. Specifically, ST-SGG requires only the addition of missing annotations and the computation of an extra loss component associated with pseudo-labeled predicates. These attributes underscore ST-SGG’s potential as a versatile and potent tool in the field of SGG.


[1] Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation. CVPR’21. Ronjie Li et al.

Reviewer comment (Experiments):
Zero-shot evaluation missing. Since unannotated triplets are being accounted in the training, this method might improve the zero-shot performance of the baseline model as well. A separate study with zero-shot evaluation would strengthen the claim of the paper.

Author response:
**(W2) Zero-shot performance of the baseline model as well**

We thank the reviewer for giving insightful feedback, which has illuminated aspects of our work that were previously unaddressed in the paper. We conducted an additional experiment to evaluate the zero-shot performance. Following the zero-shot settings [1],  we compute the recall@K values for specific <subject, predicate, object> pairs, which are not included in the training set but emerge in the testing phase. This experiment is conducted under the PredCls task.


| Method | zR@50 / 100 | Method | zR@50 / 100 |
|----------|----------|----------|----------|
| Motif | 16.3 / 19.4 | VCTree | 16.7 / 19.7 |
| Motif + Resampling | 17.0 / 19.5 | VCTree + Resampling | 16.3 / 19.4 |
| Motif + IE-Trans | 13.0 / 15.9 | Motif + IE-Trans | 14.0 / 16.5 |
| **Motif + ST-SGG** | **18.0 / 21.0** | **Motif + ST-SGG** | **19.8 / 21.0** |



The above table reveals that debiasing techniques, including resampling and IE-Trans, do not enhance the zero-shot performance of SGG models. To elaborate, the limitation of the resampling approach lies in its simplistic alteration of the number of instances involved in training for each predicate class, which does not augment any variance in the data. As for IE-Trans, it generates pseudo-labels only for tail predicates classes, which restricts the ability of finding zero-shot triplets. On the other hand, the application of ST-SGG shows improvements in the zero-shot performance for both Motif and VCTree models. This improvement suggests that ST-SGG can generate pseudo-labels that are instrumental in enhancing the model's generalization to unseen triplet patterns. The effectiveness of ST-SGG in this context underscores its potential as a robust tool for improving the generalization capabilities of SGG models, where the ability to predict novel patterns is crucial.

We would like to extend our appreciation to the reviewer once again for suggesting the point that strengthens the advantage of our work. We have updated these results in Appendix E.5 of **the modified PDF file.**


[1] Unbiased Scene Graph Generation from Biased Training. Tang et al. CVPR’20.

Reviewer comment (Theory):
The authors determine the threshold in an object-agnostic fashion considering only the relationship classes. However, the same relationship label has different semantic meaning to different pair and therefore an object-conditional viewpoint of relationship distribution may improve the result more.

Author response:
**(Q1) Object-conditional relation class thresholding. (based on commonsense knowledge)**

We agree with the reviewer’s suggestion that object-conditional relation pseudo-labeling is also a promising direction of self-training for SGG. The object-conditional pseudo-labels are expected to strengthen our approach by reflecting commonsense knowledge to decide whether pseudo-labeling or not. We suggest two approaches based on the SGG works with commonsense knowledge [1, 2].

- One approach would be to utilize the frequency of triplets as a commonsense knowledge, which is obtained from counting the appearance of <subject, predicate, object> in the training dataset [1]. For example, if a predicate “on” appears more frequently between “man-horse” than between “man-pizza” in the training data, then we encourage the pseudo-label <man, on, horse> to contribute more to the loss than the pseudo-label <man, on, pizza>, given that these two pseudo-labels are obtained by ST-SGG. However, although this simple approach can enhance the accuracy of pseudo-label generation, it is not expected to improve the zero-shot performance as it allows pseudo labels to be assigned only when the generated triplets emerge in the training set.
- Another approach would be to utilize the knowledge graph from external sources. By comparing the triplet in the commonsense knowledge graph, such as ConceptNet [3], we can confirm the generated pseudo-labels are valid or not. In this case, we can reduce the number of incorrect pseudo-labels by filtering the pseudo-label based on commonsense.

We express our gratitude to the reviewer for bringing to light issues that we had overlooked, thanks to their valuable insights. We acknowledge the significance of these issues and will consider them as part of our future work.

[1] Neural Motifs: Scene Graph Parsing with Global Context. Zellers et al. CVPR’18. 

[2] Visual Distant Supervision for Scene Graph Generation. Yuan Yao et al. ICCV’21. 

[3] Conceptnet 5.5: An open multilingual graph of general knowledge. Robyn Speer et al. AAAI-17

Reviewer comment (Experiments):
In the loss calculation phase, if we consider the viability of a prediction conditioned on its object, we might have better choice on the threshold. For example, the prediction probability of 'on' will be higher for 'building-window' pair, however, it will be extremely low for 'man-pizza' pair. Is it possible to use such commonsense knowledge in the threshold choice of the loss equation?

Author response:
**(Q1) Object-conditional relation class thresholding. (based on commonsense knowledge)**

We agree with the reviewer’s suggestion that object-conditional relation pseudo-labeling is also a promising direction of self-training for SGG. The object-conditional pseudo-labels are expected to strengthen our approach by reflecting commonsense knowledge to decide whether pseudo-labeling or not. We suggest two approaches based on the SGG works with commonsense knowledge [1, 2].

- One approach would be to utilize the frequency of triplets as a commonsense knowledge, which is obtained from counting the appearance of <subject, predicate, object> in the training dataset [1]. For example, if a predicate “on” appears more frequently between “man-horse” than between “man-pizza” in the training data, then we encourage the pseudo-label <man, on, horse> to contribute more to the loss than the pseudo-label <man, on, pizza>, given that these two pseudo-labels are obtained by ST-SGG. However, although this simple approach can enhance the accuracy of pseudo-label generation, it is not expected to improve the zero-shot performance as it allows pseudo labels to be assigned only when the generated triplets emerge in the training set.
- Another approach would be to utilize the knowledge graph from external sources. By comparing the triplet in the commonsense knowledge graph, such as ConceptNet [3], we can confirm the generated pseudo-labels are valid or not. In this case, we can reduce the number of incorrect pseudo-labels by filtering the pseudo-label based on commonsense.

We express our gratitude to the reviewer for bringing to light issues that we had overlooked, thanks to their valuable insights. We acknowledge the significance of these issues and will consider them as part of our future work.

[1] Neural Motifs: Scene Graph Parsing with Global Context. Zellers et al. CVPR’18. 

[2] Visual Distant Supervision for Scene Graph Generation. Yuan Yao et al. ICCV’21. 

[3] Conceptnet 5.5: An open multilingual graph of general knowledge. Robyn Speer et al. AAAI-17

Reviewer comment (Evaluation):
There's a concern if the proposed framework might, in some scenarios, sacrifice a considerable amount of overall performance to achieve improvement in fine-grained predicates.

Author response:
**(W2) Concern of sacrificing a considerable amount of overall performance.**

As we have mentioned in our response to W1, we could deliberately increase R@K at the expense of a decrease in mR@K, if we are concerned about too low R@K. However, since doing so would make it hard to correctly evaluate the performance of our model, we instead relied on F@K. Once again, we reported R@K and mR@K when the validation F@K is the best.

Nevertheless, we would like to draw attention again to the experimental result on the OI-V6 dataset. In Table 8 of **the modified PDF**, applying ST-SGG to SGG models leads to improvement in both R@K and mR@K. This indicates that our proposed approach is effective in enhancing model’s generalization across both head and tail predicate classes, implying that our approach does not compromise the overall performance. Instead, we argue that ST-SGG fulfills the primary objective of self-training: boosting the model’s generalization ability through the utilization of unlabeled instances. This achievement underscores the robustness and effectiveness of our proposed methodology.

Reviewer comment (Presentation):
The paper lacks a clear depiction of the overall framework structure. Please clearly demonstrate how the different components interact and how pseudo-labels are generated and applied.

Author response:
**(W3) Depiction of the overall framework structure**

We appreciate the reviewer’s suggestion to enhance the presentation quality of our paper.  In response, we have revised the description of ST-SGG with Algorithm 1 of Appendix C.1 in **the modified PDF** with **BLUE** text.