Paper: Label-Focused Inductive Bias over Latent Object Features in Visual Classification (ICLR-2024)

Reviewer comment (Evaluation):
The experimental improvements over the baselines are relatively modest, suggesting that the significance of Output-Domain focused Biasing (ODB) may be limited.

Author response:
Our results show consistent and statistically significant improvement. We hope that the contribution on raising a new problem and its generality are more considered rather than the significance of performance gain determined by an unclear threshold.

Reviewer comment (Experiments):
Is it possible for the authors to validate the efficacy of ODB in the context of domain adaptation?

Author response:
We can validate the efficacy in domain adaptation, but it requires more study, because domain adaptation allows not only tasks using the shared world knowledge, but also task-specific knowledge. 
We hope that the knowledge of ODB is sufficiently general to work in even target domains for adaptation, but the conflict of task-specific knowledge may dominate the validation results. Unfortunately, recognizing which knowledge causes the conflict in neural networks is still challenging as far as we know.

Reviewer comment (Evaluation):
Is there a meaningful application of ODB in addressing input-domain bias when dealing with diverse visual domains?

Author response:
We think that potential applications of ODB are most supervised learning in visual domains where labeling may rely on some implicit knowledge of humans. Particularly, if a domain uses an input containing many latent objects to determine its label, it can be a more suitable application of ODB.

Reviewer comment (Writing):
The term of output-domain knowledge is very vague/difficult to interpret. At one point in the introduction you instead write 'undescribed world knowledge', which I think refers to the same thing, and is more clear to me. Would it make sense to use another name than ODB? Specifically to change the OD to something else. This would make the paper clearer. Output domain is the domain where you test, which I think does not really do the job here.

Author response:
We agree that the term  ‘output-domain knowledge’ does not intuitively align with our paper's interest. We revised terminology to avoid using 'OD' as follows

Label-focused inductive bias is the inductive bias determined by categorization of latent objects based on only labels, while undescribed world knowledge indicates all unexpressed world knowledge on the objects in human labeling. We use the term ‘UWK’ for ‘Undescribed World Knowledge over latent objects in human labeling’ 

We revised related sentences (including titles) to use the terminologies, too. 
1)  ‘Output-domain focused inductive biasing’ (the method) -> Label-focused Latent-object Biasing (LLB)’ 
2) ‘output-domain focused inductive bias’ -> label-focused inductive bias
3) implicit relations over latent objects used for  human labeling(output-domain) -> undescribed world knowledge

Reviewer comment (Novelty):
In the related work, a knowledge graph is mentioned briefly while describing the method of one work. However, I miss a small section about the general idea of including hierarchical information or knowledge-graph representations into visual representations, which these authors are not the first to explore. For that reason, it becomes a bit difficult to buy when the authors claim they are the first to raise the issue of the implicit 'output-domain' knowledge missing during training, and a more complete related work section here would make the paper stronger (see for example the related work of Pan et al.).

Author response:
We appreciate your valuable suggestion, as it notably helps to strengthen our proposal.
We added a paragraph and revised other paragraphs in the related work. In the second paragraph (Conflict Reduction via Knowledge Graph) in the related work section, we discuss previous works of using structured information such as knowledge graph in order to reduce input-domain focused bias.  It discusses the following references as follows. 
* Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2673–2681, 2017.
* Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6857–6866, 2018.
* Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11487–11496, 2019.
* Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 35:9125–9138, 2022
* Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, and Marios Savvides. Semantic relation reasoning for shot-stable few-shot object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8782–8791, 2021.

To the best of our knowledge, previous works focus on the effective use of external knowledge, while undescribed world knowledge over latent objects in human labeling of target training data has not been utilized and even recognized. Please check the related work section for more detailed discussions.

Reviewer comment (Presentation):
The conclusion and future work section does not really contain recommendations for future work.

Author response:
In the conclusion of our paper's revised edition, we implemented the following revisions:
Beyond using the plug-in model, effectively harmonizing the undescribed world knowledge with input-domain focused bias on a simple network still remains an open question. Also, we hope that our approach will evoke further research on diverse visual domains.

Reviewer comment (Presentation):
In A.1, you refer to Tables 7a and 7c although I believe you mean Figure 7 and 7c.

Author response:
* ‘Table 7a’ and ‘Table 7c’ is revised into ‘Figure 7a’ and ‘ Figure 7c’

Reviewer comment (Writing):
Conclusion: the phrasing "harbors unseen knowledge from human labeling" was difficult to parse and ambiguous.

Author response:
With the redefinition of ‘ODB’, we revised the following sentence. 
In this paper, we highlighted the conflict of input-domain focused inductive bias and undescribed world knowledge over latent objects in human labeling.”

Reviewer comment (Writing):
Conclusion: "on vision transformer architecture" >> "on the Vision transformer architecture".

Author response:
* In the conclusion section, “on vision transformer architecture” is revised into “on Vision transformer architecture”

Reviewer comment (Writing):
Conclusion: "in qualitative and quantitative analysis on its results" >> "through qualitative and quantitative analysis of its results".

Author response:
* In the conclusion section, "in qualitative and quantitative analysis on its results" is revised into “through qualitative and quantitative analysis of its results”

Reviewer comment (Evaluation):
Table 2: Why not show the standard deviation if in effect these results were run using 5 random seeds? It seems relevant in your comparison to the baseline which is quite close (84.80 vs. 84.40). You cannot use the term 'significant' in the Ablation study section of Section 5.1 if you do not show these standard deviations.

Author response:
We have revised the results in Table 2 to include the standard deviation. 
We also corrected a minor numerical discrepancy in Table 1, where the result of ViT B/16 on IN1K differed from Table 2, despite originating from the same experiment. Note that this modification does not impact our paper's main contention.

Reviewer comment (Presentation):
Table 2: explain what w/o Pos. stands for in table caption, even if you also say it in the text (if somebody glances quickly at the table.).

Author response:
In the caption of Table 2, we have added the following explanation: “We study the impact of visual dependency disconnection (Visual Disc.), diversity loss (Diversity), not using positional encoding (w/o Pos.), and integration module with visual features (Integration).”

Reviewer comment (Presentation):
Formatting of references generally needed, use {} around text which should be case-sensitive in the .bib-file. (e.g., "mixup" >> "MixUp" for Zhang et al.).

Author response:
We also agree with this formatting, but formally published versions of the paper use mixup without upper scaling. For this reason, we left ‘mixup’ as its original paper. To be clear, we check the missed uppercase words in references.

Reviewer comment (Presentation):
In the end of Section 5.2, it would be great to specify either in the text or in the figure caption which classes numbers 1173 and 1813 correspond to. Are they the quill and paperknife or other visually similar classes?

Author response:
The numbers in each tile correspond to the assigned latent object index to an image patch described in Section 3.2 (Method Description - Latent Object Extraction from Visual Features). 1173 and 1813 are indexes of O latent objects, and we set O(=2048) as a hyperparameter. 
We revised the caption in Figure 6 and also revised related sentences to use one notation ‘O’ to represent the number of the latent object.

Reviewer comment (Writing):
5.1 title: quantiTative*.

Author response:
* Section 5.1 title is revised into “Quantitative Analysis”

Reviewer comment (Writing):
Section 4: "doesn’t" >> "does not" (too informal).

Author response:
* In Section 4, "doesn’t require" is revised into “does not require”

Reviewer comment (Writing):
Section 4: "provide series of analysis the impact" >> "provide a series of analysEs OF the impact".

Author response:
* In Section 4, "provide series of analysis the impact" is revised into “provide series of analyses of the impact”

Reviewer comment (Writing):
The second paragraph of the related work contains a duplicate sentence "Lemesle et al...". Should be removed.

Author response:
* The duplicated sentence is Related work section is removed

Reviewer comment (Presentation):
Fig. 3 is a nice figure. Howeve,r it currently says "L_diveristy" whereas I think you want to say "L_diversity".

Author response:
* In Figure 3, "L_diveristy" in Figure 3b is revised into "L_diversity"

Reviewer comment (Novelty):
Humans benefit from world information being able to avoid input domain bias. This actually benefits from humans having more prior information and being able to build better semantic relationships between classes. While Output-Domain focused Biasing (ODB) is more like a feature enhancement method, which improves performance through decoupling and enhancement of features.

Author response:
We totally agree with the benefit of human knowledge. However, we hold a differing view that the absence of human knowledge is a weakness. because this paper aimed to improve neural networks and actually intended not to use external resources. This is one of the common goals of AI literature to reduce human efforts in building better AI for the last decades.

Reviewer comment (Writing):
The paper is not easy to follow, especially the descriptions of Visual Dependency Disconnection and Non-visual Feature Structuring need more details.

Author response:
To improve the readability, we revised Section 3.2 including  ‘Visually Dependency Disconnection’ and ‘Non-visual Feature Structuring’  paragraphs. 

The revisions are briefly shown as follows. 

1) Notation definition of visual and non-visual features
We revised the notation for ‘visual features’ in Section 3.1 and ‘non-visual features’ in Section 3.2.

2) Section 3.2 Visually Dependency Disconnection
For clear readability, we defined a notation for the output of the disconnect network as patch-wise non-visual features in the third paragraph of Section 3.2. Also, we added explanations of each notation in Equation (3).

3) Emphasized points
To improve the clarity, we generally clarified the emphasized points in Section 3.2. We updated details at main components in our implementation.

Please check Section 3.1 and Section 3.2 for more details.

Reviewer comment (Evaluation):
The proposed method has limited improvement in performance.

Author response:
Our results show consistent and statistically significant improvement. We hope that the contribution on raising a new problem and its generality are more considered rather than the significance of performance gain determined by an unclear threshold.

Reviewer comment (Experiments):
Is the ODB method equally effective in convolutional networks?

Author response:
The effect of the ODB (revised to LLB) method is basically not relying on specific architecture if its representations are directly affected by loss, so we think it is also effective in convolutional networks. The only reason why we selected ViT is because of its wide use and state-of-the-art performance.
To provide more insights, as in Section 3.1 preliminaries, we visualized the distribution of centroids of all output features in each class of ImageNet, extracted from ResNet50 as shown in Section C.1  in Appendix. We found that adjacent classes from Figure 2b, which are visually similar but semantically unrelated, were also closely positioned in CNN (Broom, Puck, Crutch, Mop) (Figure 11c) . We think that the dominance of the input-domain focused bias also exists in CNNs, and therefore ODB can be effective in CNNs. Check Section C.1 in Appendix for more details.