Paper: LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies (ICLR-2024)

Reviewer comment (Writing):
The Introduction Section is not clear. The authors indicate that most methods involve modeling correlations with in-domain accuracy or agreement. And many studies evaluate generalization on OOD datasets that feature limited visual shifts. These interpretations are very unclear. I am not clear the concrete meaning. I recommend the authors modify their paper carefully.

Author response:
**Q: The Introduction Section is not clear, especially about the description of ‘most methods involve modeling correlations with in-domain accuracy or agreement. And many studies evaluate generalization on OOD datasets that feature limited visual shifts.’ These interpretations are very unclear. I am not clear the concrete meaning.**

- Please let us know if there’s any other place remain unclear and we are happy to take the advice to rephrase those sentences in paper at the end of the discussion!

- We want to echo that our main motivation is to establish ‘a unified benchmarking metric robustly applicable across both VM and VLM, to assess model generalization (use in domain measurement to predict model’s OOD performance). 

- **most methods involve modeling correlations with in-domain accuracy or agreement.** -> 
Prior work, notable, ‘accuracy-on-the-line’[1] and ‘agreement-on-the-line’[2], are two highly influenced papers in the field. The former is the first work to establish a strong baseline that the models with better ID accuracy are likely to have better OOD accuracy, while the latter adopted model ensemble and argued that OOD agreement between the predictions of any two pairs of neural networks shows linear correlation with their ID agreement. Those two works setup important base tune for followup works. However, those two work did not explicitly evaluated models of different modality(VM & VLM) in comparison. Our results shows that those two well-acknowledged measurement are less robust when comparing models under different settings.

- **many studies evaluate generalization on OOD datasets that feature limited visual shifts.** -> Many prior work( as referred in the introduction), where they found strong linear relationship between ID and OOD accuracy, are adopting OOD dataset that are visually similar to ID dataset, for instance, ImageNet-v2 or ImageNet-C. Our work argue that those dataset, did’t reflect more realistic distribution shrift, make then more like ‘another pool of ID dataset rather than realistic OOD dataset’(please also refer to section 4 for discussion.)  We instead focus on a more realistic distribution shift involving semantic concept shrift(like cartoon Dog and animal Dog) or orientation variance(like fallen chair in ObjectNet). We shows that LCA as a benchmark could predict model’s generalization on those realistic OOD dataset across both VM and VLMs..

Reviewer comment (Writing):
The authors indicate that to address the analyzed issues, they introduce a method to benchmark model generalization, i.e., using the taxonomy loss. Firstly, the authors do not interpret whether the research is meaningful clearly. Secondly, the authors do not sufficiently introduce the advantages of the taxonomy loss. I recommend the authors draw a figure to clearly describe the motivation.

Author response:
**Q: Firstly, the authors do not interpret whether the research is meaningful clearly. Secondly, the authors do not sufficiently introduce the advantages of the taxonomy loss.**

- We thanks reviewer for raising concerns of our motivation. We would like to echo here to the main motivation of our work. 
- 1. Since the early year of ImageNet (2011) datasets,  LCA distance was proposed but largely dismissed as it was widely believed to follow the same ordering as Top 1 accuracy. This metric have been revisit many times in follow-up papers of hierarchical classification but the concept that ‘LCA don’t provide much information besides Top1 accuracy’ are still largely accepted. In our work, we want to challenge this prevailing notion and shows that LCA could provide strong signal to model’s feature learning in the era of Vision Language Model. This could provide some pointer of why VLM with lower ID accuracy are having a higher OOD compared to SOTA VM model.
- 2. We then shows how LCA could solve the problem that haven’t been solved in prior work. In summary, prior work predicting model generalization are using 1.(VLM or VM) and 2.(visually similar dataset like ImageNet-v2 as OOD for ImageNet). This even include two highly influenced work [1][2]. We targeting on a more realistic setup of 1(VLM and VM) and 2. Various realistic OOD datasets like ImageNet-R/S/A/ObjectNet. 
- 3. We then provide some exploration on how‘LCA-on-the-line’ could be used to improve model’s generalization by applying taxonomy loss. This is introduced in 3.4 ENHANCING GENERALIZATION THROUGH CLASS TAXONOMY ALIGNMENT. We consider applying taxonomy loss similar to ‘alignment to human knowledge’ during model training.
- For adding a figure indicating motivation as reviewer suggestion, Figure 2 is well motivated by our main motivation.

Reviewer comment (Experiments):
The paper does not evaluate multiple OOD scoring methods like energy scores [1], ODIN [2], Mahalanobis distance [3], and ReAct [4], which would have provided insights into the validity of the key conclusions across different anomaly scoring approaches. The interaction between the choice of scoring method and LCA distance remains unclear. Understanding how these scoring methods affect OOD performance from the perspective of the LCA distance is important. It should be noted that the calculation of this LCA distance is limited to variants of ImageNet considering the WordNet hierarchy used.

Author response:
For the (2) in strengths, we believe there’s some misunderstandings about our experiment setup. **Our paper didn’t try to target OOD detection or anomalies detection(separate ID data from and OOD data, binary classification), but to predict whether a model could perform better on OOD dataset by only looking at its in-domain measurement**(correlation of model top1 accuracy between ID and OOD). There’s a difference in the problem setup. We could like to point reviewer to Table 2 for reference. 
- **it is quantitatively shown how images from OOD datasets with more distant LCA relationships to the training data tend to be easier to detect as anomalies.** -> We actually trying to show model that have a lower LCA distance on ID will have a higher top 1 accuracy on OOD. 

- **The intuitive justification provided is that greater separation between the semantics of the origin dataset and OOD dataset in the hierarchical LCA structure leads to more separable distributions.** -> Please refer to our answer below about assumption. 

**Q: Understanding how these scoring methods affect OOD performance from the perspective of the LCA distance is important. It should be noted that the calculation of this LCA distance is limited to variants of ImageNet considering the WordNet hierarchy used.**: 

- We thanks reviewer for point to scoring function. We will make sure to cite those work in our revision. Here’s our interpretation of suggested work and please feel free to suggest more!

- The suggested works are targeting out-of-distribution detection, which trying to tell which data are OOD and which data have been seemed during training. This is a different task from our paper. Pleaser refer to answer above.
- Different from our focus, the main targets of those methods is to **separate feature between ID and OOD data**, which our focus is more detail, to identify within ID feature, how does model learned to separate class semantically? Under the assumption that **ID and OOD data should lands on a share feature distribution, rather than two separate ones**. Assuming two separate distribution is what being assumed in OOD detection. However, we still believe this is an interesting difference worth to mention in the discussion as related work and will make sure to include citation of those work.

**Q: It should be noted that the calculation of this LCA distance is limited to variants of ImageNet considering the WordNet hierarchy used.**
- We want to point reviewer to section 3.4. In order to extends our metric beyond ImageNet and WordNet, we proposed to infer pseudo latent class taxonomy using K-Means Clustering as a replacement to replying on WordNet hierarchy. Our result in table 4 shows that our latent taxonomy are equally robust to the experiment in Table 2 among 75 different latent taxonomy.

Reviewer comment (Evaluation):
The claim is made that the findings offer "invaluable insights and actionable techniques" to enhance robustness and generalization. However, no concrete solutions leveraging the LCA distance are proposed or analyzed. Further theoretical or empirical analysis that establishes a connection between these insights and improved generalization would be beneficial.

Author response:
**Q: However, no concrete solutions leveraging the LCA distance are proposed or analyzed.**
- We want to point reviewer to section 3.4 and F.1 in appendix. In section 4 we have adopt our idea LCA-on-the-line as a taxonomy loss,  and explore how it could used to improve model’s generalization. Similarly in F.1, we shows that how this idea could be used on prompting. We have also analyzed the underlying connections between why LCA are suitable generalization metric in 3.3. Please feel free to ask more question where you consider less clear.

Reviewer comment (Writing):
“Given two classes, y (the ground truth class) and y′, we define the LCA distance according to (Bertinetto et al., 2020) as lcad(y′, y) := f(y) − f(lca(y, y′), where f(y) ≥ f(lca(y, y′) and lca((y′, y) denotes…” All formulations lose the right brackets.

Author response:
**Q: Minor error:**
- Thanks for identifying typo in brackets, we will update accordingly.

Reviewer comment (Writing):
“As highlighted in Fig 1 (indicated in red), when adhering to ’accuracy on the line’,” Wrong quotes.

Author response:
**Q: Minor error:**
- For “As highlighted in Fig 1 (indicated in red), when adhering to ’accuracy on the line’,” Wrong quotes.”, sorry we don’t understand. Could you provide more detail on why this is wrong quotes? Thanks!

Reviewer comment (Presentation):
Explanation of LCA is complicated. A visual illustration of LCA computation with a small part of the WordNet hierarchy and 2–3 example pairs of classes would help readers to quickly and better understand the LCA distance measure.

Author response:
**Q: Explanation of LCA is complicated**
- Thanks for the suggestion! We will point reader more directly to plot in appendix, and provide an actual class example in appendix plot. 
A simple intuition would be, 
animal have children of cat and dog;
Dog have children of chihuahua and Husky;
Etc.

Reviewer comment (Presentation):
Fig. 1 is very difficult to read and understand. The legend is too small. It would be better to show only top‑1 here (first row) with larger font sizes (match caption size, roughly) and show the full figure in supplementary.

Author response:
**Q: Fig. 1 is very difficult to read and understand**
- Thanks for identify the lack of legibility of Fig1. We have included original png of Figure 1 in the supplementary zip folder for reference. We will also make sure to enlarge the text in our revision.

Reviewer comment (Writing):
Implementation of inferred class taxonomy is difficult to understand. It is unclear what "establishing the cluster level where both classes share the same cluster as the height of LCA" means. Please clarify and try to simplify it.

Author response:
**Q: Implementation of inferred class taxonomy is difficult to understand**
- Thanks for the suggestion, we will make sure to include a algorithm pseudocode in our revision. 
- **establishing the cluster level where both classes share the same cluster as the height of LCA** -> Similar to idea of LCA in FIgure3 in appendix, we form a hierarchical tree by using different number of clustering in each layers. For instance, in level 1 we have all the class, in level 2 we have two tree node(result from K-mean of 2 cluster), in level 3 we have 4 tree node(result from K-mean of 4 cluster), etc.