Original: CH I LS
Revised: CH I LS

Original: : Z ERO - SHOT I MAGE C LASSIFICATION WITH
Revised: : Z ERO - SHOT I MAGE C LASSIFICATION WITH

Original: H IERARCHICAL L ABEL S ETS
Revised: H IERARCHICAL L ABEL S ETS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Open vocabulary models (e.g. CLIP) have shown strong performance on zeroshot classification through their ability generate embeddings for each class based on their (natural language) names.
Revised: Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names.

Original: Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning).
Revised: Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning).

Original: In this paper, we propose
Revised: In this paper, we aim to tackle classification problems with coarsely-defined class labels.

Original:  C lassification with Hi erarchical L abel S ets (or CHiLS), an alternative strategy that proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction.
Revised: We propose C lassification with Hi erarchical L abel S ets (or CHiLS), an alternative strategy that proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction.

Original: Across numerous datasets, CHiLS leads to improved accuracy yielding gains of over 30% in situations where known hierarchies are available and more modest gains when they are not.
Revised: Across numerous datasets (with implicit semantic hierarchies), CHiLS leads to improved accuracy yielding gains of over 30% in situations where known hierarchies are available and more modest gains when they are not.

Original: CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost.
Revised: CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, machine learning researchers have become captivated by the remarkable capabilities of pretrained open vocabulary models (Radford et al., 2021; Wortsman et al., 2021;
Revised: Recently, machine learning researchers have become captivated by the remarkable capabilities of pretrained open vocabulary models (Radford et al., 2021; Wortsman et al., 2021;

Original: Jia et al., 2021;
Revised: Jia et al., 2021;

Original: Gao et al., 2021; Pham et al., 2021; Cho et al., 2022; Pratt et al., 2022).
Revised: Gao et al., 2021; Pham et al., 2021; Cho et al., 2022; Pratt et al., 2022).

Original: These models, like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), learn to map images and captions into shared embedding spaces such that images are close in embedding space to their corresponding captions but far from randomly sampled captions.
Revised: These models, like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), learn to map images and captions into shared embedding spaces such that images are close in embedding space to their corresponding captions but far from randomly sampled captions.

Original: The resulting models can then used to assess the relative compatibility of a given image with an arbitrary set of textual “prompts”.
Revised: The resulting models can then used to assess the relative compatibility of a given image with an arbitrary set of textual “prompts”.

Original: Notably, Radford et al.
Revised: Notably, Radford et al.

Original: (2021) observed that by inserting each class name directly within a natural language prompt, one can then use CLIP embeddings to assess the compatibility of an images with each among the possible classes.
Revised: (2021) observed that by inserting each class name directly within a natural language prompt, one can then use CLIP embeddings to assess the compatibility of an images with each among the possible classes.

Original: Thus, open vocabulary models are able to perform zero-shot image classification, and do so with high rates of success (Radford et al., 2021; Zhang et al., 2021b).
Revised: Thus, open vocabulary models are able to perform zero-shot image classification, and do so with high rates of success (Radford et al., 2021; Zhang et al., 2021b).

Original: Despite the documented successes, the current interest in open vocabulary models poses a new question: How should we represent our classes for a given problem in natural language?
Revised: Despite the documented successes, the current interest in open vocabulary models poses a new question: How should we represent our classes for a given problem in natural language?

Original: As class names are now part of the inferential pipeline (as opposed to mostly an afterthought in traditional scenarios) for models like CLIP in the zero-shot setting, CLIP’s performance is now directly tied to the descriptiveness of the class “prompts” (Santurkar et al., 2022).
Revised: As class names are now part of the inferential pipeline (as opposed to mostly an afterthought in traditional scenarios) for models like CLIP in the zero-shot setting, CLIP’s performance is now directly tied to the descriptiveness of the class “prompts” (Santurkar et al., 2022).

Original: While many researchers have focused on improving the quality of the prompts into which class names are embedded (Radford et al., 2021;
Revised: While many researchers have focused on improving the quality of the prompts into which class names are embedded (Radford et al., 2021;

Original: Pratt et al., 2022; Zhou et al., 2022b;a; Huang et al., 2022), surprisingly little attention has been paid to improving the richness of the class names themselves .
Revised: Pratt et al., 2022; Zhou et al., 2022b;a; Huang et al., 2022), surprisingly little attention has been paid to improving the richness of the class names themselves .

Original: This can be particularly crucial in cases where class names are not very informative or are too broad tosound match the sort of descriptions that might arise in natural captions.
Revised: This can be particularly crucial in cases where class names are not very informative or are too coarsely-defined to match the sort of descriptions that might arise in natural captions.

Original: Consider, for an example, the class “large man-made outdoor things”  in the CIFAR20 dataset (Krizhevsky, 2009)).
Revised: Consider, for an example, the classes “large man-made outdoor things” and “reptiles” in the CIFAR20 dataset (Krizhevsky, 2009).

Original: In this paper, we introduce a new method for Zero-Shot prediction with CLIP models, which we refer to  as C lassification with Hi erarchical L abel S ets (CHiLS for short).
Revised: In this paper, we introduce a new method to tackle zero-shot classification with CLIP models for problems with coarsely-defined class labels. We refer to our method as C lassification with Hi erarchical L abel S ets (CHiLS for short).

Original: Our method utilizes a hierarchical map to convert each class into a list of subclasses, performs normal CLIP zero-shot prediction across the union set of all subclasses , and finally uses the inverse mapping to convert the subclass prediction to the requisite superclass.
Revised: Our method utilizes a hierarchical map to convert each class into a list of subclasses, performs normal CLIP zero-shot prediction across the union set of all subclasses , and finally uses the inverse mapping to convert the subclass prediction to the requi- site superclass.

Original: We additionally include a reweighting step wherein we leverage the raw superclass probabilities in order to make our method robust to less-confident predictions at the superclass and subclass level.
Revised: We additionally include a reweighting step wherein we leverage the raw superclass probabilities in order to make our method robust to less-confident predictions at the superclass and subclass level.

Original: We evaluate CHiLS on a wide array of image classification benchmarks with and without available hierarchical information.
Revised: We evaluate CHiLS on a wide array of image classification benchmarks with and without available hierarchical information.

Original: In the former case, leveraging preexisting hierarchies leads to strong accuracy gains across all datasets.
Revised: In the former case, leveraging preexisting hierarchies leads to strong accuracy gains across all datasets.

Original: In the latter, we show that rather than enumerating the hierarchy by hand, using GPT-3 to query a list of possible subclasses for each class (whether or not they are actualy present in the dataset) still leads to consistent improved accuracy over raw superclass prediction.
Revised: In the latter, we show that rather than enumerating the hierarchy by hand, using GPT-3 to query a list of possible subclasses for each class (whether or not they are actualy present in the dataset) still leads to consistent improved accuracy over raw superclass prediction.

Original: We summarize our main contributions below:
Revised: We summarize our main contributions below:

Original: • We propose CHiLS, a new method for improving zero-shot CLIP performance, which requires no labeled data or training time and is flexible to both existing and synthetically generated hierarchies.
Revised: • We propose CHiLS, a new method for improving zero-shot CLIP performance in scenarios with ill-defined and/or overly general class structures, which requires no labeled data or training time and is flexible to both existing and synthetically generated hierarchies.

Original: • We show that CHiLS consistently performs as well or better than standard practices in situations with only synthetic hierarchies, and that CHiLS can achieve up to 30% accuracy gains when ground truth hierarchies are available.
Revised: • We show that CHiLS consistently performs as well or better than standard practices in situations with only synthetic hierarchies, and that CHiLS can achieve up to 30% accuracy gains when ground truth hierarchies are available.

Original:  R ELATED W ORK 2. T RANSFER L EARNING
Revised:  R ELATED W ORK 2. T RANSFER L EARNING

Original: While the focus of this paper is to improve CLIP models in the zero-shot regime, there is a large body of work exploring improvements to CLIP’s few-shot capabilities.
Revised: While the focus of this paper is to improve CLIP models in the zero-shot regime, there is a large body of work exploring improvements to CLIP’s few-shot capabilities.

Original: In the standard fine-tuning paradigm for CLIP models, practitioners discard the text encoder and only use the image embeddings as inputs for some additional training layers.
Revised: In the standard fine-tuning paradigm for CLIP models, practitioners discard the text encoder and only use the image embeddings as inputs for some additional training layers.

Original: This however, leads to certain problems.
Revised: This however, leads to certain problems.

Revised: One particular line of work on improving the fine-tuned capabilities of CLIP models leverages model weight interpolation.

Original: Wortsman et al.
Revised: Wortsman et al.

Original: (2021)  and Ding et al. (2022) focus on robustness issues that arise when finetuning CLIP models, specifically through the lenses of distribution shift and catastrophic forgetting respectively.
Revised: (2021) proposes to linear interpolate the weights of a finetuned and a zero-shot CLIP model to improve the fine-tuned model under distribution shifts.

Original:  Wortsman et al.
Revised: This idea is extended by Wortsman et al.

Original: (2021) proposes to linear interpolate the weights of a  fine-tuned anda zero-shot CLIP model, which succeeds in making fine-tuned CLIP models robust under standard distribution shifts.
Revised: into a general purpose paradigm for ensembling models’ weights in order to improve robustness. Ilharco et al. (2022) then builds on both these works and puts forth a method to “patch” fine-tuned and zero-shot CLIP weights together in order to avoid the issue of catastrophic forgetting.

Revised: Among all the works in this section, our paper is perhaps most similar to this vein of work (albeit in spirit), as CHiLS too seeks to combine two different predictive methods.

Original: In Ding et al.
Revised: Ding et al.

Original: , they propose to fine-tune both the image encoder and the text encoder, where the latter draws from a replay vocabulary of text concepts from the original 
Revised: (2022) also tackles catastrophic forgetting, though they propose an orthogonal direction and fine-tune both the image encoder and the text encoder, where the latter draws from a replay vocabulary of text concepts from the original CLIP database.

Original: CLIP database. There is another line of work leveraging the adapter framework (Houlsby et al., 2019) from parameter-efficient learning; specifically, in Ding et al.
Revised: There is another line of work that seeks to improve CLIP models by injecting a small amount of learnable parameters into the frozen CLIP backbone. This has been commonly achieved through the adapter framework (Houlsby et al., 2019) from parameter-efficient learning; specifically, in Gao et al.

Original: (2022) they fine-tune a small number of additional weights on top of the encoder blocks, which is then connected with the original embeddings through residual connections.
Revised: (2021) they fine-tune a small number of additional weights on top of the encoder blocks, which is then connected with the original embeddings through residual connections.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021a) builds on this method by removing the need for additional training and simply uses a cached model. 
Revised: (2021a) builds on this method by removing the need for additional training and simply uses a cached model. In contrast to these works, Jia et al. (2022) forgoes the adapter framework when using a Vision

Revised: Transformer backbone for inserting learnable “prompt” vectors into the transformer’s input layers, which shows superior performance over the aforementioned methods.

Original: Additionally, some have looked at circumventing the entire process of prompt engineering.
Revised: Additionally, some have looked at circumventing the entire process of prompt engineering.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2022a) and Zhou et al.
Revised: (2022a) and Zhou et al.

Original: (2022b) tackle this by treating the tokens within each prompt as learnable vectors, which are then optimized within only a few images per class.
Revised: (2022b) tackle this by treating the tokens within each prompt as learnable vectors, which are then optimized within only a few images per class.

Original: Huang et al.
Revised: Huang et al.

Original: (2022) echoes these works, but instead does not utilize any labeled data and learns the prompt representations in an unsupervised manner.
Revised: (2022) echoes these works, but instead does not utilize any labeled data and learns the prompt representations in an unsupervised manner.

Revised: Zhai et al.

Revised: (2022) completely forgoes the notion of fine-tuning in the first place, instead proposing to reframe the pre-training process as only training a language model to match a pre-trained and frozen image model.

Original: In all the above situations, some amount of data, whether labeled or not, is used in order to improve the predictive accuracy of the CLIP model.
Revised: In all the above situations, some amount of data, whether labeled or not, is used in order to improve the predictive accuracy of the CLIP model.

Original: 2. Z ERO -S HOT P REDICTION
Revised: 2. Z ERO -S HOT P REDICTION

Original: The field of Zero-Shot Learning (ZSL) has existed well before the emergence of open vocabularly models, with its inception traced to Larochelle et al.
Revised: The field of Zero-Shot Learning (ZSL) has existed well before the emergence of open vocabularly models, with its inception traced to Larochelle et al.

Original: With regards to non-CLIP related methods, the ZSL paradigm has shown success in improving multilingual question answering (Kuo & Chen, 2022) with large language models (LLMs), and also in image classification tasks where wikipedia-like context is used in order to perform the classification without access to the training labels (Bujwid & Sullivan, 2021;
Revised: With regards to non-CLIP related methods, the ZSL paradigm has shown success in improving multilingual question answering (Kuo & Chen, 2022) with large language models (LLMs), and also in image classification tasks where wikipedia-like context is used in order to perform the classification without access to the training labels (Bujwid & Sullivan, 2021;

Original: Shen et al., 2022).
Revised: Shen et al., 2022).

Original: With CLIP models, ZSL success has been found in a variety of tasks.
Revised: With CLIP models, ZSL success has been found in a variety of tasks.

Original: Namely, Zhang et al.
Revised: Namely, Zhang et al.

Original: (2021b) expands the CLIP 2D paradigm for 3D point clouds.
Revised: (2021b) expands the CLIP 2D paradigm for 3D point clouds.

Original: Tewel et al.
Revised: Tewel et al.

Original: (2021) shows that CLIP models can be retrofitted to perform the reverse task of image-to-text generation, with a strong ability for semantic arithmetic in the image domain.
Revised: (2021) shows that CLIP models can be retrofitted to perform the reverse task of image-to-text generation, and Shen et al. (2021) likewise display’s CLIP’s ability to improve performance on an array of Vision&Language tasks.

Original: Both Yu et al.
Revised: Both Yu et al.

Original: (2022) and Cho et al.
Revised: (2022) and Cho et al.

Original: (2022) expand CLIP’s zero-shot abilities through techniques drawn from reinforcement learning, with the former using 
Revised: (2022) expand CLIP’s zero-shot abilities through techniques drawn from reinforcement learning (RL), with the former using CLIP for the task of audio captioning.

Original:  CLIP for the  task of audio captioning.
Revised: Gadre et al. (2022) similarly works with the RL literature and retrofits CLIP to improve the embodied AI task of object navigation without any additional training.

Original: Zeng et al.
Revised: Zeng et al.

Original: (2022) in particular show the capabilities of composing CLIP-like models and LLMs together to extend the zero-shot capabilities tonovel tasks like assitive dialogue and open-ended reasoning.
Revised: (2022) shows the capabilities of composing CLIP-like models and LLMs together to extend the zero-shot capabilities to tasks like assitive dialogue and open-ended reasoning.

Original: Unlike our work here, these prior directions mostly focus on generative problems or, in the case of Bujwid & Sullivan (2021) and Shen et al.
Revised: Unlike our work here, these prior directions mostly focus on generative problems or, in the case of Bujwid & Sullivan (2021) and Shen et al.

Original: (2022), require rich external knowledge databases to employ their methods.
Revised: (2022), require rich external knowledge databases to employ their methods.

Original: In the realm of improving CLIP’s zero-shot capabilities for image classification, we particularly note the contemporary work of Pratt et al.
Revised: In the realm of improving CLIP’s zero-shot capabilities for image classification, we particularly note the contemporary work of Pratt et al.

Original: Here, authors explore using GPT-3 to generate rich textual prompts for each class rather than using preexisting prompt templates, and show improvements in zero-shot accuracy across a variety of image classification baselines.
Revised: Here, authors explore using GPT-3 to generate rich textual prompts for each class rather than using preexisting prompt templates, and show improvements in zero-shot accuracy across a variety of image classification baselines.

Original: In another work, Ren et al.
Revised: In another work, Ren et al.

Original: (2022) proposes leveraging preexisting captions in order to improve performance, though this is restricted to querying the pre-training set of captions.
Revised: (2022) proposes leveraging preexisting captions in order to improve performance, though this is restricted to querying the pre-training set of captions.

Original: In contrast, our work explores a complementary direction of leveraging hierarchy in class names to improve zero-shot performance of CLIP.
Revised: In contrast, our work explores a complementary direction of leveraging hierarchy in class names to improve zero-shot performance of CLIP.

Original: with a fixed set of preexisting prompt templates. 
Revised: with a fixed set of preexisting prompt templates. 2. H IERARCHICAL C LASSIFICATION

Revised: Our present work is related to the domain of Hierarchical Classification (Silla & Freitas, 2010), i.e. classification tasks when the set of labels can be arranged in a DAG-like class hierarchy.

Revised: Methodologies from this domain have been extensively used for multi-label classification (Dimitrovski et al., 2011; Liu et al., 2021; Chalkidis et al., 2020), and recent works have shown that this paradigm can aid in zero-shot learning by attempting to uncover hierarchical relations between classes (Chen et al., 2021; Mensink et al., 2014) and/or leveraging existing hierarchical information during training (Yi et al., 2022; Cao et al., 2020).

Original: P ROPOSED M ETHODS
Revised: Our line of inquiry is orthogonal to these approaches, as CHiLS is input f 1: Set of subclasses for subclass prediction 2: ▷ Subclass probabilities 3: ▷ Superclass probabilities 4: for = to

Revised: C do 5:

Revised: S c i =

Revised: G ( c i do 7: probability 8:

Revised: 9: end output easily composable into standard CLIP pipelines, only models unseen class hyponyms (rather than relationships between known classes), and requires no training of hierarchical embeddings.

Original: In this paper, we are primarily concerned with the problem of zero-shot image classification in CLIP models.
Revised: In this paper, we are primarily concerned with the problem of zero-shot image classification in CLIP models (see App.

Original:  For CLIP models, zero-shot classification involves using both a pretrained image encoder and a pretrained text encoder (see the left part of Figure 1).
Revised: B for an introduction to CLIP and relavent terminology). For CLIP models, zeroshot classification involves using both a pretrained image encoder and a pretrained text encoder (see the left part of Figure 1).

Original: To perform a zero-shot classification, we need a predefined set of classes written in natural language.
Revised: To perform a zero-shot classification, we need a predefined set of classes written in natural language.

Original: Let C
Revised: Let C

Original: = { c 1 , c 2 , . .
Revised: = { c 1 , c 2 , . .

Original: . , c k } be such a set.
Revised: . , c k } be such a set.

Original: Given an image and set of classes, each class is embedded within a natural language prompt (through some function T ( · ) ) to produce a “caption” for each class (e.g. one standard prompt mentioned in Radford et al.
Revised: Given an image and set of classes, each class is embedded within a natural language prompt (through some function T ( · ) ) to produce a “caption” for each class (e.g. one standard prompt mentioned in Radford et al.

Original: (2021) is “A photo of a {} ” ).
Revised: (2021) is “A photo of a {} .” ).

Original: These prompts are then fed into the text encoder and after passing the image through the image encoder, we calculate the cosine similarity between the image embedding and each class-prompt embedding.
Revised: These prompts are then fed into the text encoder and after passing the image through the image encoder, we calculate the cosine similarity between the image embedding and each class-prompt embedding.

Original: These similarity scores form the output “logits” of the CLIP model, which can be passed through a softmax to generate the class probabilities.
Revised: These similarity scores form the output “logits” of the CLIP model, which can be passed through a softmax to generate the class probabilities.

Original: As noted in Section 2, previous work has focused on improving the T ( · ) for each class label c i .
Revised: As noted in Section 2, previous work has focused on improving the T ( · ) for each class label c i .

Original: With

Original:  CHiLS, we instead focus on the complementary task of directly modifying the set of classes C , while keeping T ( · ) fixed.
Revised: With CHiLS, we instead focus on the complementary task of directly modifying the set of classes C when C is ill-formed or overly general, while keeping T ( · ) fixed.

Original: At a high level, our method involves into two main steps: (1) using hierarchical information to perform inference across subclasses , and (2) leveraging raw superclass probabilities to combine the best of subclass and superclass prediction probabilities.
Revised: Our method involves into two main steps: (1) using hierarchical information to perform inference across subclasses , and (2) leveraging raw superclass probabilities to combine the best of subclass and superclass prediction probabilities.

Original: 3. Z ERO -S HOT P REDICTION WITH H IERARCHICAL L ABEL S ETS
Revised: 3. Z ERO -S HOT P REDICTION WITH H IERARCHICAL L ABEL S ETS

Original: Our method CHiLS slightly modifies the standard approach for zero-shot CLIP prediction.
Revised: Our method CHiLS slightly modifies the standard approach for zero-shot CLIP prediction.

Original: As each class label c i represents some concept in natural language (e.g. the label “dog”), we acquire a subclass set S c i
Revised: As each class label c i represents some concept in natural language (e.g. the label “dog”), we acquire a subclass set S c i

Original: = { s c i , 1 , s c
Revised: = { s c i , 1 , s c

Original: i , 2 , . . .
Revised: i , 2 , . . .

Original: , s c i ,m i } through some mapping function G , where each s c
Revised: , s c i ,m i } through some mapping function G , where each s c

Original: i ,j is a linguistic hyponym , or subclass, of c
Revised: i ,j is a linguistic hyponym , or subclass, of c

Original: i (e.g. corgi for dogs) and m i is the size of the set S c i .
Revised: i (e.g. corgi for dogs) and m i is the size of the set S c i .

Original: Given a label set S c
Revised: Given a label set S c

Original: i for each class, we proceed with the standard process for zero-shot prediction, but now using the union of all label sets as the set of classes.
Revised: i for each class, we proceed with the standard process for zero-shot prediction, but now using the union of all label sets as the set of classes.

Original: Through this, CHiLS will now produce its guess for the most likely subclass .
Revised: Through this, CHiLS will now produce its guess for the most likely subclass .

Original: We then leverage the inverse mapping function G − 11 to coarse-grain our prediction back into the corresponding superclass.
Revised: We then leverage the inverse mapping function G − 1 to coarsegrain our prediction back into the corresponding superclass.

Original: Our method is detailed more formally in Algorithm 1.
Revised: Our method is detailed more formally in Algorithm 1.

Original: In our work, we experiment with two scenarios: (i) when hierarchy information is available and can be readily queried; and (ii) when hierarchy information is not available and the label set for each class must be generated, which we do so by prompting GPT-3.
Revised: In our work, we experiment with two scenarios: (i) when hierarchy information is available and can be readily queried; and (ii) when hierarchy information is not available and the label set for each class must be generated, which we do so by prompting GPT-3.

Original: 3. R EWEIGHTING PROBABILITIES WITH S UPERCLASS C ONFIDENCE
Revised: 3. R EWEIGHTING PROBABILITIES WITH S UPERCLASS C ONFIDENCE

Original: While the above method is able to effectively utilize CLIP’s ability to identify relatively fine-grained concepts, by predicting on only subclass labels we lose any positive benefits of the superclass label,  and performance may vary widely based on the quality of the subclass labels.
Revised: While the above method is able to effectively utilize CLIP’s ability to identify relatively fine-grained concepts, by predicting on only subclass labels we lose any positive benefits of the superclass label, [Table] and performance may vary widely based on the quality of the subclass labels.

Original: Given recent evidence (Kadavath et al., 2022) that large language models (like the text encoder in CLIP)  generally predict correct labels with high probability, we modify our initial algorithm to leverage this behavior and utilize both superclass and subclass information.
Revised: Given recent evidence (Minderer et al., 2021; Kadavath et al., 2022) that large language models (like the text encoder in CLIP) are well-calibrated and generally predict correct labels with high probability, we modify our initial algorithm to leverage this behavior and utilize both superclass and subclass information.

Original: We provide empirical evidence of this property in Appendix A.
Revised: We provide empirical evidence of this property in Appendix A.

Original: Specifically, we include an additional reweighting step within our main algorithm (see lines 4-9 in
Revised: Specifically, we include an additional reweighting step within our main algorithm (see lines 4-9 in

Original: Algorithm 1).
Revised: Algorithm 1).

Original: Here, we reweight each set of subclass probabilities by its superclass probability.
Revised: Here, we reweight each set of subclass probabilities by its superclass probability.

Original: In this way, we can attempt to avoid thebehavior in which CLIP makes an incorrect subclass prediction despite a confident andcorrect superclass prediction, and thus bias our model to never do worse than theraw superclass predictions.
Revised: Heuristically, as the prediction is now taken as the argmax over products of probabilities, large disagreements between subclass and superclass probabilities will be down-weighted (especially if one particular superclass is confident) and subclass probabilities will be more important in cases where the superclass probabilities are roughly uniform. We show ablations on the choice of the reweighting algorithm in Section 4.4.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we first lay out the experimental set-up.
Revised: In this section, we first lay out the experimental set-up.

Original: We then discuss, in order, the efficacy of our proposed method in situations with available class hierarchy information and in situations without any preexisting hierarchy.
Revised: We then discuss, in order, the efficacy of our proposed method in situations with available class hierarchy information and in situations without any preexisting hierarchy.

Original: After these main results, we present a series of ablations over various design choices showing where our method is robust and what might be crucial for its performance.
Revised: After these main results, we present a series of ablations over various design choices showing where our method is robust and what might be crucial for its performance.

Original: 4. S ETUP
Revised: 4. S ETUP

Original: Datasets.
Revised: Datasets.

Original: We test our method on the following image benchmarks: the four BREEDS imagenet subsets (living17, nonliving26, entity13, and entity30) (Santurkar et al., 2021), CIFAR20 (the to an inverse from a subclass to superclass, i.e., G − takes a subclass and superclass. [Table] al., 2017), LSUN-Scene (Yu et al., 2015), Office31 (Saenko et al., 2010), OfficeHome (Venkateswara et al., 2017),and ObjectNet (Barbu et al., 2019).
Revised: As we are primarily concerned with improving zero-shot CLIP performance in situations with ill-formed and/or semantically coarse class labels, we test our method on the following image benchmarks: the four BREEDS imagenet subsets (living17, nonliving26, entity13, and entity30) (Santurkar et al., 2021), CIFAR20 (the coarse-label version of CIFAR100) (Krizhevsky, 2009), Food-101 (Bossard et al., 2014), Fruits-360 (Mures¸an & Oltean, 2018), Fashion1M (Xiao et al., 2015), Fashion-MNIST (Xiao et al., 2017), LSUN-Scene (Yu et al., 2015), Office31 (Saenko et al., 2010), OfficeHome (Venkateswara et al., 2017), ObjectNet (Barbu et al., 2019), EuroSAT (Helber et al., 2019; 2018), and RESISC45 (Cheng et al., 2017).

Original: These datasets constitute a wide range of different image domains and include datasets with and without available hierarchy information.
Revised: These datasets constitute a wide range of different image domains and include datasets with and without available hierarchy information.

Original: We also examine
Revised: Additionally, the chosen datasets vary widely in the semantic granularity of their classes, from overly general cases (CIFAR20) to settings with a mixture of general and specific classes (Food-101,

Original:  CHiLS’s robustness to distribution shift within a dataset by averaging all results for the BREEDS datasets, Office31, and OfficeHome across different shifts (see Appendix C for more information).
Revised: OfficeHome). We also examine CHiLS’s robustness to distribution shift within a dataset by averaging all results for the BREEDS datasets, Office31, and OfficeHome across different shifts (see Appendix H for more information).

Original: We additionally modify the Fruits-360 and ObjectNet datasets to create existing taxonomies.
Revised: We additionally modify the Fruits-360 and ObjectNet datasets to create existing taxonomies.

Original: In the former we group classes together based on major fruit species (e.g. apples) and fix some ill-formed subclass labels.

Original: In the latter we use only the subset of ObjectNet that overlaps with ImageNet and utilize the resulting hierarchy to construct a coarser version of this subset.

Original: More details for dataset preparation are detailed in Appendix C.
Revised: More details for dataset preparation are detailed in Appendix H.

Original: Model Architecture.
Revised: Model Architecture.

Original: Unless otherwise specified, we use the ViTL/14@336px backbone (Radford et al., 2021) for our CLIP model, and used DaVinci-002 (with temperature fixed at 0 . 7 ) for all ablations involving GPT-3.
Revised: Unless otherwise specified, we use the ViTL/14@336px backbone (Radford et al., 2021) for our CLIP model, and used DaVinci-002 (with temperature fixed at 0 . 7 ) for all ablations involving GPT-3.

Original: For the choice of the prompt embedding function T ( · ) ,  we follow the procedure laid out for ImageNet in Radford et al. 
Revised: For the choice of the prompt embedding function T ( · ) , for each dataset we experiment (where applicable) with two different functions: (1) Using the average text embeddings of the 75 different prompts for each label used for ImageNet in Radford et al. (2021), where the prompts cover a wide array of captions and (2) Following the procedure that Radford et al. (2021) puts forth for more specialized datasets, we modify the standard prompt to be of the form “A photo of a {} , a type of [context] .” , where [context] is dataset-dependent (e.g. “food” in the case of food101). In the case that a custom prompt set exists for a dataset, as is the case with multiple datasets that the present work shares with Radford et al.

Original: by averaging the text embeddings ofdifferent prompts for  each class label.
Revised: (2021), we use the given prompt set for the latter option rather than building it from scratch. For each dataset, we use the prompt set that gives us the best baseline (i.e. superclass) zero-shot performance. More details are in Appendix C.

Original: Choice of Mapping Function G .
Revised: Choice of Mapping Function G .

Original: In our experiments, we primarily look at how the choice of the mapping function G influences the performance of CHiLS.
Revised: In our experiments, we primarily look at how the choice of the mapping function G influences the performance of CHiLS.

Original: In Section 4.2, we focus on the datasets with available hierarchy information.
Revised: In Section 4.2, we focus on the datasets with available hierarchy information.

Original: Here, G and G − 1 are simply table lookups to find the list of subclasses and corresponding superclass respectively.
Revised: Here, G and G − 1 are simply table lookups to find the list of subclasses and corresponding superclass respectively.

Original: In Section 4.3, we explore situations in which the true set of subclasses in each superclass is unknown.
Revised: In Section 4.3, we explore situations in which the true set of subclasses in each superclass is unknown.

Original: In these scenarios, we use GPTto generate our mapping function G .
Revised: In these scenarios, we use GPT-3 to generate our mapping function G .

Original: Specifically, given some label set size m and superclass name class-name ,  we query GPT-3 with the prompt:
Revised: Specifically, given some label set size m , superclass name class-name , and optional context (which we use whenever using the context-based prompt embedding), we query GPT-3 with the prompt:

Original: Generate a list of m types of the following: [Table]
Revised: Generate a list of m types of the following [context] : class-name

Original: The resulting output list from GPT-3 thus defines our mapping G from superclass to subclass.
Revised: The resulting output list from GPT-3 thus defines our mapping G from superclass to subclass.

Original: Unless otherwise specified, we fix m = 10 for all datasets.
Revised: Unless otherwise specified, we fix m = 10 for all datasets.

Original: Additionally, in 4.4 we explore situations in which hierarchical information is present but noisy, i.e. the label set for each superclass contains the  subclassespresent and  subclasses  not present in the data. [Figure]
Revised: Additionally, in 4.4 we explore situations in which hierarchical information is present but noisy, i.e. the label set for each superclass contains the true subclasses and erroneous subclasses that are not present in the dataset.

Original: 4. L EVERAGING A VAILABLE H IERARCHY I NFORMATION .
Revised: 4. L EVERAGING A VAILABLE H IERARCHY I NFORMATION .

Original: We first concern ourselves with the scenario in which there is hierarchy information already available (or readily accessible) for a given dataset.
Revised: We first concern ourselves with the scenario in which there is hierarchy information already available (or readily accessible) for a given dataset.

Original: In this situation, the set of subclasses for each superclass is exactly specified and correct (i.e. every image within each superclass falls into one of the subclasses).
Revised: In this situation, the set of subclasses for each superclass is exactly specified and correct (i.e. every image within each superclass falls into one of the subclasses).

Original: We emphasize that here we do not need information about which example belongs to which subclass, we just need a mapping of superclass to subclass.
Revised: We emphasize that here we do not need information about which example belongs to which subclass, we just need a mapping of superclass to subclass.

Original: For example, each class in the
Revised: For example, each class in the

Original: BREEDS dataset living17 is made up of 4 – 8 ImageNet subclasses at finer granularity (e.g. “parrot” includes “african grey” and “macaw”).
Revised: BREEDS dataset living17 is made up of 4 – 8 ImageNet subclasses at finer granularity (e.g. “parrot” includes “african grey” and “macaw”).

Original: Results In Table 1, we can see that our method performs better than using the baseline superclass labels alone across all 7 of the datasets with available hierarchy information, in some cases leading to +15% improvement in predictive accuracy.
Revised: Results In Table 1, we can see that our method performs better than using the baseline superclass labels alone across all 7 of the datasets with available hierarchy information, in some cases leading to +15% improvement in predictive accuracy.

Original: 4. CH I LS IN U NKNOWN H IERARCHY S ETTINGS
Revised: 4. CH I LS IN U NKNOWN H IERARCHY S ETTINGS

Original: Though we have seen considerable success in situations with access to the true hierarchical structure, in some real-world settings our dataset may not include any available information about the subclasses within each class.
Revised: Though we have seen considerable success in situations with access to the true hierarchical structure, in some real-world settings our dataset may not include any available information about the subclasses within each class.

Original: In this scenario, we turn to using GPT-3 to approximate the hierarchical map G (as specified in Section 4.1).
Revised: In this scenario, we turn to using GPT-3 to approximate the hierarchical map G (as specified in Section 4.1).

Original: It is important to note that GPT-3 may sometimes output suboptimal label sets, most notably in situations where GPT-3 chooses the wrong wordsense or when
Revised: It is important to note that GPT-3 may sometimes output suboptimal label sets, most notably in situations where GPT-3 chooses the wrong wordsense or when

Original: GPT-3 only lists modifiers on the original superclass (e.g. producing the list [red, yellow, green] for types of apples).
Revised: GPT-3 only lists modifiers on the original superclass (e.g. producing the list [red, yellow, green] for types of apples).

Original: In order to account for these issues in an out-of-the-box fashion, we automatically append the superclass name (if not already present) to each generated subclass label, and also include the superclass itself within the label set. 
Revised: In order to account for these issues in an out-of-the-box fashion, we automatically append the superclass name (if not already present) to each generated subclass label, and also include the superclass itself within the label set. For a controlled analysis about the effect of including the superclass itself in the label set, see Appendix D.

Original:  Results In this setting, our method is still able to beat the baseline performance in 11 out of 13 datasets, albeit with lower accuracy gainsgenerally (see Table 1).
Revised: [Figure] [Figure] Results In this setting, our method is still able to beat the baseline performance in most datasets, albeit with lower accuracy gains (see Table 1).

Original: Thus, while knowing the true subclass hierarchy can lead to large accuracy gains, it is enough to simply enumerate a list of possible subclasses for each class with no prior information about the dataset in order to improve the predictive accuracy. 
Revised: Thus, while knowing the true subclass hierarchy can lead to large accuracy gains, it is enough to simply enumerate a list of possible subclasses for each class with no prior information about the dataset in order to improve the predictive accuracy. In Figure 2, we show selected examples to highlight CHiLS’s behavior across three datasets.

Original: 4. A BLATIONS
Revised: 4. A BLATIONS

Original: Is Reweighting Necessary?
Revised: Is Reweighting Necessary?

Original: Though the reweighting step in CHiLS is motivated by the evidence that CLIP generally assigns higher probability to correct predictions rather than incorrect ones (see
Revised: Though the reweighting step in CHiLS is motivated by the evidence that CLIP generally assigns higher probability to correct predictions rather than incorrect ones (see

Original: Appendix A for empirical verification), it is not immediately clear whether it is truely necessary.
Revised: Appendix A for empirical verification), it is not immediately clear whether it is truely necessary.

Original: Averaged across all documented datasets, in Figure 2 (left) we show that in the true hierarchy setting, not reweighting the subclass probabilities can actually slightly boost performance (as the label sets are adequately tuned to the distribution of images).
Revised: Averaged across all documented datasets, in Figure 3 (left) we show that in the true hierarchy setting, not reweighting the subclass probabilities can actually slightly boost performance (as the label sets are adequately tuned to the distribution of images).

Original: However, in situations where the true hierarchy 
Revised: However, in situations where the true hierarchy is not present, omitting the reweighting step puts accuracy below the baseline performance.

Revised: We attribute this difference in behavior to the fact that reweighting multiplicatively combines the superclass and subclass predictions, and thus if subclass performance is sufficient on its own (as is the case when the true hierarchy is available) then combining it with superclass predictions

Original:  [Table]
Revised: can [Table]

Original: is not present, omitting the reweighting step puts accuracy below the baseline performance.
Revised: cause the model to more closely follow the behavior of the underperforming superclass predictor.

Original: Thus, as  the latter situation is considerably more likely in the wild, the reweighting step is imperative to utilizing CHiLS to its fullest.
Revised: Thus, as not having the true hierarchy is considerably more likely in the wild, the reweighting step is imperative to utilizing CHiLS to its fullest.

Revised: Different Reweighting Strategies We too experimented on whether the initial reweighting algorithm is the optimal method for combining superclass and subclass predictions.

Revised: Namely, we investigated whether superclass probabilities could be replaced by the sum over the matching subclass probabilities, and whether we can aggregate subclass probabilities and reweight them with the matching superclass probabilities (i.e. performing the normal reweighting step but in the space of superclasses).

Revised: In Figure 3 (right) we show that replacing the superclass probabilities in the reweighting step with aggregate subclass probabilities removes any accuracy gains from CHiLS, but that doing the reweighting step in superclass space does maintain CHiLS accuracy performance.

Revised: This suggests that the beneficial behavior of CHiLS may be due to successfully combining two different sets of class labels.

Revised: We also display the upper bound for combining superclass and subclass prediction (i.e. the accuracy when a datum is correctly labeled if the superclass or subclass predictions are correct) in purple, which we note is impossible in practice, and observe that even the best possible performance is not much higher than the performance of CHiLS.

Original: Noisy Available Hierarchies While the situation described in Section 4.3 is the most probable in practice, we additionally investigate the situation in which the hierarchical information is present but overestimates the set of subclasses, where each label set for a  class  includes the in-distribution subclasses and subclasses not present in the dataset.
Revised: Noisy Available Hierarchies While the situation described in Section 4.3 is the most probable in practice, we additionally investigate the situation in which the hierarchical information is present but overestimates the set of subclasses. For example, the scenario in which a dataset with the class “dog” includes huskies and corgis, but CHiLS is provided with huskies, corgis, and Labradors as possible subclasses, with the last being out-of-distribution.

Original: To do this, we return to the BREEDS datasets presented in Santurkar et al.
Revised: To do this, we return to the BREEDS datasets presented in Santurkar et al.

Original: As the BREEDS datasets were created so that each class contains the same number of subclasses (which are ImageNet classes), we modify G such that the label set for each superclass corresponds to all the ImageNet classes descended from that node in the hierarchy.
Revised: As the BREEDS datasets were created so that each class contains the same number of subclasses (which are ImageNet classes), we modify G such that the label set for each superclass corresponds to all the ImageNet classes descended from that node in the hierarchy (see Appendix G for more information).

Original: As we can see in Table 2, CHiLS is able to improve upon the baseline performance even in the presence of added noise in each label set.
Revised: As we can see in Table 2, CHiLS is able to improve upon the baseline performance even in the presence of added noise in each label set.

Original: Label Set Size
Revised: Label Set Size

Original: In previous works investigating importance of prompts in CLIP’s performance, it has been documented that the number of prompts used can have a decent effect on the overall performance (Pratt et al., 2022; Santurkar et al., 2022).
Revised: In previous works investigating importance of prompts in CLIP’s performance, it has been documented that the number of prompts used can have a decent effect on the overall performance (Pratt et al., 2022; Santurkar et al., 2022).

Original: Along this line, we investigate how the size of the subclass set generated for each class effects the overall accuracy by re-running our main experiments with varying values of m (namely,  5, 10,  and 15).
Revised: Along this line, we investigate how the size of the subclass set generated for each class effects the overall accuracy by re-running our main experiments with varying values of m (namely, 1, 5, 10, 15, and 50).

Original: In Figure 2 (right), there is overall not much variation across label set sizes that is consistent over all datasets.
Revised: In Figure 4 (right), there is little variation across label set sizes that is consistent over all datasets, though m = 1 has a few very low performing outliers due to the extremely small label set size.

Original: In all, we observe that the optimal label set size is context-specific, and depends upon the total number of classes present and the semantic granularity of the classes themselves.
Revised: We observe that the optimal label set size is context-specific, and depends upon the total number of classes present and the semantic granularity of the classes themselves.

Original: Individual dataset results are available in Appendix B.
Revised: Individual dataset results are available in Appendix E.

Original: Model Size
Revised: Model Size

Original: In order to examine whether the performance of CHiLS only exists within the best performing CLIP backbone (e.g. ViT-L/14@336), we measure the average relative change in accuracy performance between CHiLS and the baseline superclass predictions across all datasets for an array of different CLIP models.
Revised: In order to examine whether the performance of CHiLS only exists within the best performing CLIP backbone (e.g. ViT-L/14@336), we measure the average relative change in accuracy performance between CHiLS and the baseline superclass predictions across all datasets for an array of different CLIP models.

Original: Namely, we investigate the RN50, RN101, RN50x4, ViT-B/16, ViTB/32, and ViT-L/14@336 CLIP backbones (see Radford et al.
Revised: Namely, we investigate the RN50, RN101, RN50x4, ViT-B/16, ViT-B/32, and ViT-L/14@336 CLIP backbones (see Radford et al.

Original: (2021) for more information on the model specifications).
Revised: (2021) for more information on the model specifications).

Original: In Figure 3, we show that across the 6 specified CLIP backbones, CHiLS performance leads to relatively consistent relative accuracy gains, which shows that CHiLS’s benefits are not an artifact of large model size.
Revised: In Figure 4 (left), we show that across the 6 specified CLIP backbones, CHiLS performance leads to relatively consistent relative accuracy gains, with a slight (but not confidently significant) trend showing improved performance for the ResNet backbones over the ViT [Figure] r a g

Revised: [Figure]

Revised: l

Revised: [Figure]

Revised: t i v

Original:  Alternative Ensembling Methods Whilethe core of CHiLS is based on a set-based ensembling approach for subclasses and a linear averagingapproach for prompt templates (based on Radford et al.
Revised: [Figure] backbones, which is to be expected given their worse base capabilities. This shows that CHiLS’s benefits are not an artifact of large model size. Alternative Aggregating Methods While CHiLS is based on a set-based mapping approach for subclasses and a linear averaging for prompt templates (based on Radford et al.

Original: (2021)’s procedure for ImageNet)
Revised: (2021)’s procedure)

Original: , we experimented with  alternative ensembling methods for different parts of the CHiLS pipeline. Namely, we replaced the set-based mapping for subclasses with asimple linear average (i.e. every class embedding is thelinear average of subclass embeddings), and changed the ensembling method for prompts into a set-based method.
Revised: , we experimented with two alternative ensembling methods for different parts of the CHiLS pipeline: (1) Using a linear average of subclass embeddings rather than the set-based mapping (that is, every superclass’s text embedding is the average across all subclass embeddings, each themselves averaged across every prompt template) and (2) Using a set-based mapping for prompt templates rather than a linear average (i.e. instead of averaging across prompt templates, predict across each prompt template separately at inference time and then use embedded class to map back to the set of superclasses).

Original: Note in the latter case we only experiment with how this effectsraw superclass prediction (where each class maps to a set of 75 different prompt embeddings), as using set-based ensembling for both prompts and subclasses within CHiLS quickly becomes computationally expensive.
Revised: Note in the latter case we only experiment with how this effects superclass prediction (where each class maps to a set of the dataset’s chosen prompt embeddings), as using set-based ensembling for both prompts and subclasses within CHiLS quickly becomes computationally expensive.

Original: In 4, we see that using our initial ensembling methods (i.e. linear averaging for prompts and set mappings for subclasses) achieves greater accuracy on average. [Figure] [Figure]
Revised: In Figure 6 (in Appendix F), we see that using our initial aggregation methods (i.e. linear averaging for prompts and set mappings for subclasses) achieves greater accuracy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work [Figure] we demonstrated that the zero-shot image classification capabilities of CLIP models can be improved by leveraging hierarchical information for a given set of classes.
Revised: In this work, we demonstrated that the zero-shot image classification capabilities of CLIP models can be improved by leveraging hierarchical information for a given set of classes.

Original: When hierarchical structure is available in a given dataset, our method shows large improvements in zero-shot accuracy, and even when subclass information isn’t explicitly present, we showed that we can leverage a LLM (e.g., GPT-3) to generate subclasses for each class and still improve upon the baseline (superclass) accuracy.
Revised: When hierarchical structure is available in a given dataset, our method shows large improvements in zero-shot accuracy, and even when subclass information isn’t explicitly present, we showed that we can leverage GPTto generate subclasses for each class and still improve upon the baseline (superclass) accuracy.

Original: We remark that CHiLS may be quite beneficial to practitioners using CLIP as an out-of-the-box image classifier.
Revised: We remark that CHiLS may be quite beneficial to practitioners using CLIP as an out-of-the-box image classifier.

Original: Namely, we show that  even without existing hierarchical data accuracy can be improved with a fully automated pipeline (via querying GPT-3), yet CHiLS is flexible enough that any degree of hand-crafting label sets can be worked into the zero-shot pipeline.
Revised: Namely, we show that in scenarios where the class labels may be ill-formed or overly coarse, even without existing hierarchical data accuracy can be improved with a fully automated pipeline (via querying GPT-3), yet CHiLS is flexible enough that any degree of hand-crafting label sets can be worked into the zero-shot pipeline.

Original: Our method has the added benefit of being both completely zero-shot (i.e. no training or fine-tuning necessary) and is resource efficient.
Revised: Our method has the added benefit of being both completely zero-shot (i.e. no training or fine-tuning necessary) and is resource efficient.

Original: Limitations and Future Work
Revised: Limitations and Future Work

Original: As with usual zero-shot learning, we don’t have a way to validate the performance of our method.
Revised: As with usual zero-shot learning, we don’t have a way to validate the performance of our method.

Revised: Additionally, we recognize that CHiLS is suited for scenarios in which a semantic hierarchy most likely exists, and thus may not be particularly useful in classification tasks where the classes are already decently fine-grained.

Revised: We believe that this limitation will not hinder the applicability of our method, as practitioners would know if their task contains any latent semantic hierarchy and thus choose to use our method or not a priori.

Original: Given CHiLS’s empirical successes, we hope to perform more investigation to develop an understanding of why CHiLS is able to improve zero-shot accuracy and whether there is a more principled way of reconciling superclass and subclass predictions.
Revised: Given CHiLS’s empirical successes, we hope to perform more investigation to develop an understanding of why CHiLS is able to improve zero-shot accuracy and whether there is a more principled way of reconciling superclass and subclass predictions.

Original: R EPRODUCIBLITY S TATEMENT
Revised: R EPRODUCIBLITY S TATEMENT

Original: The source code for reproducing the work presented here is available at https://github.com/ anonOpenReview1/clip-hierarchy .
Revised: The source code for reproducing the work presented here is available at https://github.com/ anonOpenReview1/clip-hierarchy .

Original: We implement our method in PyTorch (Paszke et al., 2017) and provide an infrastructure to run all the experiments to generate corresponding results.
Revised: We implement our method in PyTorch (Paszke et al., 2017) and provide an infrastructure to run all the experiments to generate corresponding results.

Original: We have stored all models and logged all hyperparameters and seeds to facilitate reproducibility.
Revised: We have stored all models and logged all hyperparameters and seeds to facilitate reproducibility.

Original: Additionally, all necessary data preprocessing details are present in Appendix C.
Revised: Additionally, all necessary data preprocessing details are present in Appendix H

Original: A PPENDIX A E MPIRICAL E VIDENCE OF CLIP C ONFIDENCE
Revised: A PPENDIX A E MPIRICAL E VIDENCE OF CLIP C ONFIDENCE

Original: b a g b a ll b o a t b o d y a r m o r b
Revised: b a g b a ll b o a t b o d y a r m o r b

Original: o
Revised: o

Original: tt l e b u s c a r c h a i r
Revised: tt l e b u s c a r c h a i r

Original: c o a t d i g i t a l c o m p
Revised: c o a t d i g i t a l c o m p

Original: u t e r d w e lli n g f e n c e h a t k
Revised: u t e r d w e lli n g f e n c e h a t k

Original: e y b o a r d i n s t r u m e n t m e r c a n t il e e s t a b li s h m e n t o u t b u il d i n g p e r c u ss i o n i n s t r u m e n t p
Revised: e y b o a r d i n s t r u m e n t m e r c a n t il e e s t a b li s h m e n t o u t b u il d i n g p e r c u ss i o n i n s t r u m e n t p

Original: o t r oo f s h
Revised: o t r oo f s h

Original: i p s k
Revised: i p s k

Original: i r t s t r i n g e d i n s t r u m e n t t i m e p i e c e t r u c k w i n d i n s t r u m e n t s q u
Revised: i r t s t r i n g e d i n s t r u m e n t t i m e p i e c e t r u c k w i n d i n s t r u m e n t s q u

Original: a s h
Revised: a s h

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: o
Revised: o

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: d o
Revised: d o

Original: [Figure]
Revised: [Figure]

Original: w o l
Revised: w o l

Original: [Figure]
Revised: [Figure]

Original: f o x d o m e s t
Revised: f o x d o m e s t

Original: i c c a t b e a r b ee t l e b u tt e r f l y a p e m o n k e y
Revised: i c c a t b e a r b ee t l e b u tt e r f l y a p e m o n k e y

Original: [Figure]
Revised: [Figure]

Original: Correctly classified probabilities are on average higher than the misclassified probabilities.
Revised: Correctly classified probabilities are on average higher than the misclassified probabilities.

Original: The motivation behind the reweighting step of CHiLS primarily comes from the heuristic that LLMs make correct predictions with high estimated probabilities assigned to them (Kadavath et al., 2022).
Revised: The motivation behind the reweighting step of CHiLS primarily comes from the heuristic that LLMs make correct predictions with high estimated probabilities assigned to them (Kadavath et al., 2022), and that CLIP models themselves are well-calibrated (Minderer et al., 2021).

Original: However, we alsoverify whether there is some evidence of this behavior in CLIP models. Given g a r m e n t b i r d r e p t il e a r t h r o p o d m a m m a l a cc e ss o 
Revised: However, we also g a r m e n t b i r d r e p t il e a r t h r o p o d m a m m a l a cc e s s o r y c

Original: ry c r a f t e q u i p m e n t f u r n i t
Revised: r a f t e q u i p m e n t f u r n i t

Original: u r e i n s t r u
Revised: u r e i n s t r u

Original: m e n t m a n - m a d e s t r u c t u r e w h ee l e d v e h
Revised: m e n t m a n - m a d e s t r u c t u r e w h ee l e d v e h

Original: i c
Revised: i c

Original: l e p r o d u c e
Revised: l e p r o d u c e

Original: [Table]
Revised: [Table]

Original: e s  p a ss e
Revised: s e r p e n t e

Original: [Table]

Original: i [Table] e [Table]
Revised: s p

Original:  i a n a r a c hn i d a q u a t
Revised: a s s e r i n e s a u r i a n a r a c hn i d a q u a t

Original: i c
Revised: i c

Original: b i r d c r u s t a c e a n c a r n i v o r
Revised: b i r d c r u s t a c e a n c a r n i v o r

Original: e
Revised: e

Original: i n s e c t un g u l a t e p r
Revised: i n s e c t un g u l a t e p r

Original: i m a t e b o n
Revised: i m a t e b o n

Original: y f i s h b
Revised: y f i s h b

Original: a
Revised: a

Original: rr
Revised: rr

Original: i e r b u il d i n g e l e c t r o n i c
Revised: i e r b u il d i n g e l e c t r o n i c

Original: e q u i p m e n t f oo t w e a r g a r m e n th e a dd r e ss
Revised: e q u i p m e n t f oo t w e a r g a r m e n t

Original: h 
Revised: h e a d d r e ss h

Original: o m e a pp li a n c e k
Revised: o m e a pp l i a n c e k

Original: i t c h e n u t e n s il m e a s u r i n g i n s t r u
Revised: i t c h e n u t e n s il m e a s u r i n g i n s t r u

Original: m e n t m o t
Revised: m e n t m o t

Original: o r v e h
Revised: o r v e h

Original: i c
Revised: i c

Original: l e m u s i c a l i n s t r u m e n t n e c k w e a r s p o r t s e q u i p m e n t t a b l e w a r e t oo l v e ss e l d
Revised: l e m u s i c a l i n s t r u m e n t n e c k w e a r s p o r t s e q u i p m e n t t a b l e w a r e t oo l v e ss e l d

Original: i s h v e g e t a b l e f r u i t
Revised: i s h v e g e t a b l e f r u i t

Original: [Table] 
Revised: [Table] [Table]

Original:  [Figure]
Revised: [Table] [Table] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: t y

Original: [Figure]
Revised: [Figure]

Original:  that the output of a CLIP model is a probability distribution over the provided classes, we care specifically about the probability of the argmax class (i.e. the predicted class) when the model is correct and when it is incorrect.
Revised: verify whether there is some evidence of this behavior in CLIP models. Given that the output of a CLIP model is a probability distribution over the provided classes, we care specifically about the probability of the argmax class (i.e. the predicted class) when the model is correct and when it is incorrect.

Original: Across the BREEDS datasets for the standard ImageNet domain, in Figure 5 we show the distribution of the correct and incorrect argmax probabilities for each class (i [Table] e. for each class
Revised: Across the BREEDS datasets for the standard ImageNet domain, in Figure 5 we show the distribution of the correct and incorrect argmax probabilities for each class (i.e. for each class

Original: c i , we show the output probabilties for c
Revised: c i , we show the output probabilties for c

Original: i when it was correctly classified and the output probabilities of the predicted classes when the true class is c i ).
Revised: i when it was correctly classified and the output probabilities of the predicted classes when the true class is c i ).

Original: Whenever CLIP is correct, the associated probability is on average much higher than the probabilities associated with misclassification. 
Revised: Whenever CLIP is correct, the associated probability is on average much higher than the probabilities associated with misclassification. B CLIP P RIMER Open Vocabulary models (as termed in Pham et al.

Original: B L ABEL S ET A BLATION A CCURACY
Revised: (2021)) refer to models that are able to classify images by associating them with natural language descriptions of each class. These models are “open” in the sense that they are to predict on an arbitrary vocabulary of descriptions (as opposed to a fix set), thus allowing for arbitrary-way image classification. Popular open vocabulary models include the model of focus CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) as examples. C ontrastive L anguage I mage P retraining (CLIP) is a family of open vocabulary models, and the focus of the present work.

Original: [Table]
Revised: CLIP, which is comprised of a text encoder and an image encoder that project into the same latent space, is trained in the following way: Given a set of image-caption pairs (e.g. a photo of a dog with the caption “a photo of a dog.”), CLIP is trained to predict which caption goes with which image as a contrastive learning objective by comparing the similarity between each image embedding and each caption embedding. At inference time (in the zero-shot setting), a na¨ıve method for image classification (which is the initial baseline tried in Radford et al. (2021)) involves simply passing in the list of class names for a given dataset, and calculating the similarity between a particular image embedding and each one of these class embeddings.

Revised: However, Radford et al.

Revised: (2021) found that by taking a cue from the recent literature on prompt engineering for large language models (Gao et al., 2020), CLIP can perform significantly better as a zero-shot predictor if each class name is included in a natural language prompt that resembles some sort of image caption (as that is what CLIP was trained on).

Revised: As an example, the standard baseline prompt mentioned is “A photo of a {} .” .

Revised: In our work, we define a prompt (or prompt template, which we use interchangeably) as any caption-like phrase in natural language that a class name can be injected into.

Revised: C A DDING C ONTEXT TO P ROMPTS AND GPT-3 Q UERIES

Revised: [Table]

Revised: In order to disentangle the effect that well-formed prompt templates have on the success of CHiLS, for each dataset (besides the BREEDS datasets and ObjectNet as they are already semantically similar to ImageNet) we compare the ImageNet 75 classes against a dataset-specific set of prompt templates.

Revised: In the case of EuroSAT, RESISC45, CIFAR20 and Food-101, we directly use the prompt template set from Radford et al.

Revised: For LSUN-Scene, we use the prompt template set for

Revised: SUN397 (Xiao et al., 2010), as the two datasets are semantically similar.

Revised: For the rest of the datasets not yet mentioned (namely Fruits360, Fashion1M, Fashion-MNIST, Office31, and OfficeHome)

Revised: we add the [context] marker into the standard prompt template as mentioned in Section 4.1.

Revised: The prompt sets themselves can be directly found in the code implementation for this project.

Revised: For the GPT-3 Query with additional context, we add the respective [context] token to the query if the dataset-specific prompt template is used.

Revised: Note that we did not create [context] tokens for

Revised: EuroSAT, LSUN-Scene, or RESISC45 despite testing dataset-specific prompt templates, as there did not seem to be a concise semantic label to describe the classes in these datasets.

Revised: In Table 3, we list the dataset, the [context] token (if applicable), and the final prompt set used for all the experiments.

Revised: Here, we found that while dataset-specific prompts often improved baseline performance, they were not gauranteed to improve performance, as in both Fasion-MNIST and OfficeHome the general

Revised: ImageNet prompt set performed better.

Revised: D I NCLUDING S UPERCLASS L ABELS IN L ABEL S ETS

Revised: Dataset

Revised: CHiLS Accuracy

Revised: CHiLS Accuracy

Revised: CHiLS Accuracy

Revised: CHiLS Accuracy (Existing Map) (Existing Map+) (GPT-3 Map) (GPT-3 Map+)

Revised: Nonliving 90.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Living

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Entity

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: Table 3 displays the raw accuracy scores for CHiLS across different label set sizes.
Revised: [Table] [Table] [Table] Entity 88.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: CIFAR

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: FoodN/A

Revised: N/A 91.63 (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Fruits

Revised: [Table]

Revised: 60.87 ( +0 . 40 ) 60.63 ( +0 . 16 ) 62.48 (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Fashion1M

Revised: N/A

Revised: N/A 47.51 ( +1 . 73 ) 47.44 ( +

Revised: [Table]

Revised: FashionN/A

Revised: N/A 67.82 ( − 0 . 98 ) 70.85 ( +

Revised: [Table]

Revised: 35 ) MNIST

Revised: LSUN-Scene

Revised: N/A

Revised: N/A 88.80 ( +

Revised: [Table]

Revised: 60 ) 88.97 ( +

Revised: [Table]

Revised: Office

Revised: N/A

Revised: N/A 86.58 ( −

Revised: [Table]

Revised: [Table]

Revised: OfficeHome

Revised: N/A

Revised: N/A

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ObjectNet

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: EuroSAT

Revised: N/

Revised: [Table]

Revised: N/

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: RESISC

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: With CHiLS when the existing map is not available, we append the superclass name to each label set to account for possible noise in the GPT-generated label set.

Revised: In Table 4, we show the effect that this inclusion has in both the existing map and GPT-map cases.

Revised: Note that in the main paper, columns 1 and 4 correspond to the main results (i.e. no superclass labels in existing maps and superclass labels in GPT-3 maps).

Revised: In both cases, the presence of the superclass label more effectively strikes a balance between subclass and superclass predictions.

Revised: In the existing map case, this actually hurts performance, as the subclass labels are optimal in the given dataset.

Revised: In the GPT-3 map case, while there are some datasets where removing the superclass label improves performance (namely

Revised: Fruits360 and Entity30), in ever other case removing the superclass label hurts performance, sometimes by multiple percentage points.

Revised: E L ABEL S ET A BLATION A CCURACY

Revised: [Table]

Revised: MNISTLSUN-Scene

Revised: [Table]

Revised: ObjectNet

Revised: [Table]

Revised: EuroSAT

Revised: [Table]

Revised: Table 5 displays the raw accuracy scores for CHiLS across different label set sizes.

Revised: F A LTERNATIVE A GGREGATION R ESULTS

Revised: [Figure]

Revised: Figure 6 displays the results of our ablation on alternative ways to aggregate subclasses and prompts.

Original: C D ATASET D ETAILS
Revised: G N OISY A VAILABLE H IERARCHY D ETAILS

Revised: The ImageNet (Deng et al., 2009) dataset itself includes a rich hierarchical taxonomy, where every class is a leaf node of the hierarchy.

Revised: In the original BREEDS (Santurkar et al., 2021) work, the authors modify the structure slightly in order to place concepts at semantically-similar levels of granularity at the same depth, and additional restrict the number of subclasses within each of the

Revised: BREEDS datasets in order to balance the data.

Revised: Thus, it is possible for each BREEDS dataset to use the dataset with its superclasses and restricted set of subclasses but provide CHiLS with all the subclass labels present in the ImageNet hierarchy for each superclass (i.e. all leaf nodes descended from each superclass node).

Revised: In Table 7, we display a subset of the living17 BREEDS dataset class structure with the original subclasses and the ImageNet subclasses.

Revised: Observe that in some cases, there are many subclass labels provided to CHiLS than is present in the data.

Revised: H D ATASET D ETAILS

Original: [Table]
Revised: [Table]

Original: CHiLS Across Domain Shifts For each of the BREEDS datasets (Santurkar et al., 2021), Office (Saenko et al., 2010), and OfficeHome (Venkateswara et al., 2017), all results presented are the average over different domains.
Revised: CHiLS Across Domain Shifts For each of the BREEDS datasets (Santurkar et al., 2021), Office (Saenko et al., 2010), and OfficeHome (Venkateswara et al., 2017), all results presented are the average over different domains.

Original: The specific domains used are show in Table 4.
Revised: The specific domains used are show in Table 6.

Original: Fruits-360 For zero-shot classification with CLIP models, Fruits-360 (Mures¸an & Oltean, 2018) in its raw form is somewhat ill-formed from a class name perspective, as there are classes only differentiated by a numeric index (e.g. “Apple Golden 1” and “Apple Golden 2”) and classes at mixed granularity (e.g. “forest nut” and “hazelnut” are separate classes even though hazelnuts are a type of forest nut).
Revised: Fruits-360 For zero-shot classification with CLIP models, Fruits-360 (Mures¸an & Oltean, 2018) in its raw form is somewhat ill-formed from a class name perspective, as there are classes only differentiated by a numeric index (e.g. “Apple Golden 1” and “Apple Golden 2”) and classes at mixed granularity (e.g. “forest nut” and “hazelnut” are separate classes even though hazelnuts are a type of forest nut).

Original: We thus manually rename classes using the structure laid out in Table 6, which results in a 59-way superclass classification problem, with 102 ground-truth subclasses.
Revised: We thus manually rename classes using the structure laid out in Table 9, which results in a 59-way superclass classification problem, with 102 ground-truth subclasses.

Original: ObjectNet The ObjectNet dataset (Barbu et al., 2019) has partial overlap (113 classes) with the
Revised: ObjectNet The ObjectNet dataset (Barbu et al., 2019) has partial overlap (113 classes) with the

Original: ImageNet (Deng et al., 2009) hierarchical class structure.
Revised: ImageNet (Deng et al., 2009) hierarchical class structure.

Original: From this subset of ObjectNet, we use the
Revised: From this subset of ObjectNet, we use the

Original: BREEDS hierarchy (Santurkar et al., 2021) to generate a coarse-grained version of ObjectNet that is shown in Table 5.
Revised: BREEDS hierarchy (Santurkar et al., 2021) to generate a coarse-grained version of ObjectNet that is shown in Table 8.

Original: In this 11-way classification task, the true subclasses are the original ObjectNet classes. 
Revised: In this 11-way classification task, the true subclasses are the original ObjectNet classes. [Table] iguana [Table]

Revised: iguana

Original: [Table]
Revised: [Table]

Revised: whiptail

Original: [Table]
Revised: [Table]

Original: granny
Revised: agama

Original: [Table]
Revised: [Table]

Original: apple
Revised: alligator

Original: [Table]
Revised: [Table]

Original: Lady
Revised: vine [Table] rock

Original: [Table]
Revised: [Table]

Original: apple
Revised: cobra

Original: Apple Redred apple apple
Revised: [Table] sea [Table] [Table]

Original:  Apple Redred apple apple
Revised: Cleaned Superclass Apple Braeburn braeburn apple apple Apple Crimson Snow crimson snow apple apple Apple Goldengolden apple apple

Original: Apple Redred apple apple
Revised: Apple Goldengolden apple apple Apple Goldengolden apple apple Apple Granny Smith granny smith apple apple Apple Pink Lady pink lady apple apple

Original: Apple  Red Delicious red delicious apple apple
Revised: Apple Redred apple apple Apple Redred apple apple Apple Redred apple apple Apple Red Delicious red delicious apple apple

Original: Apple Red Yellowred yellow apple apple Apple Red Yellowred yellow apple apple
Revised: Apple Red Yellowred yellow apple apple Apple Red Yellowred yellow apple apple

Original: Apricot apricot apricot Avocado 
Revised: Apricot apricot apricot Avocado avocado avocado

Original: [Table]

Original: Avocado ripe avocado avocado
Revised: Avocado ripe avocado avocado

Original: Banana banana banana
Revised: Banana banana banana

Original: Banana Lady Finger lady finger banana banana
Revised: Banana Lady Finger lady finger banana banana

Original: Banana Red red banana banana
Revised: Banana Red red banana banana

Original: Beetroot beetroot beetroot
Revised: Beetroot beetroot beetroot

Original: Blueberry blueberry blueberry
Revised: Blueberry blueberry blueberry

Original: Cactus fruit cactus fruit cactus fruit
Revised: Cactus fruit cactus fruit cactus fruit

Original: Cantaloupemelon melon
Revised: Cantaloupemelon melon

Original: Cantaloupemelon melon
Revised: Cantaloupemelon melon

Original: Carambula star fruit star fruit
Revised: Carambula star fruit star fruit

Original: Cauliflower cauliflower cauliflower
Revised: Cauliflower cauliflower cauliflower

Original: Cherrycherry cherry
Revised: Cherrycherry cherry

Original: Cherrycherry cherry
Revised: Cherrycherry cherry

Original: Cherry Rainier rainier cherry cherry
Revised: Cherry Rainier rainier cherry cherry

Original: Cherry Wax Black black cherry cherry
Revised: Cherry Wax Black black cherry cherry

Original: Cherry Wax Red red cherry cherry
Revised: Cherry Wax Red red cherry cherry

Original: Cherry Wax Yellow yellow cherry cherry
Revised: Cherry Wax Yellow yellow cherry cherry

Original: Chestnut nut nut
Revised: Chestnut nut nut

Original: Clementine orange orange
Revised: Clementine orange orange

Original: Cocos cocos cocos
Revised: Cocos cocos cocos

Original: Corn corn corn
Revised: Corn corn corn

Original: Corn Husk corn husk corn husk
Revised: Corn Husk corn husk corn husk

Original: Cucumber Ripe cucumber cucumber
Revised: Cucumber Ripe cucumber cucumber

Original: Cucumber Ripecucumber cucumber
Revised: Cucumber Ripecucumber cucumber

Original: Dates date date
Revised: Dates date date

Original: Eggplant eggplant eggplant
Revised: Eggplant eggplant eggplant

Original: Fig fig fig
Revised: Fig fig fig

Original: Ginger Root ginger root ginger root
Revised: Ginger Root ginger root ginger root

Original: Granadilla granadilla passion fruit
Revised: Granadilla granadilla passion fruit

Original: Grape Blue blue grape grape
Revised: Grape Blue blue grape grape

Original: Grape Pink pink grape grape
Revised: Grape Pink pink grape grape

Original: Grape White white grape grape
Revised: Grape White white grape grape

Original: Grape Whitewhite grape grape Grape Whitewhite grape grape
Revised: Grape Whitewhite grape grape Grape Whitewhite grape grape

Original: Grape Whitewhite grape grape
Revised: Grape Whitewhite grape grape

Original: Grapefruit Pink pink grapefruit grapefruit
Revised: Grapefruit Pink pink grapefruit grapefruit

Original: Grapefruit White white grapefruit grapefruit
Revised: Grapefruit White white grapefruit grapefruit

Original: Guava gauva gauva
Revised: Guava gauva gauva

Original: Hazelnut nut nut
Revised: Hazelnut nut nut

Original: Huckleberry huckleberry huckleberry
Revised: Huckleberry huckleberry huckleberry

Original: Kaki kaki persimmon
Revised: Kaki kaki persimmon

Original: Kiwi kiwi kiwi
Revised: Kiwi kiwi kiwi

Original: Kohlrabi kohlrabi kohlrabi
Revised: Kohlrabi kohlrabi kohlrabi

Original: Kumquats kumquat kumquat
Revised: Kumquats kumquat kumquat

Original: Lemon lemon lemon
Revised: Lemon lemon lemon

Original: Lemon Meyer meyer lemon lemon
Revised: Lemon Meyer meyer lemon lemon

Original: Limes lime lime
Revised: Limes lime lime

Original: Lychee lychee lychee
Revised: Lychee lychee lychee

Original: Mandarine orange orange
Revised: Mandarine orange orange

Original: Mango mango mango
Revised: Mango mango mango

Original: Mango Red red mango mango
Revised: Mango Red red mango mango

Original: Mangostan mangostan mangostan
Revised: Mangostan mangostan mangostan

Original: Maracuja maracuja passion fruit
Revised: Maracuja maracuja passion fruit

Original: Melon Piel de Sapo melon melon
Revised: Melon Piel de Sapo melon melon

Original: Mulberry mulberry mulberry
Revised: Mulberry mulberry mulberry

Original: Nectarine nectarine nectarine
Revised: Nectarine nectarine nectarine

Original: Nectarine Flat flat nectarine nectarine
Revised: Nectarine Flat flat nectarine nectarine

Original: Nut Forest forest nut nut
Revised: Nut Forest forest nut nut

Original: Nut Pecan pecan nut nut
Revised: Nut Pecan pecan nut nut

Original: Onion Red red onion onion
Revised: Onion Red red onion onion

Original: Onion Red Peeled red onion onion
Revised: Onion Red Peeled red onion onion

Original: Onion White white onion onion
Revised: Onion White white onion onion

Original: Orange orange orange
Revised: Orange orange orange

Original: Papaya papaya papaya
Revised: Papaya papaya papaya

Original: Passion Fruit passion fruit passion fruit
Revised: Passion Fruit passion fruit passion fruit

Original: Peach peach peach
Revised: Peach peach peach

Original: Peachpeach peach
Revised: Peachpeach peach

Original: Peach Flat flat peach peach
Revised: Peach Flat flat peach peach

Original: Pear pear pear
Revised: Pear pear pear

Original: Pearpear pear
Revised: Pearpear pear

Original: Pear Abate abate pear pear
Revised: Pear Abate abate pear pear

Original: Pear Forelle forelle pear pear
Revised: Pear Forelle forelle pear pear

Original: Pear Kaiser kaiser pear pear
Revised: Pear Kaiser kaiser pear pear

Original: Pear Monster monster pear pear
Revised: Pear Monster monster pear pear

Original: Pear Red red pear pear
Revised: Pear Red red pear pear

Original: Pear Stone stone pear pear
Revised: Pear Stone stone pear pear

Original: Pear Williams williams pear pear
Revised: Pear Williams williams pear pear

Original: Pepino pepino pepino
Revised: Pepino pepino pepino

Original: Pepper Green green pepper pepper
Revised: Pepper Green green pepper pepper

Original: Pepper Orange orange pepper pepper
Revised: Pepper Orange orange pepper pepper

Original: Pepper Red red pepper pepper
Revised: Pepper Red red pepper pepper

Original: Pepper Yellow yellow pepper pepper
Revised: Pepper Yellow yellow pepper pepper

Original: Physalis groundcherry groundcherry
Revised: Physalis groundcherry groundcherry

Original: Physalis with Husk groundcherry groundcherry
Revised: Physalis with Husk groundcherry groundcherry

Original: Pineapple pineapple pineapple
Revised: Pineapple pineapple pineapple

Original: Pineapple Mini mini pineapple pineapple
Revised: Pineapple Mini mini pineapple pineapple

Original: Pitahaya Red dragon fruit dragon fruit
Revised: Pitahaya Red dragon fruit dragon fruit

Original: Plum plum plum
Revised: Plum plum plum

Original: Plumplum plum
Revised: Plumplum plum

Original: Plumplum plum
Revised: Plumplum plum

Original: Pomegranate pomegranate pomegranate
Revised: Pomegranate pomegranate pomegranate

Original: Pomelo Sweetie pomelo pomelo
Revised: Pomelo Sweetie pomelo pomelo

Original: Potato Red red potato potato Potato Red Washed red potato potato
Revised: Potato Red red potato potato Potato Red Washed red potato potato

Original: Potato Sweet sweet 
Revised: Potato Sweet sweet potato potato

Original: [Table]
Revised: Potato White white potato potato Quince quince quince Rambutan rambutan rambutan

Original: Raspberry raspberry raspberry
Revised: Raspberry raspberry raspberry

Original: Redcurrant redcurrant redcurrant
Revised: Redcurrant redcurrant redcurrant

Original: Salak salak snake fruit
Revised: Salak salak snake fruit

Original: Strawberry strawberry strawberry Strawberry Wedge strawberry strawberry
Revised: Strawberry strawberry strawberry Strawberry Wedge strawberry strawberry

Original: Tamarillo tamarillo
Revised: Tamarillo tamarillo

Original: tamarillo
Revised: tamarillo

Original: Tangelo tangelo tangelo
Revised: Tangelo tangelo tangelo

Original: Tomatotomato tomato
Revised: Tomatotomato tomato

Original: Tomatotomato tomato Tomatotomato tomato
Revised: Tomatotomato tomato Tomatotomato tomato

Original: Tomatotomato tomato
Revised: Tomatotomato tomato

Original: Tomato Cherry Red cherry tomato tomato
Revised: Tomato Cherry Red cherry tomato tomato

Original: Tomato Heart heart tomato tomato
Revised: Tomato Heart heart tomato tomato

Original: Tomato Maroon maroon tomato tomato
Revised: Tomato Maroon maroon tomato tomato

Original: Tomato Yellow yellow tomato tomato
Revised: Tomato Yellow yellow tomato tomato

Original: Tomato not Ripened unripe tomato tomato
Revised: Tomato not Ripened unripe tomato tomato

Original: Walnut nut nut
Revised: Walnut nut nut

Original: Watermelon melon melon
Revised: Watermelon melon melon

Revised: [Table]