Original: Semantic Supervision: Enabling Generalization over Output Spaces
Revised: Semantic Supervision: Enabling Generalization over Output Spaces

Original: Abstract
Revised: Abstract

Original: In this paper, we propose semantic supervision (S EM S UP ) – a unified paradigm for training classifiers that generalize over output spaces.
Revised: In this paper, we propose semantic supervision (S EM S UP ) – a unified paradigm for training classifiers that generalize over output spaces.

Original: In contrast to standard classification, which treats classes as discrete symbols, S EM S UP represents them as dense vector features obtained from descriptions of classes (e.g., “The cat is a small carnivorous mammal”).
Revised: In contrast to standard classification, which treats classes as discrete symbols, S EM S UP represents them as dense vector features obtained from descriptions of classes (e.g., “The cat is a small carnivorous mammal”).

Original: This allows the output space to be unbounded (in the space of descriptions) and enables models to generalize both over unseen inputs and unseen outputs.
Revised: This allows the output space to be unbounded (in the space of descriptions) and enables models to generalize both over unseen inputs and unseen outputs.

Original: Specifically, S EM S UP enables four types of generalization, to – (1) unseen class descriptions, (2) unseen classes, (3) unseen super-classes, and (4) unseen tasks.
Revised: Specifically, S EM S UP enables four types of generalization, to – (1) unseen class descriptions, (2) unseen classes, (3) unseen super-classes, and (4) unseen tasks.

Original: Through experiments on four classification datasets, two input modalities (text and images), and two output description modalities (text and JSON), we show that our S EM S UP models significantly outperform baselines.
Revised: Through experiments on four classification datasets, two input modalities (text and images), and two output description modalities (text and JSON), we show that our S EM S UP models significantly outperform baselines.

Original: For instance, our model outperforms baselines by 40% and 20% precision points on unseen descriptions and classes, respectively, on a news categorization dataset (RCV1).
Revised: For instance, our model outperforms baselines by 40% and 15% precision points on unseen descriptions and classes, respectively, on a news categorization dataset (RCV1).

Original: S EM S UP can serve as a pathway for scaling neural models to large unbounded output spaces and enabling better generalization and model reuse for unseen tasks and domains.
Revised: S EM S UP can serve as a pathway for scaling neural models to large unbounded output spaces and enabling better generalization and model reuse for unseen tasks and domains.

Original:  Introduction
Revised:  Introduction

Original: Most approaches to supervised classification have traditionally considered different output classes as abstract symbols devoid of meaning (e.g., 0 , 1 , 2 ).
Revised: Most approaches to supervised classification have traditionally considered different output classes as abstract symbols devoid of meaning (e.g., 0 , 1 , 2 ).

Original: This predefines a rigid output space that inhibits models from generalizing to unseen classes (e.g., “moth”), even if it is similar to a class seen during training (e.g., “butterfly”).
Revised: This predefines a rigid output space that inhibits models from generalizing to unseen classes (e.g., “moth”), even if it is similar to a class seen during training (e.g., “butterfly”).

Original: Some prior works have aimed to tackle this problem by predicting classes based on semantic class attributes (Palatucci et al., 2009), word vectors of class names (Frome et al., 2013), or textual class descriptions (Lei Ba et al., 2015).
Revised: Some prior works have aimed to tackle this problem by predicting classes based on semantic class attributes (Palatucci et al., 2009), word vectors of class names (Frome et al., 2013), or textual class descriptions (Lei Ba et al., 2015).

Original: These works provide solutions with a specific capability in mind, such as zeroshot generalization or domain generalization.
Revised: These works provide solutions with a specific capability in mind, such as zeroshot generalization or domain generalization.

Original: In this paper, we develop a general unifying paradigm for supervised classification called semantic supervision (S EM S UP ) to leverage rich semantic information about classes to enable better generalization over the output space.
Revised: In this paper, we develop a general unifying paradigm for supervised classification called semantic supervision (S EM S UP ) to leverage rich semantic information about classes to enable better generalization over the output space.

Original: S EM S UP allows models to learn better representations of output classes using multiple “descriptions” that capture their semantics and neural output encoders to represent them in vector space.
Revised: S EM S UP allows models to learn better representations of output classes using multiple “descriptions” that capture their semantics and neural output encoders to represent them in vector space.

Original: S EM S UP can handle different types of descriptions, and we experiment with textual descriptions (e.g., “The cat is carnivorous mammal”) and JSONs (e.g., { size:small, legs:4 } ).
Revised: S EM S UP can handle different types of descriptions, and we experiment with textual descriptions (e.g., “The cat is carnivorous mammal”) and JSONs (e.g., { size:small, legs:4 } ).

Original: Training models to predict over semantically informative choices has several advantages: (1) the number of choices can be varied during inference, (2) the choices can be described in several different ways (e.g., by different end-users), (3) new concepts can be provided as choices by describing them in known terms, and (4) the choices can span varying levels of granularity (e.g., classify between descriptions of vehicle , bus and wheel ).
Revised: Training models to predict over semantically informative choices has several advantages: (1) the number of choices can be varied during inference, (2) the choices can be described in several different ways (e.g., by different end-users), (3) new concepts can be provided as choices by describing them in known terms, and (4) the choices can span varying levels of granularity (e.g., classify between descriptions of vehicle , bus and wheel ).

Original: We demonstrate the general applicability of S EM -
Revised: We demonstrate the general applicability of S EM -

Original: S UP to any standard classification task by considering four existing benchmarks spanning text and image inputs, two different types of ‘descriptions’ (English text and structured JSON), and two paradigms – multi-class and multi-label classification.
Revised: S UP to any standard classification task by considering four existing benchmarks spanning text and image inputs, two different types of ‘descriptions’ (English text and structured JSON), and two paradigms – multi-class and multi-label classification.

Original: We show that S EM S UP models generalize over output spaces in various ways, including (1) new descriptions of seen classes, (2) unseen classes, (3) unseen high-level superclasses, and (4) unseen tasks (Figure 1).
Revised: We show that S EM S UP models generalize over output spaces in various ways, including (1) new descriptions of seen classes, (2) unseen classes, (3) unseen high-level superclasses, and (4) unseen tasks (Figure 1).

Original: In all tasks and scenarios
Revised: In all tasks and scenarios

Original: S EM S UP outperforms existing systems developed for zero-shot generalization to unseen classes while also remaining competitive with standard classifiers on seen classes.
Revised: S EM S UP outperforms existing systems developed for zero-shot generalization to unseen classes while also remaining competitive with standard classifiers on seen classes.

Original: For instance, S EM S UP achieves absolute improvements of 40% on unseen descriptions in RCV1, 15% on unseen classes in CIFAR, and 10% on unseen superclasses in 20 Newsgroups.
Revised: For instance, S EM S UP achieves absolute improvements of 40% on unseen descriptions in RCV1, 15% on unseen classes in CIFAR, and 10% on unseen superclasses in 20 Newsgroups.

Original: We recognize the importance of using multiple descriptions and pre-trained models for encoding semantic supervision ( § A), which provides a recipe for users to adopt our work.
Revised: We recognize the importance of using multiple descriptions and pre-trained models for encoding semantic supervision ( § A), which provides a recipe for users to adopt our work.

Original:  Related Work
Revised:  Related Work

Original: S EM S UP unifies these works by using multiple rich descriptions, multiple input (e.g., image and text) and output (e.g., text and JSON) modalities, and exhibiting numerous generalization capabilities to – unseen descriptions, classes, superclasses, and tasks.
Revised: S EM S UP unifies these works by using multiple rich descriptions, multiple input (e.g., image and text) and output (e.g., text and JSON) modalities, and exhibiting numerous generalization capabilities to – unseen descriptions, classes, superclasses, and tasks.

Original: We summarize S EM S UP ’s capabilities and prior work in Table 1.
Revised: We summarize S EM S UP ’s capabilities and prior work in Table 1.

Original: Zero-shot learning with auxiliary information The goal of zero-shot learning is to predict novel classes not encountered at train time by using class specifications (auxiliary information) of some form, like representative images (Larochelle et al., 2008).
Revised: Zero-shot learning with auxiliary information The goal of zero-shot learning is to predict novel classes not encountered at train time by using class specifications (auxiliary information) of some form, like representative images (Larochelle et al., 2008).

Original: At their core, these works define a joint embedding space for inputs and auxiliary information that represents classes.
Revised: At their core, these works define a joint embedding space for inputs and auxiliary information that represents classes.

Original: Subsequent papers like DeViSE (Frome et al., 2013) and others Socher et al.
Revised: Subsequent papers like DeViSE (Frome et al., 2013) and others Socher et al.

Original: ; Pappas and Henderson (2019);
Revised: ; Pappas and Henderson (2019);

Original: Dauphin et al.
Revised: Dauphin et al.

Original: (2014) used shallow averaged word embeddings corresponding to class names, which have their shortcomings because they are oblivious to the word order.
Revised: (2014) used shallow averaged word embeddings corresponding to class names, which have their shortcomings because they are oblivious to the word order.

Original: Other papers have employed text descriptions of classes, either crowdsourced or scraped from
Revised: Other papers have employed text descriptions of classes, either crowdsourced or scraped from

Original: Wikipedia.
Revised: Wikipedia.

Original: Some of these use simple bag-of-words features to encode them (Nam et al., 2016; Lei Ba et al., 2015; Qiao et al., 2016), which has the issue of ignoring word order and a portion of semantics, while others use deeper models (Reed et al., 2016; Bujwid and Sullivan, 2021).
Revised: Some of these use simple bag-of-words features to encode them (Nam et al., 2016; Lei Ba et al., 2015; Qiao et al., 2016), which has the issue of ignoring word order and a portion of semantics, while others use deeper models (Reed et al., 2016; Bujwid and Sullivan, 2021).

Original: However, all these papers focus only on a single input modality (images), a single output modality (text), or a single generalization scenario (to unseen classes), whereas S EM S UP handles a variety of these attributes.
Revised: However, all these papers focus only on a single input modality (images), a single output modality (text), or a single generalization scenario (to unseen classes), whereas S EM S UP handles a variety of these attributes.

Original: Prompting pre-trained models
Revised: Prompting pre-trained models

Original: S EM S UP is partly related to a recent area of research which uses natural language to prompt large pre-trained models (Liu et al., 2021).
Revised: S EM S UP is partly related to a recent area of research which uses natural language to prompt large pre-trained models (Liu et al., 2021).

Original: Prompting contrastive models like CLIP (Radford et al., 2021) involves providing class names at inference time (e.g., “Image of a dog”) and choosing the description with the highest similarity to the instance.
Revised: Prompting contrastive models like CLIP (Radford et al., 2021) involves providing class names at inference time (e.g., “Image of a dog”) and choosing the description with the highest similarity to the instance.

Original: However, S EM S UP differs from CLIP in major ways.
Revised: However, S EM S UP differs from CLIP in major ways.

Original: CLIP requires a large about of paired image-caption data, which necessitates that different images (potentially belonging to the same category) have different captions.
Revised: CLIP requires a large about of paired image-caption data, which necessitates that different images (potentially belonging to the same category) have different captions.

Original: However, S EM S UP requires very inexpensive class level descriptions which can be re-used for all the images belonging to that class.
Revised: However, S EM S UP requires very inexpensive class level descriptions which can be re-used for all the images belonging to that class.

Original: Further, S EM S UP can also operate on multiple types of output descriptions like NL and JSON, whereas CLIP uses only NL descriptions.
Revised: Further, S EM S UP can also operate on multiple types of output descriptions like NL and JSON, whereas CLIP uses only NL descriptions.

Original: Learning with natural language descriptions
Revised: Learning with natural language descriptions

Original: The body of work around natural language explanations (Srivastava et al., 2017, 2018; Murty et al., 2020; Hancock et al., 2018; Mu et al., 2020) aims to induce classifiers with the help of explanations that describe rationales for instances belonging to specific classes.
Revised: The body of work around natural language explanations (Srivastava et al., 2017, 2018; Murty et al., 2020; Hancock et al., 2018; Mu et al., 2020) aims to induce classifiers with the help of explanations that describe rationales for instances belonging to specific classes.

Original: Some weak supervision studies use class-specific auxiliary information (natural language descriptions or from knowledge bases) to generate labeling functions which are used to augment the training data by annotating unsupervised data (Hancock et al., 2018; Ratner et al., 2017).
Revised: Some weak supervision studies use class-specific auxiliary information (natural language descriptions or from knowledge bases) to generate labeling functions which are used to augment the training data by annotating unsupervised data (Hancock et al., 2018; Ratner et al., 2017).

Original: But unlike our paper, both these lines of work aim to improve few-shot learning on a bounded set of classes, whereas we enable even zero-shot learning on an unbounded set.
Revised: But unlike our paper, both these lines of work aim to improve few-shot learning on a bounded set of classes, whereas we enable even zero-shot learning on an unbounded set.

Original: Our work is also related to studies that learn classifiers (Andreas et al., 2018), reinforcement learning (RL) agents (Andreas et al., 2018;
Revised: Our work is also related to studies that learn classifiers (Andreas et al., 2018), reinforcement learning (RL) agents (Andreas et al., 2018;

Original: Zhong et al., 2019; Narasimhan et al., 2018; Hanjie et al., 2021), and programs (Acquaviva et al., 2021;
Revised: Zhong et al., 2019; Narasimhan et al., 2018; Hanjie et al., 2021), and programs (Acquaviva et al., 2021;

Original: Wong et al., 2021) by “reading” natural language descriptions of the task.
Revised: Wong et al., 2021) by “reading” natural language descriptions of the task.

Original: In contrast to our work, these studies typically evaluate on synthetic domains and only consider text descriptions.
Revised: In contrast to our work, these studies typically evaluate on synthetic domains and only consider text descriptions.

Original:  Methodology 3.1 Background
Revised:  Methodology 3.1 Background

Original: Our paradigm (S EM S UP ) is a modification of the standard supervised classification paradigm (S UP ), which involves using data to learn a model by minimizing a loss function .
Revised: Our paradigm (S EM S UP ) is a modification of the standard supervised classification paradigm (S UP ), which involves using data to learn a model by minimizing a loss function .

Original: The training data can be represented as D train = { ( x 1 , y 1 ) , . . .
Revised: The training data can be represented as D train = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } , where x i ∈ X and y i ∈ Y denote the input and the categorical output ( Y = { 1 , . . .
Revised: , ( x n , y n ) } , where x i ∈ X and y i ∈ Y denote the input and the categorical output ( Y = { 1 , . . .

Original: , K } ) sampled from a hidden underlying distribution ( x i , y i ) ∼ P true ( X , Y ) .
Revised: , K } ) sampled from a hidden underlying distribution ( x i , y i ) ∼ P true ( X , Y ) .

Original: We construct a model M θ which can predict the conditional probability of outputs given the input – P pred ( y | x i ) and learn it by minimizing a loss function
Revised: We construct a model M θ which can predict the conditional probability of outputs given the input – P pred ( y | x i ) and learn it by minimizing a loss function

Original: L ( P pred ( y | x i ) , y i ) like cross-entropy or max margin loss.
Revised: L ( P pred ( y | x i ) , y i ) like cross-entropy or max margin loss.

Original: Without loss of generality, let us assume M θ to be a neural network with a hidden representation of dimensionality d and the number of output classes to be K .
Revised: Without loss of generality, let us assume M θ to be a neural network with a hidden representation of dimensionality d and the number of output classes to be K .

Original: Then, we can factorize M θ to consist of (1) an input encoder f ( · ) which encodes the input as f ( x i ) ∈ R d and an (2) output matrix O ∈ R K × d .
Revised: Then, we can factorize M θ to consist of (1) an input encoder f ( · ) which encodes the input as f ( x i ) ∈ R d and an (2) output matrix O ∈ R K × d .

Original: This allows the model to represent the conditional distribution over output classes as:
Revised: This allows the model to represent the conditional distribution over output classes as:

Original: [Equation]
Revised: [Equation]

Original: Categorization
Revised: Categorization

Original: Papers
Revised: Papers

Original: Descriptions
Revised: Descriptions

Original: Multiple
Revised: Multiple

Original: [Table]
Revised: [Table]

Original: Capabilities:
Revised: Capabilities:

Original: [Table]
Revised: [Table]

Original: to unseen
Revised: to unseen

Original: [Table]
Revised: [Table]

Original: Outputs
Revised: Outputs

Original: Descriptions
Revised: Descriptions

Original: [Table]
Revised: [Table]

Original: Ours
Revised: Ours

Original: [Table]
Revised: [Table]

Original: ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Revised: ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Original: [Table]
Revised: [Table]

Original: Frome et al.
Revised: Frome et al.

Original: (2013) ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗
Revised: (2013) ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗

Original: Mittal et al
Revised: Mittal et al

Original: [Table]
Revised: [Table]

Original: ; Dauphin et al.
Revised: ; Dauphin et al.

Original: (2014) ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗
Revised: (2014) ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗

Original: [Table]
Revised: [Table]

Original: Zhang
Revised: Zhang

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓
Revised: ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓

Original: [Table]
Revised: [Table]

Original: ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗
Revised: ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗

Original: [Table]
Revised: [Table]

Original: ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✗
Revised: ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✗

Original: [Table]
Revised: [Table]

Original: ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✗
Revised: ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✗

Original: [Table]
Revised: [Table]

Original: 3.2 Semantic Supervision
Revised: 3.2 Semantic Supervision

Original: Our semantic supervision (S EM S UP ) paradigm uses the same data , loss functions , and input encoder ( f ( · ) ) used in S UP , but changes the output matrix ( O ).
Revised: Our semantic supervision (S EM S UP ) paradigm uses the same data , loss functions , and input encoder ( f ( · ) ) used in S UP , but changes the output matrix ( O ).

Original: Instead of representing the i th class (say “cat”) using a randomly initialized vector, S EM S UP encodes semantic information about the class into O
Revised: Instead of representing the i th class (say “cat”) using a randomly initialized vector, S EM S UP encodes semantic information about the class into O

Original: [ i ] (e.g., the sentence “The cat is a small carnivorous mammal”).
Revised: [ i ] (e.g., the sentence “The cat is a small carnivorous mammal”).

Original: Hereon, we will use the term “description” to refer to any kind of semantic information (text or JSON).
Revised: Hereon, we will use the term “description” to refer to any kind of semantic information (text or JSON).

Original: S EM S UP requires access to descriptions for each class C =
Revised: S EM S UP requires access to descriptions for each class C =

Original: ( C 1 , . . .
Revised: ( C 1 , . . .

Original: , C K ) , where the i th element contains a set of descriptions corresponding to the i th class: C i
Revised: , C K ) , where the i th element contains a set of descriptions corresponding to the i th class: C i

Original: = { c i 1 , . .
Revised: = { c i 1 , . .

Original: . , c iL } .
Revised: . , c iL } .

Original: Our S EM S UP model independently samples a description for each class ( c ij ) and employs an output encoder g ( · ) to encode it as g ( c ij )
Revised: Our S EM S UP model independently samples a description for each class ( c ij ) and employs an output encoder g ( · ) to encode it as g ( c ij )

Original: ∈ R
Revised: ∈ R

Original: d out , which make up rows of the output matrix O S EM S UP [ i ] =
Revised: d out , which make up rows of the output matrix O S EM S UP [ i ] =

Original: g ( c ij ) .
Revised: g ( c ij ) .

Original: Since the last dimension of
Revised: Since the last dimension of

Original: O S EM S UP ∈ R K × d out doesn’t necessarily match the dimensionality of the input encoder f
Revised: O S EM S UP ∈ R K × d out doesn’t necessarily match the dimensionality of the input encoder f

Original: ( x i ) ∈ R d , we learn an intermediate projection matrix P ∈ R d out × d .
Revised: ( x i ) ∈ R d , we learn an intermediate projection matrix P ∈ R d out × d .

Original: The final prediction is obtained as:
Revised: The final prediction is obtained as:

Original: [Equation]
Revised: [Equation]

Original: Training and testing with S EM S UP
Revised: Training and testing with S EM S UP

Original: During training, we sample a description for each class uniformly at random and construct an output matrix ( O S EM S UP ∈
Revised: During training, we sample a description for each class uniformly at random and construct an output matrix ( O S EM S UP ∈

Original: R K × d out ) such that the i th row representing the i th class is O S EM S UP
Revised: R K × d out ) such that the i th row representing the i th class is O S EM S UP

Original: [ i ] = g ( c ij ) , c ij ∼ U
Revised: [ i ] = g ( c ij ) , c ij ∼ U

Original: ( C i ) .
Revised: ( C i ) .

Original: We learn the input encoder ( f ( · ) ), the output encoder ( g ( · ) ), and the label projection matrix ( P ) together.
Revised: We learn the input encoder ( f ( · ) ), the output encoder ( g ( · ) ), and the label projection matrix ( P ) together.

Original: Descriptions are sampled uniformly at random for each class and batch.
Revised: Descriptions are sampled uniformly at random for each class and batch.

Original: During testing, we predict the class corresponding to the class description with the highest softmax probability.
Revised: During testing, we predict the class corresponding to the class description with the highest softmax probability.

Original:  Experimental Setup 4.1 Datasets
Revised:  Experimental Setup 4.1 Datasets

Original: We evaluate S EM S UP on four diverse datasets chosen to test generalization on different scenarios ( § 3).
Revised: We evaluate S EM S UP on four diverse datasets chosen to test generalization on different scenarios ( § 3).

Original: Newsgroups (20NG) (Lang, 1995) consists of 20, newsgroup documents in 20 classes.
Revised: Newsgroups (20NG) (Lang, 1995) consists of 20, newsgroup documents in 20 classes.

Original: We partition similar classes into 5 superclasses.
Revised: We partition similar classes into 5 superclasses.

Original: CIFAR-100 (Krizhevsky et al., 2009) consists of 60K images in 100 classes, each assigned to one of 20 superclasses (e.g. aquatic mammals ).
Revised: CIFAR-100 (Krizhevsky et al., 2009) consists of 60K images in 100 classes, each assigned to one of 20 superclasses (e.g. aquatic mammals ).

Original: Animals with Attributes 2 (AWA2) (Xian et al., 2018) is an animal classification dataset with 37K images and 50 classes.
Revised: Animals with Attributes 2 (AWA2) (Xian et al., 2018) is an animal classification dataset with 37K images and 50 classes.

Original: The dataset also includes 85 animal attributes (e.g. fur , swims ).
Revised: The dataset also includes 85 animal attributes (e.g. fur , swims ).

Original: Each class is annotated with binary values indicating whether the attribute is present in the class.
Revised: Each class is annotated with binary values indicating whether the attribute is present in the class.

Original: RCV1 (Lewis et al., 2004) is a multi-label news classification dataset (multiple correct classes possible) with 103 classes.
Revised: RCV1 (Lewis et al., 2004) is a multi-label news classification dataset (multiple correct classes possible) with 103 classes.

Original: We withhold 17 parent classes in the provided hierarchy to test unseen superclass generalization.
Revised: We withhold 17 parent classes in the provided hierarchy to test unseen superclass generalization.

Original: We evaluate using the label ranking average precision (L RAP ) metric
Revised: We evaluate using the label ranking average precision (L RAP ) metric

Original: (Pappas and Henderson, 2019).
Revised: (Pappas and Henderson, 2019).

Original: 4.2 Collecting output supervision
Revised: 4.2 Collecting output supervision

Original: We collect textual output descriptions for RCV1, 20NG, and CIFAR-100 by converting class names into queries: “what is a class ” and issuing them to two popular search engines, Google and DuckDuckGo.
Revised: We collect textual output descriptions for RCV1, 20NG, and CIFAR-100 by converting class names into queries: “what is a class ” and issuing them to two popular search engines, Google and DuckDuckGo.

Original: We scrape the resulting preview snippets, automatically remove scraper artifacts and partial descriptions, and manually filter any remaining off-topic descriptions.
Revised: We scrape the resulting preview snippets, automatically remove scraper artifacts and partial descriptions, and manually filter any remaining off-topic descriptions.

Original: To construct JSON descriptions of an animal in AWA2, we use types of attributes as the keys (e.g. color ) and the actual attribute lists as values (e.g., { color:[orange, black] } for a tiger).
Revised: To construct JSON descriptions of an animal in AWA2, we use types of attributes as the keys (e.g. color ) and the actual attribute lists as values (e.g., { color:[orange, black] } for a tiger).

Original: To improve the robustness of the model, we automatically augment the descriptions by randomly removing attribute values and permuting the key and value list orders, and divide them into training, validation, and test sets.
Revised: To improve the robustness of the model, we automatically augment the descriptions by randomly removing attribute values and permuting the key and value list orders, and divide them into training, validation, and test sets.

Original: We provide examples of textual and JSON descriptions and collection and in Appendix C.
Revised: We provide examples of textual and JSON descriptions and collection and in Appendix C.

Original: 4.3 Models
Revised: 4.3 Models

Original: For text datasets, we encode input features ( f ( · ) ) using the [CLS] representation from a pretrained
Revised: For text datasets, we encode input features ( f ( · ) ) using the [CLS] representation from a pretrained

Original: BERT-small model (Turc et al., 2019).
Revised: BERT-small model (Turc et al., 2019).

Original: For image datasets, we use the activations of a ResNetmodel (He et al., 2016) immediately preceding the fullyconnected final layer.
Revised: For image datasets, we use the activations of a ResNetmodel (He et al., 2016) immediately preceding the fullyconnected final layer.

Original: For each dataset, the input encoders are identical across our models and baselines.
Revised: For each dataset, the input encoders are identical across our models and baselines.

Original: While the standard supervised baseline (S UP ) uses an output matrix to obtain the logits, for our S EM S UP models, we encode output features ( g ( · ) ) using the [CLS] representations from a pretrained BERT-small model for text descriptions and a pretrained CodeBERTasmall (HuggingFace) model for AWA2 JSONs.
Revised: While the standard supervised baseline (S UP ) uses an output matrix to obtain the logits, for our S EM S UP models, we encode output features ( g ( · ) ) using the [CLS] representations from a pretrained BERT-small model for text descriptions and a pretrained CodeBERTasmall (HuggingFace) model for AWA2 JSONs.

Original: We evaluate three S EM S UP variations: S EM S UP -A LL samples from all available class descriptions during training.
Revised: We evaluate three S EM S UP variations: S EM S UP -A LL samples from all available class descriptions during training.

Original: S EM S UP -S INGLE uses a single fixed randomly selected description for each class during training.
Revised: S EM S UP -S INGLE uses a single fixed randomly selected description for each class during training.

Original: S EMS UP -N AMES is a S EM S UP variant using class names instead of descriptions (e.g. “computer graphics”).
Revised: S EMS UP -N AMES is a S EM S UP variant using class names instead of descriptions (e.g. “computer graphics”).

Original: We consider two strong baselines from prior work.
Revised: We consider two strong baselines from prior work.

Original: D EV
Revised: D EV

Original: I SE (Frome et al., 2013) and GILE (Pappas and
Revised: I SE (Frome et al., 2013) and GILE (Pappas and

Original: Henderson, 2019) which use word-embeddings of class names.
Revised: Henderson, 2019) which use word-embeddings of class names.

Original: For all models, we use the cross-entropy loss for multi-class datasets and the binary cross-entropy loss for the multi-label dataset.
Revised: For all models, we use the cross-entropy loss for multi-class datasets and the binary cross-entropy loss for the multi-label dataset.

Original: We provide additional details about training in Appendix E.
Revised: We provide additional details about training in Appendix E.

Original:  Results
Revised:  Results

Original: We now report our main results and refer the reader to Section A for ablation experiments.
Revised: We now report our main results and refer the reader to Section A for ablation experiments.

Original: 5.1 Generalizing to unseen descriptions (S1)
Revised: 5.1 Generalizing to unseen descriptions (S1)

Original: In this scenario, the input classes are identical between train and test time, but the model is given unseen descriptions at test time.
Revised: In this scenario, the input classes are identical between train and test time, but the model is given unseen descriptions at test time.

Original: The ability to generalize to novel descriptions enables users to define classification problems over subsets of train classes simply by providing their own list of class descriptions.
Revised: The ability to generalize to novel descriptions enables users to define classification problems over subsets of train classes simply by providing their own list of class descriptions.

Original: For each dataset, we keep train and test descriptions consistent for our models and baselines.
Revised: For each dataset, we keep train and test descriptions consistent for our models and baselines.

Original: Notably, for D E V I SE and GILE we use the same train and test descriptions as S EM S UP rather than class names in this scenario.
Revised: Notably, for D E V I SE and GILE we use the same train and test descriptions as S EM S UP rather than class names in this scenario.

Original: We present the results in Table 2.
Revised: We present the results in Table 2.

Original: For all models other than S EM S UP -A LL , there is a large gap between the performance when unseen descriptions ( UN ) and seen descriptions ( S ) are used, with similar trends on all datasets.
Revised: For all models other than S EM S UP -A LL , there is a large gap between the performance when unseen descriptions ( UN ) and seen descriptions ( S ) are used, with similar trends on all datasets.

Original: Interestingly, S EM S UP -S INGLE also performs poorly with a drop of 25 points compared to S EM S UPA LL on CIFAR-100.
Revised: Interestingly, S EM S UP -S INGLE also performs poorly with a drop of 25 points compared to S EM S UPA LL on CIFAR-100.

Original: This result suggests that training with a multiple descriptions is important for generalization in the output encoder.
Revised: This result suggests that training with a multiple descriptions is important for generalization in the output encoder.

Original: 5.2 Generalizing to unseen classes (S2)
Revised: 5.2 Generalizing to unseen classes (S2)

Original: In this case, classes are partitioned into training ( Y train ) and test ( Y test ) classes, where ( Y train ∩ Y test = ∅ ).
Revised: In this case, classes are partitioned into training ( Y train ) and test ( Y test ) classes, where ( Y train ∩ Y test = ∅ ).

Original: The
Revised: The

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The test classes are at the same level of hierarchy as train classes.
Revised: The test classes are at the same level of hierarchy as train classes.

Original: S EM S UP models consistently outperform D EV I SE and GILE.
Revised: S EM S UP models consistently outperform D EV I SE and GILE.

Original: Bracketed numbers are standard deviation over 3 seeds and bolded numbers are statistically significantly higher than other numbers ( p < . 05 ).
Revised: Bracketed numbers are standard deviation over 3 seeds and bolded numbers are statistically significantly higher than other numbers ( p < . 05 ).

Original: The
Revised: The

Original: GILE baseline is invalid for AWA2 because the word vectors used cannot handle JSON descriptions.
Revised: GILE baseline is invalid for AWA2 because the word vectors used cannot handle JSON descriptions.

Original: models generalize to unseen classes in Y test by using corresponding class descriptions for the test classes.
Revised: models generalize to unseen classes in Y test by using corresponding class descriptions for the test classes.

Original: We present the results in Table 3.
Revised: We present the results in Table 3.

Original: S EM S UP models significantly outperform D E V
Revised: S EM S UP models significantly outperform D E V

Original: I SE and GILE on three out of the four datasets, with performance improvements of 13 points on RCV1 and 10 points on
Revised: I SE and GILE on three out of the four datasets, with performance improvements of 13 points on RCV1 and 10 points on

Original: CIFAR.
Revised: CIFAR.

Original: S EM S UP -A LL is also able to generalize better to unseen classes in AWA2 while using JSON attributes, beating D E V I SE and GILE which use the class name, validating that our framework can use different kinds of output descriptions effectively.
Revised: S EM S UP -A LL is also able to generalize better to unseen classes in AWA2 while using JSON attributes, beating D E V I SE and GILE which use the class name, validating that our framework can use different kinds of output descriptions effectively.

Original: 5.3 Generalizing to unseen superclasses (S3)
Revised: 5.3 Generalizing to unseen superclasses (S3)

Original: In this scenario, the classes at test time are unseen superclasses of the classes from the training set.
Revised: In this scenario, the classes at test time are unseen superclasses of the classes from the training set.

Original: For example, if Y train includes foxes , lions , and frogs , Y test can consist of mammals and reptiles .
Revised: For example, if Y train includes foxes , lions , and frogs , Y test can consist of mammals and reptiles .

Original: This task is more challenging because of the change in the granularity of classification.
Revised: This task is more challenging because of the change in the granularity of classification.

Original: We present the results in Table 4.
Revised: We present the results in Table 4.

Original: Like in the previous scenarios, our S EM S UP models consistently outperform D E V I SE and GILE, with improvements ranging from 10 points on 20 NG to 16 points on CIFAR-100.
Revised: Like in the previous scenarios, our S EM S UP models consistently outperform D E V I SE and GILE, with improvements ranging from 10 points on 20 NG to 16 points on CIFAR-100.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: S EM S UP outperforms D E V
Revised: S EM S UP outperforms D E V

Original: I SE and GILE by approx
Revised: I SE and GILE by approx

Original: 5.4 Generalizing to unseen tasks
Revised: 5.4 Generalizing to unseen tasks

Original: (S4)
Revised: (S4)

Original: In this final scenario
Revised: In this final scenario

Original: [Table]
Revised: [Table]

Original: we evaluate models under the condition of task transfer by training on the source task
Revised: we evaluate models under the condition of task transfer by training on the source task

Original: RCV1 and evaluating on the target task 20NG.
Revised: RCV1 and evaluating on the target task 20NG.

Original: Table 5 shows that S EM S UP achieves over 2 × the accuracy of D E V I SE and GILE, demonstrating the strong transfer performance of our model even though RCV and 20NG are very different types of classification tasks (multi-label and multi-class respectively) and contain different sets of classes.
Revised: Table 5 shows that S EM S UP achieves over 2 × the accuracy of D E V I SE and GILE, demonstrating the strong transfer performance of our model even though RCV and 20NG are very different types of classification tasks (multi-label and multi-class respectively) and contain different sets of classes.

Original: While there is scope for improvement, S EM S UP ’s unseen task generalization which will lead to model reuse in related tasks and domains.
Revised: While there is scope for improvement, S EM S UP ’s unseen task generalization which will lead to model reuse in related tasks and domains.

Original: 5.5 Analysis: The effect of number of descriptions
Revised: 5.5 Analysis: The effect of number of descriptions

Original: We study four variants of S EM S UP which use n = 1 , 5 , 10 text descriptions at train time each.
Revised: We study four variants of S EM S UP which use n = 1 , 5 , 10 text descriptions at train time each.

Original: The models use the same set of descriptions at test time, and we evaluate them on two different scenarios, unseen descriptions and unseen classes, and report the results in Table 7.
Revised: The models use the same set of descriptions at test time, and we evaluate them on two different scenarios, unseen descriptions and unseen classes, and report the results in Table 7.

Original: Using a single description leads to poor performance for both scenarios, around 50 and 15 points lower than when 10 descriptions are used.
Revised: Using a single description leads to poor performance for both scenarios, around 50 and 15 points lower than when 10 descriptions are used.

Original: This is likely because the output encoder learns spurious discriminative features from a single description that do not generalize.
Revised: This is likely because the output encoder learns spurious discriminative features from a single description that do not generalize.

Original: However, the likelihood of learning spurious features
Revised: However, the likelihood of learning spurious features

Original: [Table]
Revised: [Table]

Original: We evaluate on two scenarios – (a) unseen descriptions and (b) unseen classes.
Revised: We evaluate on two scenarios – (a) unseen descriptions and (b) unseen classes.

Original: Increasing the number of training descriptions significantly improve generalization performance, especially to unseen descriptions.
Revised: Increasing the number of training descriptions significantly improve generalization performance, especially to unseen descriptions.

Original: decreases as we increase the number of descriptions.
Revised: decreases as we increase the number of descriptions.

Original: Indeed, for both scenarios, we observe that the performance steadily increases as we increase the number of descriptions.
Revised: Indeed, for both scenarios, we observe that the performance steadily increases as we increase the number of descriptions.

Original:  Discussion
Revised:  Discussion

Original: We proposed semantic supervision (S EM S UP ), a unified paradigm for providing semantic supervision to enable generalization over output spaces.
Revised: We proposed semantic supervision (S EM S UP ), a unified paradigm for providing semantic supervision to enable generalization over output spaces.

Original: S EM S UP represents output classes as dense feature vectors obtained from class ‘descriptions’, which allows models to generalize over unseen output spaces during training.
Revised: S EM S UP represents output classes as dense feature vectors obtained from class ‘descriptions’, which allows models to generalize over unseen output spaces during training.

Original: Our results demonstrate the ability of models trained with
Revised: Our results demonstrate the ability of models trained with

Original: S EM S UP to generalize to unseen descriptions, classes, superclasses, and tasks, while significantly outperforming prior work across four different datasets and two variants of supervision (text and JSON).
Revised: S EM S UP to generalize to unseen descriptions, classes, superclasses, and tasks, while significantly outperforming prior work across four different datasets and two variants of supervision (text and JSON).

Original: We view S EM S UP as a generalization of the standard supervised learning setup currently prevalent in the field (since classes can always be ‘described’ as abstract numbers).
Revised: We view S EM S UP as a generalization of the standard supervised learning setup currently prevalent in the field (since classes can always be ‘described’ as abstract numbers).

Original: We believe this approach will enable better re-use of trained models for new tasks, new downstream applications, and by new end users, without requiring expensive re-training or fine-tuning procedures.
Revised: We believe this approach will enable better re-use of trained models for new tasks, new downstream applications, and by new end users, without requiring expensive re-training or fine-tuning procedures.

Original: A Analysis
Revised: A Analysis

Original: We corroborate in the following sections that the reasons for strong performance of S EM S UP are three-fold – multiple diverse descriptions of classes, pre-trained output encoders, and fine-tuning the output encoder.
Revised: We corroborate in the following sections that the reasons for strong performance of S EM S UP are three-fold – multiple diverse descriptions of classes, pre-trained output encoders, and fine-tuning the output encoder.

Original: We use the
Revised: We use the

Original: RCV1 dataset for the following ablation experiments ( § A.1,A.2).
Revised: RCV1 dataset for the following ablation experiments ( § A.1,A.2).

Original: We also compare the effect of different output modalities (natural language text v.s. JSONs with attributes) on AWA2 ( § A.3).
Revised: We also compare the effect of different output modalities (natural language text v.s. JSONs with attributes) on AWA2 ( § A.3).

Original: A.1 The effect of number of descriptions
Revised: A.1 The effect of number of descriptions

Original: We study four variants of S EM S UP which use n = 1 , 5 , 10 text descriptions at train time each.
Revised: We study four variants of S EM S UP which use n = 1 , 5 , 10 text descriptions at train time each.

Original: The models use the same set of descriptions at test time, and we evaluate them on two different scenarios, unseen descriptions and unseen classes, and report the results in Table 7.
Revised: The models use the same set of descriptions at test time, and we evaluate them on two different scenarios, unseen descriptions and unseen classes, and report the results in Table 7.

Original: Using a single description leads to poor performance for both scenarios, around 50 and 15 points lower than when 10 descriptions are used.
Revised: Using a single description leads to poor performance for both scenarios, around 50 and 15 points lower than when 10 descriptions are used.

Original: This is likely because the output encoder learns spurious discriminative features from a single description that do not generalize.
Revised: This is likely because the output encoder learns spurious discriminative features from a single description that do not generalize.

Original: However, the likelihood of learning spurious features decreases as we increase the number of descriptions.
Revised: However, the likelihood of learning spurious features decreases as we increase the number of descriptions.

Original: Indeed, for both scenarios, we observe that the performance steadily increases as we increase the number of descriptions.
Revised: Indeed, for both scenarios, we observe that the performance steadily increases as we increase the number of descriptions.

Original: A.2 The effect of the output encoder
Revised: A.2 The effect of the output encoder

Original: The output encoder, which encodes the descriptions corresponding to classes, is crucial to the success of S EMS UP , because it should learn discriminative yet generalizable features.
Revised: The output encoder, which encodes the descriptions corresponding to classes, is crucial to the success of S EMS UP , because it should learn discriminative yet generalizable features.

Original: We evaluate different encoders and report the results in Table 8.
Revised: We evaluate different encoders and report the results in Table 8.

Original: We use a 4 layer pre-trained model as our reference S EM S UP
Revised: We use a 4 layer pre-trained model as our reference S EM S UP

Original: ( L = 4 ) and consider generalization to unseen classes ( § 5.2).
Revised: ( L = 4 ) and consider generalization to unseen classes ( § 5.2).

Original: Varying the encoder complexity In the first segment of the table Table 8, we notice that using a linear model S EM S UP (Linear) , which computes a bagof-vectors representation of the class descriptions, performs poorly when compared to non-linear models like
Revised: Varying the encoder complexity In the first segment of the table Table 8, we notice that using a linear model S EM S UP (Linear) , which computes a bagof-vectors representation of the class descriptions, performs poorly when compared to non-linear models like

Original: S EM S UP ( L = 2 ) .
Revised: S EM S UP ( L = 2 ) .

Original: We experiment with three differently sized (non-linear) BERT (Devlin et al., 2019) models released by (Turc et al., 2019) which have 2 , 4 , and
Revised: We experiment with three differently sized (non-linear) BERT (Devlin et al., 2019) models released by (Turc et al., 2019) which have 2 , 4 , and

Original: [Table]
Revised: [Table]

Original: layers, respectively, and notice no significant difference between their performance.
Revised: layers, respectively, and notice no significant difference between their performance.

Original: While increasing the depth of the model typically leads to gains in performance, we speculate that the small number of class descriptions used ( 10 per class for 103 , 10 × 103 = 1030 ) when compared to the number of instances ( 480 , 000 for RCV1) means that a very deep model is unnecessary.
Revised: While increasing the depth of the model typically leads to gains in performance, we speculate that the small number of class descriptions used ( 10 per class for 103 , 10 × 103 = 1030 ) when compared to the number of instances ( 480 , 000 for RCV1) means that a very deep model is unnecessary.

Original: Thus, we conclude that while it is extremely crucial to use a non-linear output encoder, its depth is less important.
Revised: Thus, we conclude that while it is extremely crucial to use a non-linear output encoder, its depth is less important.

Original: Pre-trained v.s. randomly initialized In the second segment, we notice that a 4 layer output encoder which is initialized using random weights – S EM S UP (R. I.) performs 10 points lower than our reference – S EMS UP ( L = 4 ) , a pre-trained model of a comparable size.
Revised: Pre-trained v.s. randomly initialized In the second segment, we notice that a 4 layer output encoder which is initialized using random weights – S EM S UP (R. I.) performs 10 points lower than our reference – S EMS UP ( L = 4 ) , a pre-trained model of a comparable size.

Original: This underscores the importance of pre-training, which gives models a better semantic understanding of sentences than their randomly initialized counterparts.
Revised: This underscores the importance of pre-training, which gives models a better semantic understanding of sentences than their randomly initialized counterparts.

Original: Freezing the output encoder In the third segment, we consider S EM S UP (Frozen) , which initializes the output encoder using a pre-trained model, but freezes all the weights throughout training (no gradients are passed).
Revised: Freezing the output encoder In the third segment, we consider S EM S UP (Frozen) , which initializes the output encoder using a pre-trained model, but freezes all the weights throughout training (no gradients are passed).

Original: We notice that even though this variant has the benefits of a pre-trained model, it is worse than our reference by over 10 points.
Revised: We notice that even though this variant has the benefits of a pre-trained model, it is worse than our reference by over 10 points.

Original: This highlights the importance of finetuning, which adapts the weights of the pre-trained models to the task at hand.
Revised: This highlights the importance of finetuning, which adapts the weights of the pre-trained models to the task at hand.

Original: Interestingly, S EM S UP (Frozen) performs similar to the randomly initialized model –
Revised: Interestingly, S EM S UP (Frozen) performs similar to the randomly initialized model –

Original: S EM S UP (R. I.) .
Revised: S EM S UP (R. I.) .

Original: Both these models are missing a key ingredient required for good generalization, with the former missing fine-tuning and the latter missing pretraining, and our reference (S EM S UP ( L = 4 ) ) which includes both these aspects performs the best.
Revised: Both these models are missing a key ingredient required for good generalization, with the former missing fine-tuning and the latter missing pretraining, and our reference (S EM S UP ( L = 4 ) ) which includes both these aspects performs the best.

Original: These experiments validate that the following three aspects are useful for strong generalization in S EM S UP models – (a) multiple diverse descriptions of classes, (b) pre-trained output encoders, and (c) fine-tuning the output encoder.
Revised: These experiments validate that the following three aspects are useful for strong generalization in S EM S UP models – (a) multiple diverse descriptions of classes, (b) pre-trained output encoders, and (c) fine-tuning the output encoder.

Original: [Table]
Revised: [Table]

Original: A.3 Comparing output supervision: NL vs. JSON
Revised: A.3 Comparing output supervision: NL vs. JSON

Original: To compare the effect of different types of output supervision for S EM S UP , we evaluate its performance on
Revised: To compare the effect of different types of output supervision for S EM S UP , we evaluate its performance on

Original: AWA2 test classes using natural language and JSON supervision (Table 9).
Revised: AWA2 test classes using natural language and JSON supervision (Table 9).

Original: The best performing JSON model (S EM S UP -ALL) outperforms its corresponding model trained using natural language model by 8 points, demonstrating the utility of using structured data formats to provide class descriptions on this particular task.
Revised: The best performing JSON model (S EM S UP -ALL) outperforms its corresponding model trained using natural language model by 8 points, demonstrating the utility of using structured data formats to provide class descriptions on this particular task.

Original: For AWA2, this performance gap between natural language and JSON could be due to two reasons: (1) the
Revised: For AWA2, this performance gap between natural language and JSON could be due to two reasons: (1) the

Original: JSON class descriptions directly list all of the most salient features of each class making it more information dense than natural language descriptions, and (2) its structure is more consistent between training and test descriptions (since the keys are identical for example).
Revised: JSON class descriptions directly list all of the most salient features of each class making it more information dense than natural language descriptions, and (2) its structure is more consistent between training and test descriptions (since the keys are identical for example).

Original: These results demonstrate the flexibility of the S EMS UP framework in allowing users to pick a type of semantic supervision that suits their task needs the best and hint at the possibility of developing more sophisticated forms of semantic supervision, which may even be a combination of both freeform text and more structured representations, to maximize generalization performance.
Revised: These results demonstrate the flexibility of the S EMS UP framework in allowing users to pick a type of semantic supervision that suits their task needs the best and hint at the possibility of developing more sophisticated forms of semantic supervision, which may even be a combination of both freeform text and more structured representations, to maximize generalization performance.

Original: B Experimental Setup B.1 Datasets
Revised: B Experimental Setup B.1 Datasets

Original: We evaluate S EM S UP on four diverse datasets chosen to test generalization on different scenarios ( § 3).
Revised: We evaluate S EM S UP on four diverse datasets chosen to test generalization on different scenarios ( § 3).

Original: Newsgroups (20NG) (Lang, 1995) consists of 20, newsgroup documents in 20 classes.
Revised: Newsgroups (20NG) (Lang, 1995) consists of 20, newsgroup documents in 20 classes.

Original: We partition similar classes into 5 superclasses 1 .
Revised: We partition similar classes into 5 superclasses 1 .

Original: We further partition the classes into 12 train, 4 validation and 4 test classes, and evaluate using accuracy.
Revised: We further partition the classes into 12 train, 4 validation and 4 test classes, and evaluate using accuracy.

Original: CIFAR-100 (Krizhevsky et al., 2009) consists of 60K images in 100 classes, each assigned to one of 20 superclasses (e.g. aquatic mammals ).
Revised: CIFAR-100 (Krizhevsky et al., 2009) consists of 60K images in 100 classes, each assigned to one of 20 superclasses (e.g. aquatic mammals ).

Original: We partition the dataset into 80 train, 10 validation, and 10 test classes, and evaluate using accuracy.
Revised: We partition the dataset into 80 train, 10 validation, and 10 test classes, and evaluate using accuracy.

Original: Animals with Attributes 2 (AWA2) (Xian et al., 2018) is an animal classification dataset with 37K images andclasses.
Revised: Animals with Attributes 2 (AWA2) (Xian et al., 2018) is an animal classification dataset with 37K images andclasses.

Original: The dataset also includes 85 animal attributes (e.g. fur , swims ).
Revised: The dataset also includes 85 animal attributes (e.g. fur , swims ).

Original: Each class is annotated with binary values indicating whether the attribute is present in the class.
Revised: Each class is annotated with binary values indicating whether the attribute is present in the class.

Original: We follow the split of classes into 27 train,validation, and 10 test classes provided in the dataset.
Revised: We follow the split of classes into 27 train,validation, and 10 test classes provided in the dataset.

Original: RCV1 (Lewis et al., 2004) is a multi-label news classification dataset (multiple correct classes per instance) with over 800 , 000 articles.
Revised: RCV1 (Lewis et al., 2004) is a multi-label news classification dataset (multiple correct classes per instance) with over 800 , 000 articles.

Original: We hold out 25 of theniche classes to test unseen class generalization.
Revised: We hold out 25 of theniche classes to test unseen class generalization.

Original: We withhold 17 parent classes in the provided hierarchy to test unseen superclass generalization.
Revised: We withhold 17 parent classes in the provided hierarchy to test unseen superclass generalization.

Original: We evaluate using the label ranking average precision (L RAP ) metric
Revised: We evaluate using the label ranking average precision (L RAP ) metric

Original: (Pappas and Henderson, 2019).
Revised: (Pappas and Henderson, 2019).

Original: Further details about the experimental setup are provided in
Revised: Further details about the experimental setup are provided in

Original: B.2 Collecting output supervision
Revised: B.2 Collecting output supervision

Original: Text output supervision We collect textual output descriptions for RCV1, 20NG, and CIFAR-100 by converting class names into queries: “what is a class ” and issuing them to two popular search engines 2 .
Revised: Text output supervision We collect textual output descriptions for RCV1, 20NG, and CIFAR-100 by converting class names into queries: “what is a class ” and issuing them to two popular search engines 2 .

Original: We scrape the resulting preview snippets, automatically remove scraper artifacts and partial descriptions, and manually filter any remaining off-topic descriptions.
Revised: We scrape the resulting preview snippets, automatically remove scraper artifacts and partial descriptions, and manually filter any remaining off-topic descriptions.

Original: JSON output supervision To construct JSON descriptions of an animal in AWA2, we use types of attributes as the keys (e.g. color ) and the actual attribute lists as values (e.g., { color:[orange, black] } for a tiger).
Revised: JSON output supervision To construct JSON descriptions of an animal in AWA2, we use types of attributes as the keys (e.g. color ) and the actual attribute lists as values (e.g., { color:[orange, black] } for a tiger).

Original: To improve the robustness of the model, we automatically augment the descriptions by randomly removing attribute values and permuting the key and value list orders, and divide them into training, validation, and test sets.
Revised: To improve the robustness of the model, we automatically augment the descriptions by randomly removing attribute values and permuting the key and value list orders, and divide them into training, validation, and test sets.

Original: We provide examples of textual and
Revised: We provide examples of textual and

Original: JSON descriptions and details regarding their collection and filtering in Appendix C.
Revised: JSON descriptions and details regarding their collection and filtering in Appendix C.

Original: B.3 Models
Revised: B.3 Models

Original: For text datasets, we encode input features ( f ( · ) ) using the [CLS] representation from a pretrained
Revised: For text datasets, we encode input features ( f ( · ) ) using the [CLS] representation from a pretrained

Original: BERT-small model (Turc et al., 2019).
Revised: BERT-small model (Turc et al., 2019).

Original: For image datasets, we use the activations of a ResNetmodel (He et al., 2016) immediately preceding the fullyconnected final layer.
Revised: For image datasets, we use the activations of a ResNetmodel (He et al., 2016) immediately preceding the fullyconnected final layer.

Original: For each dataset, the input encoders are identical across our models and baselines, to ensure fair comparison.
Revised: For each dataset, the input encoders are identical across our models and baselines, to ensure fair comparison.

Original: While the standard supervised baseline (S UP ) uses an output matrix to output the logits, the other models use output encoders, which we describe below.
Revised: While the standard supervised baseline (S UP ) uses an output matrix to output the logits, the other models use output encoders, which we describe below.

Original: For our S EM S UP models, we encode output features ( g ( · ) ) using the [CLS] representations from a pretrained BERT-small model (Turc et al., 2019) for text descriptions and a pretrained CodeBERTa-smallmodel for AWA2 JSONs.
Revised: For our S EM S UP models, we encode output features ( g ( · ) ) using the [CLS] representations from a pretrained BERT-small model (Turc et al., 2019) for text descriptions and a pretrained CodeBERTa-smallmodel for AWA2 JSONs.

Original: We propose three S EM S UP models:
Revised: We propose three S EM S UP models:

Original: S EM S UP -A LL samples from all available class descriptions during training
Revised: S EM S UP -A LL samples from all available class descriptions during training

Original: S EM S UP -S INGLE uses a single fixed randomly selected description for each class during training.
Revised: S EM S UP -S INGLE uses a single fixed randomly selected description for each class during training.

Original: S EM S UP -N
Revised: S EM S UP -N

Original: AMES is a S EM S UP variant using class names instead of descriptions (e.g. “computer graphics”).
Revised: AMES is a S EM S UP variant using class names instead of descriptions (e.g. “computer graphics”).

Original: We consider two strong baselines from prior work.
Revised: We consider two strong baselines from prior work.

Original: D E V
Revised: D E V

Original: I SE (Frome et al., 2013) uses the mean of the word vectors of the class names .
Revised: I SE (Frome et al., 2013) uses the mean of the word vectors of the class names .

Original: GILE (Pappas and Henderson, 2019) uses the mean of the word vectors of class descriptions , with modeling similar to D E V I SE other than the tanh activation applied to the output embeddings.
Revised: GILE (Pappas and Henderson, 2019) uses the mean of the word vectors of class descriptions , with modeling similar to D E V I SE other than the tanh activation applied to the output embeddings.

Original: For
Revised: For

Original: both the models,
Revised: both the models,

Original: we
Revised: we

Original: use GloVe (
Revised: use GloVe (

Original: Pennington et al.,
Revised: Pennington et al.,

Original: ) vectors which are fixed throughout training (Frome et al., 2013) to represent the
Revised: ) vectors which are fixed throughout training (Frome et al., 2013) to represent the

Original: output.
Revised: output.

Original: Training We train all models end-to-end and backpropagate gradients both through the input and output encoder.
Revised: Training We train all models end-to-end and backpropagate gradients both through the input and output encoder.

Original: For all models, we use the cross-entropy loss for multi-class datasets and the binary cross-entropy loss for the multi-label dataset.
Revised: For all models, we use the cross-entropy loss for multi-class datasets and the binary cross-entropy loss for the multi-label dataset.

Original: We provide additional details about training in Appendix E.
Revised: We provide additional details about training in Appendix E.

Original: C Output Supervision
Revised: C Output Supervision

Original: We show example class descriptions for the datasets in table 10.
Revised: We show example class descriptions for the datasets in table 10.

Original: The results were scraped from www.google.
Revised: The results were scraped from www.google.

Original: com and www.duckduckgo.com using a third-party scraping tool 5 .
Revised: com and www.duckduckgo.com using a third-party scraping tool 5 .

Original: Data collection was conducted between
Revised: Data collection was conducted between

Original: September 2021 and January 2022.
Revised: September 2021 and January 2022.

Original: To reduce variability, personalized results were turned off and regions were fixed to United States.
Revised: To reduce variability, personalized results were turned off and regions were fixed to United States.

Original: Safe search was enabled for www.google.com and set to moderate on www.duckduckgo.com .
Revised: Safe search was enabled for www.google.com and set to moderate on www.duckduckgo.com .

Original: The number of search returns for www.google.com was varied betweenand 50.
Revised: The number of search returns for www.google.com was varied betweenand 50.

Original: While we obtained more descriptions using a higher number of search returns, we found that the quality and relevance was often lower.
Revised: While we obtained more descriptions using a higher number of search returns, we found that the quality and relevance was often lower.

Original: [h]
Revised: [h]

Original: An example scraping target is presented in Figure 2.
Revised: An example scraping target is presented in Figure 2.

Original: We automatically filter the scraped preview blocks by removing any incomplete sentences.
Revised: We automatically filter the scraped preview blocks by removing any incomplete sentences.

Original: For multi-sentence descriptions, we only take the first sentence.
Revised: For multi-sentence descriptions, we only take the first sentence.

Original: Sentences that are less than 5 words are discarded.
Revised: Sentences that are less than 5 words are discarded.

Original: After automatic filtering, we manually inspect the descirptions and remove irrelevant descriptions.
Revised: After automatic filtering, we manually inspect the descirptions and remove irrelevant descriptions.

Original: The mean number and lengths of the collected descriptions is presented in Table 11.
Revised: The mean number and lengths of the collected descriptions is presented in Table 11.

Original: On all datasets, we divide the class descriptions into a 60-20-20 train-val-test split.
Revised: On all datasets, we divide the class descriptions into a 60-20-20 train-val-test split.

Original: D Dataset Details D.1 RCV 
Revised: D Dataset Details D.1 RCV

Original: RCV1 contains 800 , 000 articles and we create a 60:20:20 split for train, validation, and test respectively.
Revised: RCV1 contains 800 , 000 articles and we create a 60:20:20 split for train, validation, and test respectively.

Original: It contains 103 classes.
Revised: It contains 103 classes.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and mean lengths per description.
Revised: and mean lengths per description.

Original: Note that on AWA2 we automatically augment the descriptions, so there is no variance in the number of descriptions between classes.
Revised: Note that on AWA2 we automatically augment the descriptions, so there is no variance in the number of descriptions between classes.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The test superclasses are the remaining 10 superclasses not listed in the val superclasses above
Revised: The test superclasses are the remaining 10 superclasses not listed in the val superclasses above

Original: D.2 20NG
Revised: D.2 20NG

Original: We use the 18828 variant for each newsgroup.
Revised: We use the 18828 variant for each newsgroup.

Original: Since the original dataset does not define train-test splits, we construct our own 80-20 train test split.
Revised: Since the original dataset does not define train-test splits, we construct our own 80-20 train test split.

Original: We further divide the training set into training and validation sets with a porportion of 80-20.
Revised: We further divide the training set into training and validation sets with a porportion of 80-20.

Original: We present details of the 20 NG dataset splits in table 12.
Revised: We present details of the 20 NG dataset splits in table 12.

Original: When evaluating generalization to superclasses on 20 NG we remove the misc.forsale class since it is its own superclass.
Revised: When evaluating generalization to superclasses on 20 NG we remove the misc.forsale class since it is its own superclass.

Original: D.3 CIFAR- 
Revised: D.3 CIFAR-

Original: We use the provided train-test split, but divide the train set 80-20 into training and validation examples.
Revised: We use the provided train-test split, but divide the train set 80-20 into training and validation examples.

Original: D.4 AWA 
Revised: D.4 AWA

Original: We use the predefined train-val-test splits of classes provided in the paper (Xian et al., 2018).
Revised: We use the predefined train-val-test splits of classes provided in the paper (Xian et al., 2018).

Original: We use only the second of the three train-val splits provided.
Revised: We use only the second of the three train-val splits provided.

Original: We split the instances into train and test examples 80and further divide the training set 80-20 into training
Revised: We split the instances into train and test examples 80and further divide the training set 80-20 into training

Original: To construct the JSON, we first assign each attribute to a parent attribute.
Revised: To construct the JSON, we first assign each attribute to a parent attribute.

Original: The final class-level JSON consists of the parent attributes as keys, and the values are attributes that are present in the class.
Revised: The final class-level JSON consists of the parent attributes as keys, and the values are attributes that are present in the class.

Original: We augment this dataset by first adding 50 samples per class of corrupted examples, by randomly deleting attributes independently with probability 0.15, and then further multiplying this by 25 permutations.
Revised: We augment this dataset by first adding 50 samples per class of corrupted examples, by randomly deleting attributes independently with probability 0.15, and then further multiplying this by 25 permutations.

Original: E Model Training and Evaluation
Revised: E Model Training and Evaluation

Original: All models are end-to-end differentiable and we train them using the AdamW optimizer (Loshchilov and Hutter, 2017).
Revised: All models are end-to-end differentiable and we train them using the AdamW optimizer (Loshchilov and Hutter, 2017).

Original: We use a constant learning rate of 1 × 10 −for all the vision experiments on AWA2 and CIFARand a constant learning rate of 2 × 10 − 5 for all experiments on 20 NG.
Revised: We use a constant learning rate of 1 × 10 −for all the vision experiments on AWA2 and CIFARand a constant learning rate of 2 × 10 − 5 for all experiments on 20 NG.

Original: For efficiency, the class descriptions are encoded into the output matrix O S EM S UP at each minibatch, so that all instances in the batch share the same output matrix.
Revised: For efficiency, the class descriptions are encoded into the output matrix O S EM S UP at each minibatch, so that all instances in the batch share the same output matrix.

Original: We use the validation set for early stopping, and test checkpoints saved at the point of highest validation accuracy.
Revised: We use the validation set for early stopping, and test checkpoints saved at the point of highest validation accuracy.

Original: All implementation was done in
Revised: All implementation was done in

Original: PyTorch and PyTorch Lightning and experiments were run on either a single NVIDIA RTX2080 or a single
Revised: PyTorch and PyTorch Lightning and experiments were run on either a single NVIDIA RTX2080 or a single

Original: NVIDIA RTX3090.
Revised: NVIDIA RTX3090.