Original: C ORRECTION N ETWORKS : M ETA -L EARNING FOR Z ERO -S HOT L EARNING
Revised: C ORRECTION N ETWORKS : M ETA -L EARNING FOR Z ERO -S HOT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a model that learns to perform zero-shot classiﬁcation using a metalearner that is trained to anticipate and correct errors based on the learner’s training data.
Revised: We propose a model that learns to perform zero-shot classiﬁcation using a metalearner that is trained to produce a correction to the output of a previously trained learner.

Original: The model consists of two modules: a task module that supplies an initial prediction, and a correction module that updates the  prediction.
Revised: The model consists of two modules: a task module that supplies an initial prediction, and a correction module that updates the initial prediction.

Original: The task module is the learner and the correction module is the meta-learner.
Revised: The task module is the learner and the correction module is the meta-learner.

Revised: The correction module is trained in an episodic approach whereby many different task modules are trained on various subsets of the total training data, with the rest being used as unseen data for the correction module.

Original: The correction module takes as input  the task module’s training data, input and initial prediction, and updates the initial prediction to be closer to the target value.
Revised: The correction module takes as input a representation of the task module’s training data so that the predicted correction is a function of the task module’s training data. The correction module is trained to update the task module’s prediction to be closer to the target value.

Original: We demonstrate that this approach leads to state-of-the-art performance on ﬁne-grained zero-shot classiﬁcation on natural language class descriptions on the CUB and NAB datasets.
Revised: This approach leads to state-of-the-art performance for zero-shot classiﬁcation on natural language class descriptions on the CUB and NAB datasets.

Original: Correction Networks are independent of the architectures of the modules and can be used for problems for which an updated prediction can be obtained by applying a correction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to solve a task without receiving training examples – zero-shot learning – is desirable.
Revised: The ability to solve a task without receiving training examples – zero-shot learning – is desirable.

Original: This ability is particularly valuable when training data are limited, difﬁcult to obtain, and/or expensive.
Revised: This ability is particularly valuable when training data is limited, difﬁcult to obtain, and/or expensive.

Original: We as humans can learn new tasks from descriptions of the tasks, as we learn from reading encyclopedia entries, manuals, handbooks, textbooks, etc.
Revised: We as humans can learn new tasks from descriptions of the tasks, as we learn from reading encyclopedia entries, manuals, handbooks, textbooks, etc.

Original: Our artiﬁcial agents should be able to do the same.
Revised: Our artiﬁcial agents should be able to do the same.

Original: However, generalizing to novel tasks unseen during training is challenging.
Revised: However, generalizing to novel tasks unseen during training is challenging.

Original: The agent must integrate its previous training experience to solve the new task, without individual  samples from it, while avoiding over-ﬁtting on its previous training experience.
Revised: The agent must integrate its previous training experience to solve the new task, without individual training samples from the new task, while avoiding over-ﬁtting on its previous training experience.

Original: We propose a model that learns a correction on predictions in the zero-shot setting, based on the training data set used to generate the initial prediction.
Revised: We propose a model that learns a correction on predictions in the zero-shot setting, based on the training data set used to generate the initial prediction.

Original: Hence, our model is called Correction Networks.
Revised: Hence, our model is called Correction Networks.

Original: The meta-learner corrects for the zero-shot predictions of the learner.
Revised: The meta-learner corrects for the zero-shot predictions of the learner.

Original: The intuition for our model is that a zero-shot query sample that is different from samples in the training data will require a different correction than a zero-shot query sample that is similar to samples in the training data.
Revised: The intuition for our model is that a zero-shot query sample that is different from samples in the training data will require a different correction than a zero-shot query sample that is similar to samples in the training data.

Original: Consider a simple classiﬁcation model trained to identify different color classes.
Revised: Consider a simple classiﬁcation model trained to identify different color classes.

Original: The training dataset for this simple classiﬁcation problem consists of the classes of { yellow, green, indigo } .
Revised: The training dataset for this simple classiﬁcation problem consists of the classes of { yellow, green, indigo } .

Original: Let there be two zero-shot classes: pink and blue.
Revised: Let there be two zero-shot classes: pink and blue.

Original: The deviation of predicted values from true values of pink are different, possibly larger, than deviations of predicted values from true values of blue.
Revised: The deviation of predicted values from true values of pink are different, possibly larger, than deviations of predicted values from true values of blue.

Original: These deviations would again be different if the training dataset consisted of the classes of { white, red, purple } .
Revised: These deviations would again be different if the training dataset consisted of the classes of { white, red, purple } .

Original: Correction Networks update the predictions based on the training data.
Revised: Correction Networks update the predictions based on the training data.

Original: This updated prediction is  closer to the target value than the original prediction.
Revised: This updated prediction is trained to be closer to the target value than the original prediction.

Original: Correction Networks consist of two modules: a task module that supplies an initial prediction, and a correction module that provides a correction to the initial prediction.
Revised: Correction Networks consist of two modules: a task module that supplies an initial prediction, and a correction module that provides a correction to the initial prediction.

Original: The task module is the learner and the correction module is the meta-learner.
Revised: The task module is the learner and the correction module is the meta-learner.

Original: The ﬁnal prediction is the task module’s initial prediction combined with the correction module’s correction.
Revised: The ﬁnal prediction is the task module’s initial prediction combined with the correction module’s correction.

Original: This method is illustrated in Figure 1.
Revised: This method is illustrated in Figure 1.

Original: The prediction of the meta-learner is used to modify the output of the learner, unlike other metalearning algorithms that learn to update, initialize, or optimize the learner’s weights (Finn et al., 2017)
Revised: The prediction of the meta-learner is used to modify the output of the learner, unlike other metalearning algorithms that learn to update, initialize, or optimize the learner’s weights (Finn et al., 2017)

Original: (Ravi & Larochelle, 2017).
Revised: (Ravi & Larochelle, 2017).

Original: Our approach is for zero-shot learning while previous metalearning approaches focus on few shot learn and require a few samples.
Revised: Our approach is for zero-shot learning while previous metalearning approaches focus on few shot learning and require a few training samples for the new task.

Original: Another novelty is that the correction module, the meta-learner, takes as input the dataset used to train the task module.
Revised: Another novelty is that the correction module (the meta-learner) takes as input the dataset used to train the task module.

Original: Correction Networks can be used for problems for which an updated prediction can be obtained by applying a correction.
Revised: Correction Networks can be used for problems for which an updated prediction can be obtained by applying a correction.

Original: Correction Networks are independent of the representation of the task module.
Revised: Correction Networks are independent of the representation of the task module.

Original: Existing models that provide predictions can be treated as task modules.
Revised: Existing models that provide predictions can be treated as task modules.

Original: The correction module only need the inputs to the task module, the predictions of the task module, and the training data of the task module.
Revised: The correction module only need the inputs to the task module, the predictions of the task module, and the training data of the task module.

Original: While our focus is on neural network architectures, both the task module and correction modules can be, but needs not be, neural networks.
Revised: While our focus is on neural network architectures, both the task module and correction modules can be, but needs not be, neural networks.

Original: Training proceeds by randomly partitioning the training data D to create disjoint sets D S and D U .
Revised: Training proceeds by randomly partitioning the training data D to create disjoint sets D S and D U .

Original: In the case of zero-shot classiﬁcation, the partition is done by class so that the classes in D S and D U are disjoint.
Revised: In the case of zero-shot classiﬁcation, the partition is done by class so that the classes in D S and D U are disjoint.

Original: The task module is trained on D S .
Revised: The task module is trained on D S .

Original: After the task module is trained, D U is used as the ’zero-shot’ queries for the task module.
Revised: After the task module is trained, D U is used as the ‘zero-shot’ queries for the task module.

Original: The task module supplies initial predictions on D U .
Revised: The task module supplies initial predictions on D U .

Original: The correction module is trained to predict the corrections that need to be applied to the task module’s initial predictions to obtain the true values associated with D U .
Revised: The correction module is trained to predict the corrections that need to be applied to the task module’s initial predictions to obtain the true values associated with D U .

Original: One novelty is that the correction module, the meta-learner, also takes as input the dataset used to train the task module.
Revised: One novelty is that the correction module, the meta-learner, also takes as input the dataset used to train the task module.

Original: The prediction of the meta-learner is used to modify the output of the learner, unlike other meta-learning algorithms that learn to update or initialize the learner’s weights (Finn et al., 2017)
Revised: The prediction of the meta-learner is used to modify the output of the learner, unlike other meta-learning algorithms that learn to update or initialize the learner’s weights (Finn et al., 2017)

Original: (Ravi & Larochelle, 2017).
Revised: (Ravi & Larochelle, 2017).

Original: We demonstrate Correction Networks on the problem of zero-shot image classiﬁcation based on natural language class descriptions.
Revised: We demonstrate Correction Networks on the problem of zero-shot image classiﬁcation based on natural language class descriptions.

Original: Zero-shot imageimage classiﬁcation involves classifying images into classes that were unseen in the training data, given a natural language description of the new class.
Revised: Zero-shot image classiﬁcation involves classifying images into classes that were unseen in the training data, given a natural language description of the new class.

Original: The main contribution of this paper is a zero-shot learning model that corrects zero-shot predictions based on training data.
Revised: The main contribution of this paper is a zero-shot learning model that corrects zero-shot predictions based on training data.

Original: We demonstrate our model on zero-shot image classiﬁcation with class text descriptions.
Revised: We demonstrate our model on zero-shot image classiﬁcation with class text descriptions.

Original: Our evaluation shows that Correction Networks compares favorably to state-of-the-art zero-shot learning methods.
Revised: Our evaluation shows that Correction Networks compares favorably to state-of-the-art zero-shot learning methods.

Original: The rest of this paper is as follows: Section 2 covers related work, Section 3 describes our Correction Network model, Section 4 summarizes the experiments and results,Section 4.5 outlines future directions, and Section 5 ﬁnishes with conclusions.
Revised: The rest of this paper is as follows: Section 2 covers related work, Section 3 describes our Correction Network model, Section 4 summarizes the experiments and results, and Section 5 ﬁnishes with conclusions.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: For zero-shot learning for images, the majority of state-of-the-art methods are embedding-based methods.
Revised: For zero-shot learning for images, the majority of state-of-the-art methods are embedding-based methods.

Original: This often involves learning a mapping from the visual space to the semantic space of class labels or vice versa.
Revised: This often involves learning a mapping from the visual space to the semantic space of class labels or vice versa.

Original: Alternatively, the embedding function between the visual and semantic spaces is jointly learned through a latent space.
Revised: Alternatively, the embedding function between the visual and semantic spaces is jointly learned through a latent space.

Original: To represent the new, novel classes, hand crafted attributes are often used.
Revised: To represent the new, novel classes, hand crafted attributes are often used.

Original: The idea is that given a deﬁned attribute ontology, each class name can be converted to an attribute vector against which image features are compared (Lampert et al., 2009)
Revised: The idea is that given a deﬁned attribute ontology, each class name can be converted to an attribute vector against which image features are compared (Lampert et al., 2009)

Original: (Palatucci et al., 2009)
Revised: (Palatucci et al., 2009)

Original: (Farhadi et al., 2009)
Revised: (Farhadi et al., 2009)

Original: Akata et al.
Revised: (Akata et al., 2013).

Original: Attribute based embedding spaces more recently gave way to text based embeddings.
Revised: Attribute based embedding spaces more recently gave way to text based embeddings.

Original: These text-based embeddings learn from a large external text corpus e.g. Wikipedia or WordNet on a natural language task to learn word embeddings into which class names are then projected (Oquab et al., 2014)
Revised: These text-based embeddings learn from a large external text corpus e.g. Wikipedia or WordNet on a natural language task to learn word embeddings into which class names are then projected (Oquab et al., 2014)

Original: (Socher et al., 2013) (Frome et al., 2013)
Revised: (Socher et al., 2013) (Frome et al., 2013)

Original: (Fu et al., 2014).
Revised: (Fu et al., 2014).

Original: Semantic representations for zero-shot classes have also been created from text documents of the classes eg.
Revised: Semantic representations for zero-shot classes have also been created from text documents of the classes eg.

Original: Wikipedia articles corresponding to the class.
Revised: Wikipedia articles corresponding to the class.

Original: This includes learning attribute representations from a web source by ranking the visualness of attribute candidates (Berg et al., 2010).
Revised: This includes learning attribute representations from a web source by ranking the visualness of attribute candidates (Berg et al., 2010).

Original: Internet sources have also been mined for semantic relatedness of attribute-class association (Rohrbach et al., 2013).
Revised: Internet sources have also been mined for semantic relatedness of attribute-class association (Rohrbach et al., 2013).

Original: Directly measuring thethe compatibility between documents describing the class and visual features have also been modeled using deep learning (Zhu et al., 2018)
Revised: Directly measuring the compatibility between documents describing the class and visual features have also been modeled using deep learning (Zhu et al., 2018)

Original: (Ba et al., 2015) and non-deep learning models (Elhoseiny et al., 2013) (Romera-Paredes & Torr, 2015)
Revised: (Ba et al., 2015) and non-deep learning models (Elhoseiny et al., 2013) (Romera-Paredes & Torr, 2015)

Original: (Fu et al., 2015) (Qiao et al., 2016)
Revised: (Fu et al., 2015) (Qiao et al., 2016)

Original: (Elhoseiny et al., 2017).
Revised: (Elhoseiny et al., 2017a).

Original: A more recent strategy frames zero-shot recognition as a conventional supervised classiﬁcation problem by hallucinating samples for unseen classes.
Revised: A more recent strategy frames zero-shot recognition as a conventional supervised classiﬁcation problem by hallucinating samples for unseen classes.

Original: Classiﬁcation performance thus depends on the quality of the hallucinated samples.
Revised: Classiﬁcation performance thus depends on the quality of the hallucinated samples.

Original: Guo et al.
Revised: Guo et al.

Original: (2017a) assumes the visual features of each class is distributed as a Gaussian and estimates the distribution of unseen classes by linearly combining the distributions of seen classes.
Revised: (2017a) assumes the visual features of each class is distributed as a Gaussian and estimates the distribution of unseen classes by linearly combining the distributions of seen classes.

Original: Long et al.
Revised: Long et al.

Original: (2017) synthesizes visual data from attributes of classes in a one-to-one mapping.
Revised: (2017) synthesizes visual data from attributes of classes in a one-to-one mapping.

Original: Guo et al.
Revised: Guo et al.

Original: (2017b) assigns psuedo labels to samples from seen classes.
Revised: (2017b) assigns pseudo labels to samples from seen classes.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2018) uses generative adversarial training with the generator learning to generate image features from text and  query samples are classiﬁed based on k-nearest neighbors to labeled or generated samples.
Revised: (2018) uses generative adversarial training with the generator learning to generate image features from text and then query samples are classiﬁed based on k-nearest neighbors to labeled or generated samples.

Original: In contrast, our approach avoids the data generation phase and instead, directly predicts the center of classes in image feature space.
Revised: In contrast, our approach avoids the data generation phase and instead, directly predicts the center of classes in image feature space.

Original: Classiﬁcation regions are thus deﬁned based on the class of the nearest class center.
Revised: Classiﬁcation regions are thus deﬁned based on the class of the nearest class center.

Original: Our work is similar to gradient boosting which produces a prediction model using an ensemble of weak models, where each additional model adds an estimator to provide a better model.
Revised: Our work is similar to gradient boosting which produces a prediction model using an ensemble of weak models, where each additional model adds an estimator to provide a better model.

Original: Unlike gradient boosting, our approach takes in the training data used to create the initial estimate so that the model learns how the initial estimate deviates due to the training data.
Revised: Unlike gradient boosting, our approach takes in the training data used to create the initial estimate so that the model learns how the initial estimate deviates due to the training data.

Original: While the initial learner in gradient boosting is weak, we use a strong learner.
Revised: While the initial learner in gradient boosting is a weak learner, we use a strong learner.

Original:  C ORRECTION N ETWORKS 3. N OTATION
Revised:  C ORRECTION N ETWORKS 3. N OTATION

Original: Let D S denote our training data and D U our testing data.
Revised: Let D S denote our training data and D U our testing data.

Original: D S is subdivided into disjoint sets D sS and
Revised: D S is subdivided into disjoint sets D sS and

Original: D uS .
Revised: D uS .

Original: For classiﬁcation, the classes in D sS are disjoint from the classes in D uS .
Revised: For classiﬁcation, the classes in D sS are disjoint from the classes in D uS .

Original: 3. M ODEL
Revised: 3. M ODEL

Original: Correction Networks M consists of two modules: a task module M T and a correction module M C .
Revised: Correction Networks M consists of two modules: a task module M T and a correction module M C .

Original: The model components and their inputs and outputs are illustrated in Figure 2.
Revised: The model components and their inputs and outputs are illustrated in Figure 2.

Original: The task module M T is so-called because it is task speciﬁc and related to the application.
Revised: The task module M T is so-called because it is task speciﬁc and related to the application.

Original: The task module is trained on D sS .
Revised: The task module is trained on D sS .

Original: The output of M T is an estimate ˆ µ of a target µ .
Revised: The output of M T is an estimate ˆ µ of a target µ .

Original: The predictions of the task module on its training data D sS is ˆ µ sS .
Revised: The predictions of the task module on its training data D sS is ˆ µ sS .

Original: Training the task module proceeds by minimizing the distance between ˆ µ sS and the ground truth µ sS .
Revised: Training the task module proceeds by minimizing the distance between ˆ µ sS and the ground truth µ sS .

Original: A sparsity regularization term is added to the input layer.
Revised: A sparsity regularization term is added to the input layer of M T .

Original: The loss is:
Revised: The loss is:

Original: [Equation]
Revised: [Equation]

Original: where T is the class text description, µ is the empirical mean of samples that belong to the class, and d is a distance function such as the L2 norm.
Revised: where T is the class text description, µ is the empirical mean of samples that belong to the class, and d is a distance function. We use the L2 norm as the distance function.

Original: The task module M T has not trained on D uS nor D U .
Revised: The task module M T is not trained on D uS nor D U .

Original: The task module’s predictions of D uS are ˆ µ uS .
Revised: The task module’s predictions on D uS are ˆ µ uS .

Original: Likewise, the task module’s predictions on D U are ˆ µ U .
Revised: Likewise, the task module’s predictions on D U are ˆ µ U .

Original: Thetask module is trained to minimize. The correction module M C computes a correction ˆ (cid:15)  that is applied to the prediction ˆ µ U ˆ µ U of the task module M T , where ˆ (cid:15)  is calculated based on the data used to train M T , such that the corrected prediction (ˆ µ U + ˆ (cid:15)  ) is closer than ˆ µ U to the ground truth µ U .
Revised: The correction module M C computes a correction ˆ (cid:15) uS that is applied to the prediction ˆ µ uS of the task module M T , where ˆ (cid:15) uS is calculated based on the data used to train M T , such that the corrected prediction (ˆ µ uS + ˆ (cid:15) uS ) is closer than ˆ µ uS to the ground truth µ uS .

Original: Training the correction module proceeds by minimizing the distance between µ uS and (ˆ µ uS + ˆ (cid:15) uS ) . 
Revised: Training the correction module proceeds by minimizing the distance between µ uS and (ˆ µ uS + ˆ (cid:15) uS ) . We use the L2 norm.

Original: The training data for the task module D uS is inputted into the correction module by representing the training data D uS as an unordered collection of data by using a pooling function. 
Revised: The training data for the task module D sS is input into the correction module by representing the training data D sS as an un-ordered collection of data by using a pooling function. We use linear layers followed by sum pooling.

Original: The objective function of the correction module is to minimize:
Revised: The objective function of the correction module is to minimize:

Original: [Equation]
Revised: [Equation]

Original: We adopt the meta-learning sampling strategy during training as in Snell et al.
Revised: We adopt the meta-learning sampling strategy for training as in Snell et al.

Original: Training data for Correction Networks are formed by randomly selecting a subset D s S from the training data D S .
Revised: Training data for Correction Networks is formed by randomly selecting a subset D sS from the training data D S .

Original: Then, the task module M T , is trained on D sS .
Revised: Then, the task module M T , is trained on D sS .

Original: The remaining tasks that the task module M T does not train on are treated as D uS for M T .
Revised: The remaining tasks that the task module M T does not train on are treated as D uS for M T .

Original: The algorithm is detailed in Algorithm 1.
Revised: The algorithm is detailed in Algorithm 1.

Original: To use Correction Networks for evaluation, the task module M Tis trained on all the available training data D S . The task module M T outputs ˆ µ U and the correction network supplies ˆ (cid:15) U .
Revised: To use Correction Networks for evaluation, the task module M T outputs ˆ µ U and the correction network supplies ˆ (cid:15) U .

Original: The output of the Correction Networks is ¯ µ U
Revised: The output of the Correction Networks is ¯ µ U

Original: = ˆ µ U + ˆ (cid:15) U .
Revised: = ˆ µ U + ˆ (cid:15) U .

Original: The evaluation algorithm is outlined in Algorithm 2.
Revised: The evaluation algorithm is outlined in Algorithm 2.

Original: Correction Networks can be used for zero-shot problems for which an updated prediction can be obtained by applying a correction.
Revised: Correction Networks can be used for zero-shot problems for which an updated prediction can be obtained by applying a correction.

Original: Appropriate problems are those with continuous valued outputs.
Revised: Appropriate problems are those with continuous valued outputs.

Original: FOr classiﬁcation, samples can be represented as continuous values in a feature space and then classiﬁed based on the nearest class center.
Revised: For classiﬁcation, samples can be represented as continuous values in a feature space and then classiﬁed based on the nearest class center.

Original: In our experiments, both the base network and the
Revised: In our experiments, both the base network and the

Original: Correction Network are deep neural networks.
Revised: Correction Network are deep neural networks.

Original: 3. C ORRECTION N ETWORKS FOR Z ERO -S HOT C LASSIFICATION
Revised: 3. C ORRECTION N ETWORKS FOR Z ERO -S HOT C LASSIFICATION

Original: In zero-shot classiﬁcation, there are seen classes S available during training and unseen classes U during evaluation, where S and U are disjoint.
Revised: In zero-shot classiﬁcation, there are seen classes S available during training and unseen classes U during evaluation, where S and U are disjoint.

Original: 1: for k to K do 2: µ k ← MEAN( D k ) 3: end for 4: while not done do 5:

Original: S s ← RANDOMSAMPLE( { 1 , ..., K } , s ) 6:

Original: S u ← { 1 , ..., K }\ S s 7: while M T has not ﬁnished training do 8: ˆ µ sS ← M T ( T sS ) 9:

Original: J M T ← d (ˆ µ sS , µ sS ) + α || w M T ||10: w M T ← OPTIMIZER( ∇ w MT J M T , w M T ) 11: end while 12: ˆ µ uS ← M T ( T uS ) 13: ˆ (cid:15) uS ← M C ( T uS , ˆ µ uS , T S ) 14: (cid:15) uS ← µ uS − ˆ µ uS 15:

Original: J M C ← d (ˆ (cid:15) uS , (cid:15) uS ) 16: w M C ← OPTIMIZER( ∇ w MC J M C , w M C ) 17: end while

Original: Require Trained M T (

Original: Output: Prediction ¯ µ U 1: ˆ µ U ← M

Original: We are given data D S
Revised: We are given data D S

Original: = { ( x n , y n ) , t m } ) containing samples { x n , n = 1 , ..., N } ∈ X and class labels { y n , i = 1 , ..., N, y n ∈ Y } such that y n ∈ S , along with class text descriptions { t m , m = 1 , ..., M } ∈ T S where there exists exactly one t m for each class in S .
Revised: = { ( x n , y n ) , t m } ) containing samples { x n , n = 1 , ..., N } ∈ X and class labels { y n , i = 1 , ..., N, y n ∈ Y } such that y n ∈ S , along with class text descriptions { t m , m = 1 , ..., M } ∈ T S where there exists exactly one t m for each class in S .

Original: For image classiﬁcation, each sample x n is an image’s features extracted from a pre-trained model.
Revised: For image classiﬁcation, each sample x n is an image’s features extracted from a pre-trained model.

Original: There is no textual data per image.
Revised: There is no textual data per image.

Original: The only text data is t m with one text description per class.
Revised: The only text data is t m with one text description per class.

Original: The model does not assume a particular text representation.

Original: At evaluation, we are given new class descriptions for classes in U , denoted by T U .
Revised: At evaluation, we are given new class descriptions for classes in U , denoted by T U .

Original: We seek to learn a function f to minimize the 0-1 loss between the predicted f ( x i ) and true class labels y
Revised: We seek to learn a function f to minimize the 0-1 loss between the predicted f ( x i ) and true class labels y

Original: i for each sample x i where the samples and class labels can be from the new classes in U .
Revised: i for each sample x i where the samples and class labels can be from the new classes in U .

Original: Correction Networks will map the class text descriptions T U to their corresponding class centers in image feature space  µ U .
Revised: Correction Networks will map the class text descriptions T U to their corresponding class centers in image feature space ˆ µ U .

Original: Classiﬁcation of a single image is done by assigning it the class label of the closest class center, measured by L2 distance.
Revised: Classiﬁcation of a single image is done by assigning it the class label of the closest class center, measured by L2 distance.

Original: We subdivide the training classes S into S s and S u .
Revised: We subdivide the training classes S into S s and S u .

Original: Then, D sS corresponds to the data from D S that correspond to the classes in S s .
Revised: Then, D sS corresponds to the data from D S that belong to the classes in S s .

Original: Similarly,
Revised: Similarly,

Original: D uS corresponds to the data from D S that correspond to the classes in S u .
Revised: D uS corresponds to the data from D S that belong to the classes in S u .

Original: Training  then proceeds as described in the previous section.
Revised: Training and evaluation then proceed as described in the previous section.

Original: Algorithm 2 also lays out the training procedure.

Original: Correction Networks predict class cluster centers for zero-shot classes.
Revised: Correction Networks predict class cluster centers for zero-shot classes.

Original: Individual samples are classiﬁed to the nearest class cluster center using L2 distance.
Revised: Individual samples are classiﬁed to the nearest class cluster center using L distance.

Original: For classes in the training data, the class cluster center is the empirical mean of samples from the respective class.
Revised: For classes in the training data, the class cluster center is the empirical mean of samples from the respective class.

Original: In terms of implementation, both the task module and the correction module are feed-forward neural networks consisting of linear layers, activation functions, and dropout.
Revised: In terms of implementation, both the task module and the correction module are feed-forward neural networks consisting of linear layers, activation functions, and dropout.

Original: We found the task module performance improves slightly when the output of the task module is feed into a classiﬁer with a single hidden layer that is also trained to classify samples from the task model’s training dataset. 
Revised: We found the task module performance improves slightly when the output of the task module is fed into a classiﬁer with a single hidden layer that is also trained to classify samples from the task model’s training dataset. Additional model and training details are in Appendix A. Algorithm class descriptions of classes in S individual samples of

Revised: ) for classes in

Original: E XPERIMENTS 4. T ASK S ETUP
Revised: S subset D all samples from in S s . D k is the subset of D S containing the individual elements ( x i y i ) such that y i =

Revised: k .

Revised: MEAN is a function that returns the mean of its inputs.

Revised: OPTIMIZER is an optimizer.

Revised: d is a distance function, for example, L2 distance.

Revised: M T is the task module.

Revised: M C is the correction module.

Revised: 1: for k in 1 to K do 2: µ k ← MEAN( D k ) 3: end for 4: while not done do 5:

Revised: S s ← RANDOMSAMPLE( { 1 , ..., K } , s ) 6:

Revised: S u ← { 1 , ..., K }\ S s 7: while M T has not ﬁnished training do 8: ˆ µ sS ← M T ( T sS ) 9:

Revised: J M T ← d (ˆ µ sS , µ sS ) + α || w M T ||10: w M T ← OPTIMIZER( ∇ w MT J M T , w M T ) 11: end while 12: ˆ µ uS ← M T ( T uS ) 13: ˆ (cid:15) uS ← M C ( T uS , ˆ µ uS , T sS ) 14: (cid:15) uS ← µ uS − ˆ µ uS 15:

Revised: J M C ← d (ˆ (cid:15) uS , (cid:15) uS ) 16: w M C ← OPTIMIZER( ∇ w MC J M C , w M C ) 17: end while

Revised: Require Trained M T on T s S , trained M C

Revised: Output: Prediction ¯ µ U 1: ˆ µ U ← M T ( T U ) 2: ˆ (cid:15) ← M C ( T U , ˆ µ U , T sS ) 3: ¯ µ U =

Original:  We demonstrate Correction Networks on ﬁne-grained zero-shot classiﬁcation based on natural language text descriptions of the class.
Revised: ˆ µ U + ˆ (cid:15) U  E XPERIMENTS 4. T ASK S ETUP We demonstrate Correction Networks on ﬁne-grained zero-shot classiﬁcation based on natural language text descriptions of the class.

Original: Fine-grained image classiﬁcation is image classiﬁcation when classes are very similar.
Revised: Fine-grained image classiﬁcation is image classiﬁcation when classes are very similar.

Original: This requires the ability to distinguish between minute details and subtle differences between classes.
Revised: This requires the ability to distinguish between minute details and subtle differences between classes.

Original: An example of ﬁne-grained classiﬁcation is distinguishing between different birds that may be very similar.
Revised: An example of ﬁne-grained classiﬁcation is distinguishing between different birds that may be very similar.

Original: Features of a class described in a text description may not be visible in all images belonging to that class.
Revised: Features of a class described in a text description may not be visible in all images belonging to that class.

Original: This loss of visibility can be due to image cropping, posture, camera angle, age of the bird, and gender of the bird, among other reasons.
Revised: This loss of visibility can be due to image cropping, posture, camera angle, age, and gender of the bird, among other reasons.

Original: For example, an image shown from the front results in obfuscation of the back.
Revised: For example, an image shown from the front results in obfuscation of the back.

Original: An image may only be a head shot.
Revised: An image may only be a head shot.

Original: Other objects such as branches or leaves may also block the object of interest.
Revised: Other objects such as branches or leaves may also block the object of interest.

Revised: This is illustrated in Figure 5.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: Image Data We evaluate our method on Caltech UCSD Birds 2011 (CUB)
Revised: Image Data We evaluate our method on Caltech UCSD Birds 2011 (CUB)

Original: (Wah et al., 2011) and
Revised: (Wah et al., 2011) and

Original: North America Birds (NAB) (Van Horn et al., 2015).
Revised: North America Birds (NAB) (Van Horn et al., 2015).

Original: The CUB dataset contains 200 classes of birds with a total of 11,788 images.
Revised: The CUB dataset contains 200 classes of birds with a total of 11,788 images.

Original: We only use the images and their associated labels.
Revised: We only use the images and their associated labels.

Original: We use a newer version of the NAB dataset that contains 404 classes with a total of 48,562 images.
Revised: We use a newer version of the NAB dataset that contains 404 classes with a total of 48,562 images.

Original: Two splits were proposed for each dataset (Mohamed Elhoseiny & Elgammal, 2017).
Revised: Two splits were proposed for each dataset (Elhoseiny et al., 2017b).

Original: The splits are named SuperCategory-Shared (SCS) and Super-Category-Exclusive (SCE).
Revised: The splits are named Super-Category-Shared (SCS) and Super-Category-Exclusive (SCE).

Original: In the SCS split, unseen classes and seen classes share the same parent category.
Revised: In the SCS split, unseen classes and seen classes share the same parent category.

Original: In the SCE split, the parent categories are disjoint between seen and unseen classes.
Revised: In the SCE split, the parent categories are disjoint between seen and unseen classes.

Original: The SCE splits are more difﬁcult than the SCS splits because the unseen classes are more different than the seen classes.
Revised: The SCE splits are more difﬁcult than the SCS splits because the unseen classes are more different than the seen classes.

Original: Where published validation sets are not available, we create our own validation sets by holding out part of the training set.
Revised: Where published validation sets are not available, we create our own validation sets by holding out part of the training set, where the classes in the validation set are randomly sampled from among the training classes.

Original: Visual Features The visual features are activations extracted from the FC layer of a pre-trainedparts detector (Mohamed Elhoseiny & Elgammal, 2017) and are the same as used in (Zhu et al., 2018).
Revised: Visual Features The visual features are activations extracted from the FC layer of a pre-trained detector network (Elhoseiny et al., 2017b). We use the published features from (Zhu et al., 2018).

Original: Text Features Each class is associated with a Wikipedia article.
Revised: Text Features Each class is associated with a Wikipedia article.

Original: The articles are tokenized into words, stop words are removed, and words are reduced to their word stems.
Revised: The articles are tokenized into words, stop words are removed, and words are reduced to their word stems.

Original: Then, the processed text is represented by TF-IDF features.
Revised: Then, the processed text is represented by TF-IDF features.

Original: We use the same text features as (Zhu et al., 2018).
Revised: We use the published text features from (Zhu et al., 2018).

Revised: Examples of the text and images are shown in Figure 3.

Original: [Table]
Revised: [Table]

Original: Examples of the text and images are shown in Figure 4.
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: . 4. C ONVENTIONAL Z ERO -S HOT R ECOGNITION
Revised: C ONVENTIONAL Z ERO -S HOT R ECOGNITION

Original: The top-1 accuracy of our method and eight state-of-the-art algorithms for the CUB and NAB datasets for both the SCS  and  SCE splits are tabulated in Table 1.
Revised: The top-1 accuracy of our method and eight state-of-the-art algorithms for the CUB and NAB datasets for both the SCS split and the SCE split are tabulated in Table 1.

Original: The eight comparison models are MCZSL (Akata et al., 2016), ZSLNS (Qiao et al., 2016), SJE (Akata et al., 2015), WAC (Elhoseiny et al., 2017), SynC (Changpinyo et al., 2016), ZSLPP (Mohamed Elhoseiny & Elgammal, 2017), and GAZSL (Zhu et al., 2018) and the performance numbers are copied from (Zhu et al., 2018).
Revised: The eight comparison models are MCZSL (Akata et al., 2016), ZSLNS (Qiao et al., 2016), SJE (Akata et al., 2015), WAC (Elhoseiny et al., 2017a), SynC (Changpinyo et al., 2016), ZSLPP (Elhoseiny et al., 2017b), and GAZSL (Zhu et al., 2018). The performance numbers are copied from (Zhu et al., 2018).

Original: MCZSL directly uses manual part annotations to extract visual representations.
Revised: MCZSL directly uses manual part annotations to extract visual representations.

Original: On the otherhand our approach, GAZSL, and ZSLPP merely use detected parts for both training and testing.
Revised: On the other hand, our approach, GAZSL, and ZSLPP merely use detected parts for both training and testing.

Original: Thus,performance using methods that use detected parts are expected to be poorer than performance on MCZSL that uses manual parts annotations. 
Revised: Thus, methods that use detected parts are expected to perform poorer than MCZSL, which uses manual parts annotations. Our model performs favorably against the other models, showing a relative improvement of up to 10% over the previous state-of-the-art. Qualitative results are shown in Figure 4 in Appendix B.

Original: Our model increases conventional zero-shot  classiﬁcation accuracy by more than 1% on the SCSsplits.
Revised: The conventional zero-shot learning setting considers queries that come from unseen classes D U and classiﬁcation of queries is restricted to be among the unseen classes U . In contrast, the generalized zero-shot learning setting classiﬁes queries from both seen D S and unseen classes D U into S ∪ U .

Original:  [Table]
Revised: A metric for generalized zero-shot learning performance is the area under the seen-unseen curve (Chao et al., 2016). This is tabulated in Table 2 with values from competitors copied from Zhu et al. For Correction Networks, Correction Networks is used to predict ˆ µ for the unseen classes [Table]

Revised: while the empirical µ is used for the seen classes.

Revised: Correction Networks consistently outperforms previous approaches in the generalized zero-shot learning setting.

Revised: This improvement is between 3-31% relative to the runner-up.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: To examine the contributions of separate components of our model, we conduct ablation studies by removing selected components.
Revised: To examine the contributions of separate components of our model, we conduct ablation studies by removing selected components.

Original: Then, the model is retrained and evaluated on the test set.
Revised: Then, the model is retrained and evaluated on the test set.

Original: The resulting performance of the ablated models are reported in Table 2.
Revised: The resulting performance of the ablated models are reported in Table 3.

Original: Removing the correction module degrades the performance but the task module itself already achieves the state of the art.
Revised: Removing the correction module degrades the performance, but the task module itself already achieves the state of the art.

Original: The correction module improves the performance by about 1.4%.
Revised: The correction module improves the performance by 2% in absolute accuracy.

Original: This suggests that the correction modules makes a slight change to the output of the task module.
Revised: This suggests that the correction modules makes a slight change to the output of the task module.

Original: When we remove the task module’s training data as input into the correction module, the zero-shot accuracy decreases by 1.7%.
Revised: When we remove the task module’s training data as input into the correction module, the zero-shot accuracy decreases by 2.4%.

Original: This demonstrates that the task module’s training data is important to the correction module’s prediction.
Revised: This demonstrates that the task module’s training data is important to the correction module’s prediction.

Original: 4. E XTENSIONS
Revised: 4. E XTENSIONS

Original:  Additional investigations include applying Correction Networks to other tasks with outputs that can be updated, including for example, estimates for regression problems, classiﬁcation probabilities, or probability distributions for reinforcement learning policies.
Revised: Correction Networks are independent of the architectures of the modules and can be used for problems for which an updated prediction can be obtained by applying a correction. Additional investigations include applying Correction Networks to other tasks with outputs that can be updated, including for example, estimates for regression problems, classiﬁcation probabilities, or probability distributions for reinforcement learning policies.

Original: The type of update can also be multiplicative, weighted sum, or another function, in addition to the summation presented here.
Revised: The type of update can also be multiplicative, weighted sum, or another function. In addition to the zero-shot classiﬁcation problem presented here, Correction Networks can also be applied to correct outputs in the few shot setting.

Original: The magnitude of the correction vector predicted by the Correction Network can be interpreted as the size of correction needed.
Revised: The magnitude of the correction vector predicted by the Correction Network can be interpreted as the size of the update applied.

Revised: It would be interesting to use an interpretable output space where different dimensions correspond to human-understandable variables.

Revised: Then, the size and direction of corrections in individual dimensions can be interpreted.

Original: The size of the correction can be compared against the number of training samples given to the base network, deﬁned as the number of seen classes.
Revised: The size of the correction can be compared against the number of training samples given to the base network, deﬁned as the number of seen classes.

Original: One would expect that the size of  correction decreases as the number of seen classes increases.
Revised: One would expect that the size of the correction decreases as the number of seen classes increases.

Original: Also, one would expect that the accuracy stays constant or else improves with the number of seen classes.
Revised: Also, one would expect that the accuracy stays constant or else improves with the number of seen classes.

Original: It would be interesting to see if correction networks can be used to correct for bias in the training data.
Revised: It would be interesting to see if correction networks can be used to understand bias in the training data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a zero-shot learning model that consists of a task module and a correction module that is trained to extrapolate from training data to unseen data.
Revised: We propose a zero-shot learning model that consists of a task module and a correction module that is trained to extrapolate from training data to unseen data.

Original: The training data is partitioned into a set of data used to train the task module and a disjoint set of data used to train the correction module.
Revised: The training data is partitioned into a set of data used to train the task module and a disjoint set of data used to train the correction module.

Original: This later data is zero-shot with respect to the task module and the correction module is trained to update the task module’s zero-shot predictions to create a better prediction.
Revised: This later data is zero-shot with respect to the task module and the correction module is trained to update the task module’s zero-shot predictions to create a better prediction.

Original: We demonstrate our model  on zero-shot ﬁne-grained classiﬁcation using only a single natural language description per zero-shot class.
Revised: Our model is demonstrated on zero-shot ﬁne-grained classiﬁcation using only a single natural language description per zero-shot class.

Original: Our model performs favorably against the state-of-the-art on zero-shot classiﬁcation.
Revised: Our model performs favorably against the state-of-the-art on zero-shot classiﬁcation.

Original: This framework is ﬂexible to different representations of the task and correction modules and can be extended to other types of model predictions that can be updated to improve predictions. 
Revised: This framework is ﬂexible to different representations of the task and correction modules and can be extended to other types of model predictions that can be updated to improve predictions. A PPENDIX A: E XPERIMENTAL D ETAILS Architecture

Revised: The task module is a feed-forward network consisting of linear layers.

Revised: The input is the TF-IDF feature for a class.

Revised: There are two outputs: the predicted mean in image embedding space and softmaxes over the classes in T S .

Revised: The loss function is the sum of the L2 loss between ˆ µ uS and µ uS and the classiﬁcation loss, with a sparsity regualarizer on the input layer.

Revised: The correction module is also a feed-forward network consisting of linear layers.

Revised: Each sample in T sS is fed through linear layers, then sum pooled across classes, and fed through an additional layer.

Revised: The ﬁnal transformation of T sS is concatenated with T uS and ˆ µ uS and then passed through linear layers to output ˆ (cid:15) uS .

Revised: The loss function is L2 loss between ˆ (cid:15) uS and (cid:15) uS .

Revised: Architectures are illustrated in Figure 4.

Revised: [Figure]

Revised: Training

Revised: The publicly available SCS and SCE splits (Elhoseiny et al., 2017b) deﬁne a test set.

Revised: The remaining classes are randomly divided into a validation set and a training set.

Revised: The size of the validation set is the same size as the training set.

Revised: For example, CUB has 200 classes with the conventional SCS split consisting of 50 test classes.

Revised: We divided the remaining 150 classes into a 50 class validation set and a 100 class training set.

Revised: The Correction Network is ﬁrst trained by considering the training set as T S .

Revised: Training proceeds as in

Revised: Algorithm 1.

Revised: That is, T S is randomly divided into disjoint sets T sS and T uS .

Revised: A task module is trained on T s S , stopping based on performance on the validation set.

Revised: The trained task module predicts ˆ µ u S based on T uS .

Revised: The correction module is updated using T uS , ˆ µ uS , and T sS .

Revised: The correction module is trained across multiple episodes of task module training, where the task module in each episode is trained on a randomly sampled T sS from T S .

Revised: Training of different episodes of task modules is parallelizable as they are independent of one another and independent of the correction module.

Revised: Training of the correction module stops based on performance on the validation set.

Revised: The stopping iterations of the task modules and correction module on the validation set is used to determine the stopping iterations of the task modules and the correction module on the test set.

Revised: To evaluate on the test set, the validation and training sets are combined and treated as T S while the test set becomes T U .

Revised: Training again proceeds as in Algorithm 1.

Revised: The Adam optimizer is used to train the models.

Revised: A PPENDIX B: R ESULTS V ISUALIZATIONS

Revised: Correctly and incorrectly classiﬁed images are shown in Figure 4 below.

Revised: Each row is a different class.

Revised: The left-most three images are correctly classiﬁed.

Revised: The middle three images are false negatives, and the right-most images are false positives.