Original: A CTIVE L EARNING WITH P ARTIAL F EEDBACK
Revised: A CTIVE L EARNING WITH P ARTIAL F EEDBACK

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes / no binary feedback).
Revised: While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes / no binary feedback).

Original: To annotate examples corpora for multiclass classiﬁcation, we might need to ask multiple
Revised: To annotate examples corpora for multiclass classiﬁcation, we might need to ask multiple

Original: yes / no questions, exploiting a label hierarchy if one is available.
Revised: yes / no questions, exploiting a label hierarchy if one is available.

Original: To address this more realistic setting, we propose active learning with partial feedback (ALPF), where the learner must actively choose both which example to label and which binary question to ask .
Revised: To address this more realistic setting, we propose active learning with partial feedback (ALPF), where the learner must actively choose both which example to label and which binary question to ask .

Original: At each step, the learner selects an example, asking if it belongs to a chosen (possibly composite) class.
Revised: At each step, the learner selects an example, asking if it belongs to a chosen (possibly composite) class.

Original: Each answer eliminates some classes, leaving the learner with a partial label .
Revised: Each answer eliminates some classes, leaving the learner with a partial label .

Original: The learner may then either ask more questions about the same example (until an exact label is uncovered) or move on immediately, leaving the ﬁrst example partially labeled.
Revised: The learner may then either ask more questions about the same example (until an exact label is uncovered) or move on immediately, leaving the ﬁrst example partially labeled.

Original: Active learning with partial labels requires (i) a sampling strategy to choose (example, class) pairs, and (ii) learning from partial labels between rounds.
Revised: Active learning with partial labels requires (i) a sampling strategy to choose (example, class) pairs, and (ii) learning from partial labels between rounds.

Original: Experiments on Tiny ImageNet demonstrate that our most effective method improves 26% (relative) in top-1 classiﬁcation accuracy compared to i.i.d.
Revised: Experiments on Tiny ImageNet demonstrate that our most effective method improves 26% (relative) in top-1 classiﬁcation accuracy compared to i.i.d.

Original: baselines and standard active learners given 30% of the annotation budget that would be required (naively) to annotate the dataset.
Revised: baselines and standard active learners given 30% of the annotation budget that would be required (naively) to annotate the dataset.

Original: Moreover, ALPF-learners fully annotate TinyImageNet at 42% lower cost.
Revised: Moreover, ALPF-learners fully annotate TinyImageNet at 42% lower cost.

Original: Surprisingly, we observe that accounting for per-example annotation costs can alter the conventional wisdom that active learners should solicit labels for hard examples.
Revised: Surprisingly, we observe that accounting for per-example annotation costs can alter the conventional wisdom that active learners should solicit labels for hard examples.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Given a large set of unlabeled images, and a budget to collect annotations, how can we learn an accurate image classiﬁer most economically?
Revised: Given a large set of unlabeled images, and a budget to collect annotations, how can we learn an accurate image classiﬁer most economically?

Original: Active Learning (AL) seeks to increase data efﬁciency by strategically choosing which examples to annotate.
Revised: Active Learning (AL) seeks to increase data efﬁciency by strategically choosing which examples to annotate.

Original: Typically, AL treats the labeling process as atomic: every annotation costs the same and produces a correct label.
Revised: Typically, AL treats the labeling process as atomic: every annotation costs the same and produces a correct label.

Original: However, large-scale multi-class annotation is seldom atomic; we can’t simply ask a crowd-worker to select one among 1000 classes if they aren’t familiar with our ontology.
Revised: However, large-scale multi-class annotation is seldom atomic; we can’t simply ask a crowd-worker to select one among 1000 classes if they aren’t familiar with our ontology.

Original: Instead, annotation pipelines typically solicit feedback through simpler mechanisms such as yes/no questions.
Revised: Instead, annotation pipelines typically solicit feedback through simpler mechanisms such as yes/no questions.

Original: For example, to construct the 1000 -class
Revised: For example, to construct the 1000 -class

Original: ImageNet dataset, researchers ﬁrst ﬁltered candidates for each class via Google Image Search, then asking crowd-workers questions like “Is there a Burmese cat in this image?”
Revised: ImageNet dataset, researchers ﬁrst ﬁltered candidates for each class via Google Image Search, then asking crowd-workers questions like “Is there a Burmese cat in this image?”

Original: Deng et al.
Revised: (Deng et al., 2009).

Original: For tasks where the Google trick won’t work, we might exploit class hierarchies to drill down to the exact label.
Revised: For tasks where the Google trick won’t work, we might exploit class hierarchies to drill down to the exact label.

Original: Costs scale with the number of questions asked.
Revised: Costs scale with the number of questions asked.

Original: Thus, real-world annotation costs can vary per example Settles (2011).
Revised: Thus, real-world annotation costs can vary per example (Settles, 2011).

Original: We propose Active Learning with Partial Feedback (ALPF)1 , asking, can we cut costs by actively choosing both which examples to annotate, and which questions to ask?
Revised: We propose Active Learning with Partial Feedback (ALPF) , asking, can we cut costs by actively choosing both which examples to annotate, and which questions to ask?

Original: Say that for a new image, our current classiﬁer places 99% of the predicted probability mass on various dog breeds.
Revised: Say that for a new image, our current classiﬁer places 99% of the predicted probability mass on various dog breeds.

Original: Why start at the top of the tree – “is this an artiﬁcial object?”
Revised: Why start at the top of the tree – “is this an artiﬁcial object?”

Original: – when we can cut costs by jumping straight to dog breeds (Figure 1)?
Revised: – when we can cut costs by jumping straight to dog breeds (Figure 1)?

Original: ALPF proceeds as follows:
Revised: ALPF proceeds as follows:

Original: In addition to the class labels, the learner possesses a pre-deﬁned collection of composite classes , e.g. dog ⊃ bulldog , mastiff , ....
Revised: In addition to the class labels, the learner possesses a pre-deﬁned collection of composite classes , e.g. dog ⊃ bulldog , mastiff , ....

Original: At each round, the learner selects an (example, class) pair.
Revised: At each round, the learner selects an (example, class) pair.

Original: The annotator responds with binary feedback, leaving the learner with a partial label .
Revised: The annotator responds with binary feedback, leaving the learner with a partial label .

Original: If only the atomic class label remains, the learner has obtained an exact label .
Revised: If only the atomic class label remains, the learner has obtained an exact label .

Original: For simplicity, we focus on hierarchically-organized collections—trees with atomic classes as leaves and composite classes as internal nodes.
Revised: For simplicity, we focus on hierarchically-organized collections—trees with atomic classes as leaves and composite classes as internal nodes.

Original: [Figure]
Revised: [Figure]

Original: For this to work, we need a hierarchy of concepts familiar to the annotator.
Revised: For this to work, we need a hierarchy of concepts familiar to the annotator.

Original: Imagine asking an annotator “is this a foo ?”
Revised: Imagine asking an annotator “is this a foo ?”

Original: where foo represents a category comprised of 500 random ImageNet classes.
Revised: where foo represents a category comprised of 500 random ImageNet classes.

Original: Determining class membership would be onerous for the same reason that providing an exact label is: It requires the annotator be familiar with an enormous list of seemingly-unrelated options before answering.
Revised: Determining class membership would be onerous for the same reason that providing an exact label is: It requires the annotator be familiar with an enormous list of seemingly-unrelated options before answering.

Original: On the other hand, answering “is this an animal?” is easy despite animal being an extremely coarse-grained category —because most people already know what an animal is.
Revised: On the other hand, answering “is this an animal?” is easy despite animal being an extremely coarse-grained category —because most people already know what an animal is.

Original: We use active questions in a few ways.
Revised: We use active questions in a few ways.

Original: To start, in the simplest setup, we can select samples at random but then once each sample is selected, choose questions actively until ﬁnding the label:
Revised: To start, in the simplest setup, we can select samples at random but then once each sample is selected, choose questions actively until ﬁnding the label:

Original: [Equation] [Figure]
Revised: ML: “Is it a dog?”

Original: [Equation]
Revised: Human:

Original: [Figure]
Revised: Yes!

Original: [Equation] [Figure]
Revised: ML: “Is it a poodle?” Human: No! ML: “Is it a hound?”

Original: [Equation]
Revised: Human: Yes!

Original: [Figure]
Revised: ML: “Is it a Rhodesian ?”

Original: [Equation]
Revised: Human:

Original: [Figure]
Revised: No! ML: “Is it a Dachsund?”

Original: [Equation]
Revised: Human Yes!

Original: In ALPF, we go one step further.
Revised: In ALPF, we go one step further.

Original: Since our goal is to produce accurate classiﬁers on tight budget, should we necessarily label each example to completion?
Revised: Since our goal is to produce accurate classiﬁers on tight budget, should we necessarily label each example to completion?

Original: After each question, ALPF learners have the option of choosing a different example for the next binary query.
Revised: After each question, ALPF learners have the option of choosing a different example for the next binary query.

Original: Efﬁcient learning under ALPF requires (i) good strategies for choosing (example, class) pairs, and (ii) techniques for learning from the partially-labeled data that results when labeling examples to completion isn’t required.
Revised: Efﬁcient learning under ALPF requires (i) good strategies for choosing (example, class) pairs, and (ii) techniques for learning from the partially-labeled data that results when labeling examples to completion isn’t required.

Original: We ﬁrst demonstrate an effective scheme for learning from partial labels.
Revised: We ﬁrst demonstrate an effective scheme for learning from partial labels.

Original: The predictive distribution is parameterized by a softmax over all classes.
Revised: The predictive distribution is parameterized by a softmax over all classes.

Original: On a per-example basis, we convert the multiclass problem to a binary classiﬁcation problem, where the two classes correspond to the subsets of potential and eliminated classes.
Revised: On a per-example basis, we convert the multiclass problem to a binary classiﬁcation problem, where the two classes correspond to the subsets of potential and eliminated classes.

Original: We determine the total probability assigned to potential classes by summing over their softmax probabilities.
Revised: We determine the total probability assigned to potential classes by summing over their softmax probabilities.

Original: For active learning with partial feedback, we introduce several acquisition functions for soliciting partial labels, selecting questions among all (example, class) pairs.
Revised: For active learning with partial feedback, we introduce several acquisition functions for soliciting partial labels, selecting questions among all (example, class) pairs.

Original: One natural method, expected information gain (EIG) generalizes the classic maximum entropy heuristic to the ALPF setting.
Revised: One natural method, expected information gain (EIG) generalizes the classic maximum entropy heuristic to the ALPF setting.

Original: Our two other heuristics, EDC and ERC, select based on the number of labels that we expect to see eliminated from and remaining in a given partial label, respectively.
Revised: Our two other heuristics, EDC and ERC, select based on the number of labels that we expect to see eliminated from and remaining in a given partial label, respectively.

Original: We evaluate ALPF learners on CIFAR10, CIFAR100, and Tiny ImageNet datasets.
Revised: We evaluate ALPF learners on CIFAR10, CIFAR100, and Tiny ImageNet datasets.

Original: In all cases, we use
Revised: In all cases, we use

Original: WordNet to impose a hierarchy on our labels.
Revised: WordNet to impose a hierarchy on our labels.

Original: Each of our experiments simulates rounds of active learning, starting with a small amount of i.i.d.
Revised: Each of our experiments simulates rounds of active learning, starting with a small amount of i.i.d.

Original: data to warmstart the models, and proceeding until all examples are exactly labeled.
Revised: data to warmstart the models, and proceeding until all examples are exactly labeled.

Original: We compare models by their test-set accuracy after various amounts of annotation.
Revised: We compare models by their test-set accuracy after various amounts of annotation.

Original: Experiments show that ERC sampling performs best.
Revised: Experiments show that ERC sampling performs best.

Original: On TinyImageNet , with a budget of 250k binary questions,
Revised: On TinyImageNet , with a budget of 250k binary questions,

Original: ALPF improves in accuracy by 26% (relative) and 8.1% (absolute) over the i.i.d. baseline.
Revised: ALPF improves in accuracy by 26% (relative) and 8.1% (absolute) over the i.i.d. baseline.

Original: Additionally,
Revised: Additionally,

Original: ERC & EDC fully annotate the dataset with just 491 k and 484 k examples binary questions, respectively (vs 827k), a 42% reduction in annotation cost.
Revised: ERC & EDC fully annotate the dataset with just 491 k and 484 k examples binary questions, respectively (vs 827k), a 42% reduction in annotation cost.

Original: Surprisingly, we observe that taking disparate annotation costs into account may alter the conventional wisdom that active learners should solicit labels for hard examples.
Revised: Surprisingly, we observe that taking disparate annotation costs into account may alter the conventional wisdom that active learners should solicit labels for hard examples.

Original: In ALPF, easy examples might yield less information, but are cheaper to annotate.
Revised: In ALPF, easy examples might yield less information, but are cheaper to annotate.

Original:  A CTIVE L EARNING WITH P ARTIAL F EEDBACK
Revised:  A CTIVE L EARNING WITH P ARTIAL F EEDBACK

Original: By x ∈ R d and y ∈ Y for Y = {{ 1 } , ..., { k }} , we denote feature vectors and labels.
Revised: By x ∈ R d and y ∈ Y for Y = {{ 1 } , ..., { k }} , we denote feature vectors and labels.

Original: Here d is the feature dimension and k is the number of atomic classes.
Revised: Here d is the feature dimension and k is the number of atomic classes.

Original: By atomic class, we mean that they are indivisible.
Revised: By atomic class, we mean that they are indivisible.

Original: As in conventional AL, the agent starts off with an unlabeled training set D = { x 1 , ..., x n } .
Revised: As in conventional AL, the agent starts off with an unlabeled training set D = { x 1 , ..., x n } .

Original: Composite classes We also consider a pre-speciﬁed collection of composite classes C
Revised: Composite classes We also consider a pre-speciﬁed collection of composite classes C

Original: = { c 1 , ..., c m } , where each composite class c i ⊂ { 1 , ..., k } is a subset of labels such that
Revised: = { c 1 , ..., c m } , where each composite class c i ⊂ { 1 , ..., k } is a subset of labels such that

Original: | c i | ≥ 1 .
Revised: | c i | ≥ 1 .

Original: Note that C includes both the atomic and composite classes.
Revised: Note that C includes both the atomic and composite classes.

Original: In this paper’s empirical section, we generate composite classes by imposing an existing lexical hierarchy on the class labels (Miller, 1995).
Revised: In this paper’s empirical section, we generate composite classes by imposing an existing lexical hierarchy on the class labels (Miller, 1995).

Original: Partial labels For an example i , we use partial label to describe any element ˜ y
Revised: Partial labels For an example i , we use partial label to describe any element ˜ y

Original: i ⊂ { 1 , ..., k }
Revised: i ⊂ { 1 , ..., k }

Original: such that ˜ y i ⊃
Revised: such that ˜ y i ⊃

Original: y i .
Revised: y i .

Original: We call ˜ y
Revised: We call ˜ y

Original: i a partial label because it may rule out some classes, but doesn’t fully indicate underlying atomic class.
Revised: i a partial label because it may rule out some classes, but doesn’t fully indicate underlying atomic class.

Original: For example, dog = { akita, beagle, bulldog, ... } is a valid partial label when the true label is { bulldog } .
Revised: For example, dog = { akita, beagle, bulldog, ... } is a valid partial label when the true label is { bulldog } .

Original: An ALPF learner eliminates classes, obtaining successively smaller partial labels, until only one (the exact label ) remains.
Revised: An ALPF learner eliminates classes, obtaining successively smaller partial labels, until only one (the exact label ) remains.

Original: To simplify notation, in this paper, by an example’s partial label , we refer to the smallest partial label available based on the already-eliminated classes.
Revised: To simplify notation, in this paper, by an example’s partial label , we refer to the smallest partial label available based on the already-eliminated classes.

Original: At any step t and for any example i , we use ˜ y ( t ) i to denote the current partial label.
Revised: At any step t and for any example i , we use ˜ y ( t ) i to denote the current partial label.

Original: The initial partial label for every example is ˜ y 

Revised: The initial partial label for every example is ˜ y

Original: = { 1 , ..., k }
Revised: = { 1 , ..., k }

Original: An exact label is achieved when the partial label ˜ y
Revised: An exact label is achieved when the partial label ˜ y

Original: i =
Revised: i =

Original: y i .
Revised: y i .

Original: Partial Feedback
Revised: Partial Feedback

Original: The set of possible questions Q = X
Revised: The set of possible questions Q = X

Original: × C includes all pairs of examples and composite classes.
Revised: × C includes all pairs of examples and composite classes.

Original: An ALPF learner interacts with annotators by choosing questions q ∈ Q .
Revised: An ALPF learner interacts with annotators by choosing questions q ∈ Q .

Original: Informally, we pick a question q =
Revised: Informally, we pick a question q =

Original: ( x i , c j ) and ask the annotator, does x i contain a c j ?
Revised: ( x i , c j ) and ask the annotator, does x i contain a c j ?

Original: If the queried example’s label belongs to the queried composite class ( y i ⊂ c j ), the answer is 1, else 0 .
Revised: If the queried example’s label belongs to the queried composite class ( y i ⊂ c j ), the answer is 1, else 0 .

Original: Let α q denote the binary answer to question q ∈ Q .
Revised: Let α q denote the binary answer to question q ∈ Q .

Original: Based on the partial feedback, we can compute the new partial label ˜
Revised: Based on the partial feedback, we can compute the new partial label ˜

Original: y ( t +1) according to Eq.
Revised: y ( t +1) according to Eq.

Original: equation 1,
Revised: equation 1,

Original: [Equation]
Revised: [Equation]

Original: Note that here ˜ y
Revised: Note that here ˜ y

Original: ( t ) and c are sets, α is a bit, c is a set complement, and that ˜ y ( t ) \ c and ˜ y ( t ) \ c are set subtractions to eliminate classes from the partial label based on the answer.
Revised: ( t ) and c are sets, α is a bit, c is a set complement, and that ˜ y ( t ) \ c and ˜ y ( t ) \ c are set subtractions to eliminate classes from the partial label based on the answer.

Original: Learning Process
Revised: Learning Process

Original: The learning process is simple:
Revised: The learning process is simple:

Original: At each round t , the learner selects a pair ( x , c ) for labeling.
Revised: At each round t , the learner selects a pair ( x , c ) for labeling.

Original: Note that a rational agent will never select either (i) an example for which the exact label is known, or (ii) a pair ( x , c ) for which the answer is already known, e.g., if c ⊃ ˜ y ( t ) or c ∩ ˜ y ( t ) = ∅ .
Revised: Note that a rational agent will never select either (i) an example for which the exact label is known, or (ii) a pair ( x , c ) for which the answer is already known, e.g., if c ⊃ ˜ y ( t ) or c ∩ ˜ y ( t ) = ∅ .

Original: After receiving binary feedback, the agent updates the corresponding partial label ˜
Revised: After receiving binary feedback, the agent updates the corresponding partial label ˜

Original: y ( t ) →
Revised: y ( t ) →

Original: ˜ y ( t +1) , using Equation 1.
Revised: ˜ y ( t +1) , using Equation 1.

Original: The agent then re-estimates its model, using all available non-trivial partial labels and selects another question q .
Revised: The agent then re-estimates its model, using all available non-trivial partial labels and selects another question q .

Original: In batch-mode, the ALPF learner re-estimates its model once per T queries which is necessary when training is expensive (e.g. deep learning).
Revised: In batch-mode, the ALPF learner re-estimates its model once per T queries which is necessary when training is expensive (e.g. deep learning).

Original: We summarize the workﬂow of a ALPF learner in Algorithm 1.
Revised: We summarize the workﬂow of a ALPF learner in Algorithm 1.

Original: Objectives We state two goals for ALPF learners.
Revised: Objectives We state two goals for ALPF learners.

Original: First, we want to learn predictors with low error (on exactly labeled i.i.d.
Revised: First, we want to learn predictors with low error (on exactly labeled i.i.d.

Original: holdout data), given a ﬁxed annotation budget.
Revised: holdout data), given a ﬁxed annotation budget.

Original: Second, we want to fully annotate datasets at the lowest cost.
Revised: Second, we want to fully annotate datasets at the lowest cost.

Original: In our experiments (Section 3), a ALPF strategy dominates on both tasks.
Revised: In our experiments (Section 3), a ALPF strategy dominates on both tasks.

Original: Input: X ← ( x 1 , . . .
Revised: Input: X ← ( x 1 , . . .

Original: , x N ) , Q ← ( q 1 , . .
Revised: , x N ) , Q ← ( q 1 , . .

Original: . , q M ) ,
Revised: . , q M ) ,

Original: K , T .
Revised: K , T .

Original: Input: D ← [ x i ] Ni =1 , C ← [ c j ] Mj =1 , k , T
Revised: Input: D ← [ x i ] Ni =1 , C ← [ c j ] Mj =1 , k , T

Original: Initialize: ˜ y (0) i ← { 1 , . . .
Revised: Initialize: ˜ y (0) i ← { 1 , . . .

Original: , k } , θ ← θ (0) , t ←repeat
Revised: , k } , θ ← θ (0) , t ←repeat

Original: Score every ( x i , c j ) with θ repeat
Revised: Score every ( x i , c j ) with θ repeat

Original: Select ( x i ∗ , c
Revised: Select ( x i ∗ , c

Original: j ∗ ) with the best score
Revised: j ∗ ) with the best score

Original: Query c j ∗ on data x i ∗
Revised: Query c j ∗ on data x i ∗

Original: Receive feedback α
Revised: Receive feedback α

Original: Update ˜
Revised: Update ˜

Original: y
Revised: y

Original: ( t +1)
Revised: ( t +1)

Original: i ∗ according to α t ← t +until ( t mod T = 0 ) or ( ∀ i, | ˜ y ( t ) i | = 1 ) θ ← arg min θ L ( θ ) until ∀ i, | ˜ y ( t ) i | = 1 or t exhausts budget 
Revised: i ∗ according to α t ← t +until ( t mod T = 0 ) or ( ∀ i, | ˜ y ( t ) i | = 1 ) θ ← arg min θ L ( θ ) until ∀ i, | ˜ y ( t ) i | = 1 or t exhausts budget [Table]

Original: [Table]
Revised: [Table]

Original: 2. L EARNING FROM PARTIAL LABELS
Revised: L EARNING FROM PARTIAL LABELS

Original: We now address the task of learning a multiclass classiﬁer from partial labels, a fundamental requirement of ALPF, regardless of the choice of sampling strategy.
Revised: We now address the task of learning a multiclass classiﬁer from partial labels, a fundamental requirement of ALPF, regardless of the choice of sampling strategy.

Original: At time t , our model ˆ y
Revised: At time t , our model ˆ y

Original: ( y, x , θ ( t ) ) parameterised by parameters θ ( t ) estimates the conditional probability of an atomic class y .
Revised: ( y, x , θ ( t ) ) parameterised by parameters θ ( t ) estimates the conditional probability of an atomic class y .

Original: For simplicity, when the context is clear, we will use ˆ y to designate the full vector of predicted probabilities over all classes.
Revised: For simplicity, when the context is clear, we will use ˆ y to designate the full vector of predicted probabilities over all classes.

Original: The probability assigned to a partial label ˜
Revised: The probability assigned to a partial label ˜

Original: y can be expressed by marginalizing over the atomic classes that it contains: ˆ p (˜ y ( t ) , x , θ ( t ) )
Revised: y can be expressed by marginalizing over the atomic classes that it contains: ˆ p (˜ y ( t ) , x , θ ( t ) )

Original: =
Revised: =

Original: (cid:80) y ∈ ˜
Revised: (cid:80) y ∈ ˜

Original: y ( t )
Revised: y ( t )

Original: ˆ y
Revised: ˆ y

Original: ( y, x , θ ( t ) ) .
Revised: ( y, x , θ ( t ) ) .

Original: We optimize our model by minimizing the log loss:
Revised: We optimize our model by minimizing the log loss:

Original: [Equation]
Revised: [Equation]

Original: Note that when every example is exactly labeled, our loss function simpliﬁes to the standard cross entropy loss often used for multi-class classiﬁcation.
Revised: Note that when every example is exactly labeled, our loss function simpliﬁes to the standard cross entropy loss often used for multi-class classiﬁcation.

Original: Also note that when every partial label contains the full set of classes, all partial labels have probability 1 and the update is a no-op.
Revised: Also note that when every partial label contains the full set of classes, all partial labels have probability 1 and the update is a no-op.

Original: Finally, if the partial label indicates a composite class such as dog , and the predictive probability mass is exclusively allocated among various breeds of dog, our loss will be 0 .
Revised: Finally, if the partial label indicates a composite class such as dog , and the predictive probability mass is exclusively allocated among various breeds of dog, our loss will be 0 .

Original: Models are only updated when their predictions disagree (to some degree) with the current partial label.
Revised: Models are only updated when their predictions disagree (to some degree) with the current partial label.

Original: 2. S AMPLING STRATEGIES
Revised: 2. S AMPLING STRATEGIES

Original: Expected Information Gain (EIG): Per classic uncertainty sampling, we can quantify a classifer’s uncertainty via the entropy of the predictive distribution.
Revised: Expected Information Gain (EIG): Per classic uncertainty sampling, we can quantify a classifer’s uncertainty via the entropy of the predictive distribution.

Original: In AL, each query returns an exact label, and thus the post-query entropy is always 0 .
Revised: In AL, each query returns an exact label, and thus the post-query entropy is always 0 .

Original: In our case, each answer to the query yields a different partial label.
Revised: In our case, each answer to the query yields a different partial label.

Original: We use the notation ˆ y 0 , and ˆ y 1 to denote consequent predictive distributions for each answer (no or yes).
Revised: We use the notation ˆ y 0 , and ˆ y 1 to denote consequent predictive distributions for each answer (no or yes).

Original: We generalize maximum entropy to ALPF by selecting questions with greatest expected reduction in entropy .
Revised: We generalize maximum entropy to ALPF by selecting questions with greatest expected reduction in entropy .

Original: [Equation]
Revised: [Equation]

Original: ) whereS ( · ) is the entropy function. It’s easy to prove that EIG is maximized when ˆ p ( c, x , θ ) = 0 .
Revised: where

Original: Expected Remaining Classes (ERC): Next, we propose ERC, a heuristic that suggests arriving as quickly as possible at exactly-labeled examples.
Revised: Expected Remaining Classes (ERC): Next, we propose ERC, a heuristic that suggests arriving as quickly as possible at exactly-labeled examples.

Original: At each round, ERC selects those examples for which the expected number of remaining classes is fewest:
Revised: At each round, ERC selects those examples for which the expected number of remaining classes is fewest:

Original: [Equation]
Revised: [Equation]

Original: where || ˆ y α || is the size of the partial label following given answer α .
Revised: where || ˆ y α || is the size of the partial label following given answer α .

Original: ERC is minimized when the result of the feedback will produce an exact label with probability 1 .
Revised: ERC is minimized when the result of the feedback will produce an exact label with probability 1 .

Original: For a given example x i , if || ˆ
Revised: For a given example x i , if || ˆ

Original: y i
Revised: y i

Original: ||
Revised: ||

Original: = 2 containing only the potential classes (e.g.) dog and cat , then with certainty, ERC will produce an exact label by querying the class { dog } (or equivalently { cat } ).
Revised: = 2 containing only the potential classes (e.g.) dog and cat , then with certainty, ERC will produce an exact label by querying the class { dog } (or equivalently { cat } ).

Original: This heuristic is inspired by Cour et al.
Revised: This heuristic is inspired by Cour et al.

Original: (2011), which shows that the partial classiﬁcation loss (what we optimize with partial labels) is an upper bound of the true classiﬁcation loss (as if true labels are available) with a linear factor of 11 − ε , where ε is ambiguity degree and ε ∝
Revised: (2011), which shows that the partial classiﬁcation loss (what we optimize with partial labels) is an upper bound of the true classiﬁcation loss (as if true labels are available) with a linear factor of 11 − ε , where ε is ambiguity degree and ε ∝

Original: | ˜ y | .
Revised: | ˜ y | .

Original: By selecting q ∈ Q that leads to the smallest | ˜ y
Revised: By selecting q ∈ Q that leads to the smallest | ˜ y

Original: | , we can tighten the bound to make optimization with partial labels more effective.
Revised: | , we can tighten the bound to make optimization with partial labels more effective.

Original: Expected Decrease in Classes (EDC): More in keeping with the traditional goal of minimizing uncertainty, we might choose EDC, the sampling strategy which we expect to result in the greatest reduction in the number of potential classes.
Revised: Expected Decrease in Classes (EDC): More in keeping with the traditional goal of minimizing uncertainty, we might choose EDC, the sampling strategy which we expect to result in the greatest reduction in the number of potential classes.

Original: We can express EDC as the difference between the number of potential labels (known) and the expected number of potential labels remaining: EDC ( x ,c )
Revised: We can express EDC as the difference between the number of potential labels (known) and the expected number of potential labels remaining: EDC ( x ,c )

Original: = |
Revised: = |

Original: ˜ y ( t ) | − ERC ( x ,c ) .
Revised: ˜ y ( t ) | − ERC ( x ,c ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate ALPF algorithms on the CIFAR10, CIFAR100, and Tiny ImageNet datasets, with training sets of 50k, 50k, and 100k examples, and 10, 100, and 200 classes respectively.
Revised: We evaluate ALPF algorithms on the CIFAR10, CIFAR100, and Tiny ImageNet datasets, with training sets of 50k, 50k, and 100k examples, and 10, 100, and 200 classes respectively.

Original: After imposing the Wordnet hierarchy on the label names, the size of the set of possible binary questions |C| for each dataset are 27 , 261 , and 304 , respectively.
Revised: After imposing the Wordnet hierarchy on the label names, the size of the set of possible binary questions |C| for each dataset are 27 , 261 , and 304 , respectively.

Original: The number of binary questions between re-trainings are 5k, 15k, and 30k, respectively.
Revised: The number of binary questions between re-trainings are 5k, 15k, and 30k, respectively.

Original: By default, we warm-start each learner with the same 5% of training examples selected i.i.d. and exactly labeled.
Revised: By default, we warm-start each learner with the same 5% of training examples selected i i d. and exactly labeled.

Original: Warm-starting has proven essential in other papers combining deep and active learning Shen et al.
Revised: Warm-starting has proven essential in other papers combining deep and active learning (Shen et al., 2018).

Original: Our own analysis (Section 3.3) conﬁrms the importance of warm-starting although the affect appears variable across acquisition strategies.
Revised: Our own analysis (Section 3.3) conﬁrms the importance of warm-starting although the affect appears variable across acquisition strategies.

Original: Model For each experiment, we adopt the widely-popular ResNet-18 architecture He et al.
Revised: Model For each experiment, we adopt the widely-popular ResNet-18 architecture (He et al., 2016).

Original: Because we are focused on active learning and thus seek fundamental understanding of this new problem formulation, we do not complicate the picture with any ﬁne-tuning techniques.
Revised: Because we are focused on active learning and thus seek fundamental understanding of this new problem formulation, we do not complicate the picture with any ﬁne-tuning techniques.

Original: Note that some leaderboard scores circulating on the Internet appear to have far superior numbers.
Revised: Note that some leaderboard scores circulating on the Internet appear to have far superior numbers.

Original: This owes to pre-training on the full ImageNet dataset (from which Tiny-ImageNet was subsampled and downsampled), constituting a target leak.
Revised: This owes to pre-training on the full ImageNet dataset (from which Tiny-ImageNet was subsampled and downsampled), constituting a target leak.

Original: We initialize weights with the Xavier technique (Glorot & Bengio, 2010) and minimize our loss using the Adam Kingma & Ba (2014) optimizer, ﬁnding that it outperforms SGD signiﬁcantly when learning from partial labels.
Revised: We initialize weights with the Xavier technique (Glorot and Bengio, 2010) and minimize our loss using the Adam (Kingma and Ba, 2014) optimizer, ﬁnding that it outperforms SGD signiﬁcantly when learning from partial labels.

Original: We use the same learning rate of 0 .
Revised: We use the same learning rate of 0 .

Original: 001 for all experiments, ﬁrst-order momentum decay ( β 1 ) of 0 .
Revised: 001 for all experiments, ﬁrst-order momentum decay ( β 1 ) of 0 .

Original: 9 , and second-order momentum decay ( β 2 ) of 0 .
Revised: 9 , and second-order momentum decay ( β 2 ) of 0 .

Original: Finally, we train with mini-batches ofexamples and perform standard data augmentation techniques including random cropping, resizing, and mirror-ﬂipping.
Revised: Finally, we train with mini-batches ofexamples and perform standard data augmentation techniques including random cropping, resizing, and mirror-ﬂipping.

Original: We implement all models in MXNet and have posted our code publicly.
Revised: We implement all models in MXNet and have posted our code publicly 1 .

Original: Re-training Ideally, we might update models after each query, but this is too costly.
Revised: Re-training Ideally, we might update models after each query, but this is too costly.

Original: Instead, following
Revised: Instead, following

Original: Shen et al.
Revised: Shen et al.

Original: (2018) and others, we alternately query labels and update our models in rounds.
Revised: (2018) and others, we alternately query labels and update our models in rounds.

Original: We warm-start all experiments with 5% labeled data and iterate until every example is exactly labeled.
Revised: We warm-start all experiments with 5% labeled data and iterate until every example is exactly labeled.

Original: At each round, we re-train our classiﬁer from scratch with random initialization.
Revised: At each round, we re-train our classiﬁer from scratch with random initialization.

Original: While we could initialize the new classiﬁer with the previous best one (as in Shen et al.
Revised: While we could initialize the new classiﬁer with the previous best one (as in Shen et al.

Original: , preliminary experiments showed that this faster convergence comes at the cost of worse performance, perhaps owing to severe over-ﬁtting to labels acquired early in training.
Revised: , preliminary experiments showed that this faster convergence comes at the cost of worse performance, perhaps owing to severe over-ﬁtting to labels acquired early in training.

Original: In all experiments, for simplicity, we terminate the optimization after 75 epochs.
Revised: In all experiments, for simplicity, we terminate the optimization after 75 epochs.

Original: Since  30 k questions per re-training (for TinyImagenet) seems infrequent, we compared against 10x more frequent re-training More frequent training conferred no beneﬁt (Appendix B).[Figure]
Revised: Since [Figure] 30 k questions per re-training (for TinyImagenet) seems infrequent, we compared against 10x more frequent re-training More frequent training conferred no beneﬁt (Appendix B).

Original: 3. L EARNING FROM PARTIAL LABELS
Revised: 3. L EARNING FROM PARTIAL LABELS

Original: Since the success of ALPF depends in part on learning from partial labels, we ﬁrst demonstrate the efﬁcacy of learning from partial labels with our loss function when the partial labels are given a priori.
Revised: Since the success of ALPF depends in part on learning from partial labels, we ﬁrst demonstrate the efﬁcacy of learning from partial labels with our loss function when the partial labels are given a priori.

Original: In these experiments we simulate a partially labeled dataset and show that the learner achieves signiﬁcantly better accuracy when learning from partial labels than if it excluded the partial labels and focused only on exactly annotated examples.
Revised: In these experiments we simulate a partially labeled dataset and show that the learner achieves signiﬁcantly better accuracy when learning from partial labels than if it excluded the partial labels and focused only on exactly annotated examples.

Original: Using our WordNet-derived hierarchy, we conduct experiments with partial labels at different levels of granularity.
Revised: Using our WordNet-derived hierarchy, we conduct experiments with partial labels at different levels of granularity.

Original: Using partial labels from one level above the leaf, German shepherd becomes dog .
Revised: Using partial labels from one level above the leaf, German shepherd becomes dog .

Original: Going up two levels, it becomes animal .
Revised: Going up two levels, it becomes animal .

Original: We ﬁrst train a standard multi-class classiﬁer with γ (%) exactly labeled training data and then another classiﬁer with the remaining (1 − γ )% partially labeled at a different granularity (level of hierarchy).
Revised: We ﬁrst train a standard multi-class classiﬁer with γ (%) exactly labeled training data and then another classiﬁer with the remaining (1 − γ )% partially labeled at a different granularity (level of hierarchy).

Original: We compare the classiﬁer performance on holdout data both with and without adding partial labels in Table 1.
Revised: We compare the classiﬁer performance on holdout data both with and without adding partial labels in Table 1.

Original: We make two key observations: (i) additional coarse-grained partial labels improve model accuracy (ii) as expected, the improvement diminishes as partial label gets coarser.
Revised: We make two key observations: (i) additional coarse-grained partial labels improve model accuracy (ii) as expected, the improvement diminishes as partial label gets coarser.

Original: These observations suggest we can learn effectively given a mix of exact and partial labels.
Revised: These observations suggest we can learn effectively given a mix of exact and partial labels.

Original: 3. S AMPLING STRATEGIES
Revised: 3. S AMPLING STRATEGIES

Original: Baseline This learner samples examples at random.
Revised: Baseline This learner samples examples at random.

Original: Once an example is sampled, the learner applies topdown binary splitting—choosing the question that most evenly splits the probability mass, see Related Work for details— with a uniform prior over the classes until that example is exactly labeled.
Revised: Once an example is sampled, the learner applies topdown binary splitting—choosing the question that most evenly splits the probability mass, see Related Work for details— with a uniform prior over the classes until that example is exactly labeled.

Original: AL
Revised: AL

Original: To disentangle the effect of active sampling of questions and samples, we compare to conventional AL approaches selecting examples with uncertainty sampling but selecting questions as baseline.
Revised: To disentangle the effect of active sampling of questions and samples, we compare to conventional AL approaches selecting examples with uncertainty sampling but selecting questions as baseline.

Original: AQ Active questions learners, choose examples at random but use partial feedback strategies to efﬁciently label those examples, moving on to the next example after ﬁnding an example’s exact label.
Revised: AQ Active questions learners, choose examples at random but use partial feedback strategies to efﬁciently label those examples, moving on to the next example after ﬁnding an example’s exact label.

Original: ALPF ALPF learners are free to choose any (example, question) pair at each turn, Thus, unlike AL and AQ,
Revised: ALPF ALPF learners are free to choose any (example, question) pair at each turn, Thus, unlike AL and AQ,

Original: ALPF learners commonly encounter partial labels during training.
Revised: ALPF learners commonly encounter partial labels during training.

Original: Results We run all experiments until fully annotating the training set.
Revised: Results We run all experiments until fully annotating the training set.

Original: We then evaluate each method from two perspectives: classiﬁcation and annotation.
Revised: We then evaluate each method from two perspectives: classiﬁcation and annotation.

Original: We measure each classiﬁers’ top-1 accuracy at each annotation budget.
Revised: We measure each classiﬁers’ top-1 accuracy at each annotation budget.

Original: To quantify annotation performance, we count the number questions required to exactly label all training examples.
Revised: To quantify annotation performance, we count the number questions required to exactly label all training examples.

Original: We compile our results in Table 2, rounding costs to 10%, 20% etc.
Revised: We compile our results in Table 2, rounding costs to 10%, 20% etc.

Original: The budget includes[Table] the (5%) i.i [Table]
Revised: The budget includes the (5%) i.i.d.

Original: d. data for warm-starting.
Revised: data for warm-starting.

Original: Some key results: (i) vanilla active learning does not improve over i.i
Revised: Some key results: (i) vanilla active learning does not improve over i.i.d.

Original: [Table]

Original: d. baselines, conﬁrming similar observations on image classiﬁcation by Sener & Savarese (2017); (ii) AQ provides a dramatic improvement over baseline.
Revised: baselines, conﬁrming similar observations on image classiﬁcation by Sener and Savarese (2017); (ii) AQ provides a dramatic improvement over baseline.

Original: The advantage persists throughout training.
Revised: The advantage persists throughout training.

Original: These learners sample examples randomly and label to completion (until an exact label is produced) before moving on, differing only in how efﬁciently they annotate data.
Revised: These learners sample examples randomly and label to completion (until an exact label is produced) before moving on, differing only in how efﬁciently they annotate data.

Original: (iii)
Revised: (iii)

Original: On Tiny ImageNet, at 30% of budget, ALPF-ERC outperforms AQ methods by 4.5% and outperforms the i.i.d.
Revised: On Tiny ImageNet, at 30% of budget, ALPF-ERC outperforms AQ methods by 4.5% and outperforms the i.i.d.

Original: baseline by 8.1%.
Revised: baseline by 8.1%.

Original: [Figure]

Original: [Figure]

Original: 3. D IAGNOSTIC ANALYSES
Revised: 3. D IAGNOSTIC ANALYSES

Original: First, we study how different amounts of warm-starting affects ALPF learners’ performance with a small set of i.i [Figure]
Revised: First, we study how different amounts of warm-starting affects ALPF learners’ performance with a small set of i.i.d.

Original: d. labels.
Revised: labels.

Original: Second, we compare the selections due to ERC and EDC to those produced through uncertainty sampling.
Revised: Second, we compare the selections due to ERC and EDC to those produced through uncertainty sampling.

Original: Third, we note that while EDC and ERC appear to perform best on our problems, they may be vulnerable to excessively focusing on classes that are trivial to recognize.
Revised: Third, we note that while EDC and ERC appear to perform best on our problems, they may be vulnerable to excessively focusing on classes that are trivial to recognize.

Original: We examine this setting via an adversarial dataset intended to break the heuristics.
Revised: We examine this setting via an adversarial dataset intended to break the heuristics.

Original: Warm-starting We compare the performance of each strategy under different percentages (0%, 5%, and 10%) of pre-labeled i.i.d.
Revised: Warm-starting We compare the performance of each strategy under different percentages (0%, 5%, and 10%) of pre-labeled i.i.d.

Original: data (Figure 5, Appendix A).
Revised: data (Figure 5, Appendix A).

Original: Results show that ERC works properly even without warm-starting, while EIG beneﬁts from a 5% warm-start and EDC suffers badly without warm-starting.
Revised: Results show that ERC works properly even without warm-starting, while EIG beneﬁts from a 5% warm-start and EDC suffers badly without warm-starting.

Original: We observe that 10% warm-starting yields no further improvement.
Revised: We observe that 10% warm-starting yields no further improvement.

Original: Sample uncertainty Classic uncertainty sampling chooses data of high uncertainty.
Revised: Sample uncertainty Classic uncertainty sampling chooses data of high uncertainty.

Original: This question is worth re-examining in the context of ALPF.
Revised: This question is worth re-examining in the context of ALPF.

Original: To analyze the behavior of ALPF learners vis-a-vis uncertainty we plot average prediction entropy of sampled data for ALPF learners with different sampling strategies (Figure 3).
Revised: To analyze the behavior of ALPF learners vis-a-vis uncertainty we plot average prediction entropy of sampled data for ALPF learners with different sampling strategies (Figure 3).

Original: Note that ALPF learners using EIG pick high-entropy data, while ALPF learners with EDC and
Revised: Note that ALPF learners using EIG pick high-entropy data, while ALPF learners with EDC and

Original: ERC choose examples with lower entropy predictions.
Revised: ERC choose examples with lower entropy predictions.

Original: The (perhaps) surprising performance of EDC and
Revised: The (perhaps) surprising performance of EDC and

Original: ERC may owe to the cost structure of ALPF.
Revised: ERC may owe to the cost structure of ALPF.

Original: While labels for examples with low-entropy predictions confer less information, they also come at lower cost.
Revised: While labels for examples with low-entropy predictions confer less information, they also come at lower cost.

Original: Adversarial setting Because ERC goes after “easy” examples, we test its behavior on a simulated dataset where 2 of the CIFAR10 classes (randomly chosen) are trivially easy.
Revised: Adversarial setting Because ERC goes after “easy” examples, we test its behavior on a simulated dataset where 2 of the CIFAR10 classes (randomly chosen) are trivially easy.

Original: We set all pixels white for one class all pixels black for the other.
Revised: We set all pixels white for one class all pixels black for the other.

Original: We plot the label distribution among the selected data over rounds of selection in against that on the unperturbed CIFAR10 in Figure 4.
Revised: We plot the label distribution among the selected data over rounds of selection in against that on the unperturbed CIFAR10 in Figure 4.

Original: As we can see, in the normal case, EIG splits its budget among all classes roughly evenly while EDC and ERC focus more on different classes at different stages.
Revised: As we can see, in the normal case, EIG splits its budget among all classes roughly evenly while EDC and ERC focus more on different classes at different stages.

Original: In the adversarial case, EIG quickly learns the easy classes, thereafter focusing on the others until they are exhausted, while EDC and ERC concentrate on exhausting the easy ones ﬁrst.
Revised: In the adversarial case, EIG quickly learns the easy classes, thereafter focusing on the others until they are exhausted, while EDC and ERC concentrate on exhausting the easy ones ﬁrst.

Original: Although EDC and
Revised: Although EDC and

Original: ERC still manage to label all data with less total cost than EIG, this behavior might cost us when we have trivial classes, especially when our unlabeled dataset is enormous relative to our budget.
Revised: ERC still manage to label all data with less total cost than EIG, this behavior might cost us when we have trivial classes, especially when our unlabeled dataset is enormous relative to our budget.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Binary identiﬁcation: Efﬁciently ﬁnding answers with yes/no questions is a classic problem (Garey, 1972) dubbed binary identiﬁcation .
Revised: Binary identiﬁcation: Efﬁciently ﬁnding answers with yes/no questions is a classic problem (Garey, 1972) dubbed binary identiﬁcation .

Original: Hyaﬁl & Rivest (1976) proved that ﬁnding the optimal strategy given an arbitrary set of binary tests is NP-complete.
Revised: Hyaﬁl and Rivest (1976) proved that ﬁnding the optimal strategy given an arbitrary set of binary tests is NP-complete.

Original: A well-known greedy algorithm called binary splitting (Garey & Graham, 1974; Loveland, 1985), chooses questions that most evenly split the probability mass.
Revised: A well-known greedy algorithm called binary splitting (Garey and Graham, 1974; Loveland, 1985), chooses questions that most evenly split the probability mass.

Original: Active learning: Our work builds upon the AL framework (Box & Draper, 1987; Cohn et al., 1996;Settles, 2010)
Revised: Active learning: Our work builds upon the AL framework (Box and Draper, 1987; Cohn et al., 1996;

Original:  (vs. i.i.d labeling).
Revised: Settles, 2010) (vs. i.i.d labeling).

Original: Classical AL methods select examples for which the current predictor is  most uncertain, according to various notions of uncertainty:
Revised: Classical AL methods select examples for which the current predictor is [Table] most uncertain, according to various notions of uncertainty:

Original: Dagan & Engelson (1995) selects examples with maximum entropy (ME) predictive distributions, while Culotta & McCallum (2005) uses the least conﬁdence (LC) heuristic, sorting examples in ascending order by the probability assigned to the argmax.
Revised: Dagan and Engelson (1995) selects examples with maximum entropy (ME) predictive distributions, while Culotta and McCallum (2005) uses the least conﬁdence (LC) heuristic, sorting examples in ascending order by the probability assigned to the argmax.

Original: Settles et al.
Revised: Settles et al.

Original: (2008) notes that annotation costs may vary across data points suggesting cost-aware sampling heuristics but doesn’t address the setting when costs change dynamically during training as a classiﬁer grows stronger.
Revised: (2008) notes that annotation costs may vary across data points suggesting cost-aware sampling heuristics but doesn’t address the setting when costs change dynamically during training as a classiﬁer grows stronger.

Original: Luo et al.
Revised: Luo et al.

Original: (2013) incorporates structure among outputs into an active learning scheme in the context of structured prediction.
Revised: (2013) incorporates structure among outputs into an active learning scheme in the context of structured prediction.

Original: Mo et al.
Revised: Mo et al.

Original: (2016) addresses hierarchical label structure in active learning interestingly in a setting where subclasses are easier to learn.
Revised: (2016) addresses hierarchical label structure in active learning interestingly in a setting where subclasses are easier to learn.

Original: Thus they query classes more ﬁne-grained than the targets, while we solicit feedback on more general categories.
Revised: Thus they query classes more ﬁne-grained than the targets, while we solicit feedback on more general categories.

Original:  Deep Active Learning Deep Active Learning (DAL) has recently emerged as an active research area.
Revised: [Figure] [Figure] Deep Active Learning Deep Active Learning (DAL) has recently emerged as an active research area.

Original: Wang et al.
Revised: Wang et al.

Original: (2016) explores a scheme that combines traditional heuristics with pseudo-labeling.
Revised: (2016) explores a scheme that combines traditional heuristics with pseudo-labeling.

Original: Gal et al.
Revised: Gal et al.

Original: (2017) notes that the softmax outputs of neural networks do not capture epistemic uncertainty (Kendall & Gal, 2017), proposing instead to use Monte Carlo samples from a dropout-regularized neural network to produce uncertainty estimates.
Revised: (2017) notes that the softmax outputs of neural networks do not capture epistemic uncertainty (Kendall and Gal, 2017), proposing instead to use Monte Carlo samples from a dropout-regularized neural network to produce uncertainty estimates.

Original: DAL has demonstrated success on NLP tasks.
Revised: DAL has demonstrated success on NLP tasks.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017) explores AL for sentiment classiﬁcation, proposing a new sampling heuristic, choosing examples for which the expected update to the word embeddings is largest.
Revised: (2017) explores AL for sentiment classiﬁcation, proposing a new sampling heuristic, choosing examples for which the expected update to the word embeddings is largest.

Original: Recently, Shen et al.
Revised: Recently, Shen et al.

Original: (2018) matched state of the art performance on named entity recognition, using just 25% of the training data.
Revised: (2018) matched state of the art performance on named entity recognition, using just 25% of the training data.

Original: Kampffmeyer et al.
Revised: Kampffmeyer et al.

Original: (2016) and Kendall et al.
Revised: (2016) and Kendall et al.

Original: (2015) explore other measures of uncertainty over neural network predictions.
Revised: (2015) explore other measures of uncertainty over neural network predictions.

Original: Learning from partial labels Many papers on learning from partial labels (Grandvalet & Bengio, 2004;
Revised: Learning from partial labels Many papers on learning from partial labels (Grandvalet and Bengio, 2004;

Original: Nguyen & Caruana, 2008;
Revised: Nguyen and Caruana, 2008;

Original: Cour et al., 2011) assume that partial labels are given a priori and ﬁxed.
Revised: Cour et al., 2011) assume that partial labels are given a priori and ﬁxed.

Original: Grandvalet & Bengio (2004) formalizes the partial labeling problem in the probabilistic framework and proposes a minimum entropy based solution.
Revised: Grandvalet and Bengio (2004) formalizes the partial labeling problem in the probabilistic framework and proposes a minimum entropy based solution.

Original: Nguyen & Caruana (2008) proposes an efﬁcient algorithm to learn classiﬁers from partial labels within the max-margin framework.
Revised: Nguyen and Caruana (2008) proposes an efﬁcient algorithm to learn classiﬁers from partial labels within the max-margin framework.

Original: Cour et al.
Revised: Cour et al.

Original: (2011) addresses desirable properties of partial labels that allow us to learn from them effectively.
Revised: (2011) addresses desirable properties of partial labels that allow learning from them effectively.

Original: While these papers assume a ﬁxed set of partial labels, we actively solicit partial feedback.
Revised: While these papers assume a ﬁxed set of partial labels, we actively solicit partial feedback.

Original: This presents new algorithmic challenges: (i) the partial labels for each data point changes across training rounds; (ii) the partial labels result from active selection, which introduces bias; and (iii) our problem setup requires a sampling strategy to choose questions.
Revised: This presents new algorithmic challenges: (i) the partial labels for each data point changes across training rounds; (ii) the partial labels result from active selection, which introduces bias; and (iii) our problem setup requires a sampling strategy to choose questions.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our experiments validate the active learning with partial feedback framework on large-scale classiﬁcation benchmarks.
Revised: Our experiments validate the active learning with partial feedback framework on large-scale classiﬁcation benchmarks.

Original: The best among our proposed ALPF learners fully labels the data with 42% fewer binary questions as compared to traditional active learners.
Revised: The best among our proposed ALPF learners fully labels the data with 42% fewer binary questions as compared to traditional active learners.

Original: Our diagnostic analysis suggests that in ALPF, it’s sometimes more efﬁcient to start with “easier” examples that can be cheaply annotated rather than with “harder” data as often suggested by traditional active learning.
Revised: Our diagnostic analysis suggests that in ALPF, it’s sometimes more efﬁcient to start with “easier” examples that can be cheaply annotated rather than with “harder” data as often suggested by traditional active learning.

Original: A W ARM 
Revised: A W ARM - STARTING P LOT

Original: [Figure]

Original: Figure 5 compares our strategies under various amounts of warm-starting with pre-labeled i.i.d. data.
Revised: Figure 5 compares our strategies under various amounts of warm-starting with pre-labeled i.i.d. data.

Original: [Figure]
Revised: [Figure]

Original: B U PDATING MODELS MORE FREQUENTLY
Revised: B U PDATING MODELS MORE FREQUENTLY

Original: On Tiny ImageNet, we normally re-initialize and train models from scratch for 75 epochs after every 30K questions.
Revised: On Tiny ImageNet, we normally re-initialize and train models from scratch for 75 epochs after every 30K questions.

Original: Since we found re-initialization is crucial for good performance, to ensure a fair comparison, we keep the same re-initialization frequency (i.e. every 30K questions) while updating the model by ﬁne-tuning 5 epochs after every 3K questions.
Revised: Since we found re-initialization is crucial for good performance, to ensure a fair comparison, we keep the same re-initialization frequency (i [Figure] e. every 30K questions) while updating the model by ﬁne-tuning 5 epochs after every 3K questions.

Original: This results in 10X faster model updating frequency.
Revised: This results in 10X faster model updating frequency.

Original: As in Figureand Table 3, results show only ALPF-EDC and ALPF-ERC seem to beneﬁt from updating 10 times more frequently 
Revised: As in Figureand Table 3, results show only ALPF-EDC and ALPF-ERC seem to beneﬁt from updating 10 times more frequently [Table] [Table]

Original: [Table]
Revised: [Table]

Original: - 491k
Revised: - 491k

Original: ALPF [Table]
Revised: [Table] [Table]

Original: - 581k
Revised: - 581k

Original: ALPF [Table]
Revised: [Table] [Table]

Original: - 521k
Revised: - 521k

Original: ALPF
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: - 521k
Revised: - 521k

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: EIG
Revised: - (EIG

Original: [Figure]
Revised: [Figure]

Original: EDC
Revised: EDC

Original: [Figure]
Revised: [Figure]

Original: ERC
Revised: ERC

Original: [Figure]
Revised: [Figure]