Original: Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning
Revised: Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Unlabeled examples awaiting annotations contain open-set noise inevitably.
Revised: Unlabeled examples awaiting annotations contain open-set noise inevitably.

Original: A fewactive learning studies have attempted to deal with this open-set noise in activelearning by filtering out the noisy examples.
Revised: A fewactive learning studies have attempted to deal with this open-set noise in activelearning by filtering out the noisy examples.

Original: However, because focusing on thepurity of examples in a query set leads to overlooking the informativeness of theexamples, the best balancing of purity and informativeness remains as an importantquestion.
Revised: However, because focusing on thepurity of examples in a query set leads to overlooking the informativeness of theexamples, the best balancing of purity and informativeness remains as an importantquestion.

Original: In this paper, to solve this purity-informativeness dilemma in open setactive learning, we propose a novel Meta-Query-Net (MQ-Net) that adaptivelyfinds the best balancing between the two factors.
Revised: In this paper, to solve this purity-informativeness dilemma in open setactive learning, we propose a novel Meta-Query-Net (MQ-Net) that adaptivelyfinds the best balancing between the two factors.

Original: Specifically, by leveragingthe multi-round property of active learning, we train MQ-Net using a query setwithout an additional validation set.
Revised: Specifically, by leveragingthe multi-round property of active learning, we train MQ-Net using a query setwithout an additional validation set.

Original: Furthermore, a clear dominance relationshipbetween unlabeled examples is effectively captured by MQ-Net through a novelskyline regularization.
Revised: Furthermore, a clear dominance relationshipbetween unlabeled examples is effectively captured by MQ-Net through a novelskyline regularization.

Original: Extensive experiments on multiple open-set active learningscenarios demonstrate that the proposed MQ-Net achieves 20.14 % improvement interms of accuracy, compared with the state-of-the-art methods.
Revised: Extensive experiments on multiple open-set active learningscenarios demonstrate that the proposed MQ-Net achieves 20.14 % improvement interms of accuracy, compared with the state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: The success of deep learning in many complex tasks highly depends on the availability of massive datawith well-annotated labels, which are often very costly to obtain in practice [1].
Revised: The success of deep learning in many complex tasks highly depends on the availability of massive datawith well-annotated labels, which are often very costly to obtain in practice [1].

Original: Active learning (AL)is one of the popular learning frameworks to reduce the high human-labeling cost, where a smallnumber of maximally-informative examples are selected by a query strategy and labeled by an oraclerepeatedly [2].
Revised: Active learning (AL)is one of the popular learning frameworks to reduce the high human-labeling cost, where a smallnumber of maximally-informative examples are selected by a query strategy and labeled by an oraclerepeatedly [2].

Original: Numerous query ( i e. , sample selection) strategies, mainly categorized into uncertaintybased sampling [3, 4, 5] and diversity -based sampling [6, 7, 8], have succeeded in effectively reducingthe labeling cost while achieving high performance.
Revised: Numerous query ( i e. , sample selection) strategies, mainly categorized into uncertaintybased sampling [3, 4, 5] and diversity -based sampling [6, 7, 8], have succeeded in effectively reducingthe labeling cost while achieving high performance.

Original: Despite their success, most standard AL approaches rely on a strict assumption that all unlabeledexamples should be cleanly collected from a pre-defined domain called in-distribution (IN) , evenbefore being labeled [9].
Revised: Despite their success, most standard AL approaches rely on a strict assumption that all unlabeledexamples should be cleanly collected from a pre-defined domain called in-distribution (IN) , evenbefore being labeled [9].

Original: This assumption is unrealistic in practice since the unlabeled examplesare mostly collected from a rather casual data curation processes such as web-crawling.
Revised: This assumption is unrealistic in practice since the unlabeled examplesare mostly collected from a rather casual data curation processes such as web-crawling.

Original: Notably,in Google search engine, the precision of image retrieval is reported to be 82 % on average, and itis worsened to 48 % for unpopular entities [10, 11].
Revised: Notably,in Google search engine, the precision of image retrieval is reported to be 82 % on average, and itis worsened to 48 % for unpopular entities [10, 11].

Original: That is, such collected unlabeled data naturallyinvolves open-set noise , which is defined as a set of the examples collected from different domainscalled out-of-distribution (OOD) [12].
Revised: That is, such collected unlabeled data naturallyinvolves open-set noise , which is defined as a set of the examples collected from different domainscalled out-of-distribution (OOD) [12].

Original: In general, standard AL approaches favor the examples either highly uncertain in predictions or highlydiverse in representations as a query for labeling.
Revised: In general, standard AL approaches favor the examples either highly uncertain in predictions or highlydiverse in representations as a query for labeling.

Original: However, the addition of open-set noise makes thesetwo measures fail to identify informative examples; the OOD examples also exhibit high uncertaintyand diversity because they share neither class-distinctive features nor other inductive biases with INexamples [14, 15].
Revised: However, the addition of open-set noise makes thesetwo measures fail to identify informative examples; the OOD examples also exhibit high uncertaintyand diversity because they share neither class-distinctive features nor other inductive biases with INexamples [14, 15].

Original: As a result, an active learner is confused and likely to query the OOD examples toa human-annotator for labeling.
Revised: As a result, an active learner is confused and likely to query the OOD examples toa human-annotator for labeling.

Original: Human annotators would disregard the OOD examples because they
Revised: Human annotators would disregard the OOD examples because they

Original: [Figure]
Revised: [Figure]

Revised: Unlabeled Examples ( )

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 10 Informativeness

Original: Rounds P u r it y Unlabeled Examples ( ) 65% 55% 45% 35%
Revised: [Figure] [Figure]

Original: [-HP-HI 1 
-]
Revised: [-HP-HI 1 
-]HI

Original: [Figure]
Revised: [Figure]

Original: 10 Rounds

Original: HP-dominance HI-dominance
Revised: HP [Figure] dominance HI

Original: HP-dominance
Revised: [Figure]

Original: HI-dom 65% 55% 45% 35% 
Revised: dom 55% 45% 35% 25%

Original: LL (HI-focused)
Revised: LL (HI [Figure] focused)

Original: CCAL (HP-focused)
Revised: CCAL (HP-focused)

Original: MQ-Net LP
Revised: MQ

Original: [Figure]
Revised: [Figure]

Original: HI Low
Revised: Net

Original: High ?
Revised: [Figure]

Original: HP-LI LP-LI
Revised: [Figure]

Original: IN,
Revised: IN,

Original: OOD
Revised: OOD

Original: [Figure]
Revised: [Figure]

Original: are unnecessary for the target task, thereby wasting the labeling budget.
Revised: are unnecessary for the target task, thereby wasting the labeling budget.

Original: Therefore, the problem ofactive learning with open-set noise, which we call open-set active learning , has emerged as a newimportant challenge for real-world applications.
Revised: Therefore, the problem ofactive learning with open-set noise, which we call open-set active learning , has emerged as a newimportant challenge for real-world applications.

Original: Recently, a few studies have attempted to deal with the open-set noise for active learning [13, 16].
Revised: Recently, a few studies have attempted to deal with the open-set noise for active learning [13, 16].

Original: They commonly try to increase the purity of examples in a query set, which is defined as the proportionof IN examples, by effectively filtering out the OOD examples.
Revised: They commonly try to increase the purity of examples in a query set, which is defined as the proportionof IN examples, by effectively filtering out the OOD examples.

Original: However, whether focusing on thepurity is needed throughout the entire training period remains a question.
Revised: However, whether focusing on thepurity is needed throughout the entire training period remains a question.

Original: In Figure 1(a), let’s considerthe examples of high purity and low informativeness (HP-LI) and those of low purity and highinformativeness (LP-HI).
Revised: In Figure 1(a), let’s considerthe examples of high purity and low informativeness (HP-LI) and those of low purity and highinformativeness (LP-HI).

Original: The dominance between these two groups of examples in usefulness foropen-set AL is not clear but is determined by the learning stage as well as the ratio of OOD examples.
Revised: The dominance between these two groups of examples in usefulness foropen-set AL is not clear but is determined by the learning stage as well as the ratio of OOD examples.

Original: Thus, we coin a new term “purity-informativeness dilemma” to call attention to the best balancing ofpurity and informativeness.
Revised: Thus, we coin a new term “purity-informativeness dilemma” to call attention to the best balancing ofpurity and informativeness.

Original: Figures 1(b) and 1(c) illustrate the purity-informativeness dilemma.
Revised: Figures 1(b) and 1(c) illustrate the purity-informativeness dilemma.

Original: The standard AL approach,
Revised: The standard AL approach,

Original: LL[5], puts more weight on the examples of high informativeness (denoted as HI-focused), whilethe existing open-set AL approach, CCAL [13], puts more weight on those of high purity (denotedas HP-focused).
Revised: LL[5], puts more weight on the examples of high informativeness (denoted as HI-focused), whilethe existing open-set AL approach, CCAL [13], puts more weight on those of high purity (denotedas HP-focused).

Original: The HP-focused approach improves the test accuracy more significantly than the
Revised: The HP-focused approach improves the test accuracy more significantly than the

Original: HP-focused one at earlier AL rounds, meaning that pure as well as easy examples are more beneficial.
Revised: HI-focused one at earlier AL rounds, meaning that pure as well as easy examples are more beneficial.

Original: In contrast, the HI-focused approach outperforms the HP-focused one at later AL rounds, meaningthat highly informative examples should be selected even at the expense of purity.
Revised: In contrast, the HI-focused approach outperforms the HP-focused one at later AL rounds, meaningthat highly informative examples should be selected even at the expense of purity.

Original: Furthermore,comparing a low OOD (noise) ratio in Figure 1(b) and a high OOD ratio in Figure 1(c), the shiftfrom HP-dominance to HI-dominance tends to occur later at a higher OOD ratio, which renders thisdilemma more difficult.
Revised: Furthermore,comparing a low OOD (noise) ratio in Figure 1(b) and a high OOD ratio in Figure 1(c), the shiftfrom HP-dominance to HI-dominance tends to occur later at a higher OOD ratio, which renders thisdilemma more difficult.

Original: In this paper, to solve the purity-informativeness dilemma in open-set AL, we propose a novel metamodel Meta-Query-Net (MQ-Net) that adaptively finds the best balancing between the two factors.
Revised: In this paper, to solve the purity-informativeness dilemma in open-set AL, we propose a novel metamodel Meta-Query-Net (MQ-Net) that adaptively finds the best balancing between the two factors.

Original: Akey challenge is the best balancing is unknown in advance.
Revised: Akey challenge is the best balancing is unknown in advance.

Original: The meta-model is trained to assign higherpriority for in-distribution examples over OOD examples as well as for more informative examplesamong in-distribution ones.
Revised: The meta-model is trained to assign higherpriority for in-distribution examples over OOD examples as well as for more informative examplesamong in-distribution ones.

Original: The input to the meta-model, which includes the target and OOD labels,is obtained for free from each AL round’s query set by leveraging the multi-round property of AL.
Revised: The input to the meta-model, which includes the target and OOD labels,is obtained for free from each AL round’s query set by leveraging the multi-round property of AL.

Original: Moreover, the meta-model is optimized more stably through a novel regularization inspired by theskyline query [17, 18] popularly used in multi-objective optimization.
Revised: Moreover, the meta-model is optimized more stably through a novel regularization inspired by theskyline query [17, 18] popularly used in multi-objective optimization.

Original: As a result, MQ-Net can guidethe learning of the target model by providing the best balancing between purity and informativenessthroughout the entire training period.
Revised: As a result, MQ-Net can guidethe learning of the target model by providing the best balancing between purity and informativenessthroughout the entire training period.

Original: Overall, our main contributions are summarized as follows:
Revised: Overall, our main contributions are summarized as follows:

Original: We formulate the purity-informativeness dilemma , which hinders the usability of open-set AL inreal-world applications.
Revised: We formulate the purity-informativeness dilemma , which hinders the usability of open-set AL inreal-world applications.

Original: As our answer to the dilemma, we propose a novel AL framework, MQ-Net, which keeps findingthe best trade-off between purity and informativeness.
Revised: As our answer to the dilemma, we propose a novel AL framework, MQ-Net, which keeps findingthe best trade-off between purity and informativeness.

Original: Extensive experiments on CIFAR10, CIFAR100, and ImageNet show that MQ-Net improvesthe classification accuracy consistently when the OOD ratio changes from 10% to 60% by up to20 .
Revised: Extensive experiments on CIFAR10, CIFAR100, and ImageNet show that MQ-Net improvesthe classification accuracy consistently when the OOD ratio changes from 10% to 60% by up to20 .

Original:  Related Work2.1 Active Learning and Open-set Recognition
Revised:  Related Work2.1 Active Learning and Open-set Recognition

Original: Active Learning is a learning framework to reduce the human labeling cost by finding the mostinformative examples given unlabeled data [9].
Revised: Active Learning is a learning framework to reduce the human labeling cost by finding the mostinformative examples given unlabeled data [9].

Original: One representative directions is uncertainty-basedsampling.
Revised: One popular direction is uncertainty-based sampling.

Original: Conventional approaches have been developed based on the prediction probability, e.g. ,soft-max confidence [19, 3], margin [20], and entropy [21].
Revised: Typical approaches have exploited prediction probability, e.g. , soft-max confidence [19, 3], margin [20], and entropy [21].

Original: Another approaches have exploitedthe components of DNNs for estimating uncertainty.

Original: Some of them obtain uncertainty byusing Monte Carlo Dropout throughout multiple forward passes [22, 23, 24].
Revised: Some approaches obtain uncertainty by Monte Carlo Dropout on multipleforward passes [22, 23, 24].

Original: LL [5] predicts the loss ofunlabeled examples by jointly learning asmall loss prediction module with target model.
Revised: LL [5] predicts the loss of examples by jointly learning a loss predictionmodule with target model.

Original: Meanwhile,diversity-based sampling has been widely studied.
Revised: Meanwhile, diversity-based sampling has been widely studied.

Original: To incorporate diversity, most approaches usea clusteringalgorithm [6, 25] or core-set selection algorithm 
Revised: To incorporate diversity, most methods use a clustering [6, 25] or core-set selection algorithm [7].

Original: Notably, CoreSet [7] finds the setof examples having the highest distance coverage on the entire unlabeled data.
Revised: Notably, CoreSet [7] finds the set of examples having the highest distance coverage on the entire unlabeleddata.

Original: BADGE [8] is ahybrid of uncertainty-and diversity-based sampling which uses k-MEANS clustering in the gradientembedding space.
Revised: BADGE [8] is a hybrid of uncertainty-and diversity-based sampling which uses k-MEANS++clustering in the gradient embedding space.

Original: However, this family of approaches is not appropriate for open-set AL since theydo not consider how to handle the OOD examples for query selection.
Revised: However, this family of approaches is not appropriatefor open-set AL since they do not consider how to handle the OOD examples for query selection.

Original: Open-set Recognition (OSR) is a detection task to recognize the examples outside of the targetdomain [12].
Revised: Open-set Recognition (OSR) is a detection task to recognize the examples outside of the targetdomain [12].

Original: Closely related to this purpose, OOD detection has been actively studied [26].
Revised: Closely related to this purpose, OOD detection has been actively studied [26].

Original: Recentwork can be categorized as: classifier-dependent, density-based, and self-supervised approaches.
Revised: Recentwork can be categorized as: classifier-dependent, density-based, and self-supervised approaches.

Original: The classifier-dependent approach leverages a pre-trained classifier and introduces several scoringfunctions, such as Uncertainty [27], ODIN [28], mahalanobis distance (MD) [29], and Energy[30].
Revised: The classifier-dependent approach leverages a pre-trained classifier and introduces several scoringfunctions, such as Uncertainty [27], ODIN [28], mahalanobis distance (MD) [29], and Energy[30].

Original: Recently, ReAct [31] shows that rectifying penultimate activations can enhance most of the aforementioned classifier-dependent OOD scores.
Revised: Recently, ReAct [31] shows that rectifying penultimate activations can enhance most of the aforementioned classifier-dependent OOD scores.

Original: The density-based approach learns an auxiliary generativemodel like variational auto-encoder to compute likelihood-based OOD scores [32, 33, 34].
Revised: The density-based approach learns an auxiliary generativemodel like variational auto-encoder to compute likelihood-based OOD scores [32, 33, 34].

Original: In contrast,most self-supervised approaches leverage contrastive learning [35, 36, 37].
Revised: In contrast,most self-supervised approaches leverage contrastive learning [35, 36, 37].

Original: CSI shows that contrastingwith distributionally-shifted augmentations can considerably enhance the OSR performance [35].
Revised: CSI shows that contrastingwith distributionally-shifted augmentations can considerably enhance the OSR performance [35].

Original: The OSR performance of classifier-dependent approaches degrades significantly if the classifierperforms poorly [38].
Revised: The OSR performance of classifier-dependent approaches degrades significantly if the classifierperforms poorly [38].

Original: Similarly, the performance of density-based and self-supervised approachesheavily resorts to the amount of clean IN data [34, 35].
Revised: Similarly, the performance of density-based and self-supervised approachesheavily resorts to the amount of clean IN data [34, 35].

Original: Therefore, open-set active learning is achallenging problem to be resolved by simply applying OSR approaches since it is difficult to obtainhigh-quality classifiers and sufficient IN data at early AL rounds.
Revised: Therefore, open-set active learning is achallenging problem to be resolved by simply applying OSR approaches since it is difficult to obtainhigh-quality classifiers and sufficient IN data at early AL rounds.

Original: 2.2 Open-set Active learning
Revised: 2.2 Open-set Active learning

Original: Two recent approaches have attempted to deal with the open-set noise for active learning [13, 16].
Revised: Two recent approaches have attempted to handle the open-set noise for AL [13, 16].

Original: Bothapproaches attempt to increase purity in query selection by effectively filtering out the OOD examples.
Revised: Both approachestry to increase purity in query selection by effectively filtering out the OOD examples.

Original: CCAL [13] learns two contrastive coding models where each is for calculating informativeness or OODness of an example, andthen it combines two scoresinto one using a heuristic balancing rule.
Revised: CCAL [13]learns two contrastive coding models where each is for calculating informativeness or OODness of anexample, and it combines two scores using a heuristic balancing rule.

Original: SIMILAR [16] selects a pure and core set of examples that maximize the distancecoverage on theentire unlabeled data while minimizing the distance coverage to thealready identified OOD data.
Revised: SIMILAR [16] selects a pureand core set of examples that maximize the distance on the entire unlabeled data while minimizing thedistance to the identified OOD data.

Original: However, CCAL and SIMILAR are often worse than standard AL methods because they always putmore higher weights on purity, although informativeness should be emphasized when the open-setnoise ratio is small or in later AL rounds.
Revised: However, CCAL and SIMILAR are often worse than standard AL methods because they always put higher weights on purity, although informativeness should beemphasized when the open-set noise ratio is small or in later AL rounds.

Original: This calls for the development of a new solution to carefullyfind the best trade-off between purity and informativeness in open-set AL.
Revised: This calls for developing anew solution to carefully find the best trade-off between purity and informativeness in open-set AL.

Original:  Purity-Informativeness Dilemma in Open-set Active Learning
Revised:  Purity-Informativeness Dilemma in Open-set Active Learning

Original: In this section, we define the problem of open-set active learning and then introduce the purityinformativeness delimma in query selection.

Original: 3.1 Problem Statement
Revised: 3.1 Problem Statement

Original: Let D IN and D OOD be the IN and OOD data, where the example’s label of D OOD does not belongto any of the k known (pre-defined) labels Y =
Revised: Let D IN and D OOD be the IN and OOD data, where the example’s label of D OOD does not belongto any of the k known (pre-defined) labels Y =

Original: { y i } ki =1 .
Revised: { y i } ki =1 .

Original: Thus, an unlabeled set is a mixture of INand OOD examples, U =
Revised: Thus, an unlabeled set is a mixture of INand OOD examples, U =

Original: { X IN , X OOD } , i
Revised: { X IN , X OOD } , i.e. , X IN ∼ D IN and X OOD ∼ D OOD . In the open-set

Revised: AL, a human oracle is requested to assign a known label y to an IN example x ∈ X IN with a labelingcost c IN , while an OOD example x ∈ X OOD is marked as open-set noise with a labeling cost c OOD .

Original: AL imposes restrictions on the labeling budget b every round.
Revised: AL imposes restrictions on the labeling budget b every round.

Original: It starts with a small labeled set S L ,consisting of both labeled IN and OOD examples.
Revised: It starts with a small labeled set S L ,consisting of both labeled IN and OOD examples.

Original: The initial labeled set S L improves by addinga small but maximally-informative labeled query set S Q per round, i.e. , S L ← S L ∪ S Q , where thelabeling cost for S Q by the oracle does not exceed the labeling budget b .
Revised: The initial labeled set S L improves by addinga small but maximally-informative labeled query set S Q per round, i.e. , S L ← S L ∪ S Q , where thelabeling cost for S Q by the oracle does not exceed the labeling budget b .

Original: Hence, the goal of open-set
Revised: Hence, the goal of open-set

Original: AL is defined to construct the optimal query set S ∗ Q , minimizing the loss for the unseen targetINdata T IN ∼D IN , where T IN ∩ X IN = ∅ .
Revised: AL is defined to construct the optimal query set S ∗ Q , minimizing the loss for the unseen target IN data.

Original: The difference from standard AL is that the labeling costfor OOD examples is introduced, where the labeling budget is wasted when OOD examples aremisclassified as informative ones.
Revised: The difference from standard AL is that the labeling cost for OOD examples is introduced, where thelabeling budget is wasted when OOD examples are misclassified as informative ones.

Original: Formally, let C ( · ) be the labeling cost function for a given unlabeled set; then, each round of open-set
Revised: Formally, let C ( · ) be the labeling cost function for a given unlabeled set; then, each round of open-set

Original: AL is formulated to find the best query set S ∗ Q as
Revised: AL is formulated to find the best query set S ∗ Q as

Original: [Equation]
Revised: [Equation]

Original: where C ( S Q ) = (cid:88)
Revised: where C ( S Q ) = (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  Here, f ( · ; Θ S L ∪ S Q ) denotes the target model trained for only IN examples in S L ∪ S Q , and ℓ cls is acertain loss function for classification.
Revised: Q [Equation] [Equation] 131 Here, f ( · ; Θ S L ∪ S Q ) denotes the target model trained for only IN examples in S L ∪ S Q , and ℓ cls is acertain loss function for classification.

Original: For each AL round, all the examples in S ∗ Q are removed fromthe unlabeled set U and then added to the accumulated labeled set S L along with their labels.
Revised: For each AL round, all the examples in S ∗ Q are removed fromthe unlabeled set U and then added to the accumulated labeled set S L along with their labels.

Original: Thisprocedure repeats for the total number r of rounds.
Revised: Thisprocedure repeats for the total number r of rounds.

Original: 3.2 Purity-Informativeness Dilemma
Revised: 3.2 Purity-Informativeness Dilemma

Original: An ideal approach for open-set AL would be to increase both purity and informativeness of aquery set by completely suppressing selection of OOD examples and accurately querying the mostinformative examples among the remaining IN ones.
Revised: An ideal approach for open-set AL would be to increase both purity and informativeness of aquery set by completely suppressing selection of OOD examples and accurately querying the mostinformative examples among the remaining IN ones.

Original: However, the ideal approach is infeasiblebecause overly emphasizing purity in query selection does not promote example informativeness andvice versa .
Revised: However, the ideal approach is infeasiblebecause overly emphasizing purity in query selection does not promote example informativeness andvice versa .

Original: Specifically, OOD examples with low purity scores mostly exhibit high informativenessscores because they share neither class-distinctive features nor other inductive biases with the INexamples [14, 15].
Revised: Specifically, OOD examples with low purity scores mostly exhibit high informativenessscores because they share neither class-distinctive features nor other inductive biases with the INexamples [14, 15].

Original: We call this trade-off in query selection as the purity-informativeness dilemma ,which is our new finding expected to trigger a lot of subsequent work.
Revised: We call this trade-off in query selection as the purity-informativeness dilemma ,which is our new finding expected to trigger a lot of subsequent work.

Original: To address this dilemma, we need to consider the proper weights of a purity score and an informativescore when they are combined.
Revised: To address this dilemma, we need to consider the proper weights of a purity score and an informativescore when they are combined.

Original: Let P ( x ) be a purity score of an example x which can be measuredby any existing OOD scores, e.g. , negative energy [30], and I ( x ) be an informativeness score of anexample x from any standard AL strategies, e.g. , uncertainty [3] and diversity [25].
Revised: Let P ( x ) be a purity score of an example x which can be measuredby any existing OOD scores, e.g. , negative energy [30], and I ( x ) be an informativeness score of anexample x from any standard AL strategies, e.g. , uncertainty [3] and diversity [25].

Original: Next, supposingz x = ⟨P ( x ) , I ( x ) ⟩ is a tuple1 of available purity and informativeness scores for an example x .
Revised: Next, supposingz x = ⟨P ( x ) , I ( x ) ⟩ is a tuple of available purity and informativeness scores for an example x .

Original: Then,a score combination function Φ( z x ) , where z x = ⟨P ( x ) , I ( x ) ⟩ , is defined to return an overall scorethat indicates the necessity of x being included in the query set.
Revised: Then,a score combination function Φ( z x ) , where z x = ⟨P ( x ) , I ( x ) ⟩ , is defined to return an overall scorethat indicates the necessity of x being included in the query set.

Original: Given two unlabeled examples x i and x j , if P ( x i ) >
Revised: Given two unlabeled examples x i and x j , if P ( x i ) >

Original: P ( x j ) and I ( x i ) >
Revised: P ( x j ) and I ( x i ) >

Original: I ( x j ) , it is clear to favorx i over x j based on Φ( z x i ) >
Revised: I ( x j ) , it is clear to favorx i over x j based on Φ( z x i ) >

Original: Φ( z x j ) .
Revised: Φ( z x j ) .

Original: However, due to the purity-informativeness dilemma, if
Revised: However, due to the purity-informativeness dilemma, if

Original: P ( x i ) > P ( x j ) and I ( x i ) <
Revised: P ( x i ) > P ( x j ) and I ( x i ) <

Original: I ( x j ) or P ( x i ) <
Revised: I ( x j ) or P ( x i ) <

Original: P ( x j ) and I ( x i ) >
Revised: P ( x j ) and I ( x i ) >

Original: I ( x j ) , it is very challengingto determine the dominance between Φ( z x i ) and Φ( z x j ) .
Revised: I ( x j ) , it is very challengingto determine the dominance between Φ( z x i ) and Φ( z x j ) .

Original: In order to model Φ( · ) , we mainly focuson leveraging meta-learning , which is a more agnostic approach to resolve the dilemma other thanseveral heuristic approaches, such as linear combination and multiplication.
Revised: In order to model Φ( · ) , we mainly focuson leveraging meta-learning , which is a more agnostic approach to resolve the dilemma other thanseveral heuristic approaches, such as linear combination and multiplication.

Original:  Meta-Query-Net
Revised:  Meta-Query-Net

Original: We incorporates a meta-model, named Meta-Query-Net (MQ-Net) , which aims to learn a meta-scorefunction for the purpose of identifying the query set.
Revised: We incorporates a meta-model, named Meta-Query-Net (MQ-Net) , which aims to learn a meta-scorefunction for the purpose of identifying the query set.

Original: In the presence of open-set noise, MQ-Netoutputs the meta-score for unlabeled examples to achieve the best balance between purity andinformativeness in the selected query set.
Revised: In the presence of open-set noise, MQ-Netoutputs the meta-score for unlabeled examples to achieve the best balance between purity andinformativeness in the selected query set.

Original: In this section, we introduce the notion of a self-validationset to guide the meta-model in a supervised manner and then demonstrate the meta-objective of
Revised: In this section, we introduce the notion of a self-validationset to guide the meta-model in a supervised manner and then demonstrate the meta-objective of

Original: MQ-Net for training.
Revised: MQ-Net for training.

Original: Then, we propose a novel skyline constraint used in optimization, which helps
Revised: Then, we propose a novel skyline constraint used in optimization, which helps

Original: MQ-Net capture the obvious preference among unlabeled examples when a clear dominance exists.
Revised: MQ-Net capture the obvious preference among unlabeled examples when a clear dominance exists.

Original: Next, we present a way of converting the purity and informativeness scores estimated by existingmethods for use in MQ-Net.
Revised: Next, we present a way of converting the purity and informativeness scores estimated by existingmethods for use in MQ-Net.

Original: The overview of MQ-Net is illustrated in Figure 2.
Revised: The overview of MQ-Net is illustrated in Figure 2.

Original: 4.1 Training Objective with Self-validation Set
Revised: 4.1 Training Objective with Self-validation Set

Revised: The parameters w contained in MQ-Net Φ( · ; w ) is optimized in a supervised manner.

Revised: For a cleansupervision, a validation data is required for training.

Revised: Without assuming a hard-to-obtain cleanvalidation set, we propose to use a self-validation set, which is instantaneously generated every ALround.

Original:  Unlabeled 𝑈 = {𝑋 𝐼𝑁 ,𝑋 𝑂𝑂𝐷 }
Revised: In detail, we obtain a labeled query set S Q by the oracle, consisting of a labeled IN set and anidentified OOD set in every round. As the query set S Q is unseen for the target Θ and meta models wat the current round, we can exploit it as a self-validation set to train MQ-Net. This self-validation seteliminates the need for a clean validation set in meta-learning. Unlabeled 𝑈 = {𝑋 𝐼𝑁 ,𝑋 𝑂𝑂𝐷 }

Original: Query
Revised: Query

Original: MQ-Net Θ
Revised: MQ-Net Θ

Original: Labeled 𝐰 𝑆 𝑄 𝑆 𝐿
Revised: Labeled 𝐰 𝑆 𝑄 𝑆 𝐿

Original: Self-validation
Revised: Self-validation

Original: Update
Revised: Update

Original: Update 𝑃(𝑥) , 𝐼(𝑥) 𝐰≽
Revised: Update 𝑃(𝑥) , 𝐼(𝑥) 𝐰≽

Original: The parameters w contained in MQ-Net Φ( · ; w ) isoptimized in a supervised manner.

Original: For a clean supervision, a validation data is required for training.

Original: Without assuming a hard-to-obtain clean validationset, we propose to use a self-validation set, which isinstantaneously generated every AL round.

Original: In detail,we obtain a labeled query set S Q by the oracle, consisting of a labeled IN set and an identified OOD setin every round.

Original: As the query set S Q is unseen for thetarget Θ and meta models w at the current round, we can exploit it as a self-validation set to train

Original: MQ-Net.

Original: This self-validation set eliminates the need for a clean validation set in meta-learning.

Original: Given the ground-truth labels in the self-validation set, it is feasible to guide MQ-Net to be trained toresolve the purity-informativeness dilemma by designing an appropriate meta-objective.
Revised: Given the ground-truth labels in the self-validation set,it is feasible to guide MQ-Net to be trained to resolvethe purity-informativeness dilemma by designing anappropriate meta-objective.

Original: It is basedon a cross-entropy loss for classification because a loss value of training examples has been proven tobe effective in identifying high informativeness examples [5].
Revised: It is based on the crossentropy loss for classification because the loss valueof training examples has been proven to be effectivein identifying high informativeness examples [5].

Original: The conventional loss value by a targetmodel Θ is masked to be zero if x ∈ X OOD since OOD examples are useless for AL,
Revised: Theconventional loss value by a target model Θ is maskedto be zero if x ∈ X OOD since OOD examples areuseless for AL,

Original: [Equation]
Revised: [Equation]

Original: (2) where l is a true binary IN label, i.e. , 1 for IN examples and 0 for OOD examples, which can bereliably obtained from the self-validation set.
Revised: 2) where l is a true binary IN label, i.e. , 1 for IN examples and 0 for OOD examples, which can bereliably obtained from the self-validation set.

Original: This masked loss, ℓ mce , preserves the informativenessof IN examples while excluding OOD examples.
Revised: This masked loss, ℓ mce , preserves the informativenessof IN examples while excluding OOD examples.

Original: Given a self-validation data S Q , the meta-objectiveis defined such that MQ-Net parameterized by w outputs a high (or low) meta-score Φ( z x ; w ) if anexample x ’s masked loss value is large (or small), 
Revised: Given a self-validation data S Q , the meta-objectiveis defined such that MQ-Net parameterized by w outputs a high (or low) meta-score Φ( z x ; w ) if anexample x ’s masked loss value is large (or small), [Equation] max

Original: [Equation]
Revised: [Equation]

Original: s.t.
Revised: x [Equation] (cid:1) [Equation]

Original: ∀ x i , x j , if P ( x i ) >
Revised: (cid:0) [Equation] [Equation] w

Original: P ( x j ) and I ( x i ) >
Revised: [Equation] [Equation] w [Equation]

Original:  I ( x j ) , then Φ( z x i ; w ) > Φ( z x j ; w ) , (3) where η > 0 is a constant margin for the ranking loss, and Sign ( a, b ) is an indicator function thatreturns +1 if a > b , 0 if a = b , and − 1 otherwise.
Revised: (cid:1)(cid:17) s.t. ∀ x i , x j , if P ( x i ) > P ( x j ) and I ( x i ) > I ( x j ) , then Φ( z x i ; w ) > Φ( z x j ; w ) , (3) where η > 0 is a constant margin for the ranking loss, and Sign ( a, b ) is an indicator function thatreturns +1 if a > b , 0 if a = b , and − 1 otherwise.

Original: Hence, Φ( z x i ; w ) is forced to be higher thanΦ( z x j ; w ) if ℓ mce ( x i ) > ℓ mce ( x j ) ; in contrast, Φ( z x i ; w ) is forced to be lower than Φ( z x j ; w ) ifℓ mce ( x i ) <
Revised: Hence, Φ( z x i ; w ) is forced to be higher thanΦ( z x j ; w ) if ℓ mce ( x i ) > ℓ mce ( x j ) ; in contrast, Φ( z x i ; w ) is forced to be lower than Φ( z x j ; w ) ifℓ mce ( x i ) <

Original: ℓ mce ( x j ) .
Revised: ℓ mce ( x j ) .

Original: Two OOD examples do not affect the optimization because they do not haveany priority between them, i.e. , ℓ mce ( x i ) =
Revised: Two OOD examples do not affect the optimization because they do not haveany priority between them, i.e. , ℓ mce ( x i ) =

Original: ℓ mce ( x j ) .
Revised: ℓ mce ( x j ) .

Original: In addition to the ranking loss, we add a regularization term named the skyline constraint ( i.e. , thesecond line) in the meta-objective Eq.
Revised: In addition to the ranking loss, we add a regularization term named the skyline constraint ( i.e. , thesecond line) in the meta-objective Eq.

Original: (3), which is inspired by the skyline query which aims tonarrow down a search space in a large-scale database by keeping only those items that are not worsethan any other [17, 18].
Revised: (3), which is inspired by the skyline query which aims tonarrow down a search space in a large-scale database by keeping only those items that are not worsethan any other [17, 18].

Original: Specifically, in the case of P ( x i ) > P ( x j ) and I ( x i ) >
Revised: Specifically, in the case of P ( x i ) > P ( x j ) and I ( x i ) >

Original: I ( x j ) , the conditionΦ( z x i ; w ) > Φ( z x j ; w ) must always hold in our objective, and hence we make this proposition as theskyline constraint.
Revised: I ( x j ) , the conditionΦ( z x i ; w ) > Φ( z x j ; w ) must always hold in our objective, and hence we make this proposition as theskyline constraint.

Original: This simple yet intuitive regularization is very helpful for achieving a meta-modelthat better judges the importance of purity or informativeness.
Revised: This simple yet intuitive regularization is very helpful for achieving a meta-modelthat better judges the importance of purity or informativeness.

Original: We provide an ablation study on theskyline constraint in Section 5.4.
Revised: We provide an ablation study on theskyline constraint in Section 5.4.

Original: 4.2 Architecture of MQ-Net
Revised: 4.2 Architecture of MQ-Net

Original: MQ-Net is parameterized by a multi-layer perceptron (MLP), a widely-used deep learning architecturefor meta-learning [39].
Revised: MQ-Net is parameterized by a multi-layer perceptron (MLP), a widely-used deep learning architecturefor meta-learning [39].

Original: A challenge here is that the proposed skyline constraint in Eq.
Revised: A challenge here is that the proposed skyline constraint in Eq.

Original: (3) does not holdwith a standard MLP model.
Revised: (3) does not holdwith a standard MLP model.

Original: To satisfy the skyline constraint, the meta-score function Φ( · ; w ) shouldbe a monotonic non-decreasing function because the output (meta-score) of MQ-Net for an examplex i must be higher than that for another example x j if the two factors (purity and informativeness)of
Revised: To satisfy the skyline constraint, the meta-score function Φ( · ; w ) shouldbe a monotonic non-decreasing function because the output (meta-score) of MQ-Net for an examplex i must be higher than that for another example x j if the two factors (purity and informativeness)of

Original: x
Revised: x

Original: i are both higher than those of x j .
Revised: i are both higher than those of x j .

Original: The MLP model consists of multiple matrix multiplicationswith non-linear activation functions such as ReLU and Sigmoid.
Revised: The MLP model consists of multiple matrix multiplicationswith non-linear activation functions such as ReLU and Sigmoid.

Original: In order for the MLP model to bemonotonically non-decreasing, all the parameters in w for Φ( · ; w ) should be non-negative , as provenby Theorem 4.1.
Revised: In order for the MLP model to bemonotonically non-decreasing, all the parameters in w for Φ( · ; w ) should be non-negative , as provenby Theorem 4.1.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For any MLP meta-model w with  a meta-score function Φ( z ; w ): R d → R, the skylineconstraints holds  if w ⪰ 0 and z ( ∈ R d ) ⪰ 0 , where ⪰ is the component-wise inequality.
Revised: For any MLP meta-model w with non-decreasing activation functions, a meta-scorefunction Φ( z ; w ): R d → R holds the skyline constraints if w ⪰ 0 and z ( ∈ R d ) ⪰ 0 , where ⪰ is thecomponent-wise inequality.

Original: Proof.
Revised: Proof.

Original: An MLP model is involved with matrix multiplication and composition with activationfunctions, which are characterized by three basic operators: (1) addition : h ( z ) =
Revised: An MLP model is involved with matrix multiplication and composition with activationfunctions, which are characterized by three basic operators: (1) addition : h ( z ) =

Original: f ( z ) + g ( z ) , (2)multiplication : h ( z ) =
Revised: f ( z ) + g ( z ) , (2)multiplication : h ( z ) =

Original: f ( z ) × g ( z ) , and (3) composition : h ( z ) =
Revised: f ( z ) × g ( z ) , and (3) composition : h ( z ) =

Original: f ◦ g
Revised: f ◦ g

Original: ( z ) .
Revised: ( z ) .

Original: These three operatorsare guaranteed to be non-decreasing functions if the parameters of the MLP model are all nonnegative, because the non-negative weights guarantee all decomposed scalar operations in MLP to benon-decreasing functions.
Revised: These three operatorsare guaranteed to be non-decreasing functions if the parameters of the MLP model are all nonnegative, because the non-negative weights guarantee all decomposed scalar operations in MLP to benon-decreasing functions.

Original: Combining the three operators, the MLP model Φ( z ; w ) , where w ⪰ 0 ,naturally becomes a monotonic non-decreasing function for each input dimension.
Revised: Combining the three operators, the MLP model Φ( z ; w ) , where w ⪰ 0 ,naturally becomes a monotonic non-decreasing function for each input dimension.

Original: Refer to Appendix
Revised: Refer to Appendix

Original: A for the complete proof.
Revised: A for the complete proof.

Original: In implementation, non-negative weights are guaranteed by applying a ReLU function to meta-modelparameters.
Revised: In implementation, non-negative weights are guaranteed by applying a ReLU function to meta-modelparameters.

Original: Since the ReLU function is differentiable, MQ-Net can be trained with the proposedobjective in an end-to-end manner.
Revised: Since the ReLU function is differentiable, MQ-Net can be trained with the proposedobjective in an end-to-end manner.

Original: Putting this simple modification, the skyline constraint is preservedsuccessfully without introducing any complex loss-based regularization term.
Revised: Putting this simple modification, the skyline constraint is preservedsuccessfully without introducing any complex loss-based regularization term.

Original: The only remainingcondition is that each input of MQ-Net must be a vector of non-negative entries.
Revised: The only remainingcondition is that each input of MQ-Net must be a vector of non-negative entries.

Original: 4.3 Active Learning with MQ-Net4.3.1 Meta-input Conversion
Revised: 4.3 Active Learning with MQ-Net

Revised: 4.3.1 Meta-input Conversion

Original: MQ-Net receives z x = ⟨P ( x ) , I ( x ) ⟩ and then returns a meta-score for query selection.
Revised: MQ-Net receives z x = ⟨P ( x ) , I ( x ) ⟩ and then returns a meta-score for query selection.

Original: All the scoresfor the input of MQ-Net must be positive to preserve the skyline constraint, i.e. , z ⪰ 0 .
Revised: All the scoresfor the input of MQ-Net must be positive to preserve the skyline constraint, i.e. , z ⪰ 0 .

Original: Existing
Revised: Existing

Original: OOD and AL query scores are converted to the meta-input.
Revised: OOD and AL query scores are converted to the meta-input.

Original: The methods used for calculating thescores are orthogonal to our framework.
Revised: The methods used for calculating thescores are orthogonal to our framework.

Original: The OOD score O ( · ) is conceptually the opposite of purityand varies in its scale; hence, we convert it to a purity score by P ( x )
Revised: The OOD score O ( · ) is conceptually the opposite of purityand varies in its scale; hence, we convert it to a purity score by P ( x )

Original: =
Revised: =

Original: Exp(Normalize( −O ( x ))) ,where Normalize( · ) is the z-score normalization.
Revised: Exp(Normalize( −O ( x ))) ,where Normalize( · ) is the z-score normalization.

Revised: R4Q3:

Revised: For the z-score normalization, we computethe mean and standard deviation of each scalar score O ( x ) and Q ( x ) over the unlabeled examples.

Revised: The mean and standard deviation are iteratively computed before the meta-training, and used forthe z-score normalization at that round.

Original: This conversion guarantees the purity score to bepositive.
Revised: This conversion guarantees the purity score to be positive.

Original: Similarly, for the informativeness score, we convert an existing AL query score Q ( · ) to I ( x ) =
Revised: Similarly, for the informativeness score, we convert an existing AL query score Q ( · ) to I ( x ) =

Original: Exp(Normalize( Q ( x ))) .
Revised: Exp(Normalize( Q ( x ))) .

Original: 4.3.2 Overall Procedure
Revised: 4.3.2 Overall Procedure

Original: For each AL round, a target model is trained via stochastic gradient descent (SGD) on mini-batchessampled from the IN examples in the current labeled set S L .
Revised: For each AL round, a target model is trained via stochastic gradient descent (SGD) on mini-batchessampled from the IN examples in the current labeled set S L .

Original: Based on the current target model,the purity and informative scores are computed by using certain OOD and AL query scores.
Revised: Based on the current target model,the purity and informative scores are computed by using certain OOD and AL query scores.

Original: Thequerying phase is then performed by selecting the examples S Q with the highest meta-scores withinthe labeling budget b .
Revised: Thequerying phase is then performed by selecting the examples S Q with the highest meta-scores withinthe labeling budget b .

Original: The query set S Q is used as the self-validation set for training MQ-Net at thecurrent AL round.
Revised: The query set S Q is used as the self-validation set for training MQ-Net at thecurrent AL round.

Original: The trained MQ-Net is used at the next AL round.
Revised: The trained MQ-Net is used at the next AL round.

Original: The alternating procedureof updating the target model and the meta-model repeats for a given number r of AL rounds.
Revised: The alternating procedureof updating the target model and the meta-model repeats for a given number r of AL rounds.

Original: Thepseudocode of MQ-Net can be found in Appendix B.
Revised: Thepseudocode of MQ-Net can be found in Appendix B.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Experiment Setting
Revised: 5.1 Experiment Setting

Original: Datasets.
Revised: Datasets.

Original: We perform the active learning task on three benchmark datasets; CIFAR10 [40], CIFAR100 [40], and ImageNet [41].
Revised: We perform the active learning task on three benchmark datasets; CIFAR10 [40], CIFAR100 [40], and ImageNet [41].

Original: Following the ‘split-dataset’ setup in open-world learning literature [13, 16, 42], we divide each dataset into two subsets: (1) the target set with IN classes and (2)the noise set with OOD classes.
Revised: Following the ‘split-dataset’ setup in open-world learning literature [13, 16, 42], we divide each dataset into two subsets: (1) the target set with IN classes and (2)the noise set with OOD classes.

Original: Specifically, CIFAR10 is split into the target set with four classesand the noise set with the rest six classes; CIFAR100 into the two sets with 40 and 60 classes; and
Revised: Specifically, CIFAR10 is split into the target set with four classesand the noise set with the rest six classes; CIFAR100 into the two sets with 40 and 60 classes; and

Original: ImageNet into the two sets with 50 and 950 classes.
Revised: ImageNet into the two sets with 50 and 950 classes.

Original: The entire target set is used as the unlabeled INdata, while only a part of classes in the noise set is selected as the unlabeled OOD data accordingto the given noise ratio.
Revised: The entire target set is used as the unlabeled INdata, while only a part of classes in the noise set is selected as the unlabeled OOD data accordingto the given noise ratio.

Original: In addition, following ODD detection literature [27, 32], we also considerthe ‘cross-dataset’ setup, which mixes a certain dataset with two external OOD datasets collectedfrom different domains, such as LSUN [43] and Places365 [44].
Revised: In addition, following ODD detection literature [27, 32], we also considerthe ‘cross-dataset’ setup, which mixes a certain dataset with two external OOD datasets collectedfrom different domains, such as LSUN [43] and Places365 [44].

Original: For sake of space, we present all theresults on the cross-dataset setup in Appendix D.
Revised: For sake of space, we present all theresults on the cross-dataset setup in Appendix D.

Original: Algorithms.
Revised: Algorithms.

Original: We compare MQ-Net with  four standard AL and two recent open-set AL approaches.
Revised: We compare MQ-Net with a random selection, four standard AL and two recent open-set AL approaches.

Original: • Standard AL :
Revised: • Standard AL :

Original: The four methods perform AL without any processing for open-set noise: (1)CONF [3] queries the most uncertain examples with the lowest softmax confidence in the prediction,(2)
Revised: The four methods perform AL without any processing for open-set noise: (1)CONF [3] queries the most uncertain examples with the lowest softmax confidence in the prediction,(2)

Original: CORESET [7] queries the most diverse examples with the highest coverage in the representationspace, (3) LL[5] queries the exampleshaving the largest predicted loss by jointly learning a lossprediction module
Revised: CORESET [7] queries the most diverse examples with the highest coverage in the representationspace, (3) LL queries the examples

Original: and (4) BADGE [8] considers both uncertainty and diversity by querying themost representative examples in the gradient via k -means + clustering [45].
Revised: having

Original: [Figure]
Revised: the

Original: [Figure]
Revised: largest predicted

Original: [Figure] [Figure] [Figure]
Revised: loss by jointly learning a loss

Original:  [Figure]
Revised:  prediction

Original: [Figure] [Figure]
Revised: module, and (4) BADGE [8] considers both uncertainty and diversity by querying themost representative examples in the gradient via k -means + clustering [45]

Original: • Open-set AL :
Revised: • Open-set AL :

Original: The two methods tend to put more weight on the examples with high purity: (1)CCAL [13] learns two contrastive coding models for calculating informativeness and OODness,and then it combines the two scores into one using a heuristic balancing rule, and (2) SIMILAR [16] selects a pure and core set of examples that maximize the distance coverage on the entire unlabeleddata while minimizing the distance coverage to the already labeled OOD data. 
Revised: The two methods tend to put more weight on the examples with high purity: (1)CCAL [13] learns two contrastive coding models for calculating informativeness and OODness,and then it combines the two scores into one using a heuristic balancing rule, and (2) SIMILAR [16]selects a pure and core set of examples that maximize the distance coverage on the entire unlabeleddata while minimizing the distance coverage to the already labeled OOD data. [Figure] SIMILAR

Revised: OURS

Original:  For all the experiments, regarding the two inputs of MQ-Net, we mainly use CSI [35] and LL [5]for calculating the purity and informativeness scores, respectively.
Revised: [Figure] [Figure] [Figure] For all the experiments, regarding the two inputs of MQ-Net, we mainly use CSI [35] and LL [5]for calculating the purity and informativeness scores, respectively.

Original: For CSI, as in CCAL, we train acontrastive learner on the entire unlabeled set with open-set noise since the clean in-distribution set isnot available in open-set AL.
Revised: For CSI, as in CCAL, we train acontrastive learner on the entire unlabeled set with open-set noise since the clean in-distribution set isnot available in open-set AL.

Original: The ablation study in Section 5.4 shows that MQ-Net is also effectivewith other OOD and AL scores as its input.
Revised: The ablation study in Section 5.4 shows that MQ-Net is also effectivewith other OOD and AL scores as its input.

Original: Implementation Details.
Revised: Implementation Details.

Original: We repeat the three steps—training, querying, and labeling—of AL.
Revised: We repeat the three steps—training, querying, and labeling—of AL.

Original: Thetotal number r of rounds is set to 10.
Revised: Thetotal number r of rounds is set to 10.

Original: Following the prior open-set AL setup [13, 16], we set thelabeling cost c IN = 1 for IN examples and c OOD = 1 for OOD examples.
Revised: Following the prior open-set AL setup [13, 16], we set thelabeling cost c IN = 1 for IN examples and c OOD = 1 for OOD examples.

Original: For the class-splitsetup, the labeling budget b per round is set to 500 for CIFAR10/100 and 1 , 000 for ImageNet.
Revised: For the class-splitsetup, the labeling budget b per round is set to 500 for CIFAR10/100 and 1 , 000 for ImageNet.

Original: Regarding the open-set noise ratio τ , we configure five different levels from light to heavy noisein { 10% , 20% , 40% , 60% } .
Revised: Regarding the open-set noise ratio τ , we configure four different levels from light to heavy noisein { 10% , 20% , 40% , 60% } .

Original: In the case of τ = 0% (no noise), MQ-Net naturally discard the purityscore and only use the informativeness score for query selection, since the self-validation set does notcontain any OOD examples.
Revised: In the case of τ = 0% (no noise), MQ-Net naturally discard the purityscore and only use the informativeness score for query selection, since the self-validation set does notcontain any OOD examples.

Original: The initial labeled set is randomly selected uniformly at random fromthe entire unlabeled set within the labeling budget b . 
Revised: The initial labeled set is randomly selected uniformly at random fromthe entire unlabeled set within the labeling budget b . R1Q3:

Revised: As for the architecture of MQ-Net, weuse a 2-layer MLP with the hidden dimension size of 64 and the ReLU activation fuction.

Revised: R4Q1:

Original: {+All methods are implemented with PyTorch 1.8.0 and executed on a single NVIDIA Tesla V
+}
Revised: {+All methods are implemented with PyTorch 1.8.0 and executed on a single NVIDIA Tesla V
+}

Revised: GPU.

Original: We report the average results of five runs withdifferent class splits.
Revised: We report the average results of five runs with different class splits.

Original: See Appendix C for more implementation details with training configurations.
Revised: See Appendix C formore implementation details with training configurations.

Revised: R2Q3 & R4Q1:

Revised: The code is available athttps://anonymous.4open.science/r/MQNet-43E6/ .

Original: 5.2 Open-set Noise Robustness
Revised: 5.2 Open-set Noise Robustness

Original: 5.2.1 Results over 
Revised: 5.2.1 Results over AL Rounds

Original: [Table]

Original: Figure 3 illustrates the test accuracy of the target model over AL rounds on the two CIFAR datasets.
Revised: Figure 3 illustrates the test accuracy of the target model over AL rounds on the two CIFAR datasets.

Original: MQ-Net achieves the highest test accuracy in most AL rounds, thereby reaching the best test accuracyat the final round in every case for various datasets and noise ratios.
Revised: MQ-Net achieves the highest test accuracy in most AL rounds, thereby reaching the best test accuracyat the final round in every case for various datasets and noise ratios.

Original: Compared with the two existingopen-set AL methods, CCAL and SIMILAR, MQ-Net shows a steeper improvement in test accuracyover rounds by resolving the purity-informativeness dilemma in query selection.
Revised: Compared with the two existingopen-set AL methods, CCAL and SIMILAR, MQ-Net shows a steeper improvement in test accuracyover rounds by resolving the purity-informativeness dilemma in query selection.

Original: For example, theperformance gap between MQ-Net and the two open-set AL methods gets larger after the sixth round,as shown in Figure 3(b), because CCAL and SIMILAR mainly depend on purity in query selection,which conveys less informative information to the classifier.
Revised: For example, theperformance gap between MQ-Net and the two open-set AL methods gets larger after the sixth round,as shown in Figure 3(b), because CCAL and SIMILAR mainly depend on purity in query selection,which conveys less informative information to the classifier.

Original: For a better classifier, informative
Revised: For a better classifier, informative

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: 2.40 % improve over the least 3. P u r it y ( C S I)

Original: [Figure]
Revised: Informativeness (LL) Informativeness (LL) Informativeness (LL) Informativeness (LL)

Original: [Figure] 
Revised: [Figure] [Figure] Round Round

Original:  [Figure] 
Revised: Round [Figure] [Figure] P u r it y

Original:  [Figure] 
Revised: ( C S I) P u r it y [Figure] S I )

Revised: P u r it y

Original: [Figure]
Revised: [Figure]

Original:  [Figure]
Revised: S I ) [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] y (

Original: [Figure]
Revised: [Figure]

Revised: S I)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: examples should be favored at a later AL round due to the sufficient number of IN examples in thelabeled set.
Revised: examples should be favored at a later AL round due to the sufficient number of IN examples in thelabeled set.

Original: In contrast, MQ-Net keeps improving the test accuracy even in a later AL round byfinding the best balancing between purity and informativeness in its query set.
Revised: In contrast, MQ-Net keeps improving the test accuracy even in a later AL round byfinding the best balancing between purity and informativeness in its query set.

Original: More analysis of
Revised: More analysis of

Original: MQ-Net associated with the purity-informativeness dilemma is discussed in Section 5.3.
Revised: MQ-Net associated with the purity-informativeness dilemma is discussed in Section 5.3.

Original: 5.2.2 Results with Varying Noise Ratios
Revised: 5.2.2 Results with Varying Noise Ratios

Original: Table 1 summarizes the last test accuracy at the final AL round for three datasets with varying levelsof open-set noise.
Revised: Table 1 summarizes the last test accuracy at the final AL round for three datasets with varying levelsof open-set noise.

Original: Overall, the last test accuracy of MQ-Net is the best in every case.
Revised: Overall, the last test accuracy of MQ-Net is the best in every case.

Original: This superiorityconcludes that MQ-Net successfully find the best trade-off between purity and informativeness interms of AL accuracy regardless of the noise ratio.
Revised: This superiorityconcludes that MQ-Net successfully find the best trade-off between purity and informativeness interms of AL accuracy regardless of the noise ratio.

Original: In general, the performance improvement becomeslarger with the increase in the noise ratio.
Revised: In general, the performance improvement becomeslarger with the increase in the noise ratio.

Original: On the other hand, the two open-set AL approaches areeven worse than the four standard AL approaches when noise ratio is less than 20 % .
Revised: On the other hand, the two open-set AL approaches areeven worse than the four standard AL approaches when noise ratio is less than 20 % .

Original: This trendconfirms that increasing informativeness is more crucial than increasing purity when the noise ratio issmall.
Revised: This trendconfirms that increasing informativeness is more crucial than increasing purity when the noise ratio issmall.

Original: In addition, in CIFAR10 relatively easier than others, CCAL and SIMILAR are inferior to thenon-robust AL method, LL, even with 40 % noise.
Revised: In addition, in CIFAR10 relatively easier than others, CCAL and SIMILAR are inferior to thenon-robust AL method, LL, even with 40 % noise.

Original: Highly informative examples are still beneficialwhen the performance of a classifier is saturated in the presence of open-set noise.
Revised: Highly informative examples are still beneficialwhen the performance of a classifier is saturated in the presence of open-set noise.

Original: 5.3 Answers to the Purity-Informativeness Dilemma
Revised: 5.3 Answers to the Purity-Informativeness Dilemma

Original: The high robustness of MQ-Net in Table 1 and Figure 3 is mainly attributed to its ability to keepfinding the best trade-off between purity and informativeness.
Revised: The high robustness of MQ-Net in Table 1 and Figure 3 is mainly attributed to its ability to keepfinding the best trade-off between purity and informativeness.

Original: Figure 4(a) illustrates the preferencechange of MQ-Net between purity and informativeness throughout the AL rounds. 
Revised: Figure 4(a) illustrates the preferencechange of MQ-Net between purity and informativeness throughout the AL rounds. As the round [Table]

Original: As the roundprogresses, MQ-Net automatically raises the importance of informativeness rather than purity; theslope of the tangent line keeps steepening from − 0 .
Revised: [Table] [Table] progresses, MQ-Net automatically raises the importance of informativeness rather than purity; theslope of the tangent line keeps steepening from − 0 .

Original: 74 to − 1 . 21 .
Revised: 74 to − 1 . 21 .

Original: This trend implies that more

Original: [Table] [Table] [Table] informative examples are required to be labeled when the target classifier becomes mature.
Revised: This trend implies that moreinformative examples are required to be labeled when the target classifier becomes mature.

Original: Figure4(b) describes the preference change of MQ-Net with varying noise ratios.
Revised: Figure4(b) describes the preference change of MQ-Net with varying noise ratios.

Original: Contrary to the trend over
Revised: Contrary to the trend over

Original: AL rounds [Table] as the noise ratio gets higher, MQ-Net prefers purity more over informativeness.
Revised: AL rounds, as the noise ratio gets higher, MQ-Net prefers purity more over informativeness [Table]

Original: 5.4 Ablation Studies
Revised: 5.4 Ablation Studies

Original: Various Combination of Meta-input.
Revised: Various Combination of Meta-input.

Original: MQ-Net can design its purity and informativeness scoresby leveraging diverse metrics in the existing OOD detection and AL literature.
Revised: MQ-Net can design its purity and informativeness scoresby leveraging diverse metrics in the existing OOD detection and AL literature.

Original: Table 2 shows thefinal round test accuracy on CIFAR10 for the four variants of score combinations, each of which isconstructed by a combination of two purity scores and two informativeness scores; each purity scoreis induced by the two recent OOD detection methods, ReAct [31] and CSI, while each informativenessscore is converted from the two existing AL methods, CONF and LL.
Revised: Table 2 shows thefinal round test accuracy on CIFAR10 for the four variants of score combinations, each of which isconstructed by a combination of two purity scores and two informativeness scores; each purity scoreis induced by the two recent OOD detection methods, ReAct [31] and CSI, while each informativenessscore is converted from the two existing AL methods, CONF and LL.

Original: “CONF-ReAct” denotes avariant that uses ReAct as the purity score and CONF as the informativeness score.
Revised: “CONF-ReAct” denotes avariant that uses ReAct as the purity score and CONF as the informativeness score.

Original: Overall, allthe variants perform better thancompared standard and open-set AL baselines in everynoise level.
Revised: Overall, all variants perform better than standard and open-set AL baselines in every noise level.

Original: Refer to Table 2 for detailed comparison.
Revised: Refer to Table 2 for detailed comparison.

Original: Thisresult concludes that MQ-Net can begeneralized over different types of meta-input owing to the learning flexibility of MLPs.
Revised: This concludes that MQ-Net can be generalized overdifferent types of meta-input owing to the learning flexibility of MLPs.

Original: Interestingly,the variant using CSI as the purity score is consistently better than those using ReAct.
Revised: Interestingly, the variant using CSI as the purity score is consistently better than those using ReAct.

Original: ReAct, aclassifier-dependent OOD score, performsvery poorly in earlier AL rounds where the performanceof the classifier is not sufficient.
Revised: ReAct, a classifier-dependent OOD score, performs poorly in earlier AL rounds.

Original: Amore detailed analysis of the two OOD detectors, ReAct and CSI,over AL rounds, can be found in Appendix E.
Revised: A detailed analysis of the two OOD detectors, ReAct and CSI, over AL rounds, can be found in Appendix E.

Original: Efficacy of Self-validation Set.
Revised: Efficacy of Self-validation Set.

Original: MQ-Net can be trained with an independent validation set available,instead of using the proposed self-validation set.
Revised: MQ-Net can be trained with an independent validation set, insteadof using the proposed self-validation set.

Original: We generate the independent validation set byrandomly sampling examples  with their ground-truth labels from the entire data, which is notoverlapped with the unlabeled set used for AL, where its size is exactly the same with the selfvalidation set.
Revised: We generate the independent validation set by randomlysampling the same number of examples as the self-validation set with their ground-truth labels fromthe entire data not overlapped with the unlabeled set used for AL.

Original: As can be seen from Table 3, it is of interest to see that our self-validation setperforms better than the random validation set.
Revised: As can be seen from Table 3, it isof interest to see that our self-validation set performs better than the random validation set.

Original: The two validation sets have a major difference in datadistributions; the self-validation set mainly consists of the examples with highest meta-scores amongthe remaining unlabeled data per round, while the random validation set consists of random examples.
Revised: The twovalidation sets have a major difference in data distributions; the self-validation set mainly consistsof the examples with highest meta-scores among the remaining unlabeled data per round, while therandom validation set consists of random examples.

Original: We conclude that the meta-score of MQ-Net has the potential for constructing a high-quality validationset in addition to query selection.
Revised: We conclude that the meta-score of MQ-Net hasthe potential for constructing a high-quality validation set in addition to query selection.

Original: Efficacy of Skyline Constraint.
Revised: Efficacy of Skyline Constraint.

Original: Table 4 demonstrates the final round test accuracy of MQ-Net withor without the skyline constraint.
Revised: Table 4 demonstrates the final round test accuracy of MQ-Net withor without the skyline constraint.

Original: For the latter, a standard 2-layer MLP is used as the meta-networkarchitecture without any modification.
Revised: For the latter, a standard 2-layer MLP is used as the meta-networkarchitecture without any modification.

Original: The performance of MQ-Net degrades significantly withoutthe skyline constraint, meaning that the non-constrained MLP can easily overfit to the small-sizedself-validation set, thereby assigning high output scores on less-pure and less-informative examples.
Revised: The performance of MQ-Net degrades significantly withoutthe skyline constraint, meaning that the non-constrained MLP can easily overfit to the small-sizedself-validation set, thereby assigning high output scores on less-pure and less-informative examples.

Original: Therefore, the violation of the skyline constraint in optimization makes MQ-Net hard to balancebetween the purity and informativeness scores in query selection.
Revised: Therefore, the violation of the skyline constraint in optimization makes MQ-Net hard to balancebetween the purity and informativeness scores in query selection.

Original:  Conclusion
Revised:  Conclusion

Original: We propose MQ-Net, a novel meta-model for open-set active learning that deals with the purityinformativeness dilemma.
Revised: We propose MQ-Net, a novel meta-model for open-set active learning that deals with the purityinformativeness dilemma.

Original: MQ-Net finds the best balancing between the two factors, being adaptiveto the noise ratio and target model status.
Revised: MQ-Net finds the best balancing between the two factors, being adaptive tothe noise ratio and target model status.

Original: A clean validation set for the meta-model is obtainedfor free by exploiting the procedure of active learning.
Revised: A clean validation set for the meta-model is obtained for freeby exploiting the procedure of active learning.

Original: A ranking loss with the skyline constraintoptimizes MQ-Net to make the output a legit meta-score that keeps the obvious order of two examples.
Revised: A ranking loss with the skyline constraint optimizes MQ-Net to make the output a legitimate meta-score that keeps the obvious order of two examples.

Original: MQ-Net is shown to yield the best test accuracy throughout the entire active learning rounds, therebyempirically proving the correctness of our solution to the purity-informativeness dilemma.
Revised: MQ-Net is shown to yield the best test accuracy throughout the entire active learning rounds, therebyempirically proving the correctness of our solution to the purity-informativeness dilemma.

Original: Overall,we expect that our work will raise the practical usability of active learning with open-set noise.
Revised: Overall,we expect that our work will raise the practical usability of active learning with open-set noise.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claimsmade
Revised: (a) Do the main claims

Revised: made

Original: in the abstract and introduction accurately
Revised: in the abstract and introduction accurately

Original: reflect the paper’s
Revised: reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See the supplementary material.
Revised: [Yes] See the supplementary material.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See the
Revised: [Yes] See the

Original: supplementary material.
Revised: supplementary material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sections
Revised: [Yes] See Sections

Original: and 4.
Revised: and 4.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See the supplementary
Revised: [Yes] See the supplementary

Original: material.
Revised: material.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See Section 5and the supplementary material.
Revised: [Yes] See Section 5and the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 5 and the supplementary material.
Revised: [Yes] See Section 5 and the supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [No](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]