Original: Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning
Revised: Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Unlabeled  examples awaiting annotations contain open-set noise inevitably.
Revised: Unlabeled data examples awaiting annotations contain open-set noise inevitably.

Original: A fewactive learning studies have attempted to deal with this open-set noise in activelearning by filtering out the noisy examples.
Revised: A few active learning studies have attempted to deal with this open-set noise for sample selection by filtering out the noisy examples.

Original: However, because focusing on thepurity of examples in a query set leads to overlooking the informativeness of theexamples, the best balancing of purity and informativeness remainsas an importantquestion.
Revised: However, because focusing on the purity of examples in a query set leads to overlooking the informativeness of the examples, the best balancing of purity and informativeness remains an important question.

Original: In this paper, to solve this purity-informativeness dilemma in open setactive learning, we propose a novel Meta-Query-Net (MQ-Net) that adaptivelyfinds the best balancing between the two factors.
Revised: In this paper, to solve this purity-informativeness dilemma in open-set active learning, we propose a novel Meta-Query-Net (MQ-Net) that adaptively finds the best balancing between the two factors.

Original: Specifically, by leveragingthe multi-round property of active learning, we train MQ-Net using a query setwithout an additional validation set.
Revised: Specifically, by leveraging the multi-round property of active learning, we train MQ-Net using a query set without an additional validation set.

Original: Furthermore, a clear dominance relationshipbetween unlabeled examples is effectively captured by MQ-Net through a novelskyline regularization.
Revised: Furthermore, a clear dominance relationship between unlabeled examples is effectively captured by MQ-Net through a novel skyline regularization.

Original: Extensive experiments on multiple open-set active learningscenarios demonstrate that the proposed MQ-Net achieves 20.14 % improvement interms of accuracy, compared with the state-of-the-art methods.
Revised: Extensive experiments on multiple open-set active learning scenarios demonstrate that the proposed MQ-Net achieves 20.14 % improvement in terms of accuracy, compared with the state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: The success of deep learning in many complex tasks highly depends on the availability of massive datawith well-annotated labels, which areoften very costly to obtain in practice [1].
Revised: The success of deep learning in many complex tasks highly depends on the availability of massive data with well-annotated labels, which are very costly to obtain in practice [1].

Original: Active learning (AL)is one of the popular learning frameworks to reduce the high human-labeling cost, where a smallnumber of maximally-informative examples are selected by a query strategy and labeled by an oraclerepeatedly [2].
Revised: Active learning (AL) is one of the popular learning frameworks to reduce the high human-labeling cost, where a small number of maximally-informative examples are selected by a query strategy and labeled by an oracle repeatedly [2].

Original: Numerous query ( i e. , sample selection) strategies, mainly categorized into uncertaintybased sampling [3, 4, 5] and diversity -based sampling [6, 7, 8], have succeeded in effectively reducingthe labeling cost while achieving high  performance.
Revised: Numerous query ( i.e. , sample selection) strategies, mainly categorized into uncertaintybased sampling [3, 4, 5] and diversity -based sampling [6, 7, 8], have succeeded in effectively reducing the labeling cost while achieving high model performance.

Original: Despite their success, most standard AL approaches rely on a strict assumption that all unlabeledexamples should be cleanly collected from a pre-defined domain called in-distribution (IN) , evenbefore being labeled [9].
Revised: Despite their success most standard AL approaches rely on a strict assumption that all unlabeled examples should be cleanly collected from a pre-defined domain called in-distribution (IN) , even before being labeled [9].

Original: This assumption is unrealistic in practice since the unlabeled examplesare mostly collected froma rather casual data curation processes such as web-crawling.
Revised: This assumption is unrealistic in practice since the unlabeled examples are mostly collected from rather casual data curation processes such as web-crawling.

Original: Notably,in Google search engine, the precision of image retrieval is reported to be 82 % on average, and itis worsened to 48 % for unpopular entities [10, 11].
Revised: Notably, in the Google search engine, the precision of image retrieval is reported to be 82 % on average, and it is worsened to 48 % for unpopular entities [10, 11].

Original: That is, such collected unlabeled data naturallyinvolves open-set noise , which is defined as a set of the examples collected from different domainscalled out-of-distribution (OOD) [12].
Revised: That is, such collected unlabeled data naturally involves open-set noise , which is defined as a set of the examples collected from different domains called out-of-distribution (OOD) [12].

Original: In general, standard AL approaches favor the examples either highly uncertain in predictions or highlydiverse in representations as a query for labeling.
Revised: In general, standard AL approaches favor the examples either highly uncertain in predictions or highly diverse in representations as a query for labeling.

Original: However, the addition of open-set noise makes thesetwo measures fail to identify informative examples; the OOD examples also exhibit high uncertaintyand diversity because they share neither class-distinctive features nor other inductive biases with INexamples [14, 15].
Revised: However, the addition of open-set noise makes these two measures fail to identify informative examples; the OOD examples also exhibit high uncertainty and diversity because they share neither class-distinctive features nor other inductive biases with IN examples [14, 15].

Original: As a result, an active learner is confused and likely to query the OOD examples toa human-annotator for labeling. Human annotators would disregard the OOD examples because they
Revised: As a result, an active learner is confused and likely to query the OOD examples to

Original: [Figure]
Revised: [Figure]

Original: Unlabeled Examples ( ) [Figure] [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] Rounds

Original: HI
Revised: P u r

Original: [Figure]
Revised: [Figure]

Revised: Unlabeled Examples ( ) 65% 55% 45% 35%

Original: HP
Revised: HP-HI

Original: [Figure]
Revised: Rounds

Original: dominance HI
Revised: HP-dominance HI-dominance

Original: [Figure]
Revised: HP-dominance

Original: dom 55% 45% 35% 25%
Revised: HI-dom 55% 45% 35% 25%

Original: LL (HI [Figure] focused)
Revised: LL (HI-focused)

Original: CCAL (HP-focused)
Revised: CCAL (HP-focused)

Original: MQ
Revised: MQ-Net

Revised: LP

Original: [Figure]
Revised: [Figure]

Original: Net
Revised: HI

Original: [Figure] 
Revised: [Figure] High ?

Revised: HP-LI

Original:  [Figure]
Revised: LP [Figure]

Original: IN,
Revised: IN,

Original: OOD
Revised: OOD

Original: [Figure]
Revised: [Figure]

Original:  are unnecessary for the target task, thereby wasting the labeling budget.
Revised: a human-annotator for labeling. Human annotators would disregard the OOD examples because they are unnecessary for the target task, thereby wasting the labeling budget.

Original: Therefore, the problem ofactive learning with open-set noise, which we call open-set active learning , has emerged as a newimportant challenge for real-world applications.
Revised: Therefore, the problem of active learning with open-set noise, which we call open-set active learning , has emerged as a new important challenge for real-world applications.

Original: Recently, a few studies have attempted to deal with the open-set noise for active learning [13, 16].
Revised: Recently, a few studies have attempted to deal with the open-set noise for active learning [13, 16].

Original: They commonly try to increase the purity of examples in a query set, which is defined as the proportionof IN examples, by effectively filtering out the OOD examples.
Revised: They commonly try to increase the purity of examples in a query set, which is defined as the proportion of IN examples, by effectively filtering out the OOD examples.

Original: However, whether focusing on thepurity is needed throughout the entire training period remains a question.
Revised: However, whether focusing on the purity is needed throughout the entire training period remains a question.

Original: In Figure 1(a), let’s considerthe examples of high purity and low informativeness (HP-LI) and those of low purity and highinformativeness (LP-HI). The dominance between these two groups of examplesin usefulness foropen-set AL is not clear but is determined by the learning stage as well as the ratio of OOD examples.
Revised: In Figure 1(a), let’s consider an open-set AL task with a binary classification of cats and dogs, where the images of other animals, e.g. , horses and wolves, are regarded as OOD examples. It is clear that the group of high purity and high informativeness (HP-HI) is the most preferable for sample selection. However, when comparing the group of high purity and low informativeness (HP-LI) and that of low purity and high informativeness (LP-HI), the preference between these two groups of examples is not clear, but rather contingent on the learning stage and the ratio of OOD examples.

Original: Thus, we coin a new term “purity-informativeness dilemma” to call attention to the best balancing ofpurity and informativeness.
Revised: Thus, we coin a new term “purity-informativeness dilemma” to call attention to the best balancing of purity and informativeness.

Original: Figures 1(b) and 1(c) illustrate the purity-informativeness dilemma.
Revised: Figures 1(b) and 1(c) illustrate the purity-informativeness dilemma.

Original: The standard AL approach,
Revised: The standard AL approach,

Original: LL[5], puts more weight on the examples of high informativeness (denoted as HI-focused), whilethe existing open-set AL approach, CCAL [13], puts more weight on those of high purity (denotedas HP-focused).
Revised: LL[5], puts more weight on the examples of high informativeness (denoted as HI-focused), while the existing open-set AL approach, CCAL [13], puts more weight on those of high purity (denoted as HP-focused).

Original: The HP-focused approach improves the test accuracy more significantly than the
Revised: The HP-focused approach improves the test accuracy more significantly than the

Original: HI-focused one at earlier AL rounds, meaning that pure as well as easy examples are more beneficial.
Revised: HI-focused one at earlier AL rounds, meaning that pure as well as easy examples are more beneficial.

Original: In contrast, the HI-focused approach outperforms the HP-focused one at later AL rounds, meaningthat highly informative examples should be selected even at the expense of purity.
Revised: In contrast, the HI-focused approach beats the HP-focused one at later AL rounds, meaning that highly informative examples should be selected even at the expense of purity.

Original: Furthermore,comparing a low OOD (noise) ratio in Figure 1(b) and a high OOD ratio in Figure 1(c), the shiftfrom HP-dominance to HI-dominance tends to occur later at a higher OOD ratio, which renders thisdilemma more difficult.
Revised: Furthermore, comparing a low OOD (noise) ratio in Figure 1(b) and a high OOD ratio in Figure 1(c), the shift from HP-dominance to HI-dominance tends to occur later at a higher OOD ratio, which renders this dilemma more difficult.

Original: In this paper, to solve the purity-informativeness dilemma in open-set AL, we propose a novel metamodel Meta-Query-Net (MQ-Net) that adaptively finds the best balancing between the two factors.
Revised: In this paper, to solve the purity-informativeness dilemma in open-set AL, we propose a novel metamodel Meta-Query-Net (MQ-Net) that adaptively finds the best balancing between the two factors.

Original: Akey challenge is the best balancing is unknown in advance.
Revised: A key challenge is the best balancing is unknown in advance.

Original: The meta-model is trained to assign higherpriority for in-distribution examples over OOD examples as well as for more informative examplesamong in-distribution ones.
Revised: The meta-model is trained to assign higher priority for in-distribution examples over OOD examples as well as for more informative examples among in-distribution ones.

Original: The input to the meta-model, which includes the target and OOD labels,is obtained for free from each AL round’s query set by leveraging the multi-round property of AL.
Revised: The input to the meta-model, which includes the target and OOD labels, is obtained for free from each AL round’s query set by leveraging the multi-round property of AL.

Original: Moreover, the meta-model is optimized more stably through a novel regularization inspired by theskyline query [17, 18] popularly used in multi-objective optimization.
Revised: Moreover, the meta-model is optimized more stably through a novel regularization inspired by the skyline query [17, 18] popularly used in multi-objective optimization.

Original: As a result, MQ-Net can guidethe learning of the target model by providing the best balancing between purity and informativenessthroughout the entire training period.
Revised: As a result, MQ-Net can guide the learning of the target model by providing the best balancing between purity and informativeness throughout the entire training period.

Original: Overall, our main contributions are summarized as follows:
Revised: Overall, our main contributions are summarized as follows:

Original: We formulate the purity-informativeness dilemma , which hinders the usability of open-set AL inreal-world applications.
Revised: We formulate the purity-informativeness dilemma , which hinders the usability of open-set AL in real-world applications.

Original: As our answer to the dilemma, we propose a novel AL framework, MQ-Net, which keeps findingthe best trade-off between purity and informativeness.
Revised: As our answer to the dilemma, we propose a novel AL framework, MQ-Net, which keeps finding the best trade-off between purity and informativeness.

Original: Extensive experiments on CIFAR10, CIFAR100, and ImageNet show that MQ-Net improvesthe classification accuracy consistently when the OOD ratio changes from 10% to 60% by up to20 .
Revised: Extensive experiments on CIFAR10, CIFAR100, and ImageNet show that MQ-Net improves the classifier accuracy consistently when the OOD ratio changes from 10% to 60% by up to 20 .

Original:  Related Work2.1 Active Learning and Open-set Recognition
Revised:  Related Work 2.1 Active Learning and Open-set Recognition

Original: Active Learning is a learning framework to reduce the human labeling cost by finding the mostinformative examples given unlabeled data [9].
Revised: Active Learning is a learning framework to reduce the human labeling cost by finding the most informative examples given unlabeled data [9].

Original: One popular direction is uncertainty-based sampling.
Revised: One popular direction is uncertainty-based sampling.

Original: Typical approaches have exploited prediction probability, e.g. , soft-max confidence [19, 3], margin [20], and entropy [21].
Revised: Typical approaches have exploited prediction probability, e.g. , soft-max confidence [19, 3], margin [20], and entropy [21].

Original: Some approaches obtain uncertainty by Monte Carlo Dropout on multipleforward passes [22, 23, 24].
Revised: Some approaches obtain uncertainty by Monte Carlo Dropout on multiple forward passes [22, 23, 24].

Original: LL [5] predicts the loss of examples by jointly learning a loss predictionmodule with  target model.
Revised: LL [5] predicts the loss of examples by jointly learning a loss prediction module with a target model.

Original: Meanwhile, diversity-based sampling has  been widely studied.
Revised: Meanwhile, diversity-based sampling has also been widely studied.

Original: To incorporate diversity, most methods use a clustering [6, 25] or core-set selection algorithm [7].
Revised: To incorporate diversity, most methods use a clustering [6, 25] or coreset selection algorithm [7].

Original: Notably,
Revised: Notably,

Original: CoreSet [7] finds the set of examples having the highest distance coverage on the entire unlabeleddata.
Revised: CoreSet [7] finds the set of examples having the highest distance coverage on the entire unlabeled data.

Original: BADGE [8] is a hybrid of uncertainty-and diversity-based sampling which uses k-MEANS++clustering in the gradient embedding space.
Revised: BADGE [8] is a hybrid of uncertainty- and diversity-based sampling which uses k -means++ clustering in the gradient embedding space.

Original: However, this family of approaches is not appropriatefor open-set AL since they do not consider how to handle the OOD examples for query selection.
Revised: However, this family of approaches is not appropriate for open-set AL since they do not consider how to handle the OOD examples for query selection.

Original: Open-set Recognition (OSR) is a detection task to recognize the examples outside of the targetdomain [12].
Revised: Open-set Recognition (OSR) is a detection task to recognize the examples outside of the target domain [12].

Original: Closely related to this purpose, OOD detection has been actively studied [26].
Revised: Closely related to this purpose, OOD detection has been actively studied [26].

Original: Recentwork can be categorized as: classifier-dependent, density-based, and self-supervised approaches.
Revised: Recent work can be categorized into classifier-dependent, density-based, and self-supervised approaches.

Original: The classifier-dependent approach leverages a pre-trained classifier and introduces several scoringfunctions, such as Uncertainty [27], ODIN [28], mahalanobis distance (MD) [29], and Energy[30].
Revised: The classifier-dependent approach leverages a pre-trained classifier and introduces several scoring functions, such as Uncertainty [27], ODIN [28], mahalanobis distance (MD) [29], and Energy[30].

Original: Recently, ReAct [31] shows that rectifying penultimate activations can enhance most of the aforementioned classifier-dependent OOD scores.
Revised: Recently, ReAct [31] shows that rectifying penultimate activations can enhance most of the aforementioned classifier-dependent OOD scores.

Original: The density-based approach learns an auxiliary generativemodel like  variational auto-encoder to compute likelihood-based OOD scores [32, 33, 34].
Revised: The density-based approach learns an auxiliary generative model like a variational auto-encoder to compute likelihood-based OOD scores [32, 33, 34].

Original: In contrast,most self-supervised approaches leverage contrastive learning [35, 36, 37].
Revised: Most self-supervised approaches leverage contrastive learning [35, 36, 37].

Original: CSI shows that contrastingwith distributionally-shifted augmentations can considerably enhance the OSR performance [35].
Revised: CSI shows that contrasting with distributionally-shifted augmentations can considerably enhance the OSR performance [35].

Original: The OSR performance of classifier-dependent approaches degrades significantly if the classifierperforms poorly [38].
Revised: The OSR performance of classifier-dependent approaches degrades significantly if the classifier performs poorly [38].

Original: Similarly, the performance of density-based and self-supervised approachesheavily resorts to the amount of clean IN data [34, 35].
Revised: Similarly, the performance of density-based and self-supervised approaches heavily resorts to the amount of clean IN data [34, 35].

Original: Therefore, open-set active learning is achallenging problem to be resolved by simply applying  OSR approaches since it is difficult to obtainhigh-quality classifiers and sufficient IN data at early AL rounds.
Revised: Therefore, open-set active learning is a challenging problem to be resolved by simply applying the OSR approaches since it is difficult to obtain high-quality classifiers and sufficient IN data at early AL rounds.

Original: 2.2 Open-set Active learning
Revised: 2.2 Open-set Active learning

Original: Two recent approaches have attempted to handle the open-set noise for AL [13, 16].
Revised: Two recent approaches have attempted to handle the open-set noise for AL [13, 16].

Original: Both approachestry to increase purity in query selection by effectively filtering out the OOD examples.
Revised: Both approaches try to increase purity in query selection by effectively filtering out the OOD examples.

Original: CCAL [13]learns two contrastive coding modelswhere eachis for calculating informativeness or OODness of anexample, andit combines  two scores using a heuristic balancing rule.
Revised: CCAL [13] learns two contrastive coding models each for calculating informativeness and OODness of an example, and combines the two scores using a heuristic balancing rule.

Original: SIMILAR [16] selects a pureand core set of examples that maximize the distance on the entire unlabeled data while minimizing thedistance to the identified OOD data.
Revised: SIMILAR [16] selects a pure and core set of examples that maximize the distance on the entire unlabeled data while minimizing the distance to the identified OOD data.

Original: However,  CCAL and SIMILAR are often worse than standard AL methods because they always put higher weights on purity, although informativeness should beemphasized when the open-set noise ratio is small or in later AL rounds.
Revised: However, we found that CCAL and SIMILAR are often worse than standard AL methods, since they always put higher weights on purity although informativeness should be emphasized when the open-set noise ratio is small or in later AL rounds.

Original: This calls for developing anew solution to carefully find the best trade-off between purity and informativeness in open-set AL.
Revised: This calls for developing a new solution to carefully find the best balance between purity and informativeness.

Original:  Purity-Informativeness Dilemma in Open-set Active Learning 
Revised:  Purity-Informativeness Dilemma in Open-set Active Learning 3.1 Problem Statement

Original: 3.1 Problem Statement

Original: Let D IN and D OOD be the IN and OOD data, where theexample’s label of  D OOD does not belongto any of the k known(pre-defined) labels Y =
Revised: Let D IN and D OOD be the IN and OOD data distributions, where the label of examples from D OOD does not belong to any of the k known labels Y =

Original: { y i } ki =1 .
Revised: { y i } ki =1 .

Original: Thus, an unlabeled set is a mixture of INand OOD examples, U =
Revised: Then, an unlabeled set is a mixture of IN and OOD examples, U =

Original: { X IN , X OOD } , i.e. , X IN ∼ D IN and X OOD ∼ D OOD .
Revised: { X IN , X OOD } , i.e. , X IN ∼ D IN and X OOD ∼ D OOD .

Original: In the open-set
Revised: In the open-set

Original: AL, a human oracle is requested to assign a known label y to an IN example x ∈ X IN with a labelingcost c IN , while an OOD example x ∈ X OOD is marked as open-set noise with a labeling cost c OOD .
Revised: AL, a human oracle is requested to assign a known label y to an IN example x ∈ X IN with a labeling cost c IN , while an OOD example x ∈ X OOD is marked as open-set noise with a labeling cost c OOD .

Original: AL imposes restrictions on the labeling budget b every round.
Revised: AL imposes restrictions on the labeling budget b every round.

Original: It starts with a small labeled set S L ,consisting of both labeled IN and OOD examples.
Revised: It starts with a small labeled set S L , consisting of both labeled IN and OOD examples.

Original: The initial labeled set S L improves by addinga small but maximally-informative labeled query set S Q per round, i.e. , S L ← S L ∪ S Q , where thelabeling cost for S Q by the oracle does not exceed the labeling budget b .
Revised: The initial labeled set S L improves by adding a small but maximally-informative labeled query set S Q per round, i.e. , S L ← S L ∪ S Q , where the labeling cost for S Q by the oracle does not exceed the labeling budget b .

Original: Hence, the goal of open-set
Revised: Hence, the goal of open-set

Original: AL is defined to construct the optimal query set S ∗ Q , minimizing the loss for the unseen target IN data.
Revised: AL is defined to construct the optimal query set S ∗ Q , minimizing the loss for the unseen target IN data.

Original: The difference from standard AL is that the labeling cost for OOD examples is introduced, where thelabeling budget is wasted when OOD examples are misclassified as informative ones.
Revised: The difference from standard AL is that the labeling cost for OOD examples is introduced, where the labeling budget is wasted when OOD examples are misclassified as informative ones.

Original: Formally, let C ( · ) be the labeling cost function for a given unlabeled set; then, each round of open-set
Revised: Formally, let C ( · ) be the labeling cost function for a given unlabeled set; then, each round of open-set

Original: AL is formulated to find the best query set S ∗ Q as
Revised: AL is formulated to find the best query set S ∗ Q as

Original: [Equation]
Revised: [Equation]

Original: where C ( S Q ) =(cid:88)
Revised: where C ( S Q ) =

Original: [Equation]
Revised: [Equation]

Original: Q [Equation] [Equation] 131 Here, f ( · ; Θ S L ∪ S Q ) denotes the target model trained for only IN examples in S L ∪ S Q , and ℓ cls is acertain loss function for classification.
Revised: Here, f ( · ; Θ S L ∪ S Q ) denotes the target model trained on only IN examples in S L ∪ S Q , and ℓ cls is a certain loss function, e.g. , cross-entropy, for classification.

Original: For each AL round, all the examples in S ∗ Q are removed fromthe unlabeled set U and then added to the accumulated labeled set S Lalong with their labels.
Revised: For each AL round, all the examples in S ∗ Q are removed from the unlabeled set U and then added to the accumulated labeled set S L with their labels.

Original: Thisprocedure repeats for the total number r of rounds.
Revised: This procedure repeats for the total number r of rounds.

Original: 3.2 Purity-Informativeness Dilemma
Revised: 3.2 Purity-Informativeness Dilemma

Original: An ideal approach for open-set AL would be to increase both purity and informativeness of aquery set by completely suppressing  selection of OOD examples and accurately querying the mostinformative examples among the remaining IN ones.
Revised: An ideal approach for open-set AL would be to increase both purity and informativeness of a query set by completely suppressing the selection of OOD examples and accurately querying the most informative examples among the remaining IN examples.

Original: However, the ideal approach is infeasiblebecause overly emphasizing purity in query selection does not promote example informativeness andvice versa .
Revised: However, the ideal approach is infeasible because overly emphasizing purity in query selection does not promote example informativeness and vice versa .

Original: Specifically, OOD examples with low purity scores mostly exhibit high informativenessscores because they share neither class-distinctive features nor other inductive biases with the INexamples [14, 15].
Revised: Specifically, OOD examples with low purity scores mostly exhibit high informativeness scores because they share neither class-distinctive features nor other inductive biases with the IN examples [14, 15].

Original: We call this trade-off in query selection as the purity-informativeness dilemma ,which is our new finding expected to trigger a lot of subsequent work.
Revised: We call this trade-off in query selection as the purity-informativeness dilemma , which is our new finding expected to trigger a lot of subsequent work.

Original: To address this dilemma, we need to consider the proper weights of a purity score and an informativescore when they are combined.
Revised: To address this dilemma, we need to consider the proper weights of a purity score and an informative score when they are combined.

Original: Let P ( x ) be a purity score of an example x which can be measuredby any existing OOD scores, e.g. , negative energy [30], and I ( x ) be an informativeness score of anexample x from any standard AL strategies, e.g. , uncertainty [3] and diversity [25].
Revised: Let P ( x ) be a purity score of an example x which can be measured by any existing OOD scores, e.g. , negative energy [30], and I ( x ) be an informativeness score of an example x from any standard AL strategies, e.g. , uncertainty [3] and diversity [25].

Original: Next, supposingz x = ⟨P ( x ) , I ( x ) ⟩ is a tuple of available purity and informativeness scores for an example x .
Revised: Next, supposing z x = ⟨P ( x ) , I ( x ) ⟩ is a tuple of available purity and informativeness scores for an example x .

Original: Then,a score combination function Φ( z x ) , where z x = ⟨P ( x ) , I ( x ) ⟩ , is defined to return an overall scorethat indicates the necessity of x being included in the query set.
Revised: Then, a score combination function Φ( z x ) , where z x = ⟨P ( x ) , I ( x ) ⟩ , is defined to return an overall score that indicates the necessity of x being included in the query set.

Original: Given two unlabeled examples x i and x j , if P ( x i ) >
Revised: Given two unlabeled examples x i and x j , if P ( x i ) >

Original: P ( x j ) and I ( x i ) >
Revised: P ( x j ) and I ( x i ) >

Original: I ( x j ) , it is clear to favorx i over x j based on Φ( z x i ) >
Revised: I ( x j ) , it is clear to favor x i over x j based on Φ( z x i ) >

Original: Φ( z x j ) .
Revised: Φ( z x j ) .

Original: However, due to the purity-informativeness dilemma, if
Revised: However, due to the purity-informativeness dilemma, if

Original: P ( x i ) > P ( x j ) and I ( x i ) <
Revised: P ( x i ) > P ( x j ) and I ( x i ) <

Original: I ( x j ) or P ( x i ) <
Revised: I ( x j ) or P ( x i ) <

Original: P ( x j ) and I ( x i ) >
Revised: P ( x j ) and I ( x i ) >

Original: I ( x j ) , it is very challengingto determine the dominance between Φ( z x i ) and Φ( z x j ) .
Revised: I ( x j ) , it is very challenging to determine the dominance between Φ( z x i ) and Φ( z x j ) .

Original: In order to model Φ( · ) , we mainly focuson leveraging meta-learning , which is a more agnostic approach to resolve the dilemma other thanseveral heuristic approaches, such as linear combination and multiplication.
Revised: In order to design Φ( · ) , we mainly focus on leveraging meta-learning , which is a more agnostic approach to resolve the dilemma other than several heuristic approaches, such as linear combination and multiplication.

Original:  Meta-Query-Net
Revised:  Meta-Query-Net

Original: We incorporates a meta-model, named Meta-Query-Net (MQ-Net) , which aims to learn a meta-scorefunction for the purpose of identifying the query set.
Revised: We propose a meta-model, named Meta-Query-Net (MQ-Net) , which aims to learn a meta-score function for the purpose of identifying a query set.

Original: In the presence of open-set noise, MQ-Netoutputs the meta-score for unlabeled examples to achieve the best balance between purity andinformativeness in the selected query set.
Revised: In the presence of open-set noise, MQ-Net outputs the meta-score for unlabeled examples to achieve the best balance between purity and informativeness in the selected query set.

Original: In this section, we introduce the notion of a self-validationset to guide the meta-model in a supervised manner and then demonstrate the meta-objective of MQ-Net for training.
Revised: In this section, we introduce the notion of a self-validation set to guide the meta-model in a supervised manner and then demonstrate the meta-objective of MQ-Net for training.

Original: Then, we propose a novel skyline constraint used in optimization, which helps MQ-Net capture the obvious preference among unlabeled examples when a clear dominance exists.
Revised: Then, we propose a novel skyline constraint used in optimization, which helps MQ-Net capture the obvious preference among unlabeled examples when a clear dominance exists.

Original: Next, we present a way of converting the purity and informativeness scores estimated by existingmethods for use in MQ-Net. 
Revised: Next, we present a way of converting the purity and informativeness scores estimated by existing methods for use in MQ-Net. Note that training MQ-Net is not expensive because it builds a light meta-model on a small self-validation set.

Original: The overview of MQ-Net is illustrated in Figure 2.
Revised: The overview of MQ-Net is illustrated in Figure 2.

Original: 4.1 Training Objective with Self-validation Set
Revised: 4.1 Training Objective with Self-validation Set

Original: The parameters w contained in MQ-Net Φ( · ; w ) is optimized in a supervised manner.
Revised: The parameters w contained in MQ-Net Φ( · ; w ) is optimized in a supervised manner.

Original: For a cleansupervision, a validation data is required for training.
Revised: For clean supervision, validation data is required for training.

Original: Without assuming a hard-to-obtain cleanvalidation set, we propose to use a self-validation set, which is instantaneously generated  every ALround.
Revised: Without assuming a hard-to-obtain clean validation set, we propose to use a self-validation set, which is instantaneously generated in every AL round.

Original: In detail, we obtain a labeled query set S Q by the oracle, consisting of a labeled IN set and anidentified OOD set in every round.
Revised: In detail, we obtain a labeled query set S Q by the oracle, consisting of a labeled IN set and an identified OOD set in every round.

Original: As the query set S Q is unseen for the target  Θ and meta models wat the current round, we can exploit it as a self-validation set to train MQ-Net.
Revised: Since the query set S Q is unseen for the target model Θ and the meta-model w at the current round, we can exploit it as a self-validation set to train MQ-Net.

Original: This self-validation seteliminates the need for a clean validation set in meta-learning.
Revised: This self-validation set eliminates the need for a clean validation set in meta-learning.

Original: Unlabeled 𝑈 = {𝑋 𝐼𝑁 ,𝑋 𝑂𝑂𝐷 }
Revised: Unlabeled 𝑈 = {𝑋 𝐼𝑁 ,𝑋 𝑂𝑂𝐷 }

Original: Query
Revised: Query

Original: MQ-Net Θ
Revised: MQ-Net Θ

Original: Labeled 𝐰 𝑆 𝑄 𝑆 𝐿
Revised: Labeled 𝐰 𝑆 𝑄 𝑆 𝐿

Original: Self-validation
Revised: Self-validation

Original: Update
Revised: Update

Original: Update 𝑃(𝑥) , 𝐼(𝑥) 𝐰≽
Revised: Update 𝑃(𝑥) , 𝐼(𝑥) 𝐰≽

Original: Given the ground-truth labels in the self-validation set,it is feasible to guide MQ-Net to be trained to resolvethe purity-informativeness dilemma by designing anappropriate meta-objective.
Revised: Given the ground-truth labels in the self-validation set, it is feasible to guide MQ-Net to be trained to resolve the purity-informativeness dilemma by designing an appropriate meta-objective.

Original: It is based on the crossentropy loss for classification because the loss valueof training examples has been proven to be effectivein identifying high informativeness examples [5].
Revised: It is based on the cross-entropy loss for classification because the loss value of training examples has been proven to be effective in identifying high informativeness examples [5].

Original: Theconventional loss value by a target model Θ is maskedto be zero if x ∈ X OOD since OOD examples areuseless for AL,
Revised: The conventional loss value by a target model Θ is masked to be zero if x ∈ X OOD since OOD examples are useless for AL,

Original: [Equation]
Revised: [Equation]

Original: 2) where l is a true binary IN label, i.e. , 1 for IN examples and 0 for OOD examples, which can bereliably obtained from the self-validation set.
Revised: where l is a true binary IN label, i.e. , 1 for IN examples and 0 for OOD examples, which can be reliably obtained from the self-validation set.

Original: This masked loss, ℓ mce , preserves the informativenessof IN examples while excluding OOD examples.
Revised: This masked loss, ℓ mce , preserves the informativeness of IN examples while excluding OOD examples.

Original: Given a self-validation data S Q , the meta-objectiveis defined such that MQ-Net parameterized by w outputs a high (or low) meta-score Φ( z x ; w ) if anexample x ’s masked loss value is large (or small),
Revised: Given a self-validation data S Q , the meta-objective is defined such that MQ-Net parameterized by w outputs a high (or low) meta-score Φ( z x ; w ) if an example x ’s masked loss value is large (or small),

Original: [Equation]
Revised: [Equation]

Original: max
Revised: s.t.

Original: [Equation] x [Equation] (cid:1)
Revised: ∀ x i , x j , if P ( x i ) >

Original: [Equation] (cid:0) [Equation] [Equation]
Revised: P ( x j ) and I ( x i ) >

Original:  w [Equation] [Equation] w 
Revised: I ( x j ) , then Φ( z x i ; w ) > Φ( z x j ; w ) ,

Original: [Equation]
Revised: [Equation]

Original: (cid:1)(cid:17) s.t. ∀ x i , x j , if P ( x i ) > P ( x j ) and I ( x i ) > I ( x j ) , then Φ( z x i ; w ) > Φ( z x j ; w ) , (3) where η > 0 is a constant margin for the ranking loss, and Sign ( a, b ) is an indicator function thatreturns +1 if a > b , 0 if a = b , and − 1 otherwise.
Revised: where η > 0 is a constant margin for the ranking loss, and Sign ( a, b ) is an indicator function that returns +1 if a > b , 0 if a = b , and − 1 otherwise.

Original: Hence, Φ( z x i ; w ) is forced to be higher thanΦ( z x j ; w ) if ℓ mce ( x i ) > ℓ mce ( x j ) ; in contrast, Φ( z x i ; w ) is forced to be lower than Φ( z x j ; w ) ifℓ mce ( x i ) < ℓ mce ( x j ) .
Revised: Hence, Φ( z x i ; w ) is forced to be higher than Φ( z x j ; w ) if ℓ mce ( x i ) > ℓ mce ( x j ) ; in contrast, Φ( z x i ; w ) is forced to be lower than Φ( z x j ; w ) if ℓ mce ( x i ) < ℓ mce ( x j ) .

Original: Two OOD examples do not affect the optimization because they do not haveany priority between them, i.e. , ℓ mce ( x i ) =
Revised: Two OOD examples do not affect the optimization because they do not have any priority between them, i.e. , ℓ mce ( x i ) =

Original: ℓ mce ( x j ) .
Revised: ℓ mce ( x j ) .

Original: In addition to the ranking loss, we add a regularization term named the skyline constraint ( i.e. , thesecond line) in the meta-objective Eq.
Revised: In addition to the ranking loss, we add a regularization term named the skyline constraint ( i.e. , the second line) in the meta-objective Eq.

Original: (3), which is inspired by the skyline query which aims tonarrow down a search space in a large-scale database by keeping only those items that are not worsethan any other [17, 18].
Revised: (3), which is inspired by the skyline query which aims to narrow down a search space in a large-scale database by keeping only those items that are not worse than any other [17, 18].

Original: Specifically, in the case of P ( x i ) > P ( x j ) and I ( x i ) >
Revised: Specifically, in the case of P ( x i ) > P ( x j ) and I ( x i ) >

Original: I ( x j ) , the conditionΦ( z x i ; w ) > Φ( z x j ; w ) mustalways hold in our objective, and hence we make this proposition as theskyline constraint.
Revised: I ( x j ) , the condition Φ( z x i ; w ) > Φ( z x j ; w ) must hold in our objective, and hence we make this proposition as the skyline constraint.

Original: This simple yet intuitive regularization is very helpful for achieving a meta-modelthat better judges the importance of purity or informativeness.
Revised: This simple yet intuitive regularization is very helpful for achieving a meta-model that better judges the importance of purity or informativeness.

Original: We provide an ablation study on theskyline constraint in Section 5.4.
Revised: We provide an ablation study on the skyline constraint in Section 5.4.

Original: 4.2 Architecture of MQ-Net
Revised: 4.2 Architecture of MQ-Net

Original: MQ-Net is parameterized by a multi-layer perceptron (MLP), a widely-used deep learning architecturefor meta-learning [39].
Revised: MQ-Net is parameterized by a multi-layer perceptron (MLP), a widely-used deep learning architecture for meta-learning [39].

Original: A challenge here is that the proposed skyline constraint in Eq.
Revised: A challenge here is that the proposed skyline constraint in Eq.

Original: (3) does not holdwith a standard MLP model.
Revised: (3) does not hold with a standard MLP model.

Original: To satisfy the skyline constraint, the meta-score function Φ( · ; w ) shouldbe a monotonic non-decreasing function because the output (meta-score) of MQ-Net for an examplex i must be higher than that for another examplex j if the two factors (purity and informativeness)of
Revised: To satisfy the skyline constraint, the meta-score function Φ( · ; w ) should be a monotonic non-decreasing function because the output (meta-score) of MQ-Net for an example x i must be higher than that for another example

Original: x  i are both higher than those of x j .
Revised: x j if the two factors (purity and informativeness) of x i are both higher than those of x j .

Original: The MLP model consists of multiple matrix multiplicationswith non-linear activation functions such as ReLU and Sigmoid.
Revised: The MLP model consists of multiple matrix multiplications with non-linear activation functions such as ReLU and Sigmoid.

Original: In order for the MLP model to bemonotonically non-decreasing, all the parameters in w for Φ( · ; w ) should be non-negative , as provenby Theorem 4.1.
Revised: In order for the MLP model to be monotonically non-decreasing, all the parameters in w for Φ( · ; w ) should be non-negative , as proven by Theorem 4.1.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For any MLP meta-model w with non-decreasing activation functions, a meta-scorefunction Φ( z ; w ): R d → R holds the skyline constraints if w ⪰ 0 and z ( ∈ R d ) ⪰ 0 , where ⪰ is thecomponent-wise inequality.
Revised: For any MLP meta-model w with non-decreasing activation functions, a meta-score function Φ( z ; w ): R d → R holds the skyline constraints if w ⪰ 0 and z ( ∈ R d ) ⪰ 0 , where ⪰ is the component-wise inequality.

Original: Proof.
Revised: Proof.

Original: An MLP model is involved with matrix multiplication and composition with activationfunctions, which are characterized by three basic operators: (1) addition : h ( z ) =
Revised: An MLP model is involved with matrix multiplication and composition with activation functions, which are characterized by three basic operators: (1) addition : h ( z ) =

Original: f ( z ) + g ( z ) , (2)multiplication : h ( z ) =
Revised: f ( z ) + g ( z ) , (2) multiplication : h ( z ) =

Original: f ( z ) × g ( z ) , and (3) composition : h ( z ) =
Revised: f ( z ) × g ( z ) , and (3) composition : h ( z ) =

Original: f ◦ g
Revised: f ◦ g

Original: ( z ) .
Revised: ( z ) .

Original: These three operatorsare guaranteed to be non-decreasing functions if the parameters of the MLP model are all nonnegative, because the non-negative weights guarantee all decomposed scalar operations in MLP to benon-decreasing functions.
Revised: These three operators are guaranteed to be non-decreasing functions if the parameters of the MLP model are all nonnegative, because the non-negative weights guarantee all decomposed scalar operations in MLP to be non-decreasing functions.

Original: Combining the three operators, the MLP model Φ( z ; w ) , where w ⪰ 0 ,naturally becomes a monotonic non-decreasing function for each input dimension.
Revised: Combining the three operators, the MLP model Φ( z ; w ) , where w ⪰ 0 , naturally becomes a monotonic non-decreasing function for each input dimension.

Original: Refer to Appendix
Revised: Refer to Appendix

Original: A for the complete proof.
Revised: A for the complete proof.

Original: In implementation, non-negative weights are guaranteed by applying a ReLU function to meta-modelparameters.
Revised: In implementation, non-negative weights are guaranteed by applying a ReLU function to meta-model parameters.

Original: Since the ReLU function is differentiable, MQ-Net can be trained with the proposedobjective in an end-to-end manner.
Revised: Since the ReLU function is differentiable, MQ-Net can be trained with the proposed objective in an end-to-end manner.

Original: Putting this simple modification, the skyline constraint is preservedsuccessfully without introducing any complex loss-based regularization term.
Revised: Putting this simple modification, the skyline constraint is preserved successfully without introducing any complex loss-based regularization term.

Original: The only remainingcondition is that each input of MQ-Net must be a vector of non-negative entries.
Revised: The only remaining condition is that each input of MQ-Net must be a vector of non-negative entries.

Original: 4.3 Active Learning with MQ-Net 
Revised: 4.3 Active Learning with MQ-Net 4.3.1 Meta-input Conversion

Original: 4.3.1 Meta-input Conversion

Original: MQ-Net receives z x = ⟨P ( x ) , I ( x ) ⟩ and then returns a meta-score for query selection.
Revised: MQ-Net receives z x = ⟨P ( x ) , I ( x ) ⟩ and then returns a meta-score for query selection.

Original: All the scoresfor the input of MQ-Net must be positive to preserve the skyline constraint, i.e. , z ⪰ 0 .
Revised: All the scores for the input of MQ-Net should be positive to preserve the skyline constraint, i.e. , z ⪰ 0 .

Original: Existing
Revised: Existing

Original: OOD and AL query scores are converted to the meta-input.
Revised: OOD and AL query scores are converted to the meta-input.

Original: The methods used for calculating thescores are orthogonal to our framework.
Revised: The methods used for calculating the scores are orthogonal to our framework.

Original: The OOD score O ( · ) is conceptually the opposite of purityand varies in its scale; hence, we convert it to a purity score by P ( x )
Revised: The OOD score O ( · ) is conceptually the opposite of purity and varies in its scale; hence, we convert it to a purity score by P ( x )

Original: =
Revised: =

Original: Exp(Normalize( −O ( x ))) ,where Normalize( · ) is the z-score normalization.
Revised: Exp(Normalize( −O ( x ))) , where Normalize( · ) is the z-score normalization.

Original: R4Q3:

Original: For the z-score normalization, we computethe mean and standard deviation of each scalar score O ( x ) and Q ( x ) over the unlabeled examples.

Original: The mean and standard deviation are iteratively computed before the meta-training, and used forthe z-score normalization at that round.

Original: This conversion guarantees the purity score to be positive.
Revised: This conversion guarantees the purity score to be positive.

Original: Similarly, for the informativeness score, we convert an existing AL query score Q ( · ) to I ( x ) =
Revised: Similarly, for the informativeness score, we convert an existing AL query score Q ( · ) to I ( x ) =

Original: Exp(Normalize( Q ( x ))) .
Revised: Exp(Normalize( Q ( x ))) .

Revised: For the z-score normalization, we compute the mean and standard deviation of O ( x ) or Q ( x ) over the unlabeled examples.

Revised: Such mean and standard deviation are iteratively computed before the meta-training, and used for the z-score normalization at that round.

Original: 4.3.2 Overall Procedure
Revised: 4.3.2 Overall Procedure

Original: For each AL round, a target model is trained via stochastic gradient descent (SGD) on mini-batchessampled from the IN examples in the current labeled set S L .
Revised: For each AL round, a target model is trained via stochastic gradient descent (SGD) on mini-batches sampled from the IN examples in the current labeled set S L .

Original: Based on the current target model,the purity and informative scores are computed by using certain OOD and AL query scores.
Revised: Based on the current target model, the purity and informative scores are computed by using certain OOD and AL query scores.

Original: Thequerying phase is then performed by selecting the examples S Q with the highest meta-scores withinthe labeling budget b .
Revised: The querying phase is then performed by selecting the examples S Q with the highest meta-scores within the labeling budget b .

Original: The query set S Q is used as the self-validation set for training MQ-Net at thecurrent AL round.
Revised: The query set S Q is used as the self-validation set for training MQ-Net at the current AL round.

Original: The trained MQ-Net is used at the next AL round.
Revised: The trained MQ-Net is used at the next AL round.

Original: The alternating procedureof updating the target model and the meta-model repeats for a given number r of AL rounds.
Revised: The alternating procedure of updating the target model and the meta-model repeats for a given number r of AL rounds.

Original: Thepseudocode of MQ-Net can be found in Appendix B.
Revised: The pseudocode of MQ-Net can be found in Appendix B.

Original:  Experiments 
Revised:  Experiments 5.1 Experiment Setting

Original: 5.1 Experiment Setting

Original: Datasets.
Revised: Datasets.

Original: We perform the active learning task on three benchmark datasets; CIFAR10 [40], CIFAR100 [40], and ImageNet [41].
Revised: We perform the active learning task on three benchmark datasets; CIFAR10 [40], CIFAR100 [40], and ImageNet [41].

Original: Following the ‘split-dataset’ setup in open-world learning literature [13, 16, 42], we divide each dataset into two subsets: (1) the target set with IN classes and (2)the noise set with OOD classes.
Revised: Following the ‘split-dataset’ setup in open-world learning literature [13, 16, 42], we divide each dataset into two subsets: (1) the target set with IN classes and (2) the noise set with OOD classes.

Original: Specifically, CIFAR10 is split into the target set with four classesand the noise set with the rest six classes; CIFAR100 into the two sets with 40 and 60 classes; and
Revised: Specifically, CIFAR10 is split into the target set with four classes and the noise set with the rest six classes; CIFAR100 into the two sets with 40 and 60 classes; and

Original: ImageNet into the two sets with 50 and 950 classes.
Revised: ImageNet into the two sets with 50 and 950 classes.

Original: The entire target set is used as the unlabeled INdata, while only a part of classes in the noise set is selected as the unlabeled OOD data accordingto the given noise ratio.
Revised: The entire target set is used as the unlabeled IN data, while only a part of classes in the noise set is selected as the unlabeled OOD data according to the given noise ratio.

Original: In addition, following OOD detection literature [27, 32], we also considerthe ‘cross-dataset’ setup, which mixes a certain dataset with two external OOD datasets collectedfrom different domains, such as LSUN [43] and Places365 [44].
Revised: In addition, following OOD detection literature [27, 32], we also consider the ‘cross-dataset’ setup, which mixes a certain dataset with two external OOD datasets collected from different domains, such as LSUN [43] and Places365 [44].

Original: For sake of space, we present all theresults on the cross-dataset setup in Appendix D.
Revised: For sake of space, we present all the results on the cross-dataset setup in Appendix D.

Original: Algorithms.
Revised: Algorithms.

Original: We compare MQ-Net with a random selection, four standard AL and two recent open-set AL approaches.
Revised: We compare MQ-Net with a random selection, four standard AL, and two recent open-set AL approaches.

Original: • Standard AL :
Revised: • Standard AL :

Original: The four methods perform AL without any processing for open-set noise: (1)CONF [3] queries the most uncertain examples with the lowest softmax confidence in the prediction,(2) CORESET [7] queries the most diverse examples with the highest coverage in the representationspace, (3) LL [5] queries the examples having the largest predicted loss by jointly learning a lossprediction module, and (4) BADGE [8] considers 
Revised: The four methods perform AL without any processing for open-set noise: (1) CONF [3] queries the most uncertain examples with the lowest softmax confidence in the prediction, (2) CORESET [7] queries the most diverse examples with the highest coverage in the representation space, (3) LL [5] queries the examples having the largest predicted loss by jointly learning a loss prediction module, and (4) BADGE [8] considers both uncertainty and diversity by querying the most representative examples in the gradient via k -means++ clustering [45].

Original: both

Original: uncertainty and diversity by

Original: querying

Original: the

Original: most representative examples in the gradient via k -means

Original: +

Original: clustering

Original: • Open-set AL :
Revised: • Open-set AL :

Original: The two methods tend to put more weight on the examples with high purity: (1)

Original: CCAL [13] learns two contrastive coding models for calculating informativeness and OODness,

Original: and

Original: then

Original: it

Original: combines the two

Original:  scores into one using a heuristic balancing rule, and (2) SIMILAR [16]selects a pure and core set of examples that maximize the distance coverage on the entire unlabeleddata while minimizing the distance coverage to the already labeled OOD data.
Revised: The two methods tend to put more weight on the examples with high purity: (1) CCAL [13] learns two contrastive coding models for calculating informativeness and OODness, and then it combines the two scores into one using a heuristic balancing rule, and (2) SIMILAR [16] selects a pure and core set of examples that maximize the distance coverage on the entire unlabeled data while minimizing the distance coverage to the already labeled OOD data.

Original: [Figure]
Revised: [Figure]

Original: SIMILAR
Revised: SIMILAR

Original: OURS
Revised: OURS

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] For all the experiments, regarding the two inputs of MQ-Net, we mainly use CSI [35] and LL [5]for calculating the purity and informativeness scores, respectively.
Revised: For all the experiments, regarding the two inputs of MQ-Net, we mainly use CSI [35] and LL [5] for calculating the purity and informativeness scores, respectively.

Original: For CSI, as in CCAL, we train acontrastive learner on the entire unlabeled set with open-set noise since the clean in-distribution set isnot available in open-set AL.
Revised: For CSI, as in CCAL, we train a contrastive learner on the entire unlabeled set with open-set noise since the clean in-distribution set is not available in open-set AL.

Original: The ablation study in Section 5.4 shows that MQ-Net is also effectivewith other OOD and AL scores as its input.
Revised: The ablation study in Section 5.4 shows that MQ-Net is also effective with other OOD and AL scores as its input.

Original: Implementation Details.
Revised: Implementation Details.

Original: We repeat the three steps—training, querying, and labeling—of AL.
Revised: We repeat the three steps—training, querying, and labeling—of AL.

Original: Thetotal number r of rounds is set to 10.
Revised: The total number r of rounds is set to 10.

Original: Following the prior open-set AL setup [13, 16], we set thelabeling cost c IN = 1 for IN examples and c OOD = 1 for OOD examples.
Revised: Following the prior open-set AL setup [13, 16], we set the labeling cost c IN = 1 for IN examples and c OOD = 1 for OOD examples.

Original: For the class-splitsetup, the labeling budget b per round is set to 500 for CIFAR10/100 and 1 , 000 for ImageNet.
Revised: For the class-split setup, the labeling budget b per round is set to 500 for CIFAR10/100 and 1 , 000 for ImageNet.

Original: Regarding the open-set noise ratio τ , we configure four different levels from light to heavy noisein { 10% , 20% , 40% , 60% } .
Revised: Regarding the open-set noise ratio τ , we configure four different levels from light to heavy noise in { 10% , 20% , 40% , 60% } .

Original: In the case of τ = 0% (no noise), MQ-Net naturally discard the purityscore and only use the informativeness score for query selection, since the self-validation set does notcontain any OOD examples.
Revised: In the case of τ = 0% (no noise), MQ-Net naturally discards the purity score and only uses the informativeness score for query selection, since the self-validation set does not contain any OOD examples.

Original: The initial labeled set is randomly selected uniformly at random fromthe entire unlabeled set within the labeling budget b .
Revised: The initial labeled set is randomly selected uniformly at random from the entire unlabeled set within the labeling budget b .

Original: R1Q3: As for the architecture of MQ-Net, weuse a 2-layer MLP with the hidden dimension size of 64 and the ReLU activation fuction.R4Q1:
Revised: For the architecture of MQ-Net, we use a 2-layer MLP with the hidden dimension size of 64 and the ReLU activation fuction.

Original: [-All methods are implemented with PyTorch 1.8.0 and executed on a single NVIDIA Tesla V
-]
Revised: [-All methods are implemented with PyTorch 1.8.0 and executed on a single NVIDIA Tesla V
-]

Original: GPU.

Original: We report the average results of five runs with different class splits. 
Revised: We report the average results of five runs with different class splits. We did not use any pre-trained networks.

Original: See Appendix C formore implementation details with training configurations.
Revised: See Appendix C for more implementation details with training configurations.

Original: R2Q3 & R4Q1:
Revised: All methods are implemented with PyTorch 1.8.0 and executed on a single NVIDIA Tesla V100 GPU.

Original: The code is available athttps://anonymous.4open.science/r/MQNet-43E6/ .
Revised: The code is available at https://github.com/kaist-dmlab/MQNet .

Original: 5.2 Open-set Noise Robustness 
Revised: 5.2 Open-set Noise Robustness 5.2.1 Results over AL Rounds

Original: 5.2.1 Results over AL Rounds

Original: Figure 3 illustrates the test accuracy of the target model over AL rounds on the two CIFAR datasets.
Revised: Figure 3 illustrates the test accuracy of the target model over AL rounds on the two CIFAR datasets.

Original: MQ-Net achieves the highest test accuracy in most AL rounds, thereby reaching the best test accuracyat the final round in every case for various datasets and noise ratios.
Revised: MQ-Net achieves the highest test accuracy in most AL rounds, thereby reaching the best test accuracy at the final round in every case for various datasets and noise ratios.

Original: Compared with the two existingopen-set AL methods, CCAL and SIMILAR, MQ-Net shows a steeper improvement in test accuracyover rounds by resolving the purity-informativeness dilemma in query selection.
Revised: Compared with the two existing open-set AL methods, CCAL and SIMILAR, MQ-Net shows a steeper improvement in test accuracy over rounds by resolving the purity-informativeness dilemma in query selection.

Original: For example, theperformance gap between MQ-Net and the two open-set AL methods gets larger after the sixth round,as shown in Figure 3(b), because CCAL and SIMILAR mainly depend on purity in query selection,which conveys less informative information to the classifier.
Revised: For example, the performance gap between MQ-Net and the two open-set AL methods gets larger after the sixth round, as shown in Figure 3(b), because CCAL and SIMILAR mainly depend on purity in query selection, which conveys less informative information to the classifier.

Original: For a better classifier, informative
Revised: For a better classifier, informative

Original: [Table]
Revised: [Table]

Original: 2.40 % improve over the least 3. P u r it y
Revised: [Figure]

Original: ( C S I)
Revised: [Figure]

Original: Informativeness (LL) Informativeness (LL) Informativeness (LL) Informativeness (LL)
Revised: least 3.

Original: [Figure]
Revised: [Figure]

Original: [Figure] Round Round Round
Revised: I

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: P u r it y

Original: ( C S I) P u r it y [Figure]S I )
Revised: [Figure]

Original: P u r it y

Original: [Figure]
Revised: [Figure]

Original: S I ) [Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] y (
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: S I) [Figure] [Figure]
Revised: The slope of the tangent line on the lowest scored example in the query set is displayed together; the steeper the slope, the more informativeness is emphasized in query selection.

Original: examples should be favored at a later AL round due to the sufficient number of IN examples in thelabeled set.
Revised: examples should be favored at a later AL round due to the sufficient number of IN examples in the labeled set.

Original: In contrast, MQ-Net keeps improving the test accuracy even in a later AL round byfinding the best balancing between purity and informativeness in its query set.
Revised: In contrast, MQ-Net keeps improving the test accuracy even in a later AL round by finding the best balancing between purity and informativeness in its query set.

Original: More analysis of
Revised: More analysis of

Original: MQ-Net associated with the purity-informativeness dilemma is discussed in Section 5.3.
Revised: MQ-Net associated with the purity-informativeness dilemma is discussed in Section 5.3.

Original: 5.2.2 Results with Varying Noise Ratios
Revised: 5.2.2 Results with Varying Noise Ratios

Original: Table 1 summarizes the last test accuracy at the final AL round for three datasets with varying levelsof open-set noise.
Revised: Table 1 summarizes the last test accuracy at the final AL round for three datasets with varying levels of open-set noise.

Original: Overall, the last test accuracy of MQ-Net is the best in every case.
Revised: Overall, the last test accuracy of MQ-Net is the best in every case.

Original: This superiorityconcludes that MQ-Net successfully find the best trade-off between purity and informativeness interms of AL accuracy regardless of the noise ratio.
Revised: This superiority concludes that MQ-Net successfully finds the best trade-off between purity and informativeness in terms of AL accuracy regardless of the noise ratio.

Original: In general, the performance improvement becomeslarger with the increase in the noise ratio.
Revised: In general, the performance improvement becomes larger with the increase in the noise ratio.

Original: On the other hand, the two open-set AL approaches areeven worse than the four standard AL approaches when  noise ratio is less than  20 % .
Revised: On the other hand, the two open-set AL approaches are even worse than the four standard AL approaches when the noise ratio is less than or equal to 20 % .

Original: This trendconfirms that increasing informativeness is more crucial than increasing purity when the noise ratio issmall.

Original: In addition, in CIFAR10 relatively easier than others, CCAL and SIMILAR are inferior to thenon-robust AL method, LL, even with 40 % noise.
Revised: Especially, in CIFAR10 relatively easier than others, CCAL and SIMILAR are inferior to the non-robust AL method, LL, even with 40 % noise.

Original: Highly informative examples are still beneficialwhen the performance of a classifier is saturated in the presence of open-set noise. 
Revised: This trend confirms that increasing informativeness is more crucial than increasing purity when the noise ratio is small; highly informative examples are still beneficial when the performance of a classifier is saturated in the presence of open-set noise. An in-depth analysis on the low accuracy of the existing open-set AL approaches in a low noise ratio is presented in Appendix E.

Original:  5.3 Answers to the Purity-Informativeness Dilemma
Revised: [Table] [Table] [Table] 5.3 Answers to the Purity-Informativeness Dilemma

Original: The high robustness of MQ-Net in Table 1 and Figure 3 is mainly attributed to its ability to keepfinding the best trade-off between purity and informativeness.
Revised: The high robustness of MQ-Net in Table 1 and Figure 3 is mainly attributed to its ability to keep finding the best trade-off between purity and informativeness.

Original: Figure 4(a) illustrates the preferencechange of MQ-Net between purity and informativeness throughout the AL rounds.As the round [Table] [Table]
Revised: Figure 4(a) illustrates the preference change of MQ-Net between purity and informativeness throughout the AL rounds.

Original: [Table] progresses, MQ-Net automatically raises the importance of informativeness rather than purity; theslope of the tangent line keeps steepening from − 0 .
Revised: As the round progresses, MQ-Net automatically raises the importance of informativeness rather than purity; the slope of the tangent line keeps steepening from − 0 .

Original: 74 to − 1 . 21 .
Revised: 74 to − 1 . 21 .

Original: This trend implies that moreinformative examples are required to be labeled when the target classifier becomes mature.
Revised: This trend implies that more informative examples are required to be labeled when the target classifier becomes mature.

Revised: That is, as the model performance increases, ‘fewer but highly-informative’ examples are more impactful than ‘more but less-informative’ examples in terms of improving the model performance.

Original: Figure4(b) describes the preference change of MQ-Net with varying noise ratios.
Revised: Figure 4(b) describes the preference change of MQ-Net with varying noise ratios.

Original: Contrary to the trend over

Original:  AL rounds, as the noise ratio gets higher, MQ-Net prefers purity more over informativeness [Table]
Revised: Contrary to the trend over AL rounds, as the noise ratio gets higher, MQ-Net prefers purity more over informativeness.

Original: 5.4 Ablation Studies
Revised: 5.4 Ablation Studies

Original: Various Combination of Meta-input.
Revised: Various Combination of Meta-input.

Original: MQ-Net can design its purity and informativeness scoresby leveraging diverse metrics in the existing OOD detection and AL literature.
Revised: MQ-Net can design its purity and informativeness scores by leveraging diverse metrics in the existing OOD detection and AL literature.

Original: Table 2 shows thefinal round test accuracy on CIFAR10 for the four variants of score combinations, each of which isconstructed by a combination of two purity scores and two informativeness scores; each purity scoreis induced by the two recent OOD detection methods, ReAct [31] and CSI, while each informativenessscore is converted from the two existing AL methods, CONF and LL.
Revised: Table 2 shows the final round test accuracy on CIFAR10 for the four variants of score combinations, each of which is constructed by a combination of two purity scores and two informativeness scores; each purity score is induced by the two recent OOD detection methods, ReAct [31] and CSI [35], while each informativeness score is converted from the two existing AL methods, CONF and LL.

Original: “CONF-ReAct” denotes avariant that uses ReAct as the purity score and CONF as the informativeness score.
Revised: “CONF-ReAct” denotes a variant that uses ReAct as the purity score and CONF as the informativeness score.

Original: Overall, all variants perform better than standard and open-set AL baselines in every noise level.
Revised: Overall, all variants perform better than standard and open-set AL baselines in every noise level.

Original: Refer to Table 2 for detailed comparison.
Revised: Refer to Table 2 for detailed comparison.

Original: This  concludes that MQ-Net can be generalized overdifferent types of meta-input owing to the learning flexibility of MLPs.
Revised: This result concludes that MQ-Net can be generalized over different types of meta-input owing to the learning flexibility of MLPs.

Original: Interestingly, the variant using
Revised: Interestingly, the variant using

Original: CSI as the purity score is consistently better than those using ReAct.
Revised: CSI as the purity score is consistently better than those using ReAct.

Original: ReAct, a classifier-dependent
Revised: ReAct, a classifier-dependent

Original: OOD score, performs poorly in earlier AL rounds.
Revised: OOD score, performs poorly in earlier AL rounds.

Original: A detailed analysis of the two OOD detectors,
Revised: A detailed analysis of the two OOD detectors,

Original: ReAct and CSI, over AL rounds, can be found in Appendix E.
Revised: ReAct and CSI, over AL rounds can be found in Appendix F.

Original: Efficacy of Self-validation Set.
Revised: Efficacy of Self-validation Set.

Original: MQ-Net can be trained with an independent validation set, insteadof using the proposed self-validation set.
Revised: MQ-Net can be trained with an independent validation set, instead of using the proposed self-validation set.

Original: We generate the independent validation set by randomlysampling the same number of examples as the self-validation set with their ground-truth labels fromthe entire data not overlapped with the unlabeled set used for AL.
Revised: We generate the independent validation set by randomly sampling the same number of examples as the self-validation set with their ground-truth labels from the entire data not overlapped with the unlabeled set used for AL.

Original: As can be seen from Table 3, it isof interest to see that our self-validation set performs better than the random validation set.
Revised: As can be seen from Table 3, it is of interest to see that our self-validation set performs better than the random validation set.

Original: The twovalidation sets have a major difference in data distributions; the self-validation set mainly consistsof the examples with highest meta-scores among the remaining unlabeled data per round, while therandom validation set consists of random examples.
Revised: The two validation sets have a major difference in data distributions; the self-validation set mainly consists of the examples with highest meta-scores among the remaining unlabeled data per round, while the random validation set consists of random examples.

Original: We conclude that the meta-score of MQ-Net hasthe potential for constructing a high-quality validation set in addition to query selection.
Revised: We conclude that the meta-score of MQ-Net has the potential for constructing a high-quality validation set in addition to query selection.

Original: Efficacy of Skyline Constraint.
Revised: Efficacy of Skyline Constraint.

Original: Table 4 demonstrates the final round test accuracy of MQ-Net withor without the skyline constraint.
Revised: Table 4 demonstrates the final round test accuracy of MQ-Net with or without the skyline constraint.

Original: For the latter, a standard 2-layer MLP is used as the meta-networkarchitecture without any modification.
Revised: For the latter, a standard 2-layer MLP is used as the meta-network architecture without any modification.

Original: The performance of MQ-Net degrades significantly withoutthe skyline constraint, meaning that the non-constrained MLP can easily overfit to the small-sizedself-validation set, thereby assigning high output scores on less-pure and less-informative examples. 
Revised: The performance of MQ-Net degrades significantly without the skyline constraint, meaning that the non-constrained MLP can easily overfit to the small-sized self-validation set, thereby assigning high output scores on less-pure and less-informative examples. Therefore, the violation of the skyline constraint in optimization makes MQ-Net hard to balance between the purity and informativeness scores in query selection. Efficacy of Meta-objective.

Original: Therefore, the violation of the skyline constraint in optimization makes MQ-Net hard to balancebetween the  purity and informativeness scores in query selection.
Revised: MQ-Net keeps finding the best balance between purity and informativeness over multiple AL rounds by repeatedly minimizing the meta-objective in Eq. To validate its [Table]

Revised: efficacy, we compare it with two simple alternatives based on heuristic balancing rules such as linear combination and multiplication , denoted as P ( x ) + I ( x ) and P ( x ) · I ( x ) , respectively.

Original: Conclusion
Revised: Following the default setting of MQ-Net, we use LL for P ( x ) and CSI for I ( x ) . Table 5 shows the AL performance of the two alternatives and MQ-Net for the split-dataset setup on CIFAR10 with the noise ratios of 20 % and 40% . MQ-Net beats the two alternatives after the second AL round where MQ-Net starts balancing purity and informativeness with its meta-objective. This result implies that our meta-objective successfully finds the best balance between purity and informativeness by emphasizing informativeness over purity at the later AL rounds.

Revised: 5.5 Effect of Varying OOD Labeling Cost

Revised: [Table]

Revised: The labeling cost for OOD examples could vary with respect to data domains.

Revised: To validate the robustness of

Revised: MQ-Net on diverse labeling scenarios, we conduct an additional study of adjusting the labeling cost c OOD for the OOD examples.

Revised: Table 6 summarizes the performance change with four different labeling costs ( i.e. , 0.5, 1, 2, and 4).

Revised: The two standard AL methods, CONF and CORESET, and two open-set AL methods, CCAL and SIMILAR, are compared with MQ-Net.

Revised: Overall, MQ-Net consistently outperforms the four baselines regardless of the labeling cost.

Revised: Meanwhile, CCAL and SIMILAR are more robust to the higher labeling cost than CONF and CORESET;

Revised: CCAL and SIMILAR, which favor high purity examples, query more IN examples than CONF and

Revised: CORESET, so they are less affected by the labeling cost, especially when it is high.

Original: We propose MQ-Net, a novel meta-model for open-set active learning that deals with the purityinformativeness dilemma.
Revised: We propose MQ-Net, a novel meta-model for open-set active learning that deals with the purityinformativeness dilemma.

Original: MQ-Net finds the best balancing between the two factors, being adaptive tothe noise ratio and target model status.
Revised: MQ-Net finds the best balancing between the two factors, being adaptive to the noise ratio and target model status.

Original: A clean validation set for the meta-model is obtained for freeby exploiting the procedure of active learning.
Revised: A clean validation set for the meta-model is obtained for free by exploiting the procedure of active learning.

Original: A ranking loss with the skyline constraint optimizes
Revised: A ranking loss with the skyline constraint optimizes

Original: MQ-Net to make the output a legitimate meta-score that keeps the obvious order of two examples.
Revised: MQ-Net to make the output a legitimate meta-score that keeps the obvious order of two examples.

Original: MQ-Net is shown to yield the best test accuracy throughout the entire active learning rounds, therebyempirically proving the correctness of our solution to the purity-informativeness dilemma.
Revised: MQ-Net is shown to yield the best test accuracy throughout the entire active learning rounds, thereby empirically proving the correctness of our solution to the purity-informativeness dilemma.

Original: Overall,we expect that our work will raise the practical usability of active learning with open-set noise. 
Revised: Overall, we expect that our work will raise the practical usability of active learning with open-set noise. Acknowledgement This work was supported by Institute of Information & Communications Technology Planning &

Revised: Evaluation (IITP) grant funded by the Korea government (MSIT)

Revised: (No. 2020-0-00862, DB4DL: HighUsability and Performance In-Memory Distributed DBMS for Deep Learning).

Revised: The experiment was conducted by the courtesy of NAVER Smart Machine Learning (NSML) [46].

Original: Checklist
Revised: Checklist

Original: For all authors. (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See the supplementary material.
Revised: [Yes] See the supplementary material.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See thesupplementary material.
Revised: [Yes] See the supplementary material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sections 3and 4.
Revised: [Yes] See Sectionsand 4.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See the supplementarymaterial.
Revised: [Yes] See the supplementary material.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Section and the supplementary material.
Revised: [Yes] See Sectionand the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5 and the supplementary material.
Revised: [Yes] See Section 5 and the supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 5. 
Revised: [Yes] See Section 5. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [No] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if 
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original:  applicable? 
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning

Original: Board (IRB) approvals, if applicable? 
Revised: (Supplementary Material) A Complete Proof of Theorem 4.

Original: [N/A]
Revised: Let z x = { z ⟨ 1 ⟩ x , . . , z ⟨ d ⟩ x } be the d -dimensional meta-input for an example x consisting of d available purity and informativeness scores.

Original: [Table]
Revised: 2 A non-negative-weighted MLP Φ w can be formulated as [Equation] ) where h [0] =

Revised: z x , h [ L ] ∈ R , W [ l ] ⪰ 0 , and b [ l ] ⪰ 0 ; L is the number of layers and σ is a non-linear activation function.

Revised: We prove Theorem 4.1 by mathematical induction, as follows: (1) the first layer’s output satisfies the skyline constraint by Lemmas A.1 and A.2; and (2) the k -th layer’s output ( k ≥ 2 ) also satisfies the skyline constraint if the ( k − 1) -th layer’s output satisfies the skyline constraint.

Revised: Therefore, we conclude that the skyline constraint holds for any non-negative-weighted MLP Φ( z ; w ): R d → R by

Revised: Theorem A.4.

Revised: Lemma A.1.

Revised: Let g [1] ( z x ) =

Revised: W [1] · z x + b [1] be a non-negative-weighted single-layer MLP with m hidden units and an identity activation function, where W [1] ∈ R m × d ⪰ 0 and b [1] ∈ R m ⪰ 0 .

Revised: Given the meta-input of two different examples z x i and z x j , the function g [1] ( z x ) satisfies the skyline constraint as

Revised: [Equation]

Revised: Proof.

Revised: Let g [1] ( z x ) be g ( z x ) and W [1] be W for notation simplicity.

Revised: Consider each dimension’s scalar output of g ( z x ) , and it is denoted as g ⟨ p ⟩ ( z x ) where p is an index of the output dimension.

Revised: Similarly, let W ⟨ p,n ⟩ be a scalar element of the matrix W on the p -th row and n -th column.

Revised: With the matrix multiplication, the scalar output g ⟨ p ⟩ ( z x ) can be considered as the sum of multiple scalar linear operators W ⟨ p,n ⟩ · z ⟨ n ⟩ x .

Original: {+By this property, we show that g ⟨ p ⟩ ( z x i ) − g ⟨ p ⟩ ( z x j ) ≥ 
+}
Revised: {+By this property, we show that g ⟨ p ⟩ ( z x i ) − g ⟨ p ⟩ ( z x j ) ≥ 
+}

Revised: if z x i ⪰ z x j by

Revised: [Equation]

Revised: Therefore, without loss of generality, g ( z x i ) − g ( z x j ) ⪰ 0 if

Revised: z x i ⪰ z x j .

Revised: This concludes the proof.

Revised: Lemma A.2.

Revised: Let h ( z x )=

Revised: σ ( g ( z x )) where σ is a non-decreasing non-linear activation function.

Revised: If the skyline constraint holds by g ( · ) ∈ R d , the function h ( z x ) also satisfies the skyline constraint as

Revised: [Equation]

Revised: Proof.

Revised: By the composition rule of the non-decreasing function, applying any non-decreasing function does not change the order of its inputs.

Revised: Therefore, σ ( g ( z x i )) − σ ( g ( z x j ))

Revised: ⪰ 0 if g ( z x i ) ⪰

Revised: g ( z x j ) .

Revised: Lemma A.3.

Revised: Let h [ k ] ( z x )= σ (cid:0) W [ k ] · h [ k − 1] ( z x )+ b

Revised: [ k ] (cid:1) be the k -th layer of a non-negative-weighted

Revised: MLP ( k ≥ 2) , where W [ k ] ∈ R m ′

Revised: × m ⪰ 0 and b [ k ] ∈ R m ′

Revised: ⪰ 0 .

Revised: If h [ k − 1] ( · ) ∈ R m satisfies the skyline constraint, the function h

Revised: [ k ] ( z x ) also holds the skyline constraint as

Revised: [Equation]

Revised: Proof.

Revised: Let W [ k ] be W , h [ k ] ( z x ) be h ( z x ) , and h [ k − 1] ( z x ) be h input ( z x ) for notation simplicity.

Revised: Since an intermediate layer uses h input ( z x ) as its input rather than z , Eq.

Revised: (6) changes to

Revised: [Equation]

Revised: ( z x

Revised: [Equation]

Revised: where g ⟨ p ⟩ ( z x i ) is the p -th dimension’s output before applying non-linear activation σ .

Revised: Since h input ( · ) satisfies the skyline constraint, h ⟨ n ⟩ input ( z x i ) >

Revised: h ⟨ n ⟩ input ( z x j ) when z x i ⪰ z x j , g ⟨ p ⟩ ( z x i ) > g ⟨ p ⟩ ( z x j ) for all p ∈ { 1 , . . .

Revised: , m ′ } .

Revised: By Lemma A.2, h ⟨ p ⟩ ( z x i ) − h ⟨ p ⟩ ( z x j ) =

Revised: σ ( g ⟨ p ⟩ ( z x i )) − σ ( g ⟨ p ⟩ ( z x j ))

Revised: ≥for all p .

Revised: Therefore, z x i ⪰ z x j = ⇒

Revised: h

Revised: [ k ] ( z x i ) ⪰ h

Revised: [ k ] ( z x j ) .

Revised: Theorem A.4.

Revised: For any non-negative-weighted MLP Φ( z ; w ) : R d → R where w ⪰ 0 , the skyline constraint holds such that z x i ⪰ z x j = ⇒

Revised: Φ( z x i ) ≥ Φ( z x j ) ∀

Revised: z x i , z x j ∈ R d ⪰ 0 .

Revised: Proof.

Revised: By mathematical induction, where Lemmas A.1 and A.2 constitute the base step, and

Revised: Lemma A.3 is the inductive step, any non-negative-weighted MLP satisfies the skyline constraint.

Revised: B Detailed Procedure of MQ-Net

Revised: Mini-batch Optimization.

Revised: Mini-batch examples are sampled from the labeled query set S Q which contains both IN and OOD examples.

Revised: Since the meta-objective in Eq.

Revised: (3) is a ranking loss, a mini-batch M is a set of meta-input pairs

Revised: such that M = { ( z x i , z x j ) | x i , x j ∈ S Q } where z x = ⟨P ( x ) , I ( x ) ⟩ .

Revised: To construct a paired mini-batch M of size M , we randomly sample 2 M examples from S Q and pair the i -th example with the ( M + i ) -th

Revised: one for all i ∈ { 1 , . . .

Revised: , M } .

Revised: Then, the loss for mini-batch optimization of MQ-Net is defined as

Revised: [Equation]

Revised: Algorithm Pseudocode.

Revised: Algorithm 1 describes the overall active learning procedure with MQ-Net, which is self-explanatory.

Revised: For each AL round, a target model Θ is trained via stochastic gradient descent (SGD) using IN examples in the labeled set S L (Lines 3–5).

Revised: This trained target model is saved as the final target model at the current round.

Revised: Next, the querying phase is performed according

Revised: Algorithm

Revised: ∅ ; 8: while C ( S Q ) ≤ b do 9: if r = 1 do 10:

Revised: S Q ← S Q ∪ arg max x ∈ U

Revised: ( P ( x ) + I ( x )) ; 11: else do 12:

Revised: S Q ← S Q ∪ arg max x ∈ U (Φ( x ; w )) ; 13:

Revised: S L ← S L ∪ S Q ;

Revised: U ← U \ S Q 14: /* Training MQ-Net Φ parameterized by w */ 15: for t = 1 to meta-train-steps do 16:

Revised: Draw a mini-batch M and from S Q ; 17: w ← w − α ∇ w (cid:0) L meta ( M ) (cid:1) ; 18: return Θ ; to the order of meta-query scores from Φ given the budget b (Lines 6–13).

Revised: Then, the meta-training phase is performed, and the meta-model w is updated via SGD using the labeled query set S Q as a self-validation set (Lines 14–17).

Revised: Lines 3–17 repeat for the given number r of rounds.

Revised: At the first round, because there is no meta-model trained in the previous round, the query set is constructed by choosing the examples whose sum of purity and informativeness scores is the largest (Lines 9–10).

Revised: C Implementation Details C.1 Split-dataset Setup

Revised: Training Configurations.

Revised: We train ResNet-18 using SGD with a momentum of 0.9 and a weight decay of 0.0005, and a batch size of 64.

Revised: The initial learning rate of 0 . 1 is decayed by a factor of 0.1 at 50 % and 75 % of the total training iterations.

Revised: In the setup of open-set AL, the number of

Revised: IN examples for training differs depending on the query strategy.

Revised: We hence use a fixed number of training iterations instead of epochs for fair optimization.

Revised: The number of training iterations is set to 30,000.

Revised: We set η to 0.1 for all cases.

Revised: We train MQ-Net for 100 epochs using SGD with a weight decay of 0.0005, and a mini-batch size of 32.

Revised: An initial learning rate of 0 .

Revised: 01 is decayed by a factor of 0.1 at 50 % of the total training iterations.

Revised: Since MQ-Net is not trained at the querying phase of the first AL round, we simply use the linear combination of purity and informativeness as the query score, i.e. , Φ( x ) =

Revised: P ( x ) + I ( x ) .

Revised: For calculating the CSI-based purity score, we train a contrastive learner for CSI with 1,000 epochs under the LARS optimizer with a batch size of 32.

Revised: Following CCAL [47], we use the distance between each unlabeled example to the closest OOD example in the labeled set on the representation space of the contrastive learner as the OOD score.

Revised: The hyperparameters for other algorithms are favorably configured following the original papers.

Revised: C.2 Cross-dataset Setup

Revised: Datasets.

Revised: Each of CIFAR10, CIFAR100, and ImageNet is mixed with OOD examples sampled from an OOD dataset combined from two different domains—LSUN [43], an indoor scene understanding dataset of 59M images with 10 classes, and Places365 [44], a large collection of place scene images with 365 classes.

Revised: The resolution of LSUN and Places365 is resized into 32 × 32 after random cropping when mixing with CIFAR10 and CIFAR100.

Revised: For ImageNet, as in the split-dataset setup in Section 5.1, we use 50 randomly-selected classes as IN examples, namely ImageNet50.

Revised: Implementation Details.

Revised: For the cross-dataset setup, the budget b is set to 1 , 000 for CIFAR-10 and

Revised: ImageNet50 and 2 , 000 for CIFAR-100 following the literature [5].

Revised: Regarding the open-set noise ratio, we also configure four different levels from light to heavy noise in { 10% , 20% , 40% , 60% } .

Revised: The initial labeled set is selected uniformly at random from the entire unlabeled set within the labeling budget b .

Revised: For instance, when b is 1 , 000 and τ is 20% , 800 IN examples and 200 OOD examples are expected to be selected as the initial set.

Revised: D Experiment Results on Cross-datasets D.1 Results over AL Rounds

Revised: Figure 5 shows the test accuracy of the target model throughout AL rounds on the three cross-datasets.

Revised: Overall, as analyzed in Section 5.2, MQ-Net achieves the highest test accuracy in most AL rounds, thereby reaching the best test accuracy at the final round in every case of various datasets and noise ratios.

Revised: Compared with the two existing open-set AL methods, CCAL and SIMILAR, MQ-Net shows a steeper improvement in test accuracy over rounds by resolving the purity-informativeness dilemma in query selection, which shows that MQ-Net keeps improving the test accuracy even in a later AL round by finding the best balancing between purity and informativeness in its query set.

Revised: Together with the results in Section 5.2, we confirm that MQ-Net is robust to the two different distributions—‘split-dataset’ and ‘cross-dataset’—of open-set noise.

Revised: D.2 Results with Varying Noise Ratios

Revised: Table 7 summarizes the last test accuracy at the final AL round for three cross-datasets with varying levels of open-set noise.

Revised: Overall, the last test accuracy of MQ-Net is the best in every case, which shows that MQ-Net keeps finding the best trade-off between purity and informativeness in terms

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: three

Revised: [Table]

Revised: [Table]

Revised: of AL accuracy

Revised: regardless of the noise ratio.

Revised: The performance improvement becomes larger as the noise ratio increases.

Revised: Meanwhile, CCAL and SIMILAR are even worse than the four standard AL approaches when noise ratio is less than or equal to 20 % .

Revised: This trend indicates that focusing on informativeness is more beneficial than focusing on purity when the noise ratio is small.

Revised: E In-depth Analysis of CCAL and SIMILAR in a Low-noise Case

Revised: In the low-noise case, the standard AL method, such as CONF, can query many IN examples even without careful consideration of purity.

Revised: As shown in Table 8, with 10 % noise, the ratio of IN examples in the query set reaches 75.24 % at the last AL round in CONF.

Revised: This number is farily similar to 88.46 % and 90.24 % in CCAL and SIMILAR, respectively.

Revised: In contrast, with the high-noise case (60 % noise), the difference between CONF and CCAL or SIMILAR becomes much larger ( i.e. , from 16.28 % to 41.84 % or 67.84 % ).

Revised: That is, considering mainly on purity (not informativeness) may not be effective with the low-noise case.

Revised: Therefore, especially in the low-noise case, the two purity-focused methods,

Revised: SIMILAR and CCAL, have the potential risk of overly selecting less-informative IN examples that the model already shows high confidence, leading to lower generalization performance than the standard AL methods.

Revised: In contrast, MQ-Net outperforms the standard AL baselines by controlling the ratio of IN examples in the query set to be very high at the earlier AL rounds but moderate at the later AL rounds; MQ-Net achieves a higher ratio of IN examples in the query set than CONF at every AL round, but the gap

Revised: [Table]

Revised: keeps decreasing.

Revised: Specifically, with 10 % noise, the ratio of IN examples in the query set reaches 94.76 % at the first AL round in MQ-Net, which is higher than 87.52 % in CONF

Revised: [Table]

Revised: but it becomes 75.80 % at the last AL round, which is very similar to 75.24 % in CONF.

Revised: This observation means that MQ-Net succeeds in maintaining the high purity of the query set and avoiding the risk of overly selecting less-informative IN examples at the later learning stage

Revised: [Table]

Revised: F In-depth Analysis of Various Purity Scores

Revised: [Table]

Revised: The OSR performance of classifier-dependent OOD detection methods, e.g. , ReAct, degrades significantly if the classifier performs poorly [38].

Revised: Also, the OSR performance of self-supervised OOD detection methods, e.g. ,

Revised: CSI, highly depends on the sufficient amount of clean IN examples [34, 35].

Revised: Table 9 shows the OOD detection performance of two OOD detectors, ReAct and CSI, over AL rounds with MQ-Net.

Revised: Notably, at the earlier AL rounds,

Revised: CSI is better than ReAct, meaning that self-supervised OOD detection methods are more robust than classifier-dependent methods when the amount of labeled data is small.

Revised: Thus, the versions of MQ-Net using CSI as the purity score is better than those using ReAct, as shown in Section 5.4.

Revised: G Additional Experiment Results G.1 AL Performance with More Rounds

Revised: Figure 6 shows the test accuracy over longer AL rounds for the split-dataset setup on CIFAR10 with an open-set noise ratio of 40 % .

Revised: Owing to the ability to find the best balance between purity and informativeness, MQ-Net achieves the highest accuracy on every AL round.

Revised: The purity-focused approaches, CCAL and SIMILAR, lose their effectiveness at the later AL rounds, compared to the informativeness-focused approaches, CONF, CORESET, and BADGE; the superiority of CONF,

Revised: CORESET, and BADGE over CCAL and SIMILAR gets larger as the AL round proceeds, meaning that fewer but highly-informative examples are more beneficial than more but less-informative examples for model generalization as the model performance converges.

Revised: However, with low ( e.g. , 20%) open-set noise cases, most OOD examples are selected as a query set and removed from the unlabeled set in a few additional AL rounds, because the number of OOD examples in the unlabeled set is originally small.

Revised: Thus, the situation quickly becomes similar to the standard AL setting.

Revised: G.2 Standard Deviations of Main Results

Revised: Table 10 repeats Table 1 with the addition of the standard deviations.

Revised: Note that the standard deviations are very small, and the significance of the empirical results is sufficiently high.

Revised: BADGE

Revised: H Limitation and Potential Negative Societal Impact

Revised: Limitation.

Revised: Although MQ-Net outperforms other methods on multiple pairs of noisy datasets under the open-set AL settings, there are some issues that need to be further discussed.

Revised: First, the performance gap between standard AL without open-set noise and open-set AL still exists.

Revised: That is, we could not completely eliminate the negative effect of open-set noise.

Revised: Second, although we validated MQ-Net with many OOD datasets, its effectiveness may vary according to the types of the OOD datasets.

Revised: Formulating the effectiveness of MQ-Net based on the characteristics of a given pair of IN and OOD datasets can be an interesting research direction.

Revised: Third, we regarded the OOD examples in a query set to be completely useless in training, but recent studies have reported that

Original: the
Revised: the

Revised: OOD examples are helpful for model generalization [48, 49, 50, 15].

Revised: Therefore, analyzing how to use OOD examples for model generalization and sample selection in AL can also be an interesting research direction.

Revised: Potential Negative Societal Impact.

Original: [Table]
Revised: As in all active learning approaches, since MQ-Net requires human oracles to label each of queried data examples, the oracle can see these examples in a database, even if the proportion of the revealed examples could be very small. Then, if the oracle is not trustworthy, there may be a leak of information. Therefore, in active learning, not specifically confined to MQ-Net, this privacy breach issue should be carefully considered, especially if the database contains private or sensitive information. T a b l e 10 :

Revised: L a s t t e s t acc u r a c y ( % ) a t t h e fi n a l r ound w it h t

Revised: h e s t a nd a r d d e v i a ti on s f o r

Revised: C

Revised: I F

Revised: A R 10 ,

Revised: C

Revised: I F

Revised: A R 100 , a nd

Revised: I m a g e N e t .

Original: hourly
Revised: D a t a s e t s C I F A

Revised: R 10 ( 4 : 6 s p lit )

Original:  [Table]
Revised: C I F A [Table]

Original: paid to
Revised: ( 40 :s p lit ) I m a [Table] e

Original: [Table]
Revised: [Table]

Revised: t (

Revised: [Table]

Original: and the total
Revised: lit ) N o i s e R a ti o

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: N [Table] [Table] L

Original: [N/A
Revised: R ANDO [Table] S t a nd a r d A L

Original:  [Table]
Revised: C [Table] F [Table]

Revised: ET

Revised: [Table]