Original: S IMILARITY S EARCH FOR E FFICIENT A CTIVE L EARNING AND S EARCH OF R ARE C ONCEPTS
Revised: S IMILARITY S EARCH FOR E FFICIENT A CTIVE L EARNING AND S EARCH OF R ARE C ONCEPTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many active learning and search approaches are intractable for industrial settings with billions of unlabeled examples.
Revised: Many active learning and search approaches are intractable for industrial settings with billions of unlabeled examples.

Original: Existing approaches, such as uncertainty sampling or information density, search globally for the optimal examples to label, scaling linearly or even quadratically with the unlabeled data.
Revised: Existing approaches, such as uncertainty sampling or information density, search globally for the optimal examples to label, scaling linearly or even quadratically with the unlabeled data.

Original: However, in practice, data is often heavily skewed; only a small fraction of collected data will be relevant for a given learning task.
Revised: However, in practice, data is often heavily skewed; only a small fraction of collected data will be relevant for a given learning task.

Original: For example, when identifying rare classes, detecting malicious content, or debugging model performance, positive examples can appear in less than 1% of the data.
Revised: For example, when identifying rare classes, detecting malicious content, or debugging model performance, positive examples can appear in less than 1% of the data.

Original: In this work, we exploit this skew in large training datasets to reduce the number of unlabeled examples considered in each selection round by only looking at the nearest neighbors to the labeled examples.
Revised: In this work, we exploit this skew in large training datasets to reduce the number of unlabeled examples considered in each selection round by only looking at the nearest neighbors to the labeled examples.

Original: Empirically, we observe that learned representations can effectively cluster unseen concepts, making active learning very effective and substantially reducing the number of viable unlabeled examples.
Revised: Empirically, we observe that learned representations can effectively cluster unseen concepts, making active learning very effective and substantially reducing the number of viable unlabeled examples.

Original: We evaluate several selection strategies in this setting on three large-scale computer vision datasets: ImageNet, OpenImages, and a proprietary dataset of 10 billion images from a large internet company.
Revised: We evaluate several selection strategies in this setting on three large-scale computer vision datasets: ImageNet, OpenImages, and a proprietary dataset of 10 billion images from a large internet company.

Original: For rare classes, active learning methods need as little as 0.31% of the labeled data to match the average precision of full supervision.
Revised: For rare classes, active learning methods need as little as 0.31% of the labeled data to match the average precision of full supervision.

Original: By limiting the selection strategies to the immediate neighbors of the labeled data as candidates for labeling, we process as little as 0.1% of the unlabeled data while achieving similar reductions in labeling costs as the traditional global approach.
Revised: By limiting the selection strategies to the immediate neighbors of the labeled data as candidates for labeling, we process as little as 0.1% of the unlabeled data while achieving similar reductions in labeling costs as the traditional global approach.

Original: This process of expanding the candidate pool with the nearest neighbors of the labeled set can be done efﬁciently and reduces the computational complexity of selection by orders of magnitude.
Revised: This process of expanding the candidate pool with the nearest neighbors of the labeled set can be done efﬁciently and reduces the computational complexity of selection by orders of magnitude.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale unlabeled datasets contain millions or billions of examples spread over a wide variety of underlying concepts (Chelba et al., 2013;
Revised: Large-scale unlabeled datasets contain millions or billions of examples spread over a wide variety of underlying concepts (Chelba et al., 2013;

Original: Zhu et al., 2015; Zhang et al., 2015; Wan et al., 2019;
Revised: Zhu et al., 2015; Zhang et al., 2015; Wan et al., 2019;

Original: Russakovsky et al., 2015; Kuznetsova et al., 2020;
Revised: Russakovsky et al., 2015; Kuznetsova et al., 2020;

Original: Thomee et al., 2016; Abu-El-Haija et al., 2016;
Revised: Thomee et al., 2016; Abu-El-Haija et al., 2016;

Original: Caesar et al., 2019; Lee et al., 2019).
Revised: Caesar et al., 2019; Lee et al., 2019).

Original: Often, these massive datasets skew towards a relatively small number of common concepts, such as cats, dogs, and people (Liu et al., 2019; Zhang et al., 2017;
Revised: Often, these massive datasets skew towards a relatively small number of common concepts, such as cats, dogs, and people (Liu et al., 2019; Zhang et al., 2017;

Original: Wang et al., 2017; Van Horn & Perona, 2017).
Revised: Wang et al., 2017; Van Horn & Perona, 2017).

Original: Rare concepts, such as harbor seals, may only appear in a small fraction of the data (less than 1%).
Revised: Rare concepts, such as harbor seals, may only appear in a small fraction of the data (less than 1%).

Original: However, in many settings, performance on these rare concepts is critical.
Revised: However, in many settings, performance on these rare concepts is critical.

Original: For example, harmful or malicious content may comprise a small percentage of user-generated content, but it can have an outsize impact on the overall user experience (Wan et al., 2019).
Revised: For example, harmful or malicious content may comprise a small percentage of user-generated content, but it can have an outsize impact on the overall user experience (Wan et al., 2019).

Original: Similarly, when debugging model behavior for safety-critical applications like autonomous vehicles, or when dealing with representational biases in models, obtaining data that captures rare concepts allows machine learning practitioners to combat blind spots in model performance (Karpathy, 2018;
Revised: Similarly, when debugging model behavior for safety-critical applications like autonomous vehicles, or when dealing with representational biases in models, obtaining data that captures rare concepts allows machine learning practitioners to combat blind spots in model performance (Karpathy, 2018;

Original: Holstein et al., 2019; Ashmawy et al., 2019; Karpathy, 2020).
Revised: Holstein et al., 2019; Ashmawy et al., 2019; Karpathy, 2020).

Original: Even a simple prediction task like stop sign detection can be challenging given the diversity of real-world data.
Revised: Even a simple prediction task like stop sign detection can be challenging given the diversity of real-world data.

Original: Stop signs may appear in a variety of conditions (e.g., on a wall or held by a person), be heavily occluded, or have modiﬁers (e.g., “Except Right Turns”)
Revised: Stop signs may appear in a variety of conditions (e.g., on a wall or held by a person), be heavily occluded, or have modiﬁers (e.g., “Except Right Turns”)

Original: (Karpathy, 2020).
Revised: (Karpathy, 2020).

Original: While large-scale datasets are core to addressing these issues, ﬁnding the relevant examples for these long-tail tasks is challenging.
Revised: While large-scale datasets are core to addressing these issues, ﬁnding the relevant examples for these long-tail tasks is challenging.

Original: Active learning and search have the potential to automate the process of identifying these rare, high value data points signiﬁcantly, but existing methods become intractable at this scale.
Revised: Active learning and search have the potential to automate the process of identifying these rare, high value data points signiﬁcantly, but existing methods become intractable at this scale.

Original: Speciﬁcally, the goal of active learning is to reduce the cost of labeling (Settles, 2012).
Revised: Speciﬁcally, the goal of active learning is to reduce the cost of labeling (Settles, 2012).

Original: To this end, the learning algorithm is allowed to choose which data to label based on uncertainty (e.g., the entropy of predicted class probabilities) or other heuristics (Settles, 2011; 2012; Lewis & Gale, 1994).
Revised: To this end, the learning algorithm is allowed to choose which data to label based on uncertainty (e.g., the entropy of predicted class probabilities) or other heuristics (Settles, 2011; 2012; Lewis & Gale, 1994).

Original: Active search is a sub-area focused on ﬁnding positive examples in skewed distributions (Garnett et al., 2012).
Revised: Active search is a sub-area focused on ﬁnding positive examples in skewed distributions (Garnett et al., 2012).

Original: Because of a concentrated focus on labeling costs, existing techniques, such as uncertainty sampling (Lewis & Gale, 1994) or information density (Settles & Craven, 2008), perform multiple selection rounds and iterate over the entire unlabeled data to identify the optimal example or batch of examples to label and scale linearly or even quadratically with the size of the unlabeled data.
Revised: Because of a concentrated focus on labeling costs, existing techniques, such as uncertainty sampling (Lewis & Gale, 1994) or information density (Settles & Craven, 2008), perform multiple selection rounds and iterate over the entire unlabeled data to identify the optimal example or batch of examples to label and scale linearly or even quadratically with the size of the unlabeled data.

Original: Computational efﬁciency is becoming an impediment as the size of datasets and model complexities have increased (Amodei & Hernandez, 2018).
Revised: Computational efﬁciency is becoming an impediment as the size of datasets and model complexities have increased (Amodei & Hernandez, 2018).

Original: Recent work has tried to address this problem with sophisticated methods to select larger and more diverse batches of examples in each selection round and reduce the total number of rounds needed to reach the target labeling budget (Sener & Savarese, 2018;
Revised: Recent work has tried to address this problem with sophisticated methods to select larger and more diverse batches of examples in each selection round and reduce the total number of rounds needed to reach the target labeling budget (Sener & Savarese, 2018;

Original: Kirsch et al., 2019; Coleman et al., 2020; Pinsler et al., 2019; Jiang et al., 2018).
Revised: Kirsch et al., 2019; Coleman et al., 2020; Pinsler et al., 2019; Jiang et al., 2018).

Original: Nevertheless, these approaches still scan over all of the examples to ﬁnd the optimal examples to label in each round and can be intractable for large-scale unlabeled datasets.
Revised: Nevertheless, these approaches still scan over all of the examples to ﬁnd the optimal examples to label in each round and can be intractable for large-scale unlabeled datasets.

Original: For example, running a single inference pass over 10 billion images with ResNet-50 (He et al., 2016) would take 38 exaFLOPs.
Revised: For example, running a single inference pass over 10 billion images with ResNet-50 (He et al., 2016) would take 38 exaFLOPs.

Original: In this work, we propose Similarity search for Efﬁcient Active Learning and Search (SEALS) to restrict the candidates considered in each selection round and vastly reduce the computational complexity of active learning and search methods.
Revised: In this work, we propose Similarity search for Efﬁcient Active Learning and Search (SEALS) to restrict the candidates considered in each selection round and vastly reduce the computational complexity of active learning and search methods.

Original: Empirically, we ﬁnd that learned representations from pre-trained models can effectively cluster many unseen and rare concepts.
Revised: Empirically, we ﬁnd that learned representations from pre-trained models can effectively cluster many unseen and rare concepts.

Original: We exploit this latent structure to improve the computational efﬁciency of active learning and search methods by only considering the nearest neighbors of the currently labeled examples in each selection round.
Revised: We exploit this latent structure to improve the computational efﬁciency of active learning and search methods by only considering the nearest neighbors of the currently labeled examples in each selection round.

Original: This can be done transparently for many selection strategies making SEALS widely applicable.
Revised: This can be done transparently for many selection strategies making SEALS widely applicable.

Original: Finding the nearest neighbors for each labeled example in unlabeled data can be performed efﬁciently with sublinear retrieval times (Charikar, 2002) and sub-second latency on billion-scale datasets (Johnson et al., 2017) for approximate approaches.
Revised: Finding the nearest neighbors for each labeled example in unlabeled data can be performed efﬁciently with sublinear retrieval times (Charikar, 2002) and sub-second latency on billion-scale datasets (Johnson et al., 2017) for approximate approaches.

Original: While constructing the index for similarity search requires at least a linear pass over the unlabeled data, this computational cost is effectively amortized over many selection rounds or other applications.
Revised: While constructing the index for similarity search requires at least a linear pass over the unlabeled data, this computational cost is effectively amortized over many selection rounds or other applications.

Original: As a result, our SEALS approach enables selection to scale with the size of the labeled data rather than the size of the unlabeled data, making active learning and search tractable on datasets with billions of unlabeled examples.
Revised: As a result, our SEALS approach enables selection to scale with the size of the labeled data rather than the size of the unlabeled data, making active learning and search tractable on datasets with billions of unlabeled examples.

Original: We empirically evaluated SEALS for both active learning and search on three large scale computer vision datasets: ImageNet (Russakovsky et al., 2015), OpenImages (Kuznetsova et al., 2020), and a proprietary dataset of 10 billion images from a large internet company.
Revised: We empirically evaluated SEALS for both active learning and search on three large scale computer vision datasets: ImageNet (Russakovsky et al., 2015), OpenImages (Kuznetsova et al., 2020), and a proprietary dataset of 10 billion images from a large internet company.

Original: We selected 611 concepts spread across these datasets that range in prevalence from 0.203% to 0.002% (1 in 50,000) of the training examples.
Revised: We selected 611 concepts spread across these datasets that range in prevalence from 0.203% to 0.002% (1 in 50,000) of the training examples.

Original: We evaluated three selection strategies for each concept: max entropy uncertainty sampling (Lewis & Gale, 1994), information density (Settles & Craven, 2008), and most-likely positive (Warmuth et al., 2002; 2003; Jiang et al., 2018).
Revised: We evaluated three selection strategies for each concept: max entropy uncertainty sampling (Lewis & Gale, 1994), information density (Settles & Craven, 2008), and most-likely positive (Warmuth et al., 2002; 2003; Jiang et al., 2018).

Original: Across datasets, selection strategies, and concepts, SEALS achieved similar model quality and nearly the same recall of the positive examples as the baseline approaches, while improving the computational complexity by orders of magnitude.
Revised: Across datasets, selection strategies, and concepts, SEALS achieved similar model quality and nearly the same recall of the positive examples as the baseline approaches, while improving the computational complexity by orders of magnitude.

Original: On ImageNet with a budget of 2,000 binary labels per concept (˜0.31% of the unlabeled data), all baseline and SEALS approaches were within 0.011 mAP of full supervision and recalled over 50% of the positive examples.
Revised: On ImageNet with a budget of 2,000 binary labels per concept (˜0.31% of the unlabeled data), all baseline and SEALS approaches were within 0.011 mAP of full supervision and recalled over 50% of the positive examples.

Original: On OpenImages, SEALS reduced the candidate pool to 1% of the unlabeled data on average while remaining within 0.013 mAP and 0.1% recall of the baseline approaches.
Revised: On OpenImages, SEALS reduced the candidate pool to 1% of the unlabeled data on average while remaining within 0.013 mAP and 0.1% recall of the baseline approaches.

Original: On the proprietary dataset with 10 billion images, SEALS needed an even smaller fraction of the data, about 0.1%, to match the baseline, which allowed SEALS to run on a single machine rather than a cluster.
Revised: On the proprietary dataset with 10 billion images, SEALS needed an even smaller fraction of the data, about 0.1%, to match the baseline, which allowed SEALS to run on a single machine rather than a cluster.

Original: To the best of our knowledge, no other works have performed active learning at this scale.
Revised: To the best of our knowledge, no other works have performed active learning at this scale.

Original: We also applied SEALS to the NLP spoiler detection dataset Goodreads (Wan et al., 2019), where it achieved the same recall as the baseline approaches while only considering less than 1% of the unlabeled data.
Revised: We also applied SEALS to the NLP spoiler detection dataset Goodreads (Wan et al., 2019), where it achieved the same recall as the baseline approaches while only considering less than 1% of the unlabeled data.

Original: Together, these results demonstrate that SEALS’ improvements to computational efﬁciency make active learning and search tractable for even billion-scale datasets.
Revised: Together, these results demonstrate that SEALS’ improvements to computational efﬁciency make active learning and search tractable for even billion-scale datasets.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Active learning ’s iterative retraining combined with the high computational complexity of deep learning models has led to signiﬁcant work on computational efﬁciency (Sener & Savarese, 2018;
Revised: Active learning ’s iterative retraining combined with the high computational complexity of deep learning models has led to signiﬁcant work on computational efﬁciency (Sener & Savarese, 2018;

Original: Kirsch et al., 2019; Pinsler et al., 2019; Coleman et al., 2020; Yoo & Kweon, 2019; Mayer & Timofte, 2020; Zhu & Bento, 2017).
Revised: Kirsch et al., 2019; Pinsler et al., 2019; Coleman et al., 2020; Yoo & Kweon, 2019; Mayer & Timofte, 2020; Zhu & Bento, 2017).

Original: One branch of recent work has focused on selecting large batches of data to minimize the amount of retraining and reduce the number of selection rounds necessary to reach a target budget (Sener & Savarese, 2018;
Revised: One branch of recent work has focused on selecting large batches of data to minimize the amount of retraining and reduce the number of selection rounds necessary to reach a target budget (Sener & Savarese, 2018;

Original: Kirsch et al., 2019; Pinsler et al., 2019).
Revised: Kirsch et al., 2019; Pinsler et al., 2019).

Original: These approaches introduce novel techniques to avoid selecting highly similar or redundant examples and ensure the batches are both informative and diverse.
Revised: These approaches introduce novel techniques to avoid selecting highly similar or redundant examples and ensure the batches are both informative and diverse.

Original: In comparison, our work aims to reduce the number of examples considered in each selection round and complements existing work on batch active learning.
Revised: In comparison, our work aims to reduce the number of examples considered in each selection round and complements existing work on batch active learning.

Original: Many of these approaches sacriﬁce computational complexity to ensure diversity, and their selection methods can scale quadratically with the size of the unlabeled data.
Revised: Many of these approaches sacriﬁce computational complexity to ensure diversity, and their selection methods can scale quadratically with the size of the unlabeled data.

Original: Combined with our method, these selection methods scale with the size of the labeled data rather than the unlabeled data.
Revised: Combined with our method, these selection methods scale with the size of the labeled data rather than the unlabeled data.

Original: Outside of batch active learning, other work has tried to improve computational efﬁciency by either using much smaller models as cheap proxies during selection (Yoo & Kweon, 2019; Coleman et al., 2020) or by generating examples (Mayer & Timofte, 2020; Zhu & Bento, 2017).
Revised: Outside of batch active learning, other work has tried to improve computational efﬁciency by either using much smaller models as cheap proxies during selection (Yoo & Kweon, 2019; Coleman et al., 2020) or by generating examples (Mayer & Timofte, 2020; Zhu & Bento, 2017).

Original: Using a smaller model reduces the amount of computation per example, but unlike our approach, it still requires making multiple passes over the entire unlabeled pool of examples.
Revised: Using a smaller model reduces the amount of computation per example, but unlike our approach, it still requires making multiple passes over the entire unlabeled pool of examples.

Original: The generative approaches (Mayer & Timofte, 2020; Zhu & Bento, 2017), however, enable sub-linear runtime complexity like our approach.
Revised: The generative approaches (Mayer & Timofte, 2020; Zhu & Bento, 2017), however, enable sub-linear runtime complexity like our approach.

Original: Unfortunately, they struggle to match the label-efﬁciency of traditional approaches because the quality of the generated examples is highly variable.
Revised: Unfortunately, they struggle to match the label-efﬁciency of traditional approaches because the quality of the generated examples is highly variable.

Original: Active search is a sub-area of active learning that focuses on highly skewed class distributions (Garnett et al., 2012; Jiang et al., 2017; 2018; 2019).
Revised: Active search is a sub-area of active learning that focuses on highly skewed class distributions (Garnett et al., 2012; Jiang et al., 2017; 2018; 2019).

Original: Rather than optimizing for model quality, active search aims to ﬁnd as many examples from the minority class as possible.
Revised: Rather than optimizing for model quality, active search aims to ﬁnd as many examples from the minority class as possible.

Original: Prior work has focused on applications such as drug discovery, where the dataset sizes are limited, and labeling costs are exceptionally high.
Revised: Prior work has focused on applications such as drug discovery, where the dataset sizes are limited, and labeling costs are exceptionally high.

Original: Our work similarly focuses on skewed distributions.
Revised: Our work similarly focuses on skewed distributions.

Original: However, we consider novel active search settings in image and text where the available unlabeled datasets are much larger, and computational efﬁciency is a signiﬁcant bottleneck.
Revised: However, we consider novel active search settings in image and text where the available unlabeled datasets are much larger, and computational efﬁciency is a signiﬁcant bottleneck.

Original: k nearest neighbor ( k -NN) classiﬁers are popular models in active learning and search because they do not require an explicit training phase (Joshi et al., 2012; Wei et al., 2015; Garnett et al., 2012; Jiang et al., 2017; 2018).
Revised: k nearest neighbor ( k -NN) classiﬁers are popular models in active learning and search because they do not require an explicit training phase (Joshi et al., 2012; Wei et al., 2015; Garnett et al., 2012; Jiang et al., 2017; 2018).

Original: The prediction and score for each unlabeled example can be updated immediately after each new batch of labels.
Revised: The prediction and score for each unlabeled example can be updated immediately after each new batch of labels.

Original: In comparison, our SEALS approach uses k -NN algorithms for similarity search to create and expand the candidate pool and not as a classiﬁer.
Revised: In comparison, our SEALS approach uses k -NN algorithms for similarity search to create and expand the candidate pool and not as a classiﬁer.

Original: This is an important but subtle difference.
Revised: This is an important but subtle difference.

Original: While prior work avoids expensive training by using k -NN classiﬁers, these approaches still require evaluating all of the unlabeled examples, which can still be prohibitively expensive on large-scale datasets like the ones we consider here.
Revised: While prior work avoids expensive training by using k -NN classiﬁers, these approaches still require evaluating all of the unlabeled examples, which can still be prohibitively expensive on large-scale datasets like the ones we consider here.

Original: SEALS targets the selection phase rather than training, presenting a novel and complementary approach.
Revised: SEALS targets the selection phase rather than training, presenting a novel and complementary approach.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we outline the problems of active learning (Section 3.1) and search (Section 3.2) formally as well as the selection methods we accelerate using SEALS.
Revised: In this section, we outline the problems of active learning (Section 3.1) and search (Section 3.2) formally as well as the selection methods we accelerate using SEALS.

Original: For both, we examine the pool-based setting, where all of the unlabeled data is available at once, and examples are selected in batches to improve computational efﬁciency, as mentioned above.
Revised: For both, we examine the pool-based setting, where all of the unlabeled data is available at once, and examples are selected in batches to improve computational efﬁciency, as mentioned above.

Original: Then in Section 3.3, we describe our SEALS approach and how it further improves computational efﬁciency in both settings.
Revised: Then in Section 3.3, we describe our SEALS approach and how it further improves computational efﬁciency in both settings.

Original: 3. A CTIVE LEARNING
Revised: 3. A CTIVE LEARNING

Original: Pool-based active learning is an iterative process that begins with a large pool of unlabeled data
Revised: Pool-based active learning is an iterative process that begins with a large pool of unlabeled data

Original: U = { x 1 , . . . , x n } .
Revised: U = { x 1 , . . . , x n } .

Original: Each example is sampled from the space X with an unknown label from the label space Y = { 1 , . .
Revised: Each example is sampled from the space X with an unknown label from the label space Y = { 1 , . .

Original: , C } as ( x i , y i ) .
Revised: , C } as ( x i , y i ) .

Original: We additionally assume a feature extraction function G z to embed each x
Revised: We additionally assume a feature extraction function G z to embed each x

Original: i as a latent variable G z ( x i ) =
Revised: i as a latent variable G z ( x i ) =

Original: z i and that the C concepts are unequally distributed.
Revised: z i and that the C concepts are unequally distributed.

Original: Speciﬁcally, there are one or more valuable rare concepts R ⊂ C that appear in less than 1% of the unlabeled data.
Revised: Speciﬁcally, there are one or more valuable rare concepts R ⊂ C that appear in less than 1% of the unlabeled data.

Original: For simplicity, we frame this as | R | binary classiﬁcation problems solved independently rather than 1 multi-class classiﬁcation problem with | R | concepts.
Revised: For simplicity, we frame this as | R | binary classiﬁcation problems solved independently rather than 1 multi-class classiﬁcation problem with | R | concepts.

Original: Initially, each rare concept has a small number of positive examples and several negative examples that serve as a labeled seed set L 0 r .
Revised: Initially, each rare concept has a small number of positive examples and several negative examples that serve as a labeled seed set L 0 r .

Original: The goal of active learning is to take this seed set and select up to a budget of T examples to label that produce a model A T r that achieves low error.
Revised: The goal of active learning is to take this seed set and select up to a budget of T examples to label that produce a model A T r that achieves low error.

Original: For each round t in pool-based active learning, the most informative examples are selected according to the selection strategy φ from a pool of candidate examples P r in batches of size b and labeled, as shown in Algorithm 1.
Revised: For each round t in pool-based active learning, the most informative examples are selected according to the selection strategy φ from a pool of candidate examples P r in batches of size b and labeled, as shown in Algorithm 1.

Original: For the baseline approach, P r = { G z ( x ) | x ∈ U } , meaning that all the unlabeled examples are considered to ﬁnd the global optimal according to φ .
Revised: For the baseline approach, P r = { G z ( x ) | x ∈ U } , meaning that all the unlabeled examples are considered to ﬁnd the global optimal according to φ .

Original: Between each round, the model A tr is trained on all of the labeled data L t r , allowing the selection process to adapt.
Revised: Between each round, the model A tr is trained on all of the labeled data L t r , allowing the selection process to adapt.

Original: In this paper, we considered max entropy (MaxEnt) uncertainty sampling (Lewis & Gale, 1994):
Revised: In this paper, we considered max entropy (MaxEnt) uncertainty sampling (Lewis & Gale, 1994):

Original: [Equation]
Revised: [Equation]

Original: and information density (ID) (Settles & Craven, 2008):
Revised: and information density (ID) (Settles & Craven, 2008):

Original: [Equation]
Revised: [Equation]

Original: where sim ( z , z p ) is the cosine similarity of the embedded examples and β = 1 .
Revised: where sim ( z , z p ) is the cosine similarity of the embedded examples and β = 1 .

Original: Note that for binary classiﬁcation, max entropy is equivalent to least conﬁdence and margin sampling, which are also popular criteria for uncertainty sampling (Settles, 2009).
Revised: Note that for binary classiﬁcation, max entropy is equivalent to least conﬁdence and margin sampling, which are also popular criteria for uncertainty sampling (Settles, 2009).

Original: While max entropy uncertainty sampling only requires a linear pass over the unlabeled data, ID scales quadratically with | U | because it weights each example’s informativeness by its similarity to all other examples.
Revised: While max entropy uncertainty sampling only requires a linear pass over the unlabeled data, ID scales quadratically with | U | because it weights each example’s informativeness by its similarity to all other examples.

Original: To improve computational performance, the average similarity score for each example can be cached after the ﬁrst selection round, so subsequent rounds scale linearly.
Revised: To improve computational performance, the average similarity score for each example can be cached after the ﬁrst selection round, so subsequent rounds scale linearly.

Original: This optimization only works when G z is ﬁxed and would not apply to dynamic similarity calculations like those in Sener & Savarese (2018).
Revised: This optimization only works when G z is ﬁxed and would not apply to dynamic similarity calculations like those in Sener & Savarese (2018).

Original: We explored the greedy k-centers approach from Sener & Savarese (2018) but found that it never outperformed random sampling for our experimental setup.
Revised: We explored the greedy k-centers approach from Sener & Savarese (2018) but found that it never outperformed random sampling for our experimental setup.

Original: Unlike MaxEnt and ID, k-centers does not consider the predicted labels.
Revised: Unlike MaxEnt and ID, k-centers does not consider the predicted labels.

Original: It tries to achieve high coverage over the entire candidate pool, of which rare concepts make up a small fraction by deﬁnition, making it ineffective for our setting.
Revised: It tries to achieve high coverage over the entire candidate pool, of which rare concepts make up a small fraction by deﬁnition, making it ineffective for our setting.

Original: Algorithm
Revised: Algorithm

Original: Input: unlabeled data U , labeled seed set L 0 r , feature extractor G z , selection strategy φ ( · ) , batch size b , labeling budget T 1: L r = { ( G z ( x ) , y ) |
Revised: Input: unlabeled data U , labeled seed set L 0 r , feature extractor G z , selection strategy φ ( · ) , batch size b , labeling budget T 1: L r = { ( G z ( x ) , y ) |

Original: ( x , y ) ∈ L 0 r } 2: P r = { G z ( x ) | x ∈ U and ( x , · ) (cid:54)∈ L
Revised: ( x , y ) ∈ L 0 r } 2: P r = { G z ( x ) | x ∈ U and ( x , · ) (cid:54)∈ L

Original: r } 3: repeat 4:
Revised: r } 3: repeat 4:

Original: A r = train ( L r ) 5: for 1 to b do 6: z ∗ = arg max z ∈P r φ ( z ) 7:
Revised: A r = train ( L r ) 5: for 1 to b do 6: z ∗ = arg max z ∈P r φ ( z ) 7:

Original: L r =
Revised: L r =

Original: L r ∪ { ( z ∗ , label ( x ∗ )) } 8:
Revised: L r ∪ { ( z ∗ , label ( x ∗ )) } 8:

Original: P r = P r \ { z ∗ } 9: end for 10: until |L
Revised: P r = P r \ { z ∗ } 9: end for 10: until |L

Original: r | =
Revised: r | =

Original: T
Revised: T

Original: Input: unlabeled data U , labeled seed set
Revised: Input: unlabeled data U , labeled seed set

Original: L 0 r , feature extractor G z , selection strategy φ ( · ) , batch size b , labeling budget T , knearest neighbors implementation N ( · , · ) 1: L r = { ( G z ( x ) , y ) |
Revised: L 0 r , feature extractor G z , selection strategy φ ( · ) , batch size b , labeling budget T , knearest neighbors implementation N ( · , · ) 1: L r = { ( G z ( x ) , y ) |

Original: ( x , y ) ∈ L 0 r } 2: P r = ∪ ( z ,y ) ∈L r N ( z , k ) 3: repeat 4:
Revised: ( x , y ) ∈ L 0 r } 2: P r = ∪ ( z ,y ) ∈L r N ( z , k ) 3: repeat 4:

Original: A r = train ( L r ) 5: for 1 to b do 6: z ∗ = arg max z ∈P r φ ( z ) 7:
Revised: A r = train ( L r ) 5: for 1 to b do 6: z ∗ = arg max z ∈P r φ ( z ) 7:

Original: L r =
Revised: L r =

Original: L r ∪ { ( z ∗ , label ( x ∗ )) } 8:
Revised: L r ∪ { ( z ∗ , label ( x ∗ )) } 8:

Original: P r =
Revised: P r =

Original: ( P r \ { z ∗ } ) ∪ N ( z ∗ , k ) 9: end for 10: until |L r | =
Revised: ( P r \ { z ∗ } ) ∪ N ( z ∗ , k ) 9: end for 10: until |L r | =

Original: T
Revised: T

Original: 3. A CTIVE SEARCH
Revised: 3. A CTIVE SEARCH

Original: Active search is closely related to active learning, so much of the formalism from Section 3.1 carries over.
Revised: Active search is closely related to active learning, so much of the formalism from Section 3.1 carries over.

Original: The critical difference is that rather than selecting examples to label that minimize error, the goal of active search is to maximize the number of examples from the target concept r , expressed with the natural utility function u
Revised: The critical difference is that rather than selecting examples to label that minimize error, the goal of active search is to maximize the number of examples from the target concept r , expressed with the natural utility function u

Original: ( L r )
Revised: ( L r )

Original: = (cid:80) ( x ,y
Revised: = (cid:80) ( x ,y

Original: ) ∈
Revised: ) ∈

Original: L r 1 { y = r } ).
Revised: L r 1 { y = r } ).

Original: As a result, different selection strategies are favored, but the overall algorithm is the same as Algorithm 1.
Revised: As a result, different selection strategies are favored, but the overall algorithm is the same as Algorithm 1.

Original: In this paper, we consider an additional selection strategy to target the active search setting, mostlikely positive (MLP) (Warmuth et al., 2002; 2003; Jiang et al., 2018):
Revised: In this paper, we consider an additional selection strategy to target the active search setting, mostlikely positive (MLP) (Warmuth et al., 2002; 2003; Jiang et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: Because active learning and search are similar, we evaluate all the selection criteria from Sections 3. and 3.2 in terms of both the error the model achieves and the number of positive examples.
Revised: Because active learning and search are similar, we evaluate all the selection criteria from Sections 3. and 3.2 in terms of both the error the model achieves and the number of positive examples.

Original: 3. S IMILARITY SEARCH FOR EFFICIENT ACTIVE LEARNING AND SEARCH (SEALS)
Revised: 3. S IMILARITY SEARCH FOR EFFICIENT ACTIVE LEARNING AND SEARCH (SEALS)

Original: In this work, we propose SEALS to accelerate the inner loop of active learning and search by restricting the candidate pool of unlabeled examples.
Revised: In this work, we propose SEALS to accelerate the inner loop of active learning and search by restricting the candidate pool of unlabeled examples.

Original: To apply SEALS, we use an efﬁcient method for similarity search of the embedded examples (Charikar, 2002; Johnson et al., 2017) and make two modiﬁcations to the baseline approach, as shown in Algorithm 2:
Revised: To apply SEALS, we use an efﬁcient method for similarity search of the embedded examples (Charikar, 2002; Johnson et al., 2017) and make two modiﬁcations to the baseline approach, as shown in Algorithm 2:

Original: The candidate pool P r is restricted to the nearest neighbors of the labeled examples.
Revised: The candidate pool P r is restricted to the nearest neighbors of the labeled examples.

Original: After every example is selected, we ﬁnd its k nearest neighbors and update P r .
Revised: After every example is selected, we ﬁnd its k nearest neighbors and update P r .

Original: Computational savings.
Revised: Computational savings.

Original: Restricting the candidate pool P r to the k nearest neighbors of the labeled examples means we only apply the selection strategy to at most k | L r | examples.
Revised: Restricting the candidate pool P r to the k nearest neighbors of the labeled examples means we only apply the selection strategy to at most k | L r | examples.

Original: This can be done transparently for many selection strategies making it applicable to a wide range of active learning and search methods, even beyond the ones considered here.
Revised: This can be done transparently for many selection strategies making it applicable to a wide range of active learning and search methods, even beyond the ones considered here.

Original: Finding the k nearest neighbors for each newly labeled example adds overhead, but this can be calculated efﬁciently with sublinear retrieval times (Charikar, 2002) and sub-second latency on billion-scale datasets (Johnson et al., 2017) for approximate approaches.
Revised: Finding the k nearest neighbors for each newly labeled example adds overhead, but this can be calculated efﬁciently with sublinear retrieval times (Charikar, 2002) and sub-second latency on billion-scale datasets (Johnson et al., 2017) for approximate approaches.

Original: As a result, the computational complexity of each selection round scales with the size of the labeled dataset rather than the unlabeled dataset.
Revised: As a result, the computational complexity of each selection round scales with the size of the labeled dataset rather than the unlabeled dataset.

Original: Excluding the retrieval times for the k nearest neighbors, the computational savings from SEALS are directly proportional to the pool size reduction for φ MaxEnt and φ MLP , which is lower bounded by | U | /k
Revised: Excluding the retrieval times for the k nearest neighbors, the computational savings from SEALS are directly proportional to the pool size reduction for φ MaxEnt and φ MLP , which is lower bounded by | U | /k

Original: | L r | .
Revised: | L r | .

Original: For φ ID , the average similarity score for each example only needs to be computed once when the example is ﬁrst selected.
Revised: For φ ID , the average similarity score for each example only needs to be computed once when the example is ﬁrst selected.

Original: This caching means the ﬁrst round scales quadratically with | U | and subsequent rounds scale linearly for the baseline approach.
Revised: This caching means the ﬁrst round scales quadratically with | U | and subsequent rounds scale linearly for the baseline approach.

Original: With SEALS, each selection round scales according to
Revised: With SEALS, each selection round scales according to

Original: O ((1 + bk )
Revised: O ((1 + bk )

Original: |P
Revised: |P

Original: r | ) because the similarity scores are calculated as examples are selected rather than all at once.
Revised: r | ) because the similarity scores are calculated as examples are selected rather than all at once.

Original: The resulting computational savings of SEALS varies with the labeling budget T as the upfront cost of the baseline amortizes.
Revised: The resulting computational savings of SEALS varies with the labeling budget T as the upfront cost of the baseline amortizes.

Original: Nevertheless, for large-scale datasets with millions or billions of examples, performing that ﬁrst quadratic round for the baseline is prohibitively expensive.
Revised: Nevertheless, for large-scale datasets with millions or billions of examples, performing that ﬁrst quadratic round for the baseline is prohibitively expensive.

Original: Index construction.
Revised: Index construction.

Original: Generating the embeddings and indexing the data can be expensive and slow, requiring at least a linear pass over the unlabeled data.
Revised: Generating the embeddings and indexing the data can be expensive and slow, requiring at least a linear pass over the unlabeled data.

Original: However, this cost is effectively amortized over many selection rounds, concepts, or other applications.
Revised: However, this cost is effectively amortized over many selection rounds, concepts, or other applications.

Original: Similarity search is a critical workload for information retrieval and powers many applications, including recommendation.
Revised: Similarity search is a critical workload for information retrieval and powers many applications, including recommendation.

Original: Increasingly, embeddings from deep learning models are being used (Babenko et al., 2014; Babenko & Lempitsky, 2016; Johnson et al., 2017).
Revised: Increasingly, embeddings from deep learning models are being used (Babenko et al., 2014; Babenko & Lempitsky, 2016; Johnson et al., 2017).

Original: As a result, the embeddings and index can be generated once using a generic model trained in a weak-supervision or self-supervision fashion and reused, making our approach just one of many applications using the index.
Revised: As a result, the embeddings and index can be generated once using a generic model trained in a weak-supervision or self-supervision fashion and reused, making our approach just one of many applications using the index.

Original: Alternatively, if the data has already been passed through a predictive system (for example, to tag or classify uploaded images), the embedding could be captured and indexed at inference to avoid additional costs.
Revised: Alternatively, if the data has already been passed through a predictive system (for example, to tag or classify uploaded images), the embedding could be captured and indexed at inference to avoid additional costs.

Original:  R ESULTS
Revised:  R ESULTS

Original: We applied SEALS to three selection strategies and performed active learning and search on three datasets: ImageNet (Russakovsky et al., 2015), OpenImages (Kuznetsova et al., 2020), and a proprietary dataset of 10 billion images.
Revised: We applied SEALS to three selection strategies and performed active learning and search on three datasets: ImageNet (Russakovsky et al., 2015), OpenImages (Kuznetsova et al., 2020), and a proprietary dataset of 10 billion images.

Original: Section 4.1 details the experimental setup for each dataset and the inputs used for both the baseline approach (Algorithm 1) and our proposed method, SEALS (Algorithm 2).
Revised: Section 4.1 details the experimental setup for each dataset and the inputs used for both the baseline approach (Algorithm 1) and our proposed method, SEALS (Algorithm 2).

Original: Sections 4.2 and 4.3 present the empirical results for active learning and search.
Revised: Sections 4.2 and 4.3 present the empirical results for active learning and search.

Original: Section 4.
Revised: Section 4.

Original: explores the structure of the concepts through the nearest neighbor graphs and embeddings.
Revised: explores the structure of the concepts through the nearest neighbor graphs and embeddings.

Original: Across selection strategies, datasets, and concepts, SEALS using ResNet-50 (He et al., 2016) embeddings performed similarly to the baseline while only considering a fraction of the unlabeled data
Revised: Across selection strategies, datasets, and concepts, SEALS using ResNet-50 (He et al., 2016) embeddings performed similarly to the baseline while only considering a fraction of the unlabeled data

Original: U in the candidate pool for each concept P r .
Revised: U in the candidate pool for each concept P r .

Original: For MLP and MaxEnt, the smaller candidate pool from
Revised: For MLP and MaxEnt, the smaller candidate pool from

Original: SEALS sped-up the selection runtime by over 180 × on OpenImages.
Revised: SEALS sped-up the selection runtime by over 180 × on OpenImages.

Original: This allowed us to run active learning and search efﬁciently on an industrial scale dataset with 10 billion images.
Revised: This allowed us to run active learning and search efﬁciently on an industrial scale dataset with 10 billion images.

Original: The improvements were even larger for information density.
Revised: The improvements were even larger for information density.

Original: On ImageNet, SEALS dropped the time for the ﬁrst selection round from over 75 minutes to 1.5 seconds, over a 3000 × improvement.
Revised: On ImageNet, SEALS dropped the time for the ﬁrst selection round from over 75 minutes to 1.5 seconds, over a 3000 × improvement.

Original: On OpenImages, the baseline for information density ran for over 24 hours without completing a single round, while SEALS took less than 3 minutes to perform 19 rounds.
Revised: On OpenImages, the baseline for information density ran for over 24 hours without completing a single round, while SEALS took less than 3 minutes to perform 19 rounds.

Original: We observed similar results with selfsupervised embeddings using SimCLR (Chen et al., 2020) for ImageNet (Appendix A.3) and using
Revised: We observed similar results with selfsupervised embeddings using SimCLR (Chen et al., 2020) for ImageNet (Appendix A.3) and using

Original: SentenceBERT (Reimers & Gurevych, 2019) for Goodreads spoiler detection (Appendix A.8).
Revised: SentenceBERT (Reimers & Gurevych, 2019) for Goodreads spoiler detection (Appendix A.8).

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: Across all datasets and selection strategies, we followed the same general procedure for both active learning and search.
Revised: Across all datasets and selection strategies, we followed the same general procedure for both active learning and search.

Original: Because we are interested in rare concepts, we kept the number of initial positive examples small.
Revised: Because we are interested in rare concepts, we kept the number of initial positive examples small.

Original: We evaluated three settings, with 5, 20, and 50 positives, but only included the results with the smallest size in this section.
Revised: We evaluated three settings, with 5, 20, and 50 positives, but only included the results with the smallest size in this section.

Original: The others are shown in Appendix A.1.
Revised: The others are shown in Appendix A.1.

Original: For each setting, negative examples were randomly selected at a ratio of 19 negative examples to every positive example to form the seed set L 0 r .
Revised: For each setting, negative examples were randomly selected at a ratio of 19 negative examples to every positive example to form the seed set L 0 r .

Original: The slightly higher number of negatives in the initial seed set improved average precision on the validation set across all three datasets.
Revised: The slightly higher number of negatives in the initial seed set improved average precision on the validation set across all three datasets.

Original: The batch size b for each selection round was the same as the size of the initial seed set.
Revised: The batch size b for each selection round was the same as the size of the initial seed set.

Original: For the seed set of 5 positive andnegative examples shown below, b was 100, and the labeling budget T was 2,000 examples.
Revised: For the seed set of 5 positive andnegative examples shown below, b was 100, and the labeling budget T was 2,000 examples.

Original: As the binary classiﬁer for each concept A r , we used logistic regression trained on the embedded examples.
Revised: As the binary classiﬁer for each concept A r , we used logistic regression trained on the embedded examples.

Original: For active learning, we calculated average precision on the test data for each binary concept classiﬁer after each selection round.
Revised: For active learning, we calculated average precision on the test data for each binary concept classiﬁer after each selection round.

Original: For active search, we count the number of positive examples labeled so far.
Revised: For active search, we count the number of positive examples labeled so far.

Original: We take the mean average precision (mAP) and number of positives across concepts, run each experiment 5 times, and report the mean and standard deviation.
Revised: We take the mean average precision (mAP) and number of positives across concepts, run each experiment 5 times, and report the mean and standard deviation.

Original: For similarity search, we used locality-sensitive hashing (LSH) (Charikar, 2002) implemented in
Revised: For similarity search, we used locality-sensitive hashing (LSH) (Charikar, 2002) implemented in

Original: Faiss (Johnson et al., 2017) with Euclidean distance for all datasets aside from the 10 billion images dataset.
Revised: Faiss (Johnson et al., 2017) with Euclidean distance for all datasets aside from the 10 billion images dataset.

Original: This simpliﬁed our implementation, so the index could be created quickly and independently for each concept and conﬁguration, allowing experiments to run in parallel trivially.
Revised: This simpliﬁed our implementation, so the index could be created quickly and independently for each concept and conﬁguration, allowing experiments to run in parallel trivially.

Original: However, retrieval times for this approach were not as fast as Johnson et al.
Revised: However, retrieval times for this approach were not as fast as Johnson et al.

Original: and made up a larger part of the overall active learning loop.
Revised: and made up a larger part of the overall active learning loop.

Original: In practice, the search index can be heavily optimized and tuned for the speciﬁc data distribution, leading to computational savings closer to the improvements described in Section 3.3 and differences in the “Selection” portion of the runtimes in Table 2.
Revised: In practice, the search index can be heavily optimized and tuned for the speciﬁc data distribution, leading to computational savings closer to the improvements described in Section 3.3 and differences in the “Selection” portion of the runtimes in Table 2.

Original: We split the data, selected concepts, and created embeddings as detailed below and summarized in
Revised: We split the data, selected concepts, and created embeddings as detailed below and summarized in

Original: Table 1.
Revised: Table 1.

Original: Note that our approach does not constrain the choice of G z , which allows for many network architectures.
Revised: Note that our approach does not constrain the choice of G z , which allows for many network architectures.

Original: As representations continue to improve with new self-supervision, generative, or transfer learning techniques, SEALS is still applicable and performance will also likely improve.
Revised: As representations continue to improve with new self-supervision, generative, or transfer learning techniques, SEALS is still applicable and performance will also likely improve.

Original: ImageNet (Russakovsky et al., 2015) has 1.28 million training images spread over 1000 classes.
Revised: ImageNet (Russakovsky et al., 2015) has 1.28 million training images spread over 1000 classes.

Original: To simulate rare concepts, we split the data in half, using 500 classes to train the feature extractor
Revised: To simulate rare concepts, we split the data in half, using 500 classes to train the feature extractor

Original: G z and treating the other 500 classes as unseen concepts.
Revised: G z and treating the other 500 classes as unseen concepts.

Original: For G z , we used ResNet-50 but added a bottleneck layer before the ﬁnal output to reduce the dimension of the embeddings to 256.
Revised: For G z , we used ResNet-50 but added a bottleneck layer before the ﬁnal output to reduce the dimension of the embeddings to 256.

Original: We kept all of the other training hyperparameters the same as in He et al.
Revised: We kept all of the other training hyperparameters the same as in He et al.

Original: We extracted features from the bottleneck layer and applied l 2 normalization.
Revised: We extracted features from the bottleneck layer and applied l 2 normalization.

Original: In total, the 500 unseen concepts had 639, training examples that served as the unlabeled pool.
Revised: In total, the 500 unseen concepts had 639, training examples that served as the unlabeled pool.

Original: We used 50 concepts for validation, leaving the remaining 450 concepts for our ﬁnal experiments.
Revised: We used 50 concepts for validation, leaving the remaining 450 concepts for our ﬁnal experiments.

Original: The number of examples for each concept varied slightly, ranging from 0.114-0.203% of | U | .
Revised: The number of examples for each concept varied slightly, ranging from 0.114-0.203% of | U | .

Original: The 50,000 validation images were used as the test set.
Revised: The 50,000 validation images were used as the test set.

Original: OpenImages (Kuznetsova et al., 2020) has 7.34 million images with human-veriﬁed labels spread over 19,958 classes, taken as an unbiased sample from Flickr.
Revised: OpenImages (Kuznetsova et al., 2020) has 7.34 million images with human-veriﬁed labels spread over 19,958 classes, taken as an unbiased sample from Flickr.

Original: However, only 6.82 million images were still available in the training set at the time of writing.
Revised: However, only 6.82 million images were still available in the training set at the time of writing.

Original: As a feature extractor, we took ResNet50 pre-trained on all of ImageNet and used the l 2 normalized output from the bottleneck layer.
Revised: As a feature extractor, we took ResNet50 pre-trained on all of ImageNet and used the l 2 normalized output from the bottleneck layer.

Original: As rare concepts, we randomly selected 200 classes with between 100 to 6,817 positive training examples.
Revised: As rare concepts, we randomly selected 200 classes with between 100 to 6,817 positive training examples.

Original: We reviewed the selected classes and removed 47 classes that overlapped with ImageNet.
Revised: We reviewed the selected classes and removed 47 classes that overlapped with ImageNet.

Original: The remaining 153 classes appeared in 0.002-0.088% of the data.
Revised: The remaining 153 classes appeared in 0.002-0.088% of the data.

Original: We used the same hyperparameters as the ImageNet experiments and the OpenImages predeﬁned test split for evaluation.
Revised: We used the same hyperparameters as the ImageNet experiments and the OpenImages predeﬁned test split for evaluation.

Original: [Figure]
Revised: [Figure]

Original: 10 billion (10B) images from a large internet company were used to test SEALS’ scalability.
Revised: 10 billion (10B) images from a large internet company were used to test SEALS’ scalability.

Original: For the feature extractor, we used the same pre-trained ResNet-50 model as the OpenImages experiments.
Revised: For the feature extractor, we used the same pre-trained ResNet-50 model as the OpenImages experiments.

Original: We also selected 8 additional classes from OpenImages as rare concepts: rat, sushi, bowling, beach, hawk, cupcake
Revised: We also selected 8 additional classes from OpenImages as rare concepts: rat, sushi, bowling, beach, hawk, cupcake

Original: [Figure]
Revised: [Figure]

Original: and crowd.
Revised: and crowd.

Original: This allowed us to use the predeﬁned test split from OpenImages for evaluation.
Revised: This allowed us to use the predeﬁned test split from OpenImages for evaluation.

Original: Unlike the other datasets, we hired annotators to label images as they were selected and used a proprietary index to achieve low latency retrieval times to capture a real-world setting
Revised: Unlike the other datasets, we hired annotators to label images as they were selected and used a proprietary index to achieve low latency retrieval times to capture a real-world setting

Original: [Figure]
Revised: [Figure]

Original: 4. A CTIVE LEARNING
Revised: 4. A CTIVE LEARNING

Original: Across datasets and selection strategies, SEALS performed similarly to the baseline approaches that considered all of the unlabeled data in the candidate pool, as shown in Figures 1 and 2.
Revised: Across datasets and selection strategies, SEALS performed similarly to the baseline approaches that considered all of the unlabeled data in the candidate pool, as shown in Figures 1 and 2.

Original: ImageNet.
Revised: ImageNet.

Original: With a labeling budget of 2,000 examples per concept (˜0.31% of | U | ), all baseline and
Revised: With a labeling budget of 2,000 examples per concept (˜0.31% of | U | ), all baseline and

Original: SEALS approaches ( k = 100 ) were within 0.011 mAP of the 0.699 mAP achieved with full supervision.
Revised: SEALS approaches ( k = 100 ) were within 0.011 mAP of the 0.699 mAP achieved with full supervision.

Original: In contrast, random sampling (Random-All) only achieved 0.436 mAP.
Revised: In contrast, random sampling (Random-All) only achieved 0.436 mAP.

Original: MLP-All, MaxEnt-All, and ID-All achieved mAPs of 0.693, 0.695, and 0.688, respectively, while the SEALS equivalents were all within 0.001 mAP at 0.692, 0.695, and 0.688 respectively and considered less than 7% of the unlabeled data.
Revised: MLP-All, MaxEnt-All, and ID-All achieved mAPs of 0.693, 0.695, and 0.688, respectively, while the SEALS equivalents were all within 0.001 mAP at 0.692, 0.695, and 0.688 respectively and considered less than 7% of the unlabeled data.

Original: The resulting selection runtime for MLP-SEALS and MaxEnt-SEALS dropped by over 25 × , leading to a 3.6 × speed-up overall (Table 2).
Revised: The resulting selection runtime for MLP-SEALS and MaxEnt-SEALS dropped by over 25 × , leading to a 3.6 × speed-up overall (Table 2).

Original: The speed-up was even larger for IDSEALS, ranging from about 45 × at 2,00 labels to 3000 × at 200 labels.
Revised: The speed-up was even larger for IDSEALS, ranging from about 45 × at 2,00 labels to 3000 × at 200 labels.

Original: Even at a per-class level, the results were highly correlated with Pearson correlation coefﬁcients of 0.9998 or more (Figure 10a in the Appendix).
Revised: Even at a per-class level, the results were highly correlated with Pearson correlation coefﬁcients of 0.9998 or more (Figure 10a in the Appendix).

Original: The reduced skew from the nearest neighbor expansion of the initial seed set only accounted for a small part of the improvement, as Random-SEALS achieved an mAP of 0.498.
Revised: The reduced skew from the nearest neighbor expansion of the initial seed set only accounted for a small part of the improvement, as Random-SEALS achieved an mAP of 0.498.

Original: OpenImages.
Revised: OpenImages.

Original: The gap between the baseline approaches and SEALS widened slightly for OpenImages.
Revised: The gap between the baseline approaches and SEALS widened slightly for OpenImages.

Original: At 2,000 labels per concept (˜0.029% of | U | ), MaxEnt-All and MLP-All achieved 0.399 and 0.398 mAP, respectively, while MaxEnt-SEALS and MLP-SEALS both achieved 0.386 mAP and considered less than 1% of the data.
Revised: At 2,000 labels per concept (˜0.029% of | U | ), MaxEnt-All and MLP-All achieved 0.399 and 0.398 mAP, respectively, while MaxEnt-SEALS and MLP-SEALS both achieved 0.386 mAP and considered less than 1% of the data.

Original: This sped-up the selection time by over 180 × and the total time by over 3 × .
Revised: This sped-up the selection time by over 180 × and the total time by over 3 × .

Original: Increasing k to 1,000 signiﬁcantly narrowed this gap for MaxEnt-SEALS and MLPSEALS, improving mAP to 0.395, as shown in the Appendix (Figure 7).
Revised: Increasing k to 1,000 signiﬁcantly narrowed this gap for MaxEnt-SEALS and MLPSEALS, improving mAP to 0.395, as shown in the Appendix (Figure 7).

Original: Moreover, SEALS made
Revised: Moreover, SEALS made

Original: ID tractable on OpenImages by reducing the candidate pool to 1% of the unlabeled data, whereas
Revised: ID tractable on OpenImages by reducing the candidate pool to 1% of the unlabeled data, whereas

Original: ID-All ran for over 24 hours in wall-clock time without completing a single round (Table 2).
Revised: ID-All ran for over 24 hours in wall-clock time without completing a single round (Table 2).

Original: [Table]
Revised: [Table]

Original: 10B images.
Revised: 10B images.

Original: Despite the unprecedented scale and limited pool size, SEALS performed similarly to the baseline approaches that scanned all 10 billion images.
Revised: Despite the unprecedented scale and limited pool size, SEALS performed similarly to the baseline approaches that scanned all 10 billion images.

Original: At a budget of 1,500 labels, MaxEntSEALS ( k =10K) achieved a similar mAP to the baseline (0.504 vs. 0.508 mAP), while considering only about 0.1% of the data.
Revised: At a budget of 1,500 labels, MaxEntSEALS ( k =10K) achieved a similar mAP to the baseline (0.504 vs. 0.508 mAP), while considering only about 0.1% of the data.

Original: This reduction allowed MaxEnt-SEALS to ﬁnish selection rounds in seconds on a single 24-core machine, while MaxEnt-All took minutes on a cluster with tens of thousands of cores.
Revised: This reduction allowed MaxEnt-SEALS to ﬁnish selection rounds in seconds on a single 24-core machine, while MaxEnt-All took minutes on a cluster with tens of thousands of cores.

Original: MLP-SEALS performed poorly at this scale because, for any image, there are likely many redundant or near-duplicate examples that provide little additional value.
Revised: MLP-SEALS performed poorly at this scale because, for any image, there are likely many redundant or near-duplicate examples that provide little additional value.

Original: [Figure]
Revised: [Figure]

Original: 4. A CTIVE SEARCH
Revised: 4. A CTIVE SEARCH

Original: As shown in Figures 1 and 2, SEALS recalled nearly the same number of positive examples as the baseline approaches did for all of the considered concepts, datasets, and selection strategies.
Revised: As shown in Figures 1 and 2, SEALS recalled nearly the same number of positive examples as the baseline approaches did for all of the considered concepts, datasets, and selection strategies.

Original: ImageNet.
Revised: ImageNet.

Original: Unsurprisingly, MLP-All and MLP-SEALS signiﬁcantly outperformed all of the other selection strategies for active search.
Revised: Unsurprisingly, MLP-All and MLP-SEALS signiﬁcantly outperformed all of the other selection strategies for active search.

Original: At 2,000 labeled examples per concept, both approaches recalled over 74% of the positive examples for each concept at 74.5% and 74.2% recall, respectively.
Revised: At 2,000 labeled examples per concept, both approaches recalled over 74% of the positive examples for each concept at 74.5% and 74.2% recall, respectively.

Original: MaxEnt-All and MaxEnt-SEALS had a similar gap of 0.3%, labeling 57.2% and 56.9% of positive examples, while ID-All and ID-SEALS were even closer with a gap of only 0.1% (50.8% vs.
Revised: MaxEnt-All and MaxEnt-SEALS had a similar gap of 0.3%, labeling 57.2% and 56.9% of positive examples, while ID-All and ID-SEALS were even closer with a gap of only 0.1% (50.8% vs.

Original: Nearly all of the gains in recall are due to the selection strategies rather than the reduced skew in the initial seed, as Random-SEALS increased the recall by less than 1.0% over Random-All.
Revised: Nearly all of the gains in recall are due to the selection strategies rather than the reduced skew in the initial seed, as Random-SEALS increased the recall by less than 1.0% over Random-All.

Original: [Figure]
Revised: [Figure]

Original: OpenImages.
Revised: OpenImages.

Original: The gap between the baseline approaches and SEALS was even closer on OpenImages despite considering a much smaller fraction of the overall unlabeled pool.
Revised: The gap between the baseline approaches and SEALS was even closer on OpenImages despite considering a much smaller fraction of the overall unlabeled pool.

Original: MLP-All, MLPSEALS, MaxEnt-SEALS, and MaxEnt-All were all within 0.1% with ˜35% recall at 2,000 labels per concept.
Revised: MLP-All, MLPSEALS, MaxEnt-SEALS, and MaxEnt-All were all within 0.1% with ˜35% recall at 2,000 labels per concept.

Original: ID-SEALS had a recall of 29.3% but scaled nearly as well as the linear approaches.
Revised: ID-SEALS had a recall of 29.3% but scaled nearly as well as the linear approaches.

Original: 10B images.
Revised: 10B images.

Original: SEALS performed as well as the baseline approach despite considering less than 0.1% of the data and collected 2 orders of magnitude more positive examples than random sampling.
Revised: SEALS performed as well as the baseline approach despite considering less than 0.1% of the data and collected 2 orders of magnitude more positive examples than random sampling.

Original: 4. L ATENT STRUCTURE OF UNSEEN CONCEPTS
Revised: 4. L ATENT STRUCTURE OF UNSEEN CONCEPTS

Original: To better understand why and when SEALS works, we analyzed the nearest neighbor graph across concepts and values of k .
Revised: To better understand why and when SEALS works, we analyzed the nearest neighbor graph across concepts and values of k .

Original: Figure 3 shows the cumulative distribution functions (CDF) for the largest connected component within each concept and the average shortest paths between examples in that component.
Revised: Figure 3 shows the cumulative distribution functions (CDF) for the largest connected component within each concept and the average shortest paths between examples in that component.

Original: The 10B images dataset was excluded because only a few thousand examples were labeled.
Revised: The 10B images dataset was excluded because only a few thousand examples were labeled.

Original: The largest connected component gives a sense of how much of the concept SEALS can reach, while the average shortest path serves as a proxy for how long it will take to explore.
Revised: The largest connected component gives a sense of how much of the concept SEALS can reach, while the average shortest path serves as a proxy for how long it will take to explore.

Revised: In general, SEALS performed better for concepts that formed larger connected components and had shorter paths between examples (Figure 11 in the Appendix).

Original: For most concepts in ImageNet, the largest connected component contained the majority of examples, and the paths between examples were very short.
Revised: For most concepts in ImageNet, the largest connected component contained the majority of examples, and the paths between examples were very short.

Original: These tight clusters explain why so few examples were needed to learn accurate binary concept classiﬁers, as shown in Section 4.2, and why SEALS recovered ˜74% of positive examples on average while only labeling ˜0.31% of the data.
Revised: These tight clusters explain why so few examples were needed to learn accurate binary concept classiﬁers, as shown in Section 4.2, and why SEALS recovered ˜74% of positive examples on average while only labeling ˜0.31% of the data.

Original: If we constructed the candidate pool by randomly selecting examples, mAP and recall would drop for all strategies (Appendix A.5).
Revised: If we constructed the candidate pool by randomly selecting examples, mAP and recall would drop for all strategies (Appendix A.5).

Original: The concepts were so rare that the randomly chosen examples were not close to the decision boundary.
Revised: The concepts were so rare that the randomly chosen examples were not close to the decision boundary.

Original: For OpenImages, rare concepts were more fragmented, but each component was fairly tight, leading to short paths between examples.
Revised: For OpenImages, rare concepts were more fragmented, but each component was fairly tight, leading to short paths between examples.

Original: This fragmentation partly explains the gap between SEALS and the baselines in Section 4.2, and why increasing k closed it.
Revised: This fragmentation partly explains the gap between SEALS and the baselines in Section 4.2, and why increasing k closed it.

Original: As k increases, so does the candidate pool and the computational complexity, creating a trade-off between computational efﬁciency and the labeling efﬁciency of the underlying selection strategies.
Revised: As k increases, so does the candidate pool and the computational complexity, creating a trade-off between computational efﬁciency and the labeling efﬁciency of the underlying selection strategies.

Original: However, even for small values of k , there were signiﬁcant gains over random sampling, as shown in Figures 6 and 7 in the Appendix.
Revised: However, even for small values of k , there were signiﬁcant gains over random sampling, as shown in Figures 6 and 7 in the Appendix.

Original: This makes active learning and search tractable for almost any amount of unlabeled data and computational budget, as demonstrated with the 10B images dataset.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced Similarity search for Efﬁcient Active Learning and Search (SEALS) as a simple approach to accelerate active learning and search that can be applied to a wide range of existing algorithms.
Revised: In this work, we introduced Similarity search for Efﬁcient Active Learning and Search (SEALS) as a simple approach to accelerate active learning and search that can be applied to a wide range of existing algorithms.

Original: SEALS restricted the candidate pool for labeling to the nearest neighbors of the currently labeled set instead of scanning over all of the unlabeled data.
Revised: SEALS restricted the candidate pool for labeling to the nearest neighbors of the currently labeled set instead of scanning over all of the unlabeled data.

Original: Across three large datasets, three selection strategies, and 611 concepts, we found that SEALS achieved similar model quality and recall of positive examples while improving computational efﬁciency by orders of magnitude.
Revised: Across three large datasets, three selection strategies, and 611 concepts, we found that SEALS achieved similar model quality and recall of positive examples while improving computational efﬁciency by orders of magnitude.

Original: A A PPENDIX A. N UMBER OF INITIAL POSITIVES
Revised: A A PPENDIX A. N UMBER OF INITIAL POSITIVES

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. I MPACT OF k ON SEALS
Revised: A. I MPACT OF k ON SEALS

Original: [Figure]
Revised: [Figure]

Original: size (bottom), presenting a trade-off between labeling efﬁciency and computational efﬁciency
Revised: size (bottom), presenting a trade-off between labeling efﬁciency and computational efﬁciency

Original: [Figure]
Revised: [Figure]

Original: candidate pool size (bottom), presenting a trade-off between labeling efﬁciency and computational efﬁciency.
Revised: candidate pool size (bottom), presenting a trade-off between labeling efﬁciency and computational efﬁciency.

Original: A. S ELF - SUPERVISED EMBEDDINGS (S IM CLR) ON I MAGE N ET
Revised: A. S ELF - SUPERVISED EMBEDDINGS (S IM CLR) ON I MAGE N ET

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: In comparison to Figure 3a, the k -nearest neighbor graph for unseen concepts was still well connected, forming large connected components (left) for even moderate values of k , but the average shortest path between examples was slightly longer (right).
Revised: In comparison to Figure 3a, the k -nearest neighbor graph for unseen concepts was still well connected, forming large connected components (left) for even moderate values of k , but the average shortest path between examples was slightly longer (right).

Original: The increased path length is not too surprising considering the fully supervised model still outperformed the linear evaluation of the self-supervised embeddings in Chen et al. 
Revised: The increased path length is not too surprising considering the fully supervised model still outperformed the linear evaluation of the self-supervised embeddings in Chen et al. [Figure] [Figure] [Figure]

Revised: E A L

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: AP (*

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: A. C OMPARISON TO POOL OF RANDOMLY SELECTED EXAMPLES
Revised: A. C OMPARISON TO POOL OF RANDOMLY SELECTED EXAMPLES

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. A CTIVE LEARNING ON EACH SELECTED CLASS FROM O PEN I MAGES
Revised: A. A CTIVE LEARNING ON EACH SELECTED CLASS FROM O PEN I MAGES

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. A CTIVE SEARCH ON EACH SELECTED CLASS FROM O PEN I MAGES
Revised: A. A CTIVE SEARCH ON EACH SELECTED CLASS FROM O PEN I MAGES

Original: [Table]
Revised: [Table]

Original: Rye
Revised: Rye

Original: [Table]
Revised: [Table]

Original: Gymnast
Revised: Gymnast

Original: [Table]
Revised: [Table]

Original: Galleon
Revised: Galleon

Original: [Table]
Revised: [Table]

Original: Grapefruit
Revised: Grapefruit

Original: [Table]
Revised: [Table]

Original: /m/01ngsj
Revised: /m/01ngsj

Original: Daylily
Revised: Daylily

Original: [Table]
Revised: [Table]

Original: /m/01cz4w
Revised: /m/01cz4w

Original: Bilberry
Revised: Bilberry

Original: [Table]
Revised: [Table]

Original: /m/058hry
Revised: /m/058hry

Original: Smoothie
Revised: Smoothie

Original: [Table]
Revised: [Table]

Original: /m/02qfx
Revised: /m/02qfx

Original: Embroidery
Revised: Embroidery

Original: [Table]
Revised: [Table]

Original: /m/0fd2m
Revised: /m/0fd2m

Original: Deacon
Revised: Deacon

Original: [Table]
Revised: [Table]

Original: /m/05h5q
Revised: /m/05h5q

Original: s
Revised: s

Original: Glider
Revised: Glider

Original: [Table]
Revised: [Table]

Original: /m/04l3lm
Revised: /m/04l3lm

Original: Shooting range
Revised: Shooting range

Original: [Table]
Revised: [Table]

Original: /m/02r0zt
Revised: /m/02r0zt

Original: White-tailed deer
Revised: White-tailed deer

Original: [Table]
Revised: [Table]

Original: /m/076t48l
Revised: /m/076t48l

Original: Coral reef ﬁsh
Revised: Coral reef ﬁsh

Original: [Table]
Revised: [Table]

Original: /m/07x7yj
Revised: /m/07x7yj

Original: Pancit
Revised: Pancit

Original: [Table]
Revised: [Table]

Original: /m/

Revised: /m/

Original: lb
Revised: lb

Original: Chess
Revised: Chess

Original: [Table]
Revised: [Table]

Original: /m/03jqf
Revised: /m/03jqf

Original: Chevrolet silverado
Revised: Chevrolet silverado

Original: [Table]
Revised: [Table]

Original: /m/058qzx
Revised: /m/058qzx

Original: Kitchen knife
Revised: Kitchen knife

Original: [Table]
Revised: [Table]

Original: 53 . 9 ± 0 ./m/0cl6l
Revised: 53 . 9 ± 0 ./m/0cl6l

Original: Straw 1 . 0 ± 0 .50 .
Revised: Straw 1 . 0 ± 0 .50 .

Original: 3 ±
Revised: 3 ±

Original: [Table]
Revised: [Table]

Original: 51 . 0 ±
Revised: 51 . 0 ±

Original: [Table]
Revised: [Table]

Original: /m/0dlgzb
Revised: /m/0dlgzb

Original: Bakmi 2 . 6 ± 0 .51 .
Revised: Bakmi 2 . 6 ± 0 .51 .

Original: 8 ± 0 .51 .
Revised: 8 ± 0 .51 .

Original: 2 ± 0 ./m/0322v
Revised: 2 ± 0 ./m/0322v

Original: Lugger 4 . 9 ± 0 .53 .
Revised: Lugger 4 . 9 ± 0 .53 .

Original: 8 ± 1 .53 .
Revised: 8 ± 1 .53 .

Original: 8 ± 1 ./m/0dvg
Revised: 8 ± 1 ./m/0dvg

Original: Ancient roman architecture 0 .
Revised: Ancient roman architecture 0 .

Original: 8 ±
Revised: 8 ±

Original: [Table]
Revised: [Table]

Original: 5 ±
Revised: 5 ±

Original: [Table]
Revised: [Table]

Original: 1 ±
Revised: 1 ±

Original: [Table]
Revised: [Table]

Original: ./m/019mbd
Revised: ./m/019mbd

Original: Antenna 1 . 0 ±
Revised: Antenna 1 . 0 ±

Original: [Table]
Revised: [Table]

Original: 4 ±
Revised: 4 ±

Original: [Table]
Revised: [Table]

Original: 48 . 0 ± 0 ./m/0b97cn
Revised: 48 . 0 ± 0 ./m/0b97cn

Original: Lamian 1 . 9 ±
Revised: Lamian 1 . 9 ±

Original: [Table]
Revised: [Table]

Original: 8 ± 1 .48 .
Revised: 8 ± 1 .48 .

Original: 2 ± 0 ./m/03lnzv
Revised: 2 ± 0 ./m/03lnzv

Original: Calabaza 0 .
Revised: Calabaza 0 .

Original: 6 ±
Revised: 6 ±

Original: [Table]
Revised: [Table]

Original: 0 ± 0 .45 .
Revised: 0 ± 0 .45 .

Original: 8 ± 0 ./m/09l0j
Revised: 8 ± 0 ./m/09l0j

Original: Ring 0 .
Revised: Ring 0 .

Original: 7 ± 0 .45 .
Revised: 7 ± 0 .45 .

Original: 2 ± 0 .45 .
Revised: 2 ± 0 .45 .

Original: 4 ± 0 ./m/01y2w
Revised: 4 ± 0 ./m/01y2w

Original: Ciconiiformes 1 . 2 ± 0 .45 .
Revised: Ciconiiformes 1 . 2 ± 0 .45 .

Original: 2 ± 0 .45 .
Revised: 2 ± 0 .45 .

Original: 2 ± 0 ./m/02zkn
Revised: 2 ± 0 ./m/02zkn

Original: Log cabin 1 . 1 ± 0 .44 .
Revised: Log cabin 1 . 1 ± 0 .44 .

Original: 9 ± 1 .45 .
Revised: 9 ± 1 .45 .

Original: 7 ± 0 ./m/0l

Revised: 7 ± 0 ./m/0l

Original: Bowed string instrument 0 .
Revised: Bowed string instrument 0 .

Original: 7 ± 0 .44 .
Revised: 7 ± 0 .44 .

Original: 4 ± 0 .44 .
Revised: 4 ± 0 .44 .

Original: 7 ± 0 ./m/05z
Revised: 7 ± 0 ./m/05z

Original: Pasta 0 .
Revised: Pasta 0 .

Original: 5 ± 0 .43 .
Revised: 5 ± 0 .43 .

Original: 7 ± 0 .43 .
Revised: 7 ± 0 .43 .

Original: 8 ± 0 ./m/047fr
Revised: 8 ± 0 ./m/047fr

Original: Knitting 1 . 3 ± 0 .43 .
Revised: Knitting 1 . 3 ± 0 .43 .

Original: 5 ± 0 .42 .
Revised: 5 ± 0 .42 .

Original: 8 ±
Revised: 8 ±

Original: [Table]
Revised: [Table]

Original: ./m/01xc8d
Revised: ./m/01xc8d

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/01ll1r
Revised: /m/01ll1r

Original: Briefs
Revised: Briefs

Original: [Table]
Revised: [Table]

Original: /m/01 6hg
Revised: /m/01 6hg

Original: Delicatessen
Revised: Delicatessen

Original: [Table]
Revised: [Table]

Original: /m/0dwxr
Revised: /m/0dwxr

Original: Modern pentathlon
Revised: Modern pentathlon

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /0335ws
Revised: /0335ws

Original: [Table]
Revised: [Table]

Original: 0c0ygc
Revised: 0c0ygc

Original: Stallion
Revised: Stallion

Original: [Table]
Revised: [Table]

Original: 0176mf
Revised: 0176mf

Original: [Table]
Revised: [Table]

Original: 01czv
Revised: 01czv

Original: [Table]
Revised: [Table]

Original: /01295w
Revised: /01295w

Original: Galliformes
Revised: Galliformes

Original: [Table]
Revised: [Table]

Original: 03931h
Revised: 03931h

Original: Stele
Revised: Stele

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /03vd
Revised: /03vd

Original: Herd
Revised: Herd

Original: [Table]
Revised: [Table]

Original: /m/0hlw
Revised: /m/0hlw

Original: Algae
Revised: Algae

Original: [Table]
Revised: [Table]

Original: /m/0h8n
Revised: /m/0h8n

Original: Shelving
Revised: Shelving

Original: [Table]
Revised: [Table]

Original: /m/01d7ng
Revised: /m/01d7ng

Original: Pelecaniformes
Revised: Pelecaniformes

Original: [Table]
Revised: [Table]

Original: /m/02hnl
Revised: /m/02hnl

Original: Drums
Revised: Drums

Original: [Table]
Revised: [Table]

Original: /m/04fyb
Revised: /m/04fyb

Original: Sirloin steak
Revised: Sirloin steak

Original: [Table]
Revised: [Table]

Original: /m/025 v
Revised: /m/025 v

Original: Cactus
Revised: Cactus

Original: [Table]
Revised: [Table]

Original: /m/020dp
Revised: /m/020dp

Original: Cranberry
Revised: Cranberry

Original: [Table]
Revised: [Table]

Original: /m/

Revised: /m/

Original: g
Revised: g

Original: Factory
Revised: Factory

Original: [Table]
Revised: [Table]

Original: /m/04s5jq
Revised: /m/04s5jq

Original: [Table]
Revised: [Table]

Original: design
Revised: design

Original: [Table]
Revised: [Table]

Original: /m/05mfbz
Revised: /m/05mfbz

Original: Optical instrument
Revised: Optical instrument

Original: [Table]
Revised: [Table]

Original: /m/01jnzj
Revised: /m/01jnzj

Original: Construction
Revised: Construction

Original: [Table]
Revised: [Table]

Original: /m/01cv4r
Revised: /m/01cv4r

Original: Temperate coniferous forest
Revised: Temperate coniferous forest

Original: [Table]
Revised: [Table]

Original: /m/033cnk
Revised: /m/033cnk

Original: Egg (Food)
Revised: Egg (Food)

Original: [Table]
Revised: [Table]

Original: /m/0wgq
Revised: /m/0wgq

Original: Skating
Revised: Skating

Original: [Table]
Revised: [Table]

Original: /m/02pkr
Revised: /m/02pkr

Original: Plumbing ﬁxture
Revised: Plumbing ﬁxture

Original: [Table]
Revised: [Table]

Original: /m/0g9vs
Revised: /m/0g9vs

Original: Steamed rice
Revised: Steamed rice

Original: [Table]
Revised: [Table]

Original: /m/0cyz6w
Revised: /m/0cyz6w

Original: [Table]
Revised: [Table]

Original: 9 ± 1 .
Revised: 9 ± 1 .

Original: [Table]
Revised: [Table]

Original: 6 ± 0 .18 .
Revised: 6 ± 0 .18 .

Original: 1 ± 6 .19 .
Revised: 1 ± 6 .19 .

Original: 3 ± 4 ./m/0h8ls
Revised: 3 ± 4 ./m/0h8ls

Original: Automotive exterior 0 .
Revised: Automotive exterior 0 .

Original: 5 ± 0 .17 .
Revised: 5 ± 0 .17 .

Original: 7 ± 2 .11 .
Revised: 7 ± 2 .11 .

Original: 9 ± 10 ./m/03tqj
Revised: 9 ± 10 ./m/03tqj

Original: Icon 2 .
Revised: Icon 2 .

Original: 7 ± 0 .19 .
Revised: 7 ± 0 .19 .

Original: 9 ± 0 .17 .
Revised: 9 ± 0 .17 .

Original: 2 ± 4 ./m/0gd2v
Revised: 2 ± 4 ./m/0gd2v

Original: Marine mammal 0 .
Revised: Marine mammal 0 .

Original: 2 ± 0 .17 .
Revised: 2 ± 0 .17 .

Original: 3 ± 0 .17 .
Revised: 3 ± 0 .17 .

Original: 2 ± 0 ./m/03nxtz
Revised: 2 ± 0 ./m/03nxtz

Original: Cottage 0 .
Revised: Cottage 0 .

Original: 7 ± 0 .17 .
Revised: 7 ± 0 .17 .

Original: 6 ± 0 .17 .
Revised: 6 ± 0 .17 .

Original: 3 ± 0 ./m/099md
Revised: 3 ± 0 ./m/099md

Original: Soldier 0 .
Revised: Soldier 0 .

Original: 5 ± 0 .17 .
Revised: 5 ± 0 .17 .

Original: 3 ± 

Revised: 3 ±

Original: 8 ± 0 ./m/07k1x
Revised: 8 ± 0 ./m/07k1x

Original: Tool 0 .
Revised: Tool 0 .

Original: 3 ± 0 .17 . 0 ± 

Revised: 3 ± 0 .17 . 0 ±

Original: 9 ± 0 ./m/02zfvv
Revised: 9 ± 0 ./m/02zfvv

Original: American shorthair 0 .
Revised: American shorthair 0 .

Original: 3 ± 

Revised: 3 ±

Original: 5 ± 

Revised: 5 ±

Original: 7 ± 0 ./m/0541p
Revised: 7 ± 0 ./m/0541p

Original: Multimedia 0 .
Revised: Multimedia 0 .

Original: 7 ± 

Revised: 7 ±

Original: 8 ± 2 .17 .
Revised: 8 ± 2 .17 .

Original: 1 ± 0 ./m/0hr
Revised: 1 ± 0 ./m/0hr

Original: Asphalt 0 .
Revised: Asphalt 0 .

Original: 5 ± 0 .15 .
Revised: 5 ± 0 .15 .

Original: 1 ± 0 .11 .
Revised: 1 ± 0 .11 .

Original: 5 ± 5 ./m/09l
Revised: 5 ± 5 ./m/09l

Original: Singer 0 .
Revised: Singer 0 .

Original: 9 ± 0 .14 .
Revised: 9 ± 0 .14 .

Original: 6 ± 7 .13 .
Revised: 6 ± 7 .13 .

Original: 6 ± 7 ./m/0j6jq
Revised: 6 ± 7 ./m/0j6jq

Original: Floodplain 0 .
Revised: Floodplain 0 .

Original: 9 ± 0 .14 .
Revised: 9 ± 0 .14 .

Original: 6 ± 1 .14 . 0 ± 1 ./m/01f4td
Revised: 6 ± 1 .14 . 0 ± 1 ./m/01f4td

Original: Rural area 0 .
Revised: Rural area 0 .

Original: 6 ± 0 .14 .
Revised: 6 ± 0 .14 .

Original: 2 ± 0 .13 .
Revised: 2 ± 0 .13 .

Original: 2 ± 0 ./m/03hcy1d
Revised: 2 ± 0 ./m/03hcy1d

Original: Mitsubishi 1 . 0 ± 0 .12 .
Revised: Mitsubishi 1 . 0 ± 0 .12 .

Original: 6 ± 0 .11 .
Revised: 6 ± 0 .11 .

Original: 8 ± 1 ./m/013y0j
Revised: 8 ± 1 ./m/013y0j

Original: Organ (Biology)
Revised: Organ (Biology)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: /m/0641k
Revised: /m/0641k

Original: Paper 0 .
Revised: Paper 0 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/0jqb
Revised: /m/0jqb

Original: Annual plant
Revised: Annual plant

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/02vwbzz
Revised: /m/02vwbzz

Original: Electric blue
Revised: Electric blue

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/019cfy
Revised: /m/019cfy

Original: Stadium
Revised: Stadium

Original: [Table]
Revised: [Table]

Original: /m/04rd
Revised: /m/04rd

Original: Mural
Revised: Mural

Original: [Table]
Revised: [Table]

Original: /m/03jvv
Revised: /m/03jvv

Original: Teal
Revised: Teal

Original: [Table]
Revised: [Table]

Original: /m/09qqq
Revised: /m/09qqq

Original: Wall
Revised: Wall

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/01bmhj
Revised: /m/01bmhj

Original: Thumb
Revised: Thumb

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/01jfm
Revised: /m/01jfm

Original: Vehicle registration plate
Revised: Vehicle registration plate

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3 ±
Revised: 3 ±

Original: [Table]
Revised: [Table]

Original: /m/025sv9y
Revised: /m/025sv9y

Original: Landscaping
Revised: Landscaping

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: /m/03s7w
Revised: /m/03s7w

Original: Cirque
Revised: Cirque

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 8 ±
Revised: 8 ±

Original: [Table]
Revised: [Table]

Original: ./m/0krfg
Revised: ./m/0krfg

Original: Meal
Revised: Meal

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /m/023bbt
Revised: /m/023bbt

Original: Wilderness
Revised: Wilderness

Original: [Table]
Revised: [Table]

Original: 4 ± 0 .8 .
Revised: 4 ± 0 .8 .

Original: 5 ± 4 .9 .
Revised: 5 ± 4 .9 .

Original: 8 ± 2 ./m/06wqb
Revised: 8 ± 2 ./m/06wqb

Original: Space 0 .
Revised: Space 0 .

Original: 5 ± 0 .7 .
Revised: 5 ± 0 .7 .

Original: 8 ± 5 .6 .
Revised: 8 ± 5 .6 .

Original: 3 ± 5 ./m/04qgp
Revised: 3 ± 5 ./m/04qgp

Original: Liqueur 1 . 0 ± 0 .8 .
Revised: Liqueur 1 . 0 ± 0 .8 .

Original: 0 ± 6 .12 .
Revised: 0 ± 6 .12 .

Original: 8 ± 4 ./m/09q2t
Revised: 8 ± 4 ./m/09q2t

Original: Brown 0 .
Revised: Brown 0 .

Original: 4 ± 0 .7 .
Revised: 4 ± 0 .7 .

Original: 2 ± 5 .2 .
Revised: 2 ± 5 .2 .

Original: 6 ± 4 ./m/01sgl
Revised: 6 ± 4 ./m/01sgl

Original: Cycling 0 .
Revised: Cycling 0 .

Original: 6 ± 0 .7 .
Revised: 6 ± 0 .7 .

Original: 3 ± 2 .7 .
Revised: 3 ± 2 .7 .

Original: 8 ± 2 ./m/05nnm
Revised: 8 ± 2 ./m/05nnm

Original: Organism 0 .
Revised: Organism 0 .

Original: 4 ± 0 .6 .
Revised: 4 ± 0 .6 .

Original: 8 ± 3 .2 .
Revised: 8 ± 3 .2 .

Original: 0 ± 2 ./m/01j3sz
Revised: 0 ± 2 ./m/01j3sz

Original: Laugh 0 .
Revised: Laugh 0 .

Original: 7 ± 0 .6 .
Revised: 7 ± 0 .6 .

Original: 6 ± 2 .8 .
Revised: 6 ± 2 .8 .

Original: 6 ± 1 ./m/01dv4h
Revised: 6 ± 1 ./m/01dv4h

Original: Portrait 0 .
Revised: Portrait 0 .

Original: 2 ± 0 .5 .
Revised: 2 ± 0 .5 .

Original: 8 ± 0 .5 .
Revised: 8 ± 0 .5 .

Original: 3 ± 0 ./m/02mnkq
Revised: 3 ± 0 ./m/02mnkq

Original: Bumper 0 .
Revised: Bumper 0 .

Original: 5 ± 0 .5 .
Revised: 5 ± 0 .5 .

Original: 9 ± 4 .8 .
Revised: 9 ± 4 .8 .

Original: 3 ± 3 ./m/079bkr
Revised: 3 ± 3 ./m/079bkr

Original: Mode of transport 0 .
Revised: Mode of transport 0 .

Original: 4 ± 0 .5 .
Revised: 4 ± 0 .5 .

Original: 1 ± 2 .3 .
Revised: 1 ± 2 .3 .

Original: 6 ± 2 ./m/01ckgp
Revised: 6 ± 2 ./m/01ckgp

Original: Interaction 0 .
Revised: Interaction 0 .

Original: 6 ± 0 .4 .
Revised: 6 ± 0 .4 .

Original: 5 ± 3 .4 .
Revised: 5 ± 3 .4 .

Original: 6 ± 2 ./m/012sbd
Revised: 6 ± 2 ./m/012sbd

Original: Tournament 0 .
Revised: Tournament 0 .

Original: 6 ± 0 .4 .
Revised: 6 ± 0 .4 .

Original: 3 ± 2 .5 .
Revised: 3 ± 2 .5 .

Original: 1 ± 1 ./m/05qjc
Revised: 1 ± 1 ./m/05qjc

Original: Performing arts 0 .
Revised: Performing arts 0 .

Original: 5 ± 0 .2 .
Revised: 5 ± 0 .2 .

Original: 3 ± 0 .2 .
Revised: 3 ± 0 .2 .

Original: 5 ± 0 ./m/083jv
Revised: 5 ± 0 ./m/083jv

Original: White 0 .
Revised: White 0 .

Original: 3 ± 0 .2 .
Revised: 3 ± 0 .2 .

Original: 0 ± 1 .0 .
Revised: 0 ± 1 .0 .

Original: 5 ± 0 .
Revised: 5 ± 0 .

Original: A. S ELF - SUPERVISED EMBEDDING (S ENTENCE -BERT) ON G OODREADS
Revised: A. S ELF - SUPERVISED EMBEDDING (S ENTENCE -BERT) ON G OODREADS

Original: We followed the same general procedure described in Section 4.1, aside from the dataset speciﬁc details below.
Revised: We followed the same general procedure described in Section 4.1, aside from the dataset speciﬁc details below.

Original: Goodreads spoiler detection (Wan et al., 2019) had 17.67 million sentences with binary spoiler annotations.
Revised: Goodreads spoiler detection (Wan et al., 2019) had 17.67 million sentences with binary spoiler annotations.

Original: Spoilers made up 3.224% of the data, making them much more common than the rare concepts we evaluated in the other datasets.
Revised: Spoilers made up 3.224% of the data, making them much more common than the rare concepts we evaluated in the other datasets.

Original: Following Wan et al.
Revised: Following Wan et al.

Original: (2019), we used 3.
Revised: (2019), we used 3.

Original: million sentences for testing (20%), 10,000 sentences as the validation set, and the remaining 14. million sentences as the unlabeled pool.
Revised: million sentences for testing (20%), 10,000 sentences as the validation set, and the remaining 14. million sentences as the unlabeled pool.

Original: We also switched to the area under the ROC curve (AUC) as our primary evaluation metric for active learning to be consistent with Wan et al.
Revised: We also switched to the area under the ROC curve (AUC) as our primary evaluation metric for active learning to be consistent with Wan et al.

Original: For
Revised: For

Original: G z , we used a pre-trained Sentence-BERT model (SBERT-NLI-base)
Revised: G z , we used a pre-trained Sentence-BERT model (SBERT-NLI-base)

Original: (Reimers & Gurevych, 2019), applied PCA whitening to reduce the dimension to 256, and performed l 2 normalization.
Revised: (Reimers & Gurevych, 2019), applied PCA whitening to reduce the dimension to 256, and performed l 2 normalization.

Original: A.8. A CTIVE SEARCH
Revised: A.8. A CTIVE SEARCH

Original: SEALS achieved the same recall as the baseline approaches, but only considered less than 1% of the unlabeled data in the candidate pool, as shown in Figure 14.
Revised: SEALS achieved the same recall as the baseline approaches, but only considered less than 1% of the unlabeled data in the candidate pool, as shown in Figure 15.

Original: At a labeling budget of 2,000, MLPALL and MLP-SEALS recalled 0 .
Revised: At a labeling budget of 2,000, MLPALL and MLP-SEALS recalled 0 .

Original: 15 ± 0 .
Revised: 15 ± 0 .

Original: 02 % and 0 .
Revised: 02 % and 0 .

Original: 17 ± 0 .
Revised: 17 ± 0 .

Original: 05 %, respectively, while MaxEnt-All and MaxEnt-SEALS achieved 0 .
Revised: 05 %, respectively, while MaxEnt-All and MaxEnt-SEALS achieved 0 .

Original: 14 ± 0 .
Revised: 14 ± 0 .

Original: 04 % and 0 .
Revised: 04 % and 0 .

Original: 11 ± 0 .
Revised: 11 ± 0 .

Original: 06 % recall respectively.
Revised: 06 % recall respectively.

Original: Increasing the labeling budget to 50,000 examples, increased recall to ˜3.7% for MaxEnt and MLP but maintained a similar relative improvement over random sampling, as shown in Figure 15.
Revised: Increasing the labeling budget to 50,000 examples, increased recall to ˜3.7% for MaxEnt and MLP but maintained a similar relative improvement over random sampling, as shown in Figure 16.

Original: ID-SEALS performed worse than the other strategies.
Revised: ID-SEALS performed worse than the other strategies.

Original: However, all of the active selection strategies outperformed random sampling by up to an order of magnitude.
Revised: However, all of the active selection strategies outperformed random sampling by up to an order of magnitude.

Original: [Figure]
Revised: [Figure]

Original: A.8. A CTIVE LEARNING
Revised: A.8. A CTIVE LEARNING

Original: At a labeling budget of 2,000 examples, all the selection strategies were indistinguishable from random sampling.
Revised: At a labeling budget of 2,000 examples, all the selection strategies were indistinguishable from random sampling.

Original: Increasing the labeling budget did not help, as shown in Figure 15.
Revised: Increasing the labeling budget did not help, as shown in Figure 16.

Original: Unlike
Revised: Unlike

Original: ImageNet and OpenImages, Goodreads had a much higher fraction of positive examples (3.224%), and the examples were not tightly clustered as described in Section A.8.3.
Revised: ImageNet and OpenImages, Goodreads had a much higher fraction of positive examples (3.224%), and the examples were not tightly clustered as described in Section A.8.3.

Original: [Figure]
Revised: [Figure]

Original: Across strategies, SEALS with k = 100 performed similarly to the baseline approach in terms of both the error the model achieved for active learning (left) and the recall of positive examples for active search (right), while only considering a fraction of the data U (middle).
Revised: Across strategies, SEALS with k = 100 performed similarly to the baseline approach in terms of both the error the model achieved for active learning (left) and the recall of positive examples for active search (right), while only considering a fraction of the data U (middle).

Original: ID was excluded because of the growing pool size and computation.
Revised: ID was excluded because of the growing pool size and computation.

Original: For active search, MaxEnt and MLP continued to improve recall.
Revised: For active search, MaxEnt and MLP continued to improve recall.

Original: For active learning, all the selection strategies (both with and without SEALS) performed worse than random sampling despite the larger labeling budget.
Revised: For active learning, all the selection strategies (both with and without SEALS) performed worse than random sampling despite the larger labeling budget.

Original: This gap was likely due to spoilers being book speciﬁc and the higher fraction of positive examples in the unlabeled pool, causing relevant examples to be spread almost uniformly across the space (see Section A.8.3).
Revised: This gap was likely due to spoilers being book speciﬁc and the higher fraction of positive examples in the unlabeled pool, causing relevant examples to be spread almost uniformly across the space (see Section A.8.3).

Original: A.8. L ATENT STRUCTURE
Revised: A.8. L ATENT STRUCTURE

Original: The large number of positive examples in the Goodreads dataset limited the analysis we could perform.
Revised: The large number of positive examples in the Goodreads dataset limited the analysis we could perform.

Original: We could only calculate the size of the largest connected component in the nearest neighbor graph (Figure 16).
Revised: We could only calculate the size of the largest connected component in the nearest neighbor graph (Figure 17).

Original: For k = 10 , only 28.4% of the positive examples could be reached directly, but increasing k to 100 improved that dramatically to 96.7%.
Revised: For k = 10 , only 28.4% of the positive examples could be reached directly, but increasing k to 100 improved that dramatically to 96.7%.

Original: For such a large connected component, one might have expected active learning to perform better in Section A.8.2.
Revised: For such a large connected component, one might have expected active learning to perform better in Section A.8.2.

Original: By analyzing the embeddings, however, we found that examples are spread almost uniformly across the space with an average cosine similarity of 0.004.
Revised: By analyzing the embeddings, however, we found that examples are spread almost uniformly across the space with an average cosine similarity of 0.004.

Original: For comparison, the average cosine similarity for concepts in ImageNet and OpenImages was 0 .
Revised: For comparison, the average cosine similarity for concepts in ImageNet and OpenImages was 0 .

Original: 453 ± 0 . 077 and 0 .
Revised: 453 ± 0 . 077 and 0 .

Original: 361 ± 0 .
Revised: 361 ± 0 .

Original: respectively.
Revised: respectively.

Original: This uniformity was likely due to the higher fraction of positive examples and spoilers being book speciﬁc while
Revised: This uniformity was likely due to the higher fraction of positive examples and spoilers being book speciﬁc while

Original: Sentence-BERT is trained on generic data.
Revised: Sentence-BERT is trained on generic data.

Original: As a result, even if spoilers were tightly clustered within each book, the books were spread across a range of topics and consequently across the embedding space, illustrating a limitation and opportunity for future work.
Revised: As a result, even if spoilers were tightly clustered within each book, the books were spread across a range of topics and consequently across the embedding space, illustrating a limitation and opportunity for future work.

Original: [Figure]
Revised: [Figure]