Original: E XPLANATION -B ASED A TTENTION FOR S EMIS UPERVISED D EEP A CTIVE L EARNING
Revised: E XPLANATION -B ASED A TTENTION FOR S EMIS UPERVISED D EEP A CTIVE L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting.
Revised: We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting.

Original: The proposed attention mechanism is based on recent methods to visually explain predictions made by DNNs.
Revised: The proposed attention mechanism is based on recent methods to visually explain predictions made by DNNs.

Original: We apply the proposed explanation-based attention to MNIST and SVHN classiﬁcation.
Revised: We apply the proposed explanation-based attention to MNIST and SVHN classiﬁcation.

Original: The conducted experiments show accuracy improvements for the original and class-imbalanced datasets with the same number of training examples and faster long-tail convergence compared to uncertainty-based methods
Revised: The conducted experiments show accuracy improvements for the original and class-imbalanced datasets with the same number of training examples and faster long-tail convergence compared to uncertainty-based methods

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep active learning (AL) minimizes the number of expensive annotations needed to train DNNs by selecting a subset of relevant data points from a large unlabeled dataset (Lewis & Gale, 1994).
Revised: Deep active learning (AL) minimizes the number of expensive annotations needed to train DNNs by selecting a subset of relevant data points from a large unlabeled dataset (Lewis & Gale, 1994).

Original: This subset is annotated and added to the training dataset in a single pool of data points or, more often, in an iterative fashion.
Revised: This subset is annotated and added to the training dataset in a single pool of data points or, more often, in an iterative fashion.

Original: The goal is to maximize prediction accuracy while minimizing the product of pool size × number of iterations.
Revised: The goal is to maximize prediction accuracy while minimizing the product of pool size × number of iterations.

Original: A proxy for this goal could be the task of matching feature distributions between the validation and the AL-selected training datasets.
Revised: A proxy for this goal could be the task of matching feature distributions between the validation and the AL-selected training datasets.

Original: In density-based AL approaches, data selection is typically performed using a simple L 2 -distance metric (Sener & Savarese, 2018).
Revised: In density-based AL approaches, data selection is typically performed using a simple L 2 -distance metric (Sener & Savarese, 2018).

Original: The image retrieval ﬁeld (Zhou et al., 2017) has advanced much further in this area.
Revised: The image retrieval ﬁeld (Zhou et al., 2017) has advanced much further in this area.

Original: For example, recent state-of-the-art image retrieval systems are based on DNNbased feature extraction (Babenko & Lempitsky, 2015) with attention mechanisms (Noh et al., 2017).
Revised: For example, recent state-of-the-art image retrieval systems are based on DNNbased feature extraction (Babenko & Lempitsky, 2015) with attention mechanisms (Noh et al., 2017).

Original: The latter estimates an attention mask to weight importance of the extracted features and it is trained along with the feature extraction.
Revised: The latter estimates an attention mask to weight importance of the extracted features and it is trained along with the feature extraction.

Original: Inspired by this, we employ image retrieval techniques and propose a novel attention mechanism for deep AL.
Revised: Inspired by this, we employ image retrieval techniques and propose a novel attention mechanism for deep AL.

Original: Unlike supervised self-attention in (Noh et al., 2017; Vaswani et al., 2017), our attention mechanism is not trained with the model.
Revised: Unlike supervised self-attention in (Noh et al., 2017; Vaswani et al., 2017), our attention mechanism is not trained with the model.

Original: It relies on recent methods to generate visual explanations and to attribute feature importance values (Sundararajan et al., 2017).
Revised: It relies on recent methods to generate visual explanations and to attribute feature importance values (Sundararajan et al., 2017).

Original: We show the effectiveness of such explanation-based attention (EBA) mechanism for AL when combined with multi-scale feature extraction on a number of image classiﬁcation datasets.
Revised: We show the effectiveness of such explanation-based attention (EBA) mechanism for AL when combined with multi-scale feature extraction on a number of image classiﬁcation datasets.

Original: We also conduct experiments for distorted class-imbalanced training data which is a more realistic assumption for unlabeled data.
Revised: We also conduct experiments for distorted class-imbalanced training data which is a more realistic assumption for unlabeled data.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: AL is a well-studied approach to decrease annotation costs in a traditional machine learning pipelines (Settles, 2010).
Revised: AL is a well-studied approach to decrease annotation costs in a traditional machine learning pipelines (Settles, 2010).

Original: Recently, AL has been applied to data-demanding DNN-based systems in semi-supervised or weakly-supervised settings.
Revised: Recently, AL has been applied to data-demanding DNN-based systems in semi-supervised or weakly-supervised settings.

Original: Though AL is an attractive direction, existing methods struggle to deal with high-dimensional data e.g. images.
Revised: Though AL is an attractive direction, existing methods struggle to deal with high-dimensional data e.g. images.

Original: We believe this is related to the lack of class and instance-level feature importance information as well as the inability to capture spatially-localized features.
Revised: We believe this is related to the lack of class and instance-level feature importance information as well as the inability to capture spatially-localized features.

Original: To overcome these limitations, we are interested in estimating spatiallymultiscale features and using our EBA mechanism to select only the most discriminative features.
Revised: To overcome these limitations, we are interested in estimating spatiallymultiscale features and using our EBA mechanism to select only the most discriminative features.

Original: [Figure]
Revised: [Figure]

Original: Wang et al.
Revised: Wang et al.

Original: (2017) proposed to augment the training dataset by labeling the least conﬁdent data points and heuristically pseudo-labeling high conﬁdence predictions.
Revised: (2017) proposed to augment the training dataset by labeling the least conﬁdent data points and heuristically pseudo-labeling high conﬁdence predictions.

Original: We believe the softmax output is not a reliable proxy for the goals of AL i.e. for selecting images using feature distribution matching between validation and train data.
Revised: We believe the softmax output is not a reliable proxy for the goals of AL i.e. for selecting images using feature distribution matching between validation and train data.

Original: Unlike (Wang et al., 2017), we use pseudo labels only to estimate EBA vectors and ﬁnd similarities between discriminative features.
Revised: Unlike (Wang et al., 2017), we use pseudo labels only to estimate EBA vectors and ﬁnd similarities between discriminative features.

Original: Gal et al.
Revised: Gal et al.

Original: (2017) introduced a measure of uncertainty for approximate Bayesian inference that can be estimated using stochastic forward passes through a DNN with dropout layers.
Revised: (2017) introduced a measure of uncertainty for approximate Bayesian inference that can be estimated using stochastic forward passes through a DNN with dropout layers.

Original: An acquisition function then selects data points with the highest uncertainty which is measured at the output of softmax using several metrics.
Revised: An acquisition function then selects data points with the highest uncertainty which is measured at the output of softmax using several metrics.

Original: Recent work (Beluch et al., 2018) extended this method by using an ensemble of networks for uncertainty estimation and achieved superior accuracy.
Revised: Recent work (Beluch et al., 2018) extended this method by using an ensemble of networks for uncertainty estimation and achieved superior accuracy.

Original: Sener & Savarese (2018) formulated feature similarity-based selection as a geometric core-set approach which outperforms greedy k -center clustering.
Revised: Sener & Savarese (2018) formulated feature similarity-based selection as a geometric core-set approach which outperforms greedy k -center clustering.

Original: Though their method can complement our approach, we are focusing on the novel feature extraction.
Revised: Though their method can complement our approach, we are focusing on the novel feature extraction.

Original: For instance, they employed a simple Ldistance similarity measure for the activations of the last fully-connected layer.
Revised: For instance, they employed a simple Ldistance similarity measure for the activations of the last fully-connected layer.

Original: The most similar work to ours, by Vodrahalli et al.
Revised: The most similar work to ours, by Vodrahalli et al.

Original: (2018), uses the gradients as a measure of importance for dataset subsampling and analysis.
Revised: (2018), uses the gradients as a measure of importance for dataset subsampling and analysis.

Original: However, our approach formulates the problem as a multi-scale EBA for AL application and goes beyond a less robust single-step gradient attention.
Revised: However, our approach formulates the problem as a multi-scale EBA for AL application and goes beyond a less robust single-step gradient attention.

Original: Other related works are online importance sampling methods (Ren et al., 2018) and the inﬂuence functions approach in (Koh & Liang, 2017).
Revised: Other related works are online importance sampling methods (Ren et al., 2018) and the inﬂuence functions approach in (Koh & Liang, 2017).

Original: Online importance sampling upweights samples within the mini-batch during supervised training using gradient similarity while inﬂuence functions analyze data point importance using computationally challenging second-order gradient information.
Revised: Online importance sampling upweights samples within the mini-batch during supervised training using gradient similarity while inﬂuence functions analyze data point importance using computationally challenging second-order gradient information.

Original:  M ETHOD
Revised:  M ETHOD

Original: Pool-based AL.
Revised: Pool-based AL.

Original: Let ( X , y ) be an input-label pair.
Revised: Let ( X , y ) be an input-label pair.

Original: There is a validation dataset { ( X vi , y vi ) } i ∈ M of size M and a collection of training pairs { ( X i , y i ) } i ∈ N of size N for which, initially, only a small random subset or pool of labels indexed by N 1 is known.
Revised: There is a validation dataset { ( X vi , y vi ) } i ∈ M of size M and a collection of training pairs { ( X i , y i ) } i ∈ N of size N for which, initially, only a small random subset or pool of labels indexed by N 1 is known.

Original: The validation dataset approximates the distribution of test data.
Revised: The validation dataset approximates the distribution of test data.

Original: At every b th iteration the AL algorithm selects a pool of P new labels to be annotated and added to existing training pairs which creates a training dataset indexed by N b .
Revised: At every b th iteration the AL algorithm selects a pool of P new labels to be annotated and added to existing training pairs which creates a training dataset indexed by N b .

Original: A DNN Φ( X , θ ) is optimized by minimizing a loss function ( N b ) − 1 (cid:80) i ∈ N b L (ˆ y i , y i ) w.r.t.
Revised: A DNN Φ( X , θ ) is optimized by minimizing a loss function ( N b ) − 1 (cid:80) i ∈ N b L (ˆ y i , y i ) w.r.t.

Original: to model parameters θ .
Revised: to model parameters θ .

Original: However, the actual task is to minimize validation loss expressed by
Revised: However, the actual task is to minimize validation loss expressed by

Original: M − 1 (cid:80) i ∈
Revised: M − 1 (cid:80) i ∈

Original: M L (ˆ y vi , y vi ) .
Revised: M L (ˆ y vi , y vi ) .

Original: Therefore, an oracle AL algorithm achieves minimum of validation loss using the smallest b × P product.
Revised: Therefore, an oracle AL algorithm achieves minimum of validation loss using the smallest b × P product.

Original: In this work, we are interested not in ﬁnding an oracle acquisition function, but in a method to extract relevant features for such function.
Revised: In this work, we are interested not in ﬁnding an oracle acquisition function, but in a method to extract relevant features for such function.

Original: We use a low-complexity greedy k -center algorithm to select the data points in the unlabeled training collection which are most similar to the misclassiﬁed entries in the validation dataset.
Revised: We use a low-complexity greedy k -center algorithm to select the data points in the unlabeled training collection which are most similar to the misclassiﬁed entries in the validation dataset.

Original: Feature descriptors.
Revised: Feature descriptors.

Original: Let F ji ∈ R C × H × W , where C , H , and W are the number of channels, the height, and the width, respectively be the output of the j th layer of DNN for input image
Revised: Let F ji ∈ R C × H × W , where C , H , and W are the number of channels, the height, and the width, respectively be the output of the j th layer of DNN for input image

Original: X i .
Revised: X i .

Original: Then, a feature vector or descriptor of length L can be deﬁned as d i =
Revised: Then, a feature vector or descriptor of length L can be deﬁned as d i =

Original: φ ( F i )
Revised: φ ( F i )

Original: ∈
Revised: ∈

Original: R L × 1 , where function φ ( · ) is a conventional average pooling operation from (Babenko & Lempitsky, 2015).
Revised: R L × 1 , where function φ ( · ) is a conventional average pooling operation from (Babenko & Lempitsky, 2015).

Original: In a multi-scale case, descriptor is a concatenation of multiple feature vectors d i =
Revised: In a multi-scale case, descriptor is a concatenation of multiple feature vectors d i =

Original: [ φ j ( F j i ) , · · · , φ l ( F li )] .
Revised: [ φ j ( F j i ) , · · · , φ l ( F li )] .

Original: A descriptor matrix for the validation dataset V d ∈ R L × M andcurrent training dataset S d ∈ R L × Nb can be calculated using forward passes.
Revised: A descriptor matrix for the validation dataset V d ∈ R L × M and training dataset S d ∈ R L × N can be calculated using forward passes.

Original: Practically, descriptors can be compressed for storage efﬁciency reasons using PCA, quantization, etc.
Revised: Practically, descriptors can be compressed for storage efﬁciency reasons using PCA, quantization, etc.

Original: Then, a match kernel (Lee, 1999), e.g. cosine similarity, can be used to match features in both datasets.
Revised: Then, a match kernel (Lee, 1999), e.g. cosine similarity, can be used to match features in both datasets.

Original: Assuming that vectors d i are L 2 -normalized, the cosine-similarity matrix is simply R d = V Td S d .
Revised: Assuming that vectors d i are L 2 -normalized, the cosine-similarity matrix is simply R d = V Td S d .

Original: Explanation-based attention.
Revised: Explanation-based attention.

Original: Feature maps F i extracted by Φ( X , θ ) and pooled by φ ( · ) contain features that: a) are not class and instance-level discriminative (in other words, not disentangled), b) spatially represent features for a plurality of objects in the input.
Revised: Feature maps F i extracted by Φ( X , θ ) and pooled by φ ( · ) contain features that: a) are not class and instance-level discriminative (in other words, not disentangled), b) spatially represent features for a plurality of objects in the input.

Original: We would like to upweight discriminative features that satisfy a) and b) using an attention mechanism.
Revised: We would like to upweight discriminative features that satisfy a) and b) using an attention mechanism.

Original: One approach would be to use self-attention (Vaswani et al., 2017) at the cost of modifying network architecture and intervening into the training process.
Revised: One approach would be to use self-attention (Vaswani et al., 2017) at the cost of modifying network architecture and intervening into the training process.

Original: Instead, we propose to use EBA that is generated only for feature selection.
Revised: Instead, we propose to use EBA that is generated only for feature selection.

Original: The EBA mechanism attributes feature importance values w.r.t.
Revised: The EBA mechanism attributes feature importance values w.r.t.

Original: to the output predictions.
Revised: to the output predictions.

Original: Unlike a visual explanation task, which estimates importance heatmaps in the input (image) space, we propose to estimate feature importance tensors
Revised: Unlike a visual explanation task, which estimates importance heatmaps in the input (image) space, we propose to estimate feature importance tensors

Original: A i of the internal DNN representations F i .
Revised: A i of the internal DNN representations F i .

Original: Attention tensors
Revised: Attention tensors

Original: A i can be efﬁciently calculated using a series of backpropagation passes.
Revised: A i can be efﬁciently calculated using a series of backpropagation passes.

Original: Using one of backpropagation-based methods called integrated gradients (IG) from (Sundararajan et al., 2017),
Revised: Using one of backpropagation-based methods called integrated gradients (IG) from (Sundararajan et al., 2017),

Original: A ji can be estimated as
Revised: A ji can be estimated as

Original: [Equation]
Revised: [Equation]

Original: where K is the number of steps to approximate the continuous integral by a linear path.
Revised: where K is the number of steps to approximate the continuous integral by a linear path.

Original: Other forms of (1) are possible: from the simplest saliency method for which K = 1 (Simonyan et al., 2014) to more advanced methods with randomly sampled input features (Gudovskiy et al., 2018).
Revised: Other forms of (1) are possible: from the simplest saliency method for which K = 1 (Simonyan et al., 2014) to more advanced methods with randomly sampled input features (Gudovskiy et al., 2018).

Original: Due to lack of labels y i in (1), we use common pseudo-labeling strategy:
Revised: Due to lack of labels y i in (1), we use common pseudo-labeling strategy:

Original: y i = 1 argmax ˆy i .
Revised: y i = 1 argmax ˆy i .

Original: It is schematically shown in Figure 1.
Revised: It is schematically shown in Figure 1.

Original: Unlike (Wang et al., 2017), pseudo-labels are used only to calculate similarity without additional hyperparameters rather than to perform a threshold-selected greedy augmentation.
Revised: Unlike (Wang et al., 2017), pseudo-labels are used only to calculate similarity without additional hyperparameters rather than to perform a threshold-selected greedy augmentation.

Original: The EBA A i can be converted to multi-scale attention vector using the same processing a i =
Revised: The EBA A i can be converted to multi-scale attention vector using the same processing a i =

Original: φ ( A i ) ∈
Revised: φ ( A i ) ∈

Original: R L × 1 , which, by analogy, forms validation V a ∈ R L × M and train attention matrices S a ∈ R L × Nb .
Revised: R L × 1 , which, by analogy, forms validation V a ∈ R L × M and train attention matrices S a ∈ R L × N .

Original: The latter processing is implemented in most modern frameworks and, therefore, the complexity to generate A
Revised: The latter processing is implemented in most modern frameworks and, therefore, the complexity to generate A

Original: i is only K forward-backward passes.
Revised: i is only K forward-backward passes.

Original: Summary for the proposed method.
Revised: Summary for the proposed method.

Original: A random subset of N 1 training data points is annotated and a DNN Φ( X , θ ) optimized for this subset.
Revised: A random subset of N 1 training data points is annotated and a DNN Φ( X , θ ) optimized for this subset.

Original: Then, the AL algorithm iteratively ( b = 2 , 3 . . . )
Revised: Then, the AL algorithm iteratively ( b = 2 , 3 . . . )

Original: performs following steps: 1) generates descriptor-attention matrix pairs ( V d , V a ) , ( S d , S a ) , 2) calculates similarity matrix R = R d (cid:12) R a
Revised: performs following steps: 1) generates descriptor-attention matrix pairs ( V d , V a ) , ( S d , S a ) , 2) calculates similarity matrix R = R d (cid:12) R a

Original: = ( V Td S d ) (cid:12) ( V Ta S a ) , where (cid:12) is element-wise product, 3) selects P relevant data points from the remaining subset using acquisition function arg max
Revised: = ( V Td S d ) (cid:12) ( V Ta S a ) , where (cid:12) is element-wise product, 3) selects P relevant data points from the remaining subset using acquisition function arg max

Original: i ∈ N \ N b − 1 ( R ( X i ) , Φ) and 4) retrains Φ( X , θ ) using augmented subset N b .
Revised: i ∈ N \ N b − 1 ( R ( X i ) , Φ) and 4) retrains Φ( X , θ ) using augmented subset N b .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our method as well as uncertainty-based methods from (Gal et al., 2017) are applied to the MNIST and SVHN classiﬁcation.
Revised: Our method as well as uncertainty-based methods from (Gal et al., 2017) are applied to the MNIST and SVHN classiﬁcation.

Original: We evaluate AL with the original and distorted training data because unlabeled collection of data points cannot be a-priori perfectly selected.
Revised: We evaluate AL with the original and distorted training data because unlabeled collection of data points cannot be a-priori perfectly selected.

Original: Hence, we introduce a class imbalance which is deﬁned as the ratio of { 0 . . .
Revised: Hence, we introduce a class imbalance which is deﬁned as the ratio of { 0 . . .

Original: 4 } to { 5 . . .
Revised: 4 } to { 5 . . .

Original: 9 } digits.
Revised: 9 } digits.

Original: The following methods have been employed: random sampling, uncertainty-based (uncert), greedy selection using similarity matching without (top-P:none) and with EBA.
Revised: The following methods have been employed: random sampling, uncertainty-based (uncert), greedy selection using similarity matching without (top-P:none) and with EBA.

Original: The latter is estimated by saliency (top-P:grad) or IG (top-P:ig).
Revised: The latter is estimated by saliency (top-P:grad) or IG (top-P:ig).

Original: We rerun experiments 10 times for MNIST and 5 times for SVHN with all-randomized initial parameters.
Revised: We rerun experiments 10 times for MNIST and 5 times for SVHN with all-randomized initial parameters.

Original: Mean accuracy and standard deviation are reported.
Revised: Mean accuracy and standard deviation are reported.

Original: DNN parameters are trained from scratch initially and after each AL iteration.
Revised: DNN parameters are trained from scratch initially and after each AL iteration.

Original: Mini-batch size is chosen by cross-validation.
Revised: Mini-batch size is chosen by cross-validation.

Original: MNIST.
Revised: MNIST.

Original: The dataset train/val/test split is 50K/10K/10K.
Revised: The dataset train/val/test split is 50K/10K/10K.

Original: The LeNet is used with the following hyperparameters: epochs=50, batch-size=25, lr=0.05, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 passes and L is 20 for single-scale (before fc1 layer) and 90 for multiscale descriptors (all layers are concatenated).
Revised: The LeNet is used with the following hyperparameters: epochs=50, batch-size=25, lr=0.05, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 passes and L is 20 for single-scale (before fc1 layer) and 90 for multiscale descriptors (all layers are concatenated).

Original: Figure 2(a) shows that feature-only matching (topP:none L20) outperforms random selection by ≈ 1% while EBA (top-P:ig L90) adds another 1% of accuracy when there is no class imbalance.
Revised: Figure 2(a) shows that feature-only matching (topP:none L20) outperforms random selection by ≈ 1% while EBA (top-P:ig L90) adds another 1% of accuracy when there is no class imbalance.

Original: High class imbalance (Figure 2(c)) increases that gap: 
Revised: High class imbalance (Figure 2(c)) increases that gap: up to 20% for feature-only matching and 25% with EBA. The highest accuracy is achieved by multi-

Original: [Figure]
Revised: [Figure]

Original: 250 pool size.
Revised: 250 pool size.

Original: [Figure]
Revised: [Figure]

Original: up to 20% for feature-only matching and 25% with EBA. The highest accuracy is achieved by multiscale EBA estimated byIG. EBA-based methods outperform the best uncertainty-based variation ratio (uncert:varMC) approach for all class imbalance settings except the last one where its accuracy is higher by less than 1% when b = 4 .
Revised: scale EBA estimated by methods outperform the best uncertainty based variation ratio ( varMC) approach for all class imbalance settings except the last one where its accuracy is higher by less than 1% when b = 4 .

Original: This might be related to small-scale MNIST and pseudo-label noise for EBA.
Revised: This might be related to small-scale MNIST and pseudo-label noise for EBA.

Original: To study the effects of pseudo-labeling, we plot true-label conﬁgurations (marked by ”Abl”) as well.
Revised: To study the effects of pseudo-labeling, we plot true-label conﬁgurations (marked by ”Abl”) as well.

Original: The accuracy gap between EBA using true- and pseudo-labels is small with no class imbalance, but much larger (up to 25%) when class imbalance ratio is 100 during ﬁrst AL iterations.
Revised: The accuracy gap between EBA using true- and pseudo-labels is small with no class imbalance, but much larger (up to 25%) when class imbalance ratio is 100 during ﬁrst AL iterations.

Original: SVHN.
Revised: SVHN.

Original: The dataset train/validation/test split is 500K/104K/26K. A typical 8-layer CNN is used with the following hyperparameters: epochs=35, batch-size=25, lr=0.1, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 and L is 256 for single-scale (before fc1 layer) and 384 for two-scale descriptors (+ layer before conv7 ).
Revised: The dataset train/validation/test split is 500K/104K/26K. A typical 8-layer CNN is used with the following hyperparameters: epochs=35, batch-size=25, lr=0.1, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 and L is 256 for single-scale (before fc1 layer) and 384 for two-scale descriptors (+ layer before conv7 ).

Original: Figure 3 shows that the gap between random selection and the best EBA-based AL method grows from 2% to more than 12% when the unlabeled training collection has more class imbalance.
Revised: Figure 3 shows that the gap between random selection and the best EBA-based AL method grows from 2% to more than 12% when the unlabeled training collection has more class imbalance.

Original: The gap between full training dataset accuracy increases for larger-scale SVHN as well.
Revised: The gap between full training dataset accuracy increases for larger-scale SVHN as well.

Original: This results in even faster convergence for the proposed AL relative to random selection.
Revised: This results in even faster convergence for the proposed AL relative to random selection.

Original: Accuracies of the uncert methods are closer to each other than for MNIST, which may signal their declining effectiveness for large-scale data.
Revised: Accuracies of the uncert methods are closer to each other than for MNIST, which may signal their declining effectiveness for large-scale data.

Original: The proposed EBA-based methods outperform all uncertainty-based methods for SVHN in the ﬁrst AL iterations (up to +2.5%) and later arrive at approximately equal results.
Revised: The proposed EBA-based methods outperform all uncertainty-based methods for SVHN in the ﬁrst AL iterations (up to +2.5%) and later arrive at approximately equal results.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We applied recent image retrieval feature-extraction techniques to deep AL and introduced a novel
Revised: We applied recent image retrieval feature-extraction techniques to deep AL and introduced a novel

Original: EBA mechanism to improve feature-similarity matching.
Revised: EBA mechanism to improve feature-similarity matching.

Original: First feasibility experiments on MNIST and SVHN datasets showed advantages of EBA to improve density-based AL.
Revised: First feasibility experiments on MNIST and SVHN datasets showed advantages of EBA to improve density-based AL.

Original: Rather than performing AL for the well-picked training datasets, we also considered more realistic and challenging scenarios with class-imbalanced training collections where the proposed method emphasized the importance of additional feature supervision.
Revised: Rather than performing AL for the well-picked training datasets, we also considered more realistic and challenging scenarios with class-imbalanced training collections where the proposed method emphasized the importance of additional feature supervision.

Original: In future research, EBA could be evaluated with other types of data distortions and biases: within-class bias, adversarial examples, etc.
Revised: In future research, EBA could be evaluated with other types of data distortions and biases: within-class bias, adversarial examples, etc.

Original: Furthermore, such applications as object detection and image segmentation may beneﬁt more from EBA because multiscale attention can focus on spatially-important features.
Revised: Furthermore, such applications as object detection and image segmentation may beneﬁt more from EBA because multiscale attention can focus on spatially-important features.