Original: E XPLANATION -B ASED A TTENTION FOR S EMI - S UPERVISED D EEP A CTIVE L EARNING
Revised: E XPLANATION -B ASED A TTENTION FOR S EMIS UPERVISED D EEP A CTIVE L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting.
Revised: We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting.

Original: The proposed attention mechanism is based on recently proposed methods to visually explain predictions made by deep neural networks (DNNs).
Revised: The proposed attention mechanism is based on recent methods to visually explain predictions made by DNNs.

Original: We apply the proposed explanation-based attention to MNIST and SVHN classiﬁcation.
Revised: We apply the proposed explanation-based attention to MNIST and SVHN classiﬁcation.

Original: The conducted experiments show accuracy improvement for the original and class-imbalanced datasets with the same amount of training examples as well as faster long-tail convergence compared toconventional uncertainty-based approaches
Revised: The conducted experiments show accuracy improvements for the original and class-imbalanced datasets with the same number of training examples and faster long-tail convergence compared to uncertainty-based methods

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep active learning (AL) minimizes the number of expensive annotations needed to train deep neural networks (DNNs) by selecting a subset of relevant data points from a large unlabeled dataset (Lewis & Gale, 1994).
Revised: Deep active learning (AL) minimizes the number of expensive annotations needed to train DNNs by selecting a subset of relevant data points from a large unlabeled dataset (Lewis & Gale, 1994).

Original: This subset is annotated and added to the training dataset in a single pool of data points or, more often, in an iterative fashion.
Revised: This subset is annotated and added to the training dataset in a single pool of data points or, more often, in an iterative fashion.

Original: The goal is to maximize prediction accuracy while minimizing  pool size × number of iterations product.
Revised: The goal is to maximize prediction accuracy while minimizing the product of pool size × number of iterations.

Original: A proxy for this goal could be the task of matching feature distributions between the validation and the AL-selected training datasets.
Revised: A proxy for this goal could be the task of matching feature distributions between the validation and the AL-selected training datasets.

Original: In density-based AL approaches, data selection is typically performed using a simple Euclideandistance feature similarity metric (Sener & Savarese, 2018).
Revised: In density-based AL approaches, data selection is typically performed using a simple L 2 -distance metric (Sener & Savarese, 2018).

Original: The image retrieval ﬁeld (Zhou et al., 2017) has advanced much further in this area.
Revised: The image retrieval ﬁeld (Zhou et al., 2017) has advanced much further in this area.

Original: For example, recent state-of-the-art image retrieval systems are based on DNN-based feature extraction (Babenko & Lempitsky, 2015) and attention mechanisms (Noh et al., 2017).
Revised: For example, recent state-of-the-art image retrieval systems are based on DNNbased feature extraction (Babenko & Lempitsky, 2015) with attention mechanisms (Noh et al., 2017).

Original: The latter estimates an attention mask to weight importance of the extracted features and it is trained along with the feature extraction.
Revised: The latter estimates an attention mask to weight importance of the extracted features and it is trained along with the feature extraction.

Original: Inspired by this, we employ image retrieval techniques and propose a novel attention mechanism for deep AL.
Revised: Inspired by this, we employ image retrieval techniques and propose a novel attention mechanism for deep AL.

Original: Unlike supervised self-attention in (Noh et al., 2017) or (Vaswani et al., 2017), our attention  is not trained with the model and is based on recent methods to generate visual explanations and to attribute feature importance values (Sundararajan et al., 2017).
Revised: Unlike supervised self-attention in (Noh et al., 2017; Vaswani et al., 2017), our attention mechanism is not trained with the model. It relies on recent methods to generate visual explanations and to attribute feature importance values (Sundararajan et al., 2017).

Original: We show the effectiveness of this explanation-based attention (EBA) mechanism for AL when combined with multi-scale feature extraction on a number of image classiﬁcation datasets.
Revised: We show the effectiveness of such explanation-based attention (EBA) mechanism for AL when combined with multi-scale feature extraction on a number of image classiﬁcation datasets.

Original: We also conduct experiments for distorted class-imbalanced training data which is a more realistic assumption for unlabeled data.
Revised: We also conduct experiments for distorted class-imbalanced training data which is a more realistic assumption for unlabeled data.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: AL is a well studied approach to decrease annotation costs in a traditional machine learning pipelines (Settles, 2010).
Revised: AL is a well-studied approach to decrease annotation costs in a traditional machine learning pipelines (Settles, 2010).

Original: Recently, AL has been applied to data-demanding DNN-based systems in semisupervised or weakly-supervised settings.
Revised: Recently, AL has been applied to data-demanding DNN-based systems in semi-supervised or weakly-supervised settings.

Original: Though AL is an attractive direction, existing methods struggle to deal with high-dimensional data e.g. images.
Revised: Though AL is an attractive direction, existing methods struggle to deal with high-dimensional data e.g. images.

Original: We believe this is related to the lack of class and instance-level feature importance information as well as the inability to capture spatially-localized features.
Revised: We believe this is related to the lack of class and instance-level feature importance information as well as the inability to capture spatially-localized features.

Original: To overcome these limitations, we are interested in estimating spatiallymultiscale features and using  EBA mechanism to select only the most discriminative features. 
Revised: To overcome these limitations, we are interested in estimating spatiallymultiscale features and using our EBA mechanism to select only the most discriminative features. [Figure]

Original: Wang et al.
Revised: Wang et al.

Original: (2017) proposed to augment the training dataset by labeling the least conﬁdent data points and heuristically pseudo-labeling high conﬁdence predictions.
Revised: (2017) proposed to augment the training dataset by labeling the least conﬁdent data points and heuristically pseudo-labeling high conﬁdence predictions.

Original: We believe  softmax conﬁdence is not a reliable proxy for the goals of AL i.e. for selecting images using feature distribution[Figure] matching between validation and training data.
Revised: We believe the softmax output is not a reliable proxy for the goals of AL i.e. for selecting images using feature distribution matching between validation and train data.

Original: Unlike (Wang et al., 2017), we use pseudo-labels only to estimate EBA vectors and upweight discriminative features.
Revised: Unlike (Wang et al., 2017), we use pseudo labels only to estimate EBA vectors and ﬁnd similarities between discriminative features.

Original: Gal et al.
Revised: Gal et al.

Original: (2017) introduced a measure of uncertainty for approximate Bayesian inference that can be estimated using stochastic forward passes through a DNN with dropout layers.
Revised: (2017) introduced a measure of uncertainty for approximate Bayesian inference that can be estimated using stochastic forward passes through a DNN with dropout layers.

Original: An acquisition function then selects data points with the highest uncertainty which is measured at the output ofthe softmaxlayer using a number of the proposed methods.
Revised: An acquisition function then selects data points with the highest uncertainty which is measured at the output of softmax using several metrics.

Original: Recently, Lakshminarayanan et al. (2017) and Beluch et al. (2018) extended uncertainty-based methods by using an ensemble of networks for uncertainty estimation and achieved superior accuracy.
Revised: Recent work (Beluch et al., 2018) extended this method by using an ensemble of networks for uncertainty estimation and achieved superior accuracy.

Original: Sener & Savarese (2018) formulated feature similarity-based selection as a geometric core-set approach which outperforms conventional k -center clustering.
Revised: Sener & Savarese (2018) formulated feature similarity-based selection as a geometric core-set approach which outperforms greedy k -center clustering.

Original: Though their method can complement our approach, we are focusing on the  feature extraction part, while that work employed a simple Ldistance similarity measure for the activations of the last fully-connected layer.
Revised: Though their method can complement our approach, we are focusing on the novel feature extraction. For instance, they employed a simple Ldistance similarity measure for the activations of the last fully-connected layer.

Original: Thework that is most similar  to ours is Vodrahalli et al.
Revised: The most similar work to ours, by Vodrahalli et al.

Original: (2018) which used the gradients as a measure of importance for dataset subsampling and analysis.
Revised: (2018), uses the gradients as a measure of importance for dataset subsampling and analysis.

Original: However, our approach formulates the problem as a multi-scale EBA for AL  and goes beyond a less robust single-step gradient attention.
Revised: However, our approach formulates the problem as a multi-scale EBA for AL application and goes beyond a less robust single-step gradient attention.

Original: Other related works are online importance sampling methods (Ren et al., 2018) and the inﬂuence functions approach in (Koh & Liang, 2017).
Revised: Other related works are online importance sampling methods (Ren et al., 2018) and the inﬂuence functions approach in (Koh & Liang, 2017).

Original: Online importance sampling upweights samples within the mini-batch during supervised training using gradient similarity while inﬂuence functions analyze data point importance using computationally challenging second-order gradient information.
Revised: Online importance sampling upweights samples within the mini-batch during supervised training using gradient similarity while inﬂuence functions analyze data point importance using computationally challenging second-order gradient information.

Original:  M ETHOD
Revised:  M ETHOD

Original: Pool-based AL.
Revised: Pool-based AL.

Original: Let ( X , y ) be an input-label pair.
Revised: Let ( X , y ) be an input-label pair.

Original: There is arelatively small validation dataset { ( X vi , y vi ) } i ∈ M of size M and alarge collection of training pairs { ( X i , y i ) } i ∈ N of size N for which, initially, only a small random subset or pool of labels  N 1 is known.
Revised: There is a validation dataset { ( X vi , y vi ) } i ∈ M of size M and a collection of training pairs { ( X i , y i ) } i ∈ N of size N for which, initially, only a small random subset or pool of labels indexed by N 1 is known.

Original: We presume that the validation dataset matches the distribution of test data.
Revised: The validation dataset approximates the distribution of test data.

Original: At every b th iteration the AL algorithm selects a pool of P new labels to be  added to existing training pairs comprising a training dataset  N b .
Revised: At every b th iteration the AL algorithm selects a pool of P new labels to be annotated and added to existing training pairs which creates a training dataset indexed by N b .

Original: Then, we optimize our DNN Φ( X , θ ) subject to a loss function (1 /N b )  (cid:80) i ∈ N b L (ˆ y i , y i ) w.r.t.
Revised: A DNN Φ( X , θ ) is optimized by minimizing a loss function ( N b ) − 1 (cid:80) i ∈ N b L (ˆ y i , y i ) w.r.t.

Original: to model parameters θ .
Revised: to model parameters θ .

Original: However, the actual task is to minimize validation loss (1 /M )
Revised: However, the actual task is to minimize validation loss expressed by

Original:  (cid:80) i ∈ M L (ˆ y vi , y vi ) .
Revised: M − 1 (cid:80) i ∈ M L (ˆ y vi , y vi ) .

Original: Therefore, an oracle AL algorithm achieves minimum  validation loss while performing the smallestnumber of iterations b with the ﬁxed pool size P .
Revised: Therefore, an oracle AL algorithm achieves minimum of validation loss using the smallest b × P product.

Original: In this work, we are interested not in  an oracle AL, but in a method to extract relevant features for an AL acquisition function.
Revised: In this work, we are interested not in ﬁnding an oracle acquisition function, but in a method to extract relevant features for such function.

Original: We use a simple greedy  algorithm to select the data points in the unlabeled training collection which are most similar to the misclassiﬁed entries in the validation dataset.If the number of misclassiﬁed validation data points exceeds pool size, we apply conventional k -center clustering with P centers.
Revised: We use a low-complexity greedy k -center algorithm to select the data points in the unlabeled training collection which are most similar to the misclassiﬁed entries in the validation dataset.

Original: Feature descriptors for AL.
Revised: Feature descriptors.

Original: Let F ji ∈ R C × H × W , where C , H , and W are the number of channels, the height, and the width, respectively be the output of the j th layer of DNN for input  X i .
Revised: Let F ji ∈ R C × H × W , where C , H , and W are the number of channels, the height, and the width, respectively be the output of the j th layer of DNN for input image X i .

Original: Then, a feature vector or descriptor  can be deﬁned as d i =
Revised: Then, a feature vector or descriptor of length L can be deﬁned as d i =

Original: φ ( F i )
Revised: φ ( F i )

Original: ∈
Revised: ∈

Original: R L × 1 , where function φ () is a conventional average pooling operation from (Babenko & Lempitsky, 2015).
Revised: R L × 1 , where function φ ( · ) is a conventional average pooling operation from (Babenko & Lempitsky, 2015).

Original: In a multi-scale case, descriptor  d i =
Revised: In a multi-scale case, descriptor is a concatenation of multiple feature vectors d i =

Original: [ φ j ( F j i ); . . . ; φ l ( F li )] .
Revised: [ φ j ( F j i ) , · · · , φ l ( F li )] .

Original: A descriptor matrix for the validation dataset V d ∈ RM × L  and current training dataset S bd ∈ R  N b× L can be calculated using forward passes.
Revised: A descriptor matrix for the validation dataset V d ∈ R L × M and current training dataset S d ∈ R L × N b can be calculated using forward passes.

Original: Practically, descriptors can be compressed for storage efﬁciency reasons using PCA, quantization, etc.
Revised: Practically, descriptors can be compressed for storage efﬁciency reasons using PCA, quantization, etc.

Original: Then, a match kernel (Lee, 1999), e.g. cosine similarity, can be used to match features in both datasets.
Revised: Then, a match kernel (Lee, 1999), e.g. cosine similarity, can be used to match features in both datasets.

Original: Assuming that vectors d i are L 2 -normalized, the cosine-similarity matrix  R bd = V d ( S bd ) T .
Revised: Assuming that vectors d i are L 2 -normalized, the cosine-similarity matrix is simply R d = V Td S d .

Original: Explanation-based attention.
Revised: Explanation-based attention.

Original: Feature maps F i extracted by Φ( X , θ ) and pooled by φ () contain features that: a) are not class and instance-level discriminative, b) spatially represent features for a plurality of objects in the input.
Revised: Feature maps F i extracted by Φ( X , θ ) and pooled by φ ( · ) contain features that: a) are not class and instance-level discriminative (in other words, not disentangled), b) spatially represent features for a plurality of objects in the input.

Original: We would like to upweight discriminative features that satisfy a) and b) using an attention mechanism.
Revised: We would like to upweight discriminative features that satisfy a) and b) using an attention mechanism.

Original: One approach would be to use self-attention (Vaswani et al., 2017) at the cost of modifying network architecture and intervening in the training process.
Revised: One approach would be to use self-attention (Vaswani et al., 2017) at the cost of modifying network architecture and intervening into the training process.

Original: Instead, we propose to use EBA that is generated only for feature selection.
Revised: Instead, we propose to use EBA that is generated only for feature selection.

Original: The EBA mechanism attributes feature importance values w.r.t.
Revised: The EBA mechanism attributes feature importance values w.r.t.

Original: to the output predictions.
Revised: to the output predictions.

Original: Unlike a visual explanation task, which estimates importance heatmaps in the input (image) space, we propose to estimate feature importance tensors
Revised: Unlike a visual explanation task, which estimates importance heatmaps in the input (image) space, we propose to estimate feature importance tensors

Original: A i of the internal DNN representations F i .
Revised: A i of the internal DNN representations F i .

Original: Attention tensors
Revised: Attention tensors

Original: A i can be efﬁciently calculated using a series of backpropagation passes.
Revised: A i can be efﬁciently calculated using a series of backpropagation passes.

Original: Using one of backpropagation-based methods called integrated gradients (IG) from (Sundararajan et al., 2017), A ji can be estimated as
Revised: Using one of backpropagation-based methods called integrated gradients (IG) from (Sundararajan et al., 2017), A ji can be estimated as

Original: [Equation]
Revised: [Equation]

Original: where K is the number of steps to approximate the continuous integral by a linear path.
Revised: where K is the number of steps to approximate the continuous integral by a linear path.

Original: Other forms of (1) are possible: from the simplest saliency method for which K = 1 (Simonyan et al., 2014) to more advanced methods with randomly sampled input features (Gudovskiy et al., 2018).
Revised: Other forms of (1) are possible: from the simplest saliency method for which K = 1 (Simonyan et al., 2014) to more advanced methods with randomly sampled input features (Gudovskiy et al., 2018).

Original: Due to lack of labels  in (1), wepropose to use pseudo-labeling.
Revised: Due to lack of labels y i in (1), we use common pseudo-labeling strategy:

Original: Pseudo-labels are used only to calculate similarity  rather than to perform a  greedy augmentation in (Wang et al., 2017).
Revised: y i = 1 argmax ˆy i . It is schematically shown in Figure 1. Unlike (Wang et al., 2017), pseudo-labels are used only to calculate similarity without additional hyperparameters rather than to perform a threshold-selected greedy augmentation.

Original: The EBA A i can be converted to multi-scale attention vector using the same processing a i =
Revised: The EBA A i can be converted to multi-scale attention vector using the same processing a i =

Original: φ ( A i ) ∈
Revised: φ ( A i ) ∈

Original: R L × 1  .
Revised: R L × 1 , which, by analogy, forms validation V a ∈ R L × M and train attention matrices S a ∈ R L × N b .

Original: The latter  is implemented in most modern frameworks and, therefore, the complexity to generate A
Revised: The latter processing is implemented in most modern frameworks and, therefore, the complexity to generate A

Original: i is only K forward-backward calls.
Revised: i is only K forward-backward passes.

Original: This is schematically shown in Figure 1.

Original: Summary for the proposed method.
Revised: Summary for the proposed method.

Original: A random subset  N 1of training data points is annotated and a DNN Φ( X , θ ) optimized for this subset.
Revised: A random subset of N 1 training data points is annotated and a DNN Φ( X , θ ) optimized for this subset.

Original: Then, the AL algorithm iteratively ( b = 2 , 3 . . . )
Revised: Then, the AL algorithm iteratively ( b = 2 , 3 . . . )

Original: performs following steps: 1) generates descriptor matrix pairs ( V d , V a ) , ( S bd , S ba ) , 2) calculates similarity matrix Rb = R bd (cid:12) R ba =
Revised: performs following steps: 1) generates descriptor-attention matrix pairs ( V d , V a ) , ( S d , S a ) , 2) calculates similarity matrix R = R d (cid:12) R a

Original: V d (  S bd )T (cid:12)V a (  S ba )T , where (cid:12) is element-wise product, 3) selects P relevant data points from the remaining subset usingsimilarity matrix R b and acquisition function arg max i ∈ N \ N b − 1 ( Rb ( X i ) , Φ) and 4) retrains Φ( X , θ ) using augmented subset N b .
Revised: = ( V Td S d ) (cid:12) ( V Ta S a ) , where (cid:12) is element-wise product, 3) selects P relevant data points from the remaining subset using acquisition function arg max i ∈ N \ N b − 1 ( R ( X i ) , Φ) and 4) retrains Φ( X , θ ) using augmented subset N b .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The proposed method as well as uncertainty-based methods from Gal et al. (2017) are applied to  MNIST and SVHN.
Revised: Our method as well as uncertainty-based methods from (Gal et al., 2017) are applied to the MNIST and SVHN classiﬁcation.

Original: We evaluate AL with the original and distorted training data because unlabeled collection of data points cannot be a-priori perfectly selected.
Revised: We evaluate AL with the original and distorted training data because unlabeled collection of data points cannot be a-priori perfectly selected.

Original: Hence, we introduce a class imbalanceratio , which is deﬁned as the ratio of { 0 . . .
Revised: Hence, we introduce a class imbalance which is deﬁned as the ratio of { 0 . . .

Original: 4 }digits to { 5 . . .
Revised: 4 } to { 5 . . .

Original: 9 } digits.
Revised: 9 } digits.

Original: The following methods have been employed: random and uncertainty-based ( uncert ) selection, greedy(top-P) selection using similarity R without ( none ) and with EBA.
Revised: The following methods have been employed: random sampling, uncertainty-based (uncert), greedy selection using similarity matching without (top-P:none) and with EBA.

Original: The latter is estimated by saliency ( grad ) or IG ( ig ).
Revised: The latter is estimated by saliency (top-P:grad) or IG (top-P:ig).

Original: We rerun experiments 10 times for MNIST and 5 times for SVHN with all- initial scratch initially
Revised: We rerun experiments 10 times for MNIST and 5 times for SVHN with all-randomized initial parameters.

Revised: Mean accuracy and standard deviation are reported.

Revised: DNN parameters are trained from scratch initially and after each AL iteration.

Revised: Mini-batch size is chosen by cross-validation.

Original: MNIST.
Revised: MNIST.

Original: The dataset train/val/test split is 50K/10K/10K.
Revised: The dataset train/val/test split is 50K/10K/10K.

Original: Thestandard LeNet is used with the following hyperparameters: epochs=50, batch-size=25, lr=0.05, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 passes and L is 20 for single-scale (before fc1 layer) and 90 for multi-scale descriptors (all layers are concatenated).
Revised: The LeNet is used with the following hyperparameters: epochs=50, batch-size=25, lr=0.05, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 passes and L is 20 for single-scale (before fc1 layer) and 90 for multiscale descriptors (all layers are concatenated).

Original: Figure 2(a) shows that feature-only matching (top-P:none L20) outperforms random selection by ≈ 1% while EBA (top-P:ig L90) adds another 1% of accuracy when there is no class imbalance.
Revised: Figure 2(a) shows that feature-only matching (topP:none L20) outperforms random selection by ≈ 1% while EBA (top-P:ig L90) adds another 1% of accuracy when there is no class imbalance.

Original: High class imbalance (Figure 2(c)) widens the gap in achievable accuracy up to 20% for feature-only matching and 24% with EBA. The highest accuracy is achieved by multi-scale EBA estimated by IG. EBA-based methods outperform the best uncertainty-based variation ratio (uncertain:varMC) approach for all class imbalance settings except the last one where the accuracy of variation ratio is
Revised: High class imbalance (Figure 2(c)) increases that gap:

Original: [Figure]
Revised: [Figure]

Original: b = 10 ) are performed each with P = 250 pool size
Revised: 250 pool size.

Original: [Figure]
Revised: [Figure]

Original: slightly higher.
Revised: up to 20% for feature-only matching and 25% with EBA. The highest accuracy is achieved by multiscale EBA estimated by IG. EBA-based methods outperform the best uncertainty-based variation ratio (uncert:varMC) approach for all class imbalance settings except the last one where its accuracy is higher by less than 1% when b = 4 .

Original: This might be related to small-scale MNIST and pseudo-label noise for EBA.
Revised: This might be related to small-scale MNIST and pseudo-label noise for EBA.

Original: To study the effects of pseudo-labeling, we plot true-label conﬁgurations (marked by ”Abl”) as well.
Revised: To study the effects of pseudo-labeling, we plot true-label conﬁgurations (marked by ”Abl”) as well.

Original: When there is no class imbalance, the performance gap between EBA using true- and pseudo-labels is small, but much larger (up to 25%) when class imbalance ratio is 100 during ﬁrst AL iterations.
Revised: The accuracy gap between EBA using true- and pseudo-labels is small with no class imbalance, but much larger (up to 25%) when class imbalance ratio is 100 during ﬁrst AL iterations.

Original: SVHN.
Revised: SVHN.

Original: The dataset train/val/test split is 500K/104,388/26,032. A typical 8-layer CNN is used with the following hyperparameters: epochs=35, batch-size=25, lr=0.1, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 and L is 256 for single-scale (before fc1 layer) and 384 for two-scale descriptors (+ layer before conv7 ).
Revised: The dataset train/validation/test split is 500K/104K/26K. A typical 8-layer CNN is used with the following hyperparameters: epochs=35, batch-size=25, lr=0.1, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 and L is 256 for single-scale (before fc1 layer) and 384 for two-scale descriptors (+ layer before conv7 ).

Original: Figure 3 shows that the gap between random selection and the best EBA-based AL method grows from 2% to more than 10% when the unlabeled training collection has more class imbalance.
Revised: Figure 3 shows that the gap between random selection and the best EBA-based AL method grows from 2% to more than 12% when the unlabeled training collection has more class imbalance.

Original: The gap between full training dataset accuracy increases for larger-scale SVHN as well.
Revised: The gap between full training dataset accuracy increases for larger-scale SVHN as well.

Original: This results in even faster convergence for the proposed AL relative to random selection.
Revised: This results in even faster convergence for the proposed AL relative to random selection.

Original: The ”topP:none L256” underperforms compared to uncert and EBA-based methods.

Original: Accuracies of the uncert methods are closer to each other than for MNIST, which may signal that such methods are less effective for large-scale data.
Revised: Accuracies of the uncert methods are closer to each other than for MNIST, which may signal their declining effectiveness for large-scale data.

Original: The  EBA-based methods outperform all uncert methods, which is another argument supporting that hypothesis. Similarly to MNIST, AL methods with EBA improve accuracy with longer descriptor and more accurate attention estimation.
Revised: The proposed EBA-based methods outperform all uncertainty-based methods for SVHN in the ﬁrst AL iterations (up to +2.5%) and later arrive at approximately equal results.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We applied recent image retrieval feature-extraction techniques for deep AL and introduced a novel
Revised: We applied recent image retrieval feature-extraction techniques to deep AL and introduced a novel

Original: EBA mechanism to improve feature-similarity matching.
Revised: EBA mechanism to improve feature-similarity matching.

Original: First feasibility experiments  showed advantages of EBA to improve AL on MNIST and SVHN datasets.
Revised: First feasibility experiments on MNIST and SVHN datasets showed advantages of EBA to improve density-based AL.

Original: Rather than performing AL for the well-picked training datasets, we also conducted more realistic and challenging experiments with class-imbalanced training collections.
Revised: Rather than performing AL for the well-picked training datasets, we also considered more realistic and challenging scenarios with class-imbalanced training collections where the proposed method emphasized the importance of additional feature supervision.

Original: For the next steps, EBA should be evaluated with other types of data distortions and biases: withinclass imbalance, adversarial examples, etc.
Revised: In future research, EBA could be evaluated with other types of data distortions and biases: within-class bias, adversarial examples, etc.

Original: Furthermore, such applications as object detection and image segmentation may beneﬁt more from EBA because multi-scale attention can upweight spatially-important features.
Revised: Furthermore, such applications as object detection and image segmentation may beneﬁt more from EBA because multiscale attention can focus on spatially-important features.

Original: Lastly, a comparison with ensembled-based and density-based approaches is needed or EBA can be employed to complement them.