Original: D EEP
Revised: D EEP

Original: A CTIVE
Revised: A CTIVE

Original: L EARNING
Revised: L EARNING

Original: FOR
Revised: FOR

Original: N AMED
Revised: N AMED

Original: E NTITY
Revised: E NTITY

Original: R ECOGNITION
Revised: R ECOGNITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep learning has yielded state-of-the-art performance on many natural language processing tasks including named entity recognition (NER).
Revised: Deep learning has yielded state-of-the-art performance on many natural language processing tasks including named entity recognition (NER).

Original: However, this typically requires large amounts of labeled data.
Revised: However, this typically requires large amounts of labeled data.

Original: In this work, we demonstrate that the amount of labeled training data can be drastically reduced when deep learning is combined with active learning.
Revised: In this work, we demonstrate that the amount of labeled training data can be drastically reduced when deep learning is combined with active learning.

Original: While active learning is sample-efﬁcient, it can be computationally expensive since it requires iterative retraining.
Revised: While active learning is sample-efﬁcient, it can be computationally expensive since it requires iterative retraining.

Original: To speed this up, we introduce a lightweight architecture for NER, viz.
Revised: To speed this up, we introduce a lightweight architecture for NER, viz.

Original: , the CNN-CNN-LSTM model consisting of convolutional character and word encoders and a long short term memory (LSTM) tag decoder.
Revised: , the CNN-CNN-LSTM model consisting of convolutional character and word encoders and a long short term memory (LSTM) tag decoder.

Original: The model achieves nearly state-of-the-art performance on standard datasets for the task while being computationally much more efﬁcient than best performing models.
Revised: The model achieves nearly state-of-the-art performance on standard datasets for the task while being computationally much more efﬁcient than best performing models.

Original: We carry out incremental active learning, during the training process, and are able to nearly match state-of-the-art performance with just 25% of the original training data.
Revised: We carry out incremental active learning, during the training process, and are able to nearly match state-of-the-art performance with just 25% of the original training data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Over the past few years, papers applying deep neural networks (DNNs) to the task of named entity recognition (NER) have successively advanced the state-of-the-art (Collobert et al., 2011; Huang et al., 2015; Lample et al., 2016; Chiu & Nichols, 2015; Yang et al., 2016).
Revised: Over the past few years, papers applying deep neural networks (DNNs) to the task of named entity recognition (NER) have successively advanced the state-of-the-art (Collobert et al., 2011; Huang et al., 2015; Lample et al., 2016; Chiu & Nichols, 2015; Yang et al., 2016).

Original: However, under typical training procedures, the advantages of deep learning diminish when working with small datasets.
Revised: However, under typical training procedures, the advantages of deep learning diminish when working with small datasets.

Original: For instance, on the OntoNotes-5.0 English dataset, whose training set contains 1,088,503 words, a DNN model outperforms the best shallow model by 2.24% as measured by F1 score (Chiu &
Revised: For instance, on the OntoNotes-5.0 English dataset, whose training set contains 1,088,503 words, a DNN model outperforms the best shallow model by 2.24% as measured by F1 score (Chiu &

Original: Nichols, 2015).
Revised: Nichols, 2015).

Original: However, on the comparatively small CoNLL-2003 English dataset, whose training set contains 203,621 words, the best DNN model enjoys only a 0.4% advantage.
Revised: However, on the comparatively small CoNLL-2003 English dataset, whose training set contains 203,621 words, the best DNN model enjoys only a 0.4% advantage.

Original: To make deep learning more broadly useful, it is crucial to reduce its training data requirements.
Revised: To make deep learning more broadly useful, it is crucial to reduce its training data requirements.

Original: Generally, the annotation budget for labeling is far less than the total number of available (unlabeled) samples.
Revised: Generally, the annotation budget for labeling is far less than the total number of available (unlabeled) samples.

Original: For NER, getting unlabeled data is practically free, owing to the large amount of content that can be efﬁciently scraped off the web.
Revised: For NER, getting unlabeled data is practically free, owing to the large amount of content that can be efﬁciently scraped off the web.

Original: On the other hand, it is especially expensive to obtain annotated data for NER since it requires multi-stage pipelines with sufﬁciently well-trained annotators (Kilicoglu et al., 2016; Bontcheva et al., 2017).
Revised: On the other hand, it is especially expensive to obtain annotated data for NER since it requires multi-stage pipelines with sufﬁciently well-trained annotators (Kilicoglu et al., 2016; Bontcheva et al., 2017).

Original: In such cases, active learning offers a promising approach to efﬁciently select the set of samples for labeling.
Revised: In such cases, active learning offers a promising approach to efﬁciently select the set of samples for labeling.

Original: Unlike the supervised learning setting, in which examples are drawn and labeled at random, in the active learning setting, the algorithm can choose which examples to label.
Revised: Unlike the supervised learning setting, in which examples are drawn and labeled at random, in the active learning setting, the algorithm can choose which examples to label.

Original: Active learning aims to select a more informative set of examples in contrast to supervised learning, which is trained on a set of randomly drawn examples.
Revised: Active learning aims to select a more informative set of examples in contrast to supervised learning, which is trained on a set of randomly drawn examples.

Original: A central challenge in active learning is to determine what constitutes more informative and how the active learner can recognize this based on what it already knows.
Revised: A central challenge in active learning is to determine what constitutes more informative and how the active learner can recognize this based on what it already knows.

Original: The most common approach is uncertainty sampling , in which the model preferentially selects examples for which it’s current prediction is least conﬁdent.
Revised: The most common approach is uncertainty sampling , in which the model preferentially selects examples for which it’s current prediction is least conﬁdent.

Original: Other approaches include representativeness-based sampling where the model selects a diverse set that represent the input space without adding too much redundancy.
Revised: Other approaches include representativeness-based sampling where the model selects a diverse set that represent the input space without adding too much redundancy.

Original: In this work , we investigate practical active learning algorithms on lightweight deep neural network architectures for the NER task.
Revised: In this work , we investigate practical active learning algorithms on lightweight deep neural network architectures for the NER task.

Original: Training with active learning proceeds in multiple rounds.
Revised: Training with active learning proceeds in multiple rounds.

Original: Traditional active learning schemes are expensive for deep learning since they require complete retraining of the classiﬁer with newly annotated samples after each round.
Revised: Traditional active learning schemes are expensive for deep learning since they require complete retraining of the classiﬁer with newly annotated samples after each round.

Original: In our experiments, for example, the model must be retrained 54 times.
Revised: In our experiments, for example, the model must be retrained 54 times.

Original: Because retraining from scratch is not practical, we instead carry out incremental training with each batch of new labels: we mix newly annotated samples with the older ones, and update our neural network weights for a small number of epochs, before querying for labels in a new round.
Revised: Because retraining from scratch is not practical, we instead carry out incremental training with each batch of new labels: we mix newly annotated samples with the older ones, and update our neural network weights for a small number of epochs, before querying for labels in a new round.

Original: This modiﬁcation drastically reduces the computational requirements of active learning methods and makes it practical to deploy them.
Revised: This modiﬁcation drastically reduces the computational requirements of active learning methods and makes it practical to deploy them.

Original: We further reduce the computational complexity by selecting a lightweight architecture for NER.
Revised: We further reduce the computational complexity by selecting a lightweight architecture for NER.

Original: We propose a new CNN-CNN-LSTM architecture for NER consisting of a convolutional character-level encoder, convolutional word-level encoder, and long short term memory (LSTM) tag decoder.
Revised: We propose a new CNN-CNN-LSTM architecture for NER consisting of a convolutional character-level encoder, convolutional word-level encoder, and long short term memory (LSTM) tag decoder.

Original: This model handles out-of-vocabulary words gracefully and, owing to the greater reliance on convolutions (vs recurrent layers), trains much faster than other deep models while performing competitively.
Revised: This model handles out-of-vocabulary words gracefully and, owing to the greater reliance on convolutions (vs recurrent layers), trains much faster than other deep models while performing competitively.

Original: We introduce a simple uncertainty-based heuristic for active learning with sequence tagging.
Revised: We introduce a simple uncertainty-based heuristic for active learning with sequence tagging.

Original: Our model selects those sentences for which the length-normalized log probability of the current prediction is the lowest.
Revised: Our model selects those sentences for which the length-normalized log probability of the current prediction is the lowest.

Original: Our experiments with the Onto-Notes 5.0 English and Chinese datasets demonstrate results comparable to the Bayesian active learning by disagreement method (Gal et al., 2017).
Revised: Our experiments with the Onto-Notes 5.0 English and Chinese datasets demonstrate results comparable to the Bayesian active learning by disagreement method (Gal et al., 2017).

Original: Moreover our heuristic is faster to compute since it does not require multiple forward passes.
Revised: Moreover our heuristic is faster to compute since it does not require multiple forward passes.

Original: On the OntoNotes-5.0 English dataset, our approach matches 99 % of the F1 score achieved by the best deep models trained in a standard, supervised fashion despite using only a 24 .
Revised: On the OntoNotes-5.0 English dataset, our approach matches 99 % of the F1 score achieved by the best deep models trained in a standard, supervised fashion despite using only a 24 .

Original: 9 % of the data.
Revised: 9 % of the data.

Original: On the
Revised: On the

Original: OntoNotes-5.0 Chinese dataset, we match 99 % performance with only 30 .
Revised: OntoNotes-5.0 Chinese dataset, we match 99 % performance with only 30 .

Original: 1 % of the data.
Revised: 1 % of the data.

Original: Thus, we are able to achieve state of art performance with drastically lower number of samples.
Revised: Thus, we are able to achieve state of art performance with drastically lower number of samples.

Original:  R ELATED WORK Deep learning for named entity recognition
Revised:  R ELATED WORK Deep learning for named entity recognition

Original: The use of DNNs for NER was pioneered by
Revised: The use of DNNs for NER was pioneered by

Original: Collobert et al.
Revised: Collobert et al.

Original: (2011), who proposed an architecture based on temporal convolutional neural networks (CNNs) over the sequence of words.
Revised: (2011), who proposed an architecture based on temporal convolutional neural networks (CNNs) over the sequence of words.

Original: Since then, many papers have proposed improvements to this architecture.
Revised: Since then, many papers have proposed improvements to this architecture.

Original: Huang et al.
Revised: Huang et al.

Original: (2015) proposed to replace CNN encoder in Collobert et al.
Revised: (2015) proposed to replace CNN encoder in Collobert et al.

Original: (2011) with bidirectional LSTM encoder, while Lample et al.
Revised: (2011) with bidirectional LSTM encoder, while Lample et al.

Original: and Chiu & Nichols (2015) introduced hierarchy in the architecture by replacing hand-engineered character-level features in prior works with additional bidirectional LSTM and CNN encoders respectively.
Revised: and Chiu & Nichols (2015) introduced hierarchy in the architecture by replacing hand-engineered character-level features in prior works with additional bidirectional LSTM and CNN encoders respectively.

Original: In other related work, Mesnil et al.
Revised: In other related work, Mesnil et al.

Original: and Nguyen et al.
Revised: and Nguyen et al.

Original: (2016) pioneered the use of recurrent neural networks (RNNs) for decoding tags.
Revised: (2016) pioneered the use of recurrent neural networks (RNNs) for decoding tags.

Original: However, most recent competitive approaches rely upon CRFs as decoder (Lample et al., 2016; Chiu & Nichols, 2015; Yang et al., 2016).
Revised: However, most recent competitive approaches rely upon CRFs as decoder (Lample et al., 2016; Chiu & Nichols, 2015; Yang et al., 2016).

Original: In this work, we demonstrate that LSTM decoders outperform CRF decoders and are faster to train when the number of entity types is large.
Revised: In this work, we demonstrate that LSTM decoders outperform CRF decoders and are faster to train when the number of entity types is large.

Original: Active learning
Revised: Active learning

Original: While learning-theoretic properties of active learning algorithms are wellstudied (Dasgupta et al., 2005;
Revised: While learning-theoretic properties of active learning algorithms are wellstudied (Dasgupta et al., 2005;

Original: Balcan et al., 2009; Awasthi et al., 2014; Yan & Zhang, 2017;
Revised: Balcan et al., 2009; Awasthi et al., 2014; Yan & Zhang, 2017;

Original: Beygelzimer et al., 2009), classic algorithms and guarantees cannot be generalized to DNNs, which are currently are the state-of-the-art techniques for NER.
Revised: Beygelzimer et al., 2009), classic algorithms and guarantees cannot be generalized to DNNs, which are currently are the state-of-the-art techniques for NER.

Original: Owing to the limitations of current theoretical analysis, more practical active learning applications employ a range of heuristic procedures for selecting examples to query.
Revised: Owing to the limitations of current theoretical analysis, more practical active learning applications employ a range of heuristic procedures for selecting examples to query.

Original: For example, Tong & Koller (2001) suggests a margin-based selection criteria, while Settles & Craven (2008) while Shen et al.
Revised: For example, Tong & Koller (2001) suggests a margin-based selection criteria, while Settles & Craven (2008) while Shen et al.

Original: (2004) combines multiple criteria for NLP tasks.
Revised: (2004) combines multiple criteria for NLP tasks.

Original: Culotta & McCallum (2005) explores the application of least conﬁdence criterion for linear
Revised: Culotta & McCallum (2005) explores the application of least conﬁdence criterion for linear

Original: CRF models on sequence prediction tasks.
Revised: CRF models on sequence prediction tasks.

Original: For a more comprehensive review of the literature, we refer to Settles (2010) and Olsson (2009).
Revised: For a more comprehensive review of the literature, we refer to Settles (2010) and Olsson (2009).

Original: Deep active learning
Revised: Deep active learning

Original: While DNNs have achieved impressive empirical results across diverse applications (Krizhevsky et al., 2012; Hinton et al., 2012; Manning, 2016), active learning approaches for these models have yet to be well studied, and most current work addresses image classiﬁcation.
Revised: While DNNs have achieved impressive empirical results across diverse applications (Krizhevsky et al., 2012; Hinton et al., 2012; Manning, 2016), active learning approaches for these models have yet to be well studied, and most current work addresses image classiﬁcation.

Original: Wang et al.
Revised: Wang et al.

Original: (2016) claims to be the ﬁrst to study active learning for image classiﬁcation with CNNs and proposes methods based on uncertainty-based sampling, while Gal et al.
Revised: (2016) claims to be the ﬁrst to study active learning for image classiﬁcation with CNNs and proposes methods based on uncertainty-based sampling, while Gal et al.

Original: (2017) and Kendall &
Revised: (2017) and Kendall &

Original: Gal (2017) show that sampling based on a Bayesian uncertainty measure can be more advantageous.
Revised: Gal (2017) show that sampling based on a Bayesian uncertainty measure can be more advantageous.

Original: In one related paper, Zhang et al.
Revised: In one related paper, Zhang et al.

Original: (2017) investigate active learning for sentence classiﬁcation with
Revised: (2017) investigate active learning for sentence classiﬁcation with

Original: CNNs.
Revised: CNNs.

Original: However, to our knowledge, prior to this work, deep active learning for sequence tagging tasks, which often have structured output space and variable-length input, has not been studied.
Revised: However, to our knowledge, prior to this work, deep active learning for sequence tagging tasks, which often have structured output space and variable-length input, has not been studied.

Original:  NER M ODEL D ESCRIPTION
Revised:  NER M ODEL D ESCRIPTION

Original: Most active learning methods require frequent retraining of the model as new labeled examples are acquired.
Revised: Most active learning methods require frequent retraining of the model as new labeled examples are acquired.

Original: Therefore, it is crucial that the model can be efﬁciently retrained.
Revised: Therefore, it is crucial that the model can be efﬁciently retrained.

Original: On the other hand, we would still like to reach the level of performance rivaling state-of-the-art DNNs.
Revised: On the other hand, we would still like to reach the level of performance rivaling state-of-the-art DNNs.

Original: [Table]
Revised: [Table]

Original: To accomplish this, we ﬁrst identify that many DNN architectures for NER can be decomposed into three components: 1) the character-level encoder, which extracts features for each word from characters, 2) the word-level encoder which extracts features from the surrounding sequence of words, and 3) the tag decoder, which induces a probability distribution over any sequences of tags.
Revised: To accomplish this, we ﬁrst identify that many DNN architectures for NER can be decomposed into three components: 1) the character-level encoder, which extracts features for each word from characters, 2) the word-level encoder which extracts features from the surrounding sequence of words, and 3) the tag decoder, which induces a probability distribution over any sequences of tags.

Original: This conceptual framework allows us to view a variety of DNNs in a uniﬁed perspective; see Table 1.
Revised: This conceptual framework allows us to view a variety of DNNs in a uniﬁed perspective; see Table 1.

Original: Owing to the superior computational efﬁciency of CNNs over LSTMs, we propose a lightweight neural network architecture for NER, which we name CNN-CNN-LSTM and describe below.
Revised: Owing to the superior computational efﬁciency of CNNs over LSTMs, we propose a lightweight neural network architecture for NER, which we name CNN-CNN-LSTM and describe below.

Original: [Table]
Revised: [Table]

Original: Data Representation
Revised: Data Representation

Original: We represent each input sentence as follows; First, special [BOS] and [EOS] tokens are added at the beginning and the end of the sentence, respectively.
Revised: We represent each input sentence as follows; First, special [BOS] and [EOS] tokens are added at the beginning and the end of the sentence, respectively.

Original: In order to batch the computation of multiple sentences, sentences with similar length are grouped together into buckets, and [PAD] tokens are added at the end of sentences to make their lengths uniform inside of the bucket.
Revised: In order to batch the computation of multiple sentences, sentences with similar length are grouped together into buckets, and [PAD] tokens are added at the end of sentences to make their lengths uniform inside of the bucket.

Original: We follow an analogous procedure to represent the characters in each word.
Revised: We follow an analogous procedure to represent the characters in each word.

Original: For example, the sentence ‘Kate lives on Mars’ is formatted as shown in Table 2.
Revised: For example, the sentence ‘Kate lives on Mars’ is formatted as shown in Table 2.

Original: The formatted sentence is denoted as { x ij } , where x ij is the one-hot encoding of the j -th character in the i -th word.
Revised: The formatted sentence is denoted as { x ij } , where x ij is the one-hot encoding of the j -th character in the i -th word.

Original: Character-Level Encoder
Revised: Character-Level Encoder

Original: For each word i , we use CNNs (LeCun et al., 1995) to extract characterlevel features w char i (Figure 1).
Revised: For each word i , we use CNNs (LeCun et al., 1995) to extract characterlevel features w char i (Figure 1).

Original: While LSTM recurrent neural network (Hochreiter & Schmidhuber, 1997) slightly outperforms CNN as a character-level encoder, the improvement is not statistically signiﬁcant and the computational cost of LSTM encoders is much higher than CNNs (see Section 5, also Reimers & Gurevych (2017) for detailed analysis).
Revised: While LSTM recurrent neural network (Hochreiter & Schmidhuber, 1997) slightly outperforms CNN as a character-level encoder, the improvement is not statistically signiﬁcant and the computational cost of LSTM encoders is much higher than CNNs (see Section 5, also Reimers & Gurevych (2017) for detailed analysis).

Original: We apply ReLU nonlinearities (Nair & Hinton, 2010) and dropout (Srivastava et al., 2014) between
Revised: We apply ReLU nonlinearities (Nair & Hinton, 2010) and dropout (Srivastava et al., 2014) between

Original: CNN layers, and include a residual connection between input and output of each layer
Revised: CNN layers, and include a residual connection between input and output of each layer

Original: (He et al., 2016).
Revised: (He et al., 2016).

Original: So that our representation of the word is of ﬁxed length, we apply max-pooling on the outputs of the topmost layer of the character-level encoder (Kim, 2014).
Revised: So that our representation of the word is of ﬁxed length, we apply max-pooling on the outputs of the topmost layer of the character-level encoder (Kim, 2014).

Original: Word-Level Encoder
Revised: Word-Level Encoder

Original: To complete our representation of each word, we concatenate its characterlevel features with w emb i , a latent word embedding corresponding to that word:
Revised: To complete our representation of each word, we concatenate its characterlevel features with w emb i , a latent word embedding corresponding to that word:

Original: [Equation]
Revised: [Equation]

Original: We initialize the latent word embeddings with with word2vec training Mikolov et al.
Revised: We initialize the latent word embeddings with with word2vec training Mikolov et al.

Original: and then update the embeddings over the course of training.
Revised: and then update the embeddings over the course of training.

Original: In order to generalize to words unseen in the training data, we replace each word with a special [UNK] (unknown) token with 50% probability during training, an approach that resembles the word-drop method due to Lample et al.
Revised: In order to generalize to words unseen in the training data, we replace each word with a special [UNK] (unknown) token with 50% probability during training, an approach that resembles the word-drop method due to Lample et al.

Original: Given the sequence of word-level input features w full 1 , w full 2 , . . .
Revised: Given the sequence of word-level input features w full 1 , w full 2 , . . .

Original: , w full n , we extract word-level representations
Revised: , w full n , we extract word-level representations

Original: h Enc 1 , h Enc 2 , . .
Revised: h Enc 1 , h Enc 2 , . .

Original: , h Enc n for each word position in the sentence using a CNN.
Revised: , h Enc n for each word position in the sentence using a CNN.

Original: In Figure 2, w charh (2) h (2) h
Revised: In Figure 2, w charh (2) h (2) h

Original: (2) h (2) h
Revised: (2) h (2) h

Original: (2) h (2) h (2) h (1) h
Revised: (2) h (2) h (2) h (1) h

Original: (1) h (1) h (1) h (1) h
Revised: (1) h (1) h (1) h (1) h

Original: (1) h (1) xxxxxxx[BOW]
Revised: (1) h (1) xxxxxxx[BOW]

Original: K a t e [EOW] [PAD] max pooling we depict an instance of our architecture with two convolutional layers and kernels of width 3 .
Revised: K a t e [EOW] [PAD] max pooling we depict an instance of our architecture with two convolutional layers and kernels of width 3 .

Original: We concatenate the representation at the l -th convolutional layer h ( l ) i , with the input features w full i : h Enc i = (cid:16) h ( l ) i , w full i (cid:17)
Revised: We concatenate the representation at the l -th convolutional layer h ( l ) i , with the input features w full i : h Enc i = (cid:16) h ( l ) i , w full i (cid:17)

Original: h
Revised: h

Original: Ench Ench Ench Ench Ench Ench Ench (2) h (2) h
Revised: Ench Ench Ench Ench Ench Ench Ench (2) h (2) h

Original: (2) h (2) h
Revised: (2) h (2) h

Original: (2) h (2) h (2) h (1) h
Revised: (2) h (2) h (2) h (1) h

Original: (1) h (1) h
Revised: (1) h (1) h

Original: (1) h (1) h (1) h (1)
Revised: (1) h (1) h (1) h (1)

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: LSTM RNNs can also perform word-level encoding Huang et al.
Revised: LSTM RNNs can also perform word-level encoding Huang et al.

Original: (2015), and models with LSTM word-level encoding give a slight (but not signiﬁcant) boost over CNN word-level encoders in terms of F1 score (see Section 5).
Revised: (2015), and models with LSTM word-level encoding give a slight (but not signiﬁcant) boost over CNN word-level encoders in terms of F1 score (see Section 5).

Original: However, CNN word-level encoders are considerably faster (Strubell et al., 2017), which is crucial for the iterative retraining in our active learning scheme.
Revised: However, CNN word-level encoders are considerably faster (Strubell et al., 2017), which is crucial for the iterative retraining in our active learning scheme.

Original: Tag Decoder
Revised: Tag Decoder

Original: The tag decoder induces a probability distribution over sequences of tags, conditioned on the word-level encoder features: P (cid:2)
Revised: The tag decoder induces a probability distribution over sequences of tags, conditioned on the word-level encoder features: P (cid:2)

Original: y 2 , y 3 , . .
Revised: y 2 , y 3 , . .

Original: , y n − 1 | (cid:8)
Revised: , y n − 1 | (cid:8)

Original: h Enc i (cid:9)(cid:3) 1 .
Revised: h Enc i (cid:9)(cid:3) 1 .

Original: Chain CRF (Lafferty et al., 2001) is a popular choice for tag decoder, adopted by most modern DNNs for NER:
Revised: Chain CRF (Lafferty et al., 2001) is a popular choice for tag decoder, adopted by most modern DNNs for NER:

Original: [Equation]
Revised: [Equation]

Original: where W , A , b are learnable parameters, and {·} t i refers to the t i -th coordinate of the vector.
Revised: where W , A , b are learnable parameters, and {·} t i refers to the t i -th coordinate of the vector.

Original: To compute the partition function of (1), which is required for training, usually dynamic programming is employed, and its time complexity is O ( nT 2 ) where T is the number of entity types (Collobert et al., 2011).
Revised: To compute the partition function of (1), which is required for training, usually dynamic programming is employed, and its time complexity is O ( nT 2 ) where T is the number of entity types (Collobert et al., 2011).

Original: Alternatively, we use an LSTM RNN for the tag decoder, as depicted in Figure 3.
Revised: Alternatively, we use an LSTM RNN for the tag decoder, as depicted in Figure 3.

Original: At the ﬁrst time step, the [GO] -symbol is provided as y 1 to the decoder LSTM.
Revised: At the ﬁrst time step, the [GO] -symbol is provided as y 1 to the decoder LSTM.

Original: At each time step i , the LSTM decoder computes h
Revised: At each time step i , the LSTM decoder computes h

Original: Dec i +1 , the hidden state for decoding word i + 1 , using the last tag
Revised: Dec i +1 , the hidden state for decoding word i + 1 , using the last tag

Original: y i , the current decoder hidden state h
Revised: y i , the current decoder hidden state h

Original: Dec i , and the learned representation of next word
Revised: Dec i , and the learned representation of next word

Original: h Enc i +1 .
Revised: h Enc i +1 .

Original: Using a softmax loss function, y i +1 is decoded; this is further fed as an input to the next time step.
Revised: Using a softmax loss function, y i +1 is decoded; this is further fed as an input to the next time step.

Original: S-PER
Revised: S-PER

Original: O
Revised: O

Original: O
Revised: O

Original: S-LOC yyyyh Dech Dech
Revised: S-LOC yyyyh Dech Dech

Original: Dech Dech Decyyyy[GO]
Revised: Dech Dech Decyyyy[GO]

Original: S-PER
Revised: S-PER

Original: O
Revised: O

Original: O h
Revised: O h

Original: Ench Ench Ench Enc
Revised: Ench Ench Ench Enc

Original: Kate lives on
Revised: Kate lives on

Original: Mars
Revised: Mars

Original: Since this is a locally normalized model (Andor et al., 2016), it does not require the costly computation of partition function, and it allows us to signiﬁcantly speed up training compared to using CRFs.
Revised: Since this is a locally normalized model (Andor et al., 2016), it does not require the costly computation of partition function, and it allows us to signiﬁcantly speed up training compared to using CRFs.

Original: Also, we observed that while it is computationally intractable to ﬁnd the best sequence of tags with an LSTM decoder, greedily decoding tags from left to right yields the performance comparable to chain CRF decoder.
Revised: Also, we observed that while it is computationally intractable to ﬁnd the best sequence of tags with an LSTM decoder, greedily decoding tags from left to right yields the performance comparable to chain CRF decoder (see Appendix A).

Original: While the use of RNNs tag decoders has been explored (Mesnil et al., 2013; Nguyen et al., 2016; Zhai et al., 2017), we demonstrate for the ﬁrst time that models using RNNs instead of CRFs for tag decoder can achieve state-of-the-art performance.
Revised: While the use of RNNs tag decoders has been explored (Mesnil et al., 2013; Nguyen et al., 2016; Zhai et al., 2017), we demonstrate for the ﬁrst time that models using RNNs instead of CRFs for tag decoder can achieve state-of-the-art performance.

Original: See Section 5.
Revised: See Section 5.

Original:  A CTIVE L EARNING
Revised:  A CTIVE L EARNING

Original: Labeling data for NER usually requires manual annotations by human experts, which are costly to acquire at scale.
Revised: Labeling data for NER usually requires manual annotations by human experts, which are costly to acquire at scale.

Original: Active learning seeks to ameliorate this problem by strategically choosing which examples to annotate, in the hope of getting greater performance with fewer annotations.
Revised: Active learning seeks to ameliorate this problem by strategically choosing which examples to annotate, in the hope of getting greater performance with fewer annotations.

Original: To this end, we consider the following setup for interactively acquiring annotations.
Revised: To this end, we consider the following setup for interactively acquiring annotations.

Original: The learning process consists of multiple rounds: At the beginning of each round, the active learning algorithm chooses sentences to be annotated up to the predeﬁned budget.
Revised: The learning process consists of multiple rounds: At the beginning of each round, the active learning algorithm chooses sentences to be annotated up to the predeﬁned budget.

Original: After receiving annotations, we update the model parameters by training on the augmented dataset, and proceeds to the next round.
Revised: After receiving annotations, we update the model parameters by training on the augmented dataset, and proceeds to the next round.

Original: We assume that the cost of annotating a sentence is proportional to the number of words in the sentence and that every word in the selected sentence must be annotated at once, i.e. we do not allow or account for partially annotated sentences.
Revised: We assume that the cost of annotating a sentence is proportional to the number of words in the sentence and that every word in the selected sentence must be annotated at once, i.e. we do not allow or account for partially annotated sentences.

Original: While various existing active learning strategies suit this setup (Settles, 2010), we explore the uncertainty sampling strategy.
Revised: While various existing active learning strategies suit this setup (Settles, 2010), we explore the uncertainty sampling strategy.

Original: With the uncertainty-based sampling strategy (Lewis & Gale, 1994), we rank the unlabeled examples according to the current model’s uncertainty in its prediction of the corresponding labels.
Revised: With the uncertainty-based sampling strategy (Lewis & Gale, 1994), we rank the unlabeled examples according to the current model’s uncertainty in its prediction of the corresponding labels.

Original: We consider three ranking methods, each of which can be easily implemented in the CNN-CNN-LSTM model or most other deep neural approaches to NER.
Revised: We consider three ranking methods, each of which can be easily implemented in the CNN-CNN-LSTM model or most other deep neural approaches to NER.

Original: Least Conﬁdence (LC): Culotta & McCallum (2005) proposed to sort examples in ascending order according to the probability assigned by the model to the most likely sequence of tags:
Revised: Least Conﬁdence (LC): Culotta & McCallum (2005) proposed to sort examples in ascending order according to the probability assigned by the model to the most likely sequence of tags:

Original: [Equation]
Revised: [Equation]

Original: Exactly computing (2) requires identifying the most likely sequence of tags according to the LSTM decoder.
Revised: Exactly computing (2) requires identifying the most likely sequence of tags according to the LSTM decoder.

Original: Because determining the most likely sequence is intractable, we approximate the score by using the probability assigned to the greedily decoded sequence.
Revised: Because determining the most likely sequence is intractable, we approximate the score by using the probability assigned to the greedily decoded sequence.

Original: Maximum Normalized Log-Probability (MNLP): Preliminary analysis revealed that the LC method disproportionately selects longer sentences.
Revised: Maximum Normalized Log-Probability (MNLP): Preliminary analysis revealed that the LC method disproportionately selects longer sentences.

Original: Note that sorting unlabeled examples in descending order by (2) is equivalent to sorting in ascending order by the following scores:
Revised: Note that sorting unlabeled examples in descending order by (2) is equivalent to sorting in ascending order by the following scores:

Original: [Equation]
Revised: [Equation]

Original: Since (3) contains summation over words, LC method naturally favors longer sentences.
Revised: Since (3) contains summation over words, LC method naturally favors longer sentences.

Original: Because longer sentences requires more labor for annotation, we ﬁnd this undesirable, and propose to normalize (3) as follows, which we call Maximum Normalized Log-Probability method:
Revised: Because longer sentences requires more labor for annotation, we ﬁnd this undesirable, and propose to normalize (3) as follows, which we call Maximum Normalized Log-Probability method:

Original: [Equation]
Revised: [Equation]

Original: Bayesian Active Learning by Disagreement (BALD): We also consider sampling according to the measure of uncertainty proposed by Gal et al.
Revised: Bayesian Active Learning by Disagreement (BALD): We also consider sampling according to the measure of uncertainty proposed by Gal et al.

Original: Observing a correspondence between dropout (Srivastava et al., 2014) and deep Gaussian processes (Damianou & Lawrence, 2013), they propose that the variability of the predictions over successive forward passes due to dropout can be interpreted as a measure of the model’s uncertainty (Gal & Ghahramani, 2016).
Revised: Observing a correspondence between dropout (Srivastava et al., 2014) and deep Gaussian processes (Damianou & Lawrence, 2013), they propose that the variability of the predictions over successive forward passes due to dropout can be interpreted as a measure of the model’s uncertainty (Gal & Ghahramani, 2016).

Original: Denote P 1 , P 2 , . . .
Revised: Denote P 1 , P 2 , . . .

Original: P M as models resulting from applying M independently sampled dropout masks.
Revised: P M as models resulting from applying M independently sampled dropout masks.

Original: One measure of our uncertainty on the i th word is f i , the fraction of models which disagreed with the most popular choice:
Revised: One measure of our uncertainty on the i th word is f i , the fraction of models which disagreed with the most popular choice:

Original: [Equation]
Revised: [Equation]

Original: where |·| denotes cardinality of a set.
Revised: where |·| denotes cardinality of a set.

Original: We normalize this by the number of words as 1 n (cid:80) nj =1 f j , In this paper, we draw M = 100 independent dropout masks.
Revised: We normalize this by the number of words as 1 n (cid:80) nj =1 f j , In this paper, we draw M = 100 independent dropout masks.

Original: Other Sampling Strategies.
Revised: Other Sampling Strategies.

Original: Consider that the conﬁdence of the model can help to distinguish between hard and easy samples.
Revised: Consider that the conﬁdence of the model can help to distinguish between hard and easy samples.

Original: Thus, sampling examples where the model is uncertain might save us from sampling too heavily from regions where the model is already proﬁcient.
Revised: Thus, sampling examples where the model is uncertain might save us from sampling too heavily from regions where the model is already proﬁcient.

Original: But intuitively, when we query a batch of examples in each round, we might want to guard against querying examples that are too similar to each other, thus collecting redundant information.
Revised: But intuitively, when we query a batch of examples in each round, we might want to guard against querying examples that are too similar to each other, thus collecting redundant information.

Original: We also might worry that a purely uncertainty-based approach would oversample outliers.
Revised: We also might worry that a purely uncertainty-based approach would oversample outliers.

Original: Thus we explore techniques to guard against these problems by selecting a set of samples that is representative of the dataset.
Revised: Thus we explore techniques to guard against these problems by selecting a set of samples that is representative of the dataset.

Original: Following
Revised: Following

Original: Wei et al. (2015), we express the problem of maximizing representativeness of a labeled set as a submodular optimization problem, and provide an efﬁcient streaming algorithm adapted to use a constraint suitable to the NER task.
Revised: Wei et al. (2015), we express the problem of maximizing representativeness of a labeled set as a submodular optimization problem, and provide an efﬁcient streaming algorithm adapted to use a constraint suitable to the NER task.

Original: However, our experiments show that representativeness-based sampling does not help improve the active learning performance.
Revised: Our approach to representativeness-based sampling proceeds as follows: Denote X as the set of all samples, and X L , X U representing the set of labeled and unlabeled samples respectively. For an unlabeled set S ∈ X U , the utility f w is deﬁned as the summation of marginal utility gain over all unlabeled points, weighted by their uncertainty. More formally,

Original: The details of our explored methods for representativeness-based sampling can be found in the supplementary material (Appendix A).
Revised: [Equation] where US ( i ) is the uncertainty score on example i . In order to ﬁnd a good set S with high f w value, we exploit the submodularity of the function, and use an online algorithm under knapsack constraint. More details of this method can be found in the supplementary material (Appendix C).

Revised: In our experiments, this approach fails to match the uncertainty-based heuristics or to improve upon them when used in combination.

Original: For reference, the results are included in the Figures in the next section, named as SUBMOD .
Revised: Nevertheless, we describe the algorithm and include the negative results for their scientiﬁc value.

Original:  E XPERIMENTS 5. M ODEL EFFICIENCY AND PERFORMANCE
Revised:  E XPERIMENTS 5. M ODEL EFFICIENCY AND PERFORMANCE

Original: In order to evaluate the efﬁciency and performance CNN-CNN-LSTM as well as other alternatives, we run the experiments on two widely used NER datasets: CoNLL-2003 English (Tjong Kim Sang &
Revised: In order to evaluate the efﬁciency and performance CNN-CNN-LSTM as well as other alternatives, we run the experiments on two widely used NER datasets: CoNLL-2003 English (Tjong Kim Sang &

Original: De Meulder, 2003) and OntoNotes-5.0 English (Pradhan et al., 2013).
Revised: De Meulder, 2003) and OntoNotes-5.0 English (Pradhan et al., 2013).

Original: We use the standard split of training/validation/test sets, and use the validation set performance to determine hyperparameters such as the learning rate or the number of iterations for early stopping.
Revised: We use the standard split of training/validation/test sets, and use the validation set performance to determine hyperparameters such as the learning rate or the number of iterations for early stopping.

Original: Unlike Lample et al.
Revised: Unlike Lample et al.

Original: and Chiu & Nichols (2015), we do not train on the validation dataset.
Revised: and Chiu & Nichols (2015), we do not train on the validation dataset.

Original: We report the F1 score for each model, which is standard.
Revised: We report the F1 score for each model, which is standard.

Original: We only consider neural models in this comparison, since they outperform non-neural models for this task.
Revised: We only consider neural models in this comparison, since they outperform non-neural models for this task.

Original: Since our goal here is to compare neural network architectures, we did not experiment with gazetteers.
Revised: Since our goal here is to compare neural network architectures, we did not experiment with gazetteers.

Original: For neural architectures previously explored by others, we simply cite reported metrics.
Revised: For neural architectures previously explored by others, we simply cite reported metrics.

Original: For our own architectures, we usea single-layer LSTM with 100 hidden units for LSTM word-level encoder, single-layer LSTM with 25 hidden units for LSTM character-level encoder (following Lample et al.
Revised: For LSTM word-level encoder, we use single-layer model with 100 hidden units for CoNLL-2003 English (following Lample et al. and two-layer model with 300 hidden units for OntoNotes 5. datasets (following Chiu & Nichols (2015)). For character-level LSTM encoder, we use single-layer

Revised: LSTM with 25 hidden units (following Lample et al.

Original: (2016)), two-layer CNNs with 800 ﬁlters and kernel width 3 for CNN word-level encoder, and single-layer CNNs with 50 ﬁlters and kernel width 5 for CNN character-level encoder.
Revised: For CNN word-level encoder, we use two-layer CNNs with 800 ﬁlters and kernel width 5, and for CNN character-level encoder, we use single-layer CNNs with 50 ﬁlters and kernel width 3 (following Chiu & Nichols (2015)).

Original: Dropout probabilities are all set as 0.5.
Revised: Dropout probabilities are all set as 0.5.

Original: We use structured skip-gram model (Ling et al., 2015) trained on
Revised: We use structured skip-gram model (Ling et al., 2015) trained on

Original: Gigawords-English corpus (Graff & Cieri, 2003), which showed a good boost over vanilla skip-gram model (Mikolov et al., 2013)
Revised: Gigawords-English corpus (Graff & Cieri, 2003), which showed a good boost over vanilla skip-gram model (Mikolov et al., 2013)

Original: we do not report here.
Revised: we do not report here.

Original: We use vanilla stochastic gradient descent, since it is commonly reported in the named entity recognition literature that this outperforms more sophisticated methods at convergence (Lample et al., 2016; Chiu & Nichols, 2015).
Revised: We use vanilla stochastic gradient descent, since it is commonly reported in the named entity recognition literature that this outperforms more sophisticated methods at convergence (Lample et al., 2016; Chiu & Nichols, 2015).

Original: We uniformly set the step size as 0.001 and the batch size as 128.
Revised: We uniformly set the step size as 0.001 and the batch size as 128.

Original: When using LSTMs for the tag decoder, for inference, we only use greedy decoding; beam search gave very marginal improvement in our initial experiments.
Revised: When using LSTMs for the tag decoder, for inference, we only use greedy decoding; beam search gave very marginal improvement in our initial experiments.

Original: We repeat each experiment four times, and report mean and standard deviation.
Revised: We repeat each experiment four times, and report mean and standard deviation.

Original: In terms of measuring the training speed of our models, we compute the time spent for one iteration of training on the dataset, with eight K80 GPUs in p2.8xlarge on Amazon Web Services 2 .
Revised: In terms of measuring the training speed of our models, we compute the time spent for one iteration of training on the dataset, with eight K80 GPUs in p2.8xlarge on Amazon Web Services 2 .

Original: Table 3 and Table 4 show the comparison between our model and other best performing models.
Revised: Table 3 and Table 4 show the comparison between our model and other best performing models.

Original: LSTM tag decoder shows performance comparable to CRF tag decoder, and it works better than the CRF decoder when used with CNN encoder; compare CNN-CNN-LSTM vs. CNN-CNN-CRF on both tables.
Revised: LSTM tag decoder shows performance comparable to CRF tag decoder, and it works better than the CRF decoder when used with CNN encoder; compare CNN-CNN-LSTM vs. CNN-CNN-CRF on both tables.

Original: On the CoNLL-2003 English dataset which has only four entity types, the training speed of CNN-CNN-LSTM and CNN-CNN-CRF are comparable.
Revised: On the CoNLL-2003 English dataset which has only four entity types, the training speed of CNN-CNN-LSTM and CNN-CNN-CRF are comparable.

Original: However, on the OntoNotes 5.0 English dataset which has 18 entity types, the training speed of CNN-CNN-LSTM is twice faster than CNNCNN-CRF because the time complexity of computing the partition function for CRF is quadratic to the number of entity types.

Original: CNN-CNN-LSTM is also 44% faster than CNN-LSTM-LSTM on

Original: OntoNotes, showing the advantage of CNN over LSTM as word encoder; on CoNLL-2003, sentences tend to be shorter and this advantage was not clearly seen; its median number of words in sentences is 12 opposed 17 of OntoNotes.

Original: Therefore, the CNN-CNN-LSTM model is efﬁcient on both encoder and decoder side, and achieves comparatively high performance measured by F1 score.

Original: [Table] Reference F [Table]
Revised: However, on the OntoNotes 5.

Original: [Table] None CNN CRF 
Revised: English dataset which has 18 entity types, the training speed of CNN-CNN-LSTM is twice faster than CNN-CNN-CRF because the time complexity of computing the partition function for CRF is quadratic to the number of entity types.

Original: Collobert et al. None LSTM CRF
Revised: CNN-CNN-LSTM is also 44% faster than CNN-LSTMLSTM on OntoNotes, showing the advantage of CNN over LSTM as word encoder; on CoNLL-2003, sentences tend to be shorter and this advantage was not clearly seen; its median number of words in sentences is 12 opposed 17 of OntoNotes.

Original: Huang et al. LSTM LSTM CRF
Revised: Compared to the CNN-LSTM-CRF model, which is considered as a state-of-the-art model in terms of performance (Chiu & Nichols, 2015; Strubell et al., 2017), CNN-CNN-LSTM provides four times speedup in terms of the training speed, and achieves comparatively high performance measured by F1 score.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 5. P ERFORMANCE OF DEEP ACTIVE LEARNING
Revised: 5. P ERFORMANCE OF DEEP ACTIVE LEARNING

Original: We use OntoNotes-5.0 English and Chinese data (Pradhan et al., 2013) for our experiments.
Revised: We use OntoNotes-5.0 English and Chinese data (Pradhan et al., 2013) for our experiments.

Original: The training datasets contain 1,088,503 words and 756,063 words respectively.
Revised: The training datasets contain 1,088,503 words and 756,063 words respectively.

Original: State-of-the-art models trained on the full training sets achieve F1 scores of 86.86 Strubell et al.
Revised: State-of-the-art models trained on the full training sets achieve F1 scores of 86.86 Strubell et al.

Original: (2017) and 75.63 (our
Revised: (2017) and 75.63 (our

Original: CNN-CNN-LSTM) on the test sets. 
Revised: CNN-CNN-LSTM) on the test sets. Char

Revised: Word

Original: Comparisons of selection algorithms
Revised: [Table] Reference F [Table]

Revised: None

Original: We empirically compare selection algorithmsproposed in
Revised: [Table] [Figure] [Figure] Comparisons of selection algorithms

Original:  Section 4, as well as uniformly random baseline ( RAND ).
Revised: We empirically compare selection algorithms proposed in Section 4, as well as uniformly random baseline ( RAND ).

Original: All algorithms start with an identical 1% of original training data and a randomly initialized model.
Revised: All algorithms start with an identical 1% of original training data and a randomly initialized model.

Original: In each round, every algorithm chooses sentences from the rest of the training data until 20,000 words have been selected, adding this data to 
Revised: In each round, every algorithm chooses sentences from the rest of the training data until 20,000 words have been selected, adding this data to its training set.

Original: [Figure] its training set. We then update each model by stochastic gradient descent on its augmented training dataset for 50 passes.
Revised: We then update each model by stochastic gradient descent on its augmented training dataset forpasses.

Original: We evaluate the performance of algorithm by its F1 score on the test dataset.
Revised: We evaluate the performance of algorithm by its F1 score on the test dataset.

Original: [Figure]

Original: Figure 4 shows the results.
Revised: Figure 4 shows the results.

Original: All active learning algorithms perform signiﬁcantly better than the random baseline.
Revised: All active learning algorithms perform signiﬁcantly better than the random baseline.

Original: Among active learners,
Revised: Among active learners,

Original: MNLP  slightly outperformed others in early rounds.
Revised: MNLP and BALD slightly outperformed traditional LC in early rounds.

Revised: Note that MNLP is computationally more efﬁcient than BALD , since it only requires a single forward pass on the unlabeled dataset to compute uncertainty scores, whereas

Revised: BALD requires multiple forward passes.

Original: Impressively, active learning algorithms achieve 99% performance of the best deep model trained on full data using only 24.9% of the training data on the English dataset and 30.1% on Chinese.
Revised: Impressively, active learning algorithms achieve 99% performance of the best deep model trained on full data using only 24.9% of the training data on the English dataset and 30.1% on Chinese.

Original: Also, 12.0% and 16.9% of training data were enough for deep active learning algorithms to surpass the performance of the shallow models from Pradhan et al.
Revised: Also, 12.0% and 16.9% of training data were enough for deep active learning algorithms to surpass the performance of the shallow models from Pradhan et al.

Original: (2013) trained on the full training data.
Revised: (2013) trained on the full training data.

Revised: We repeated the experiment eight times and conﬁrmed that the trend is replicated across multiple runs; see Appendix B for details.

Original: Detection of under-explored genres
Revised: Detection of under-explored genres

Original: To better understand how active learning algorithms choose informative examples, we designed the following experiment.
Revised: To better understand how active learning algorithms choose informative examples, we designed the following experiment.

Original: The OntoNotes datasets consist of six genres: broadcast conversation (bc), braodcast news (bn), magazine genre (mz), newswire (nw), telephone conversation (tc), weblogs (wb).
Revised: The OntoNotes datasets consist of six genres: broadcast conversation (bc), braodcast news (bn), magazine genre (mz), newswire (nw), telephone conversation (tc), weblogs (wb).

Original: We created three training datasets: half-data , which contains random 50% of the original training data, nw-data , which contains sentences only from newswire (51.5% of words in the original data), and no-nw-data , which is the complement of nw-data .
Revised: We created three training datasets: half-data , which contains random 50% of the original training data, nw-data , which contains sentences only from newswire (51.5% of words in the original data), and no-nw-data , which is the complement of nwdata .

Original: Then, we trained CNN-CNN-LSTM model on each dataset.
Revised: Then, we trained CNN-CNN-LSTM model on each dataset.

Original: The model trained on half-data achieved 85.10 F1, signiﬁcantly outperforming others trained on biased datasets ( no-nw-data : 81.49, nw-only-data : 82.08).
Revised: The model trained on half-data achieved 85.10 F1, signiﬁcantly outperforming others trained on biased datasets ( no-nw-data : 81.49, nw-only-data : 82.08).

Original: This showed the importance of good genre coverage in training data.
Revised: This showed the importance of good genre coverage in training data.

Original: Then, we analyzed the genre distribution of 1, sentences MNLP chose for each model (see Figure 5).
Revised: Then, we analyzed the genre distribution of 1,000 sentences MNLP chose for each model (see Figure 5).

Original: For no-nw-data , the algorithm chose many more newswire (nw) sentences than it did for unbiased half-data (367 vs. 217).
Revised: For no-nw-data , the algorithm chose many more newswire (nw) sentences than it did for unbiased half-data (367 vs. 217).

Original: On the other hand, it undersampled newswire sentences for nw-only-data and increased the proportion of broadcast news and telephone conversation, which are genres distant from newswire.
Revised: On the other hand, it undersampled newswire sentences for nw-only-data and increased the proportion of broadcast news and telephone conversation, which are genres distant from newswire.

Original: Impressively, although we did not provide the genre of sentences to the algorithm, it was able to automatically detect underexplored genres.
Revised: Impressively, although we did not provide the genre of sentences to the algorithm, it was able to automatically detect underexplored genres.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed an efﬁcient model for NER tasks which gives high performances on well-established datasets.
Revised: We proposed an efﬁcient model for NER tasks which gives high performances on well-established datasets.

Original: Further, we use deep active learning algorithms for NER and empirically demonstrated that they achieve state-of-the-art performance with much less data than models trained in the standard supervised fashion. 
Revised: Further, we use deep active learning algorithms for NER and empirically demonstrated that they achieve state-of-the-art performance with much less data than models trained in the standard supervised fashion. A E FFECT OF B EAM S IZE ON LSTM D ECODER

Original: A R EPRESENTATIVENESS - BASED ACTIVE LEARNING
Revised: One potential concern when decoding with an LSTM decoder as compared to using a CRF decoder is that ﬁnding the best sequence of labels that maximizes the probability P (cid:2) t 2 , t 3 , . . . , t n − 1 | (cid:8) h Enc i (cid:9)(cid:3) is computationally intractable. In practice, however, we ﬁnd that simple greedy decoding, i.e., beam search with beam size 1, works surprisingly well.

Revised: Table 5 shows how changing the beam size of decoder affects the performance of the model.

Revised: It can be seen that the performance of the model changes very little with respect to the beam size.

Revised: Beam search with size 2 is marginally better than greedy decoding, and further increasing the beam size did not help at all.

Revised: Moreover, we note that while it may be computationally efﬁcient to pick the most likely tag sequence given a CRF encoder, the LSTM decoder may give more accurate predictions, owing to it’s greater representational power and ability to model long-range dependencies.

Revised: Thus even if we do not always choose the most probable tag sequence from the LSTM, we can still outperform the CRF (as our experiments demonstrate).

Revised: [Table]

Revised: B L EARNING C URVE IN A CTIVE L EARNING E XPERIMENTS ACROSS M ULTIPLE R UNS

Revised: In order to understand the variability of learning curves in Figure 4a across experiments, we repeated the active learning experiment on OntoNotes-5.0 English eight times, each of which started with different initial dataset chosen randomly.

Revised: Figure 6 shows the result in ﬁrst nine rounds of labeled data acquisition.

Revised: While MNLP, LC and BALD are all competitive against each other, there is a noticeable trend that MNLP and BALD outperforms LC in early rounds of data acquisition.

Revised: C R EPRESENTATIVENESS - BASED ACTIVE LEARNING

Original: Consider that the conﬁdence of the model can help to distinguish between hard and easy samples.
Revised: Consider that the conﬁdence of the model can help to distinguish between hard and easy samples.

Original: Thus, sampling examples where the model is uncertain might save us from sampling too heavily from regions where the model is already proﬁcient.
Revised: Thus, sampling examples where the model is uncertain might save us from sampling too heavily from regions where the model is already proﬁcient.

Original: But intuitively, when we query a batch of examples in each round, we might want to guard against querying examples that are too similar to each other, thus collecting redundant information.
Revised: But intuitively, when we query a batch of examples in each round, we might want to guard against querying examples that are too similar to each other, thus collecting redundant information.

Original: We also might worry that a purely uncertainty-based approach would oversample outliers.
Revised: We also might worry that a purely uncertainty-based approach would oversample outliers.

Original: Thus we explore techniques to guard against these problems by selecting a set of samples that is representative of the dataset.
Revised: Thus we explore techniques to guard against these problems by selecting a set of samples that is representative of the dataset.

Original: Following Wei et al.
Revised: Following Wei et al.

Original: , we express the problem of maximizing representativeness of a labeled set as a submodular optimization problem, and provide an efﬁcient streaming algorithm adapted to use a constraint suitable to the NER task.
Revised: , we express the problem of maximizing representativeness of a labeled set as a submodular optimization problem, and provide an efﬁcient streaming algorithm adapted to use a constraint suitable to the NER task.

Original: We also provide some with theoretical guarantees.
Revised: We also provide some with theoretical guarantees.

Original: Submodular utility function In order to reason about the similarity between samples, we ﬁrst embed each sample i into a ﬁxed-dimensional euclidean space as a vector x i .
Revised: Submodular utility function In order to reason about the similarity between samples, we ﬁrst embed each sample i into a ﬁxed-dimensional euclidean space as a vector x i .

Original: We consider two embedding methods: 1) the average of pre-trained word embeddings, 1 n (cid:80) nt =1 w emb t , which can be efﬁciently computed without training of the NER model, and 2) the average of activation maps at the topmost layer of the encoder 1 n (cid:80) [-n t =
-]
Revised: We consider two embedding methods: 1) the average of pre-trained word embeddings, 1 n (cid:80) nt =1 w emb t , which can be efﬁciently computed without training of the NER model, and 2) the average of activation maps at the topmost layer of the encoder 1 n (cid:80) [-n t =
-]nt

Original:  h Enc t , an embedding which might be better suited to the context of the NER task.
Revised: =1 h Enc t , an embedding which might be better suited to the context of the NER task.

Original: Then, we consider the following options for deﬁning similarity scores w ( i, j ) between each pair of samples i and j : w ( i, j ) =
Revised: Then, we consider the following options for deﬁning similarity scores w ( i, j ) between each pair of samples i and j : w ( i, j ) =

Original: d − (cid:107) x
Revised: d − (cid:107) x

Original: i − x j (cid:107) p where d = max i,j ∈ X (cid:107) x 
Revised: i − x j (cid:107) p where d = max i,j ∈ X (cid:107) x i − x j (cid:107) p round 70. t e s t

Revised: Fmethod

Original: i − x j (cid:107) p for p = 1 , 2 which corresponds to closeness in L 1 and L 2 distance (Wei et al., 2015), and w ( i, j ) = 1 +
Revised: RAND MNLP LC BALD for p = 1 , 2 which corresponds to closeness in L 1 and L 2 distance (Wei et al., 2015), and w ( i, j ) = 1 +

Original: [Equation]
Revised: [Equation]

Original: x j (cid:107) , which corresponds to cosine similarity.
Revised: , which corresponds to cosine similarity.

Original: Now, we formally deﬁne the utility function for labeling new samples.
Revised: Now, we formally deﬁne the utility function for labeling new samples.

Original: Denote X as the set of all samples which can be partitioned into two disjoint sets X L , X U representing labeled and unlabeled samples, respectively.
Revised: Denote X as the set of all samples which can be partitioned into two disjoint sets X L , X U representing labeled and unlabeled samples, respectively.

Original: Let S ⊆
Revised: Let S ⊆

Original: X U be a subset of unlabeled samples, then, the utility of labeling the set is deﬁned as follows:
Revised: X U be a subset of unlabeled samples, then, the utility of labeling the set is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where the function measures incremental gain of similarity between the labeled set and the rest.
Revised: where the function measures incremental gain of similarity between the labeled set and the rest.

Original: Given such utility function f ( · ) , choosing a set S that maximizes the function within the budget can be seen as a monotone submodular maximization problem under a knapsack constraint (Krause & Golovin, 2012):
Revised: Given such utility function f ( · ) , choosing a set S that maximizes the function within the budget can be seen as a monotone submodular maximization problem under a knapsack constraint (Krause & Golovin, 2012):

Original: [Equation]
Revised: [Equation]

Original:  where k ( S ) is the budget for the sample set S , and K is the total budget within each round.
Revised: (6) where k ( S ) is the budget for the sample set S , and K is the total budget within each round.

Original: Note that we need to consider the knapsack constraint instead of the cardinality constraint used in the prior work (Wei et al., 2015), because the entire sentence needs to be labeled once selected and sequences of length confer different labeling costs.
Revised: Note that we need to consider the knapsack constraint instead of the cardinality constraint used in the prior work (Wei et al., 2015), because the entire sentence needs to be labeled once selected and sequences of length confer different labeling costs.

Original: Combination with uncertainty sampling Representation-based sampling can beneﬁt from uncertainty-based sampling in the following two ways.
Revised: Combination with uncertainty sampling Representation-based sampling can beneﬁt from uncertainty-based sampling in the following two ways.

Original: First, we can re-weight each sample in the utility function (4) to reﬂect current model’s uncertainty on it:
Revised: First, we can re-weight each sample in the utility function (5) to reﬂect current model’s uncertainty on it:

Original: [Equation]
Revised: [Equation]

Original: where US ( i ) is the uncertainty score on example i .
Revised: where US ( i ) is the uncertainty score on example i .

Original: Second, even with the state-of-the-art submodular optimization algorithms, the optimization problem (5) can be computationally intractable.
Revised: Second, even with the state-of-the-art submodular optimization algorithms, the optimization problem (6) can be computationally intractable.

Original: To improve the computational efﬁciency, we restrict the set of unlabeled examples to top samples from uncertainty sampling within budget t · K , where t is a multiplication factor we set as 4 in our experiments.
Revised: To improve the computational efﬁciency, we restrict the set of unlabeled examples to top samples from uncertainty sampling within budget t · K , where t is a multiplication factor we set as 4 in our experiments.

Original: Streaming algorithm for sample selection Even with the reduction of candidates with uncertainty sampling, (5) is still a computationally challenging problem and requires careful design of optimization algorithms.
Revised: Streaming algorithm for sample selection Even with the reduction of candidates with uncertainty sampling, (6) is still a computationally challenging problem and requires careful design of optimization algorithms.

Original: Suppose l is the number of samples we need to consider.
Revised: Suppose l is the number of samples we need to consider.

Original: In the simplistic case in which all the samples have the same length and thus the knapsack constraint degenerates to the cardinality constraint, the greedy algorithm (Nemhauser et al., 1978) has an (1 − 1 /e ) -approximation guarantee.
Revised: In the simplistic case in which all the samples have the same length and thus the knapsack constraint degenerates to the cardinality constraint, the greedy algorithm (Nemhauser et al., 1978) has an (1 − 1 /e ) -approximation guarantee.

Original: However, it requires calculating the utility function O
Revised: However, it requires calculating the utility function O

Original: ( l 2 n ) times, where n is the number of unlabeled samples.
Revised: ( l 2 n ) times, where n is the number of unlabeled samples.

Original: In practice, both l and n are large.
Revised: In practice, both l and n are large.

Original: Alternatively, we can use lazy evaluation to decrease the computation complexity to O ( ln ) (Leskovec et al., 2007), but it requires an additional hyperparameter to be chosen in advance.
Revised: Alternatively, we can use lazy evaluation to decrease the computation complexity to O ( ln ) (Leskovec et al., 2007), but it requires an additional hyperparameter to be chosen in advance.

Original: Instead of greedily selecting elements in an ofﬂine fashion, we adopt the two-pass streaming algorithm of Badanidiyuru et al.
Revised: Instead of greedily selecting elements in an ofﬂine fashion, we adopt the two-pass streaming algorithm of Badanidiyuru et al.

Original: (2014), whose complexity is ˜
Revised: (2014), whose complexity is ˜

Original: O ( ln ) 3 , and generalize it to the knapsack constraint (shown in Alg. 2).
Revised: O ( ln ) 3 , and generalize it to the knapsack constraint (shown in Alg. 2).

Original: In the ﬁrst pass, we calculate the maximum function value of a single element normalized by its weight, which gives an estimate of the optimal value.
Revised: In the ﬁrst pass, we calculate the maximum function value of a single element normalized by its weight, which gives an estimate of the optimal value.

Original: In the second pass, we create O ( 1 (cid:15) log K ) buckets and greedily update each of the bucket according to:
Revised: In the second pass, we create O ( 1 (cid:15) log K ) buckets and greedily update each of the bucket according to:

Original: [Equation]
Revised: [Equation]

Original: where each bucket has a different value v , and ∆ g ( e | S v ) :
Revised: where each bucket has a different value v , and ∆ g ( e | S v ) :

Original: = g ( { e } ∪ S v ) − g ( S v ) is the marginal improvement of submodular function g when adding element e to set S v .
Revised: = g ( { e } ∪ S v ) − g ( S v ) is the marginal improvement of submodular function g when adding element e to set S v .

Original: The whole pipeline of the active learning algorithm is shown in Alg.
Revised: The whole pipeline of the active learning algorithm is shown in Alg.

Original: The algorithm gives the following guarantee, which is proven in Appendix.
Revised: The algorithm gives the following guarantee, which is proven in Appendix.

Original: Theorem 1.
Revised: Theorem 1.

Original: Alg.
Revised: Alg.

Original: 2 gives a (1 − (cid:15) )
Revised: 2 gives a (1 − (cid:15) )

Original: (1 − δ )-approximation guarantee for (5), where δ = max e ∈ S k
Revised: (1 − δ )-approximation guarantee for (6), where δ = max e ∈ S k

Original: ( { e } ) /K .
Revised: ( { e } ) /K .

Original: Proof sketch: The criterion (7) we use guarantees that each update we make is reasonably good.
Revised: Proof sketch: The criterion (8) we use guarantees that each update we make is reasonably good.

Original: The set S v stops updating when either the current budget is almost K , or any sample in the stream after we reach S v does not provide enough marginal improvement.
Revised: The set S v stops updating when either the current budget is almost K , or any sample in the stream after we reach S v does not provide enough marginal improvement.

Original: While it is easy to give guarantees when the budget is exhausted, it is unlikely to happen; we use a difference expression between current set S v and the optimal set, and prove the gap between the two is under control.
Revised: While it is easy to give guarantees when the budget is exhausted, it is unlikely to happen; we use a difference expression between current set S v and the optimal set, and prove the gap between the two is under control.

Original: In a practical label acquisition process, the budget we set for each round is usually much larger than the length of the longest sentence in the unlabeled set, making δ negligible.
Revised: In a practical label acquisition process, the budget we set for each round is usually much larger than the length of the longest sentence in the unlabeled set, making δ negligible.

Original: In our experiments, δ was around 0 .
Revised: In our experiments, δ was around 0 .