Original: W ORDS W ORTH S CORES FOR A TTACKING CNN S AND LSTM S FOR T EXT C LASSIFICATION
Revised: W ORDS W ORTH S CORES FOR A TTACKING CNN S AND LSTM S FOR T EXT C LASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Black box attacks on traditional deep learning models trained for text classiﬁcation target important words in a piece of text, in order to change model prediction.
Revised: Black box attacks on traditional deep learning models trained for text classiﬁcation target important words in a piece of text, in order to change model prediction.

Original: Current approaches towards highlighting important features are time consuming and require large number of model queries.
Revised: Current approaches towards highlighting important features are time consuming and require large number of model queries.

Original: We present a simple yet novel method to calculate word importance scores, based on model predictions on single words.
Revised: We present a simple yet novel method to calculate word importance scores, based on model predictions on single words.

Original: These scores, which we call WordsWorth scores, need to be calculated only once for the training vocabulary.
Revised: These scores, which we call WordsWorth scores, need to be calculated only once for the training vocabulary.

Original: They can be used to speed up any attack method that requires word importance, with negligible loss of attack performance.
Revised: They can be used to speed up any attack method that requires word importance, with negligible loss of attack performance.

Original: We run experiments on a number of datasets trained on word-level CNNs and LSTMs, for sentiment analysis and topic classiﬁcation and compare to state-of-the-art baselines.
Revised: We run experiments on a number of datasets trained on word-level CNNs and LSTMs, for sentiment analysis and topic classiﬁcation and compare to state-of-the-art baselines.

Original: Our results show the effectiveness of our method in attacking these models with success rates that are comparable to the original baselines.
Revised: Our results show the effectiveness of our method in attacking these models with success rates that are close to the original baselines.

Original: We argue that global importance scores act as a very good proxy for word importance in a local context because words are a highly informative form of data.
Revised: We argue that global importance scores act as a very good proxy for word importance in a local context because words are a highly informative form of data.

Original: This aligns with the manner in which humans interpret language, with individual words having welldeﬁned meaning and powerful connotations.
Revised: This aligns with the manner in which humans interpret language, with individual words having well-deﬁned meaning and powerful connotations.

Original: We further show that these scores can be used as a debugging tool to interpret a trained model by highlighting relevant words for each class.
Revised: We further show that these scores can be used as a debugging tool to interpret a trained model by highlighting relevant words for each class.

Original: Additionally, we demonstrate the effect of overtraining on word importance, compare the robustness of CNNs and LSTMs, and explain the transferability of adversarial examples across a CNN and an LSTM using these scores.
Revised: Additionally, we demonstrate the effect of overtraining on word importance, compare the robustness of CNNs and LSTMs, and explain the transferability of adversarial examples across a CNN and an LSTM using these scores.

Original: We highlight the fact that neural networks make highly informative predictions on single words.
Revised: We highlight the fact that neural networks make highly informative predictions on single words.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning models are vulnerable to carefully crafted adversarial examples.
Revised: Deep learning models are vulnerable to carefully crafted adversarial examples.

Original: The goal of such an attack is to fool a classiﬁer into giving incorrect prediction while the perturbed input appears normal to human observers.
Revised: The goal of such an attack is to fool a classiﬁer into giving incorrect prediction while the perturbed input appears normal to human observers.

Original: Thoroughly analyzing different kinds of vulnerabilities in neural networks would help us in creating robust models for deployment in the real world.
Revised: Thoroughly analyzing different kinds of vulnerabilities in neural networks would help us in creating robust models for deployment in the real world.

Original: In this work, we consider text classiﬁcation, where ﬁnding important words in a body of text is the ﬁrst step towards malicious modiﬁcation.
Revised: In this work, we consider text classiﬁcation, where ﬁnding important words in a body of text is the ﬁrst step towards malicious modiﬁcation.

Original: Forthisproblem, we propose a novel method for calculating word importance.
Revised: Forthisproblem, we propose a novel method for calculating word importance.

Original: After training a model, we calculate importance scores over the entire training vocabulary, word by word.
Revised: After training a model, we calculate importance scores over the entire training vocabulary, word by word.

Original: We further use these importance scores for black box attacks and demonstrate that the attack success rate is comparable to the original methods, particularly for CNNs.
Revised: We further use these importance scores for black box attacks and demonstrate that the attack success rate is comparable to the original methods, particularly for CNNs.

Original: Since these scores are global and calculated over the training vocabulary, they can also be used as a tool to interpret a trained model.
Revised: Since these scores are global and calculated over the training vocabulary, they can also be used as a tool to interpret a trained model.

Original: They provide a measure for comparing different architectures and models beyond training and validation accuracy.
Revised: They provide a measure for comparing different architectures and models beyond training and validation accuracy.

Original: Over a single training dataset, we can compare a small CNN to a large CNN, a CNN to an LSTM, or the word importance distribution of one class against another, as we outline in our experiments section.
Revised: Over a single training dataset, we can compare a small CNN to a large CNN, a CNN to an LSTM, or the word importance distribution of one class against another, as we outline in our experiments section.

Original: The motivation for our particular algorithm comes from the fact that in a piece of text,most of the time, words and phrases have a strong inﬂuence on their own.
Revised: The motivation for our particular algorithm comes from the fact that in a piece of text,most of the time, words and phrases have a strong inﬂuence on their own.

Original: This gives us a rationale for evaluating a model on single words, in direct contrast to the leave-one-out technique, which involves deleting a word from a document and measuring its importance by the change in model prediction on this modiﬁed input.
Revised: This gives us a rationale for evaluating a model on single words, in direct contrast to the leave-one-out technique, which involves deleting a word from a document and measuring its importance by the change in model prediction on this modiﬁed input.

Original: Further, we expect a well-trained network to treat a word approximately the same, irrespective of its location, when surrounding words are removed.
Revised: Further, we expect a well-trained network to treat a word approximately the same, irrespective of its location, when surrounding words are removed.

Original: Thus a particular word can occur at any position in a document with 200 words and its importance will be roughly the same.
Revised: Thus a particular word can occur at any position in a document with 200 words and its importance will be roughly the same.

Original: We expect a well-trained model to exhibit this behaviour and our experiments conﬁrm this.
Revised: We expect a well-trained model to exhibit this behaviour and our experiments conﬁrm this.

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We propose a simple and efﬁcient method for calculating word importance and evaluate it on traditional deep learning models for black box attacks.
Revised: • We propose a simple and efﬁcient method for calculating word importance and evaluate it on traditional deep learning models for black box attacks.

Original: • We argue that these scores can act as a tool for model interpretation and outline a number of use cases in this context.
Revised: • We argue that these scores can act as a tool for model interpretation and outline a number of use cases in this context.

Original:  R ELATED W ORK 2. A DVERSARIAL ATTACKS ON NLP MODELS :
Revised:  R ELATED W ORK 2. A DVERSARIAL ATTACKS ON NLP MODELS :

Original: The idea of perturbation, whether random or malicious, is rather simple in the image domain, where salt and pepper noise can be enough to images to fool models.
Revised: The idea of perturbation, whether random or malicious, is rather simple in the image domain, where salt and pepper noise can be enough to images to fool models.

Original: This kind of noise is hard for humans to detect.
Revised: This kind of noise is hard for humans to detect.

Original: However, since text data is discrete, perturbations in text are difﬁcult to quantify.
Revised: However, since text data is discrete, perturbations in text are difﬁcult to quantify.

Original: Besides, humans easily notice errors in computer-generated text.
Revised: Besides, humans easily notice errors in computer-generated text.

Original: This places additional constraints for an
Revised: This places additional constraints for an

Original: NLP attack to be counted as successful, where a successful attack is one that forces the model to give an incorrect prediction while a human would make the correct prediction on the input.
Revised: NLP attack to be counted as successful, where a successful attack is one that forces the model to give an incorrect prediction while a human would make the correct prediction on the input.

Original: We limit ourselves to text classiﬁcation problems, using sentiment analysis and topic classiﬁcation as examples.
Revised: We limit ourselves to text classiﬁcation problems, using sentiment analysis and topic classiﬁcation as examples.

Original: We only consider the attack scenarios in which speciﬁc words in the input are replaced by valid words from the dictionary.
Revised: We only consider the attack scenarios in which speciﬁc words in the input are replaced by valid words from the dictionary.

Original: Thus we are not considering attacks in which extra information is appended to input data, or where word replacements purposefully introduce spelling errors.
Revised: Thus we are not considering attacks in which extra information is appended to input data, or where word replacements purposefully introduce spelling errors.

Original: The former take an entirely different approach; the latter introduce errors and do not preserve semantics and training a neural network to be robust to spelling errors would stop these attacks.
Revised: The former take an entirely different approach; the latter introduce errors and do not preserve semantics and training a neural network to be robust to spelling errors would stop these attacks.

Original: Further, we limit ourselves to black box attacks where the attacker has no information about model architectures and parameters.
Revised: Further, we limit ourselves to black box attacks where the attacker has no information about model architectures and parameters.

Original: 2. F IND AND REPLACE ATTACKS ON TEXT CLASSIFICATION
Revised: 2. F IND AND REPLACE ATTACKS ON TEXT CLASSIFICATION

Original: Most attacks on text classiﬁcation solve the problem in two parts; by locating important words in the input, and by ﬁnding suitable replacements for these words.
Revised: Most attacks on text classiﬁcation solve the problem in two parts; by locating important words in the input, and by ﬁnding suitable replacements for these words.

Original: We only consider attacks where substitutions are valid words picked from a dictionary, so as to avoid introducing grammatical errors.
Revised: We only consider attacks where substitutions are valid words picked from a dictionary, so as to avoid introducing grammatical errors.

Original: 2.2. W HITE BOX ATTACKS
Revised: 2.2. W HITE BOX ATTACKS

Original: In the white-box setting, where an attacker has full knowledge of the model architecture, gradients serve as a good proxy for word importance.
Revised: In the white-box setting, where an attacker has full knowledge of the model architecture, gradients serve as a good proxy for word importance.

Original: Gong et al.
Revised: Gong et al.

Original: (2018) use gradient based methods to locate important words.
Revised: (2018) use gradient based methods to locate important words.

Original: Samanta & Mehta (2017) use gradients to calculate word importance, with linguistic constraints over substitution words.
Revised: Samanta & Mehta (2017) use gradients to calculate word importance, with linguistic constraints over substitution words.

Original: Lei et al.
Revised: Lei et al.

Original: (2019) carry joint word and sentence attacks, by generating sentence paraphrases in the ﬁrst stage, and resorting to greedy word substitutions if the ﬁrst stage fails.
Revised: (2019) carry joint word and sentence attacks, by generating sentence paraphrases in the ﬁrst stage, and resorting to greedy word substitutions if the ﬁrst stage fails.

Original: Again, important words are located by the magnitude of the gradient of word embedding.
Revised: Again, important words are located by the magnitude of the gradient of word embedding.

Original: 2.2. B LACK BOX ATTACKS
Revised: 2.2. B LACK BOX ATTACKS

Original: In the black box scenario, where gradients are not available, saliency maps are calculated for words through different methods.
Revised: In the black box scenario, where gradients are not available, saliency maps are calculated for words through different methods.

Original: Yang et al.
Revised: Yang et al.

Original: (2018) provide a greedy algorithm which we will outline in detail in the next section.
Revised: (2018) provide a greedy algorithm which we will outline in detail in the next section.

Original: Li et al.
Revised: Li et al.

Original: (2016) propose masking each feature with zero padding, using the decrease in the predicted probability as the score of the feature or word, and masking the top-k features as unknown.
Revised: (2016) propose masking each feature with zero padding, using the decrease in the predicted probability as the score of the feature or word, and masking the top-k features as unknown.

Original: Alzantot et al.
Revised: Alzantot et al.

Original: (2018) and Kuleshov et al.
Revised: (2018) and Kuleshov et al.

Original: (2018) propose variations of genetic algorithms.
Revised: (2018) propose variations of genetic algorithms.

Original: Kuleshov et al.
Revised: Kuleshov et al.

Original: (2018) replace words one by one until the classiﬁer is misdirected while observing a bound on the number of perturbed features.
Revised: (2018) replace words one by one until the classiﬁer is misdirected while observing a bound on the number of perturbed features.

Original: They run each new iteration on the modiﬁed input.
Revised: They run each new iteration on the modiﬁed input.

Original: For substitution, they used post processed GloVe to ﬁnd pool of suitable words.
Revised: For substitution, they used post processed GloVe to ﬁnd pool of suitable words.

Original: They also compute ’thought vectors’ for sentences and ensure that these are preserved.
Revised: They also compute ’thought vectors’ for sentences and ensure that these are preserved.

Original: Alzantot et al.
Revised: Alzantot et al.

Original: (2018) select words by random sampling, where probability of each word being selected is proportional to the number of suitable neighbours for replacement.
Revised: (2018) select words by random sampling, where probability of each word being selected is proportional to the number of suitable neighbours for replacement.

Original: They use Google 1 billion words language model to ensure that replacements match the context provided by the rest of the input.
Revised: They use Google 1 billion words language model to ensure that replacements match the context provided by the rest of the input.

Original: Ren et al.
Revised: Ren et al.

Original: (2019) propose a saliency-based greedy algorithm, calculated by deleting words during the search phase and select substitutions from WordNet.
Revised: (2019) propose a saliency-based greedy algorithm, calculated by deleting words during the search phase and select substitutions from WordNet.

Original: Another similar attack model is Jin et al.
Revised: Another similar attack model is Jin et al.

Original: (2019), which has extra semantic similarity checking when searching adversarial examples, and calculates word importance by deleting words.
Revised: (2019), which has extra semantic similarity checking when searching adversarial examples, and calculates word importance by deleting words.

Original: Zang et al.
Revised: Zang et al.

Original: (2019) propose a particle swarm optimization algorithm for the search problem.
Revised: (2019) propose a particle swarm optimization algorithm for the search problem.

Original: Gao et al.
Revised: Gao et al.

Original: (2018) deﬁne different scoring functions where they look at prediction before and after removing a particular word from a subset of input, and perform character level modiﬁcations in the second stage.
Revised: (2018) deﬁne different scoring functions where they look at prediction before and after removing a particular word from a subset of input, and perform character level modiﬁcations in the second stage.

Original: Li et al.
Revised: Li et al.

Original: (2019) use the sentence probability directly but once again, when ranking words, they try masking words in a sentence.
Revised: (2019) use the sentence probability directly but once again, when ranking words, they try masking words in a sentence.

Original: A common thread among all search methods for black box attacks is erasure or omission, where the effect of a word is observed by comparing the classiﬁer output probability for original input to that for input with this particular word removed or replaced by zero.
Revised: A common thread among all search methods for black box attacks is erasure or omission, where the effect of a word is observed by comparing the classiﬁer output probability for original input to that for input with this particular word removed or replaced by zero.

Original: 2. I NTERPRETABILITY IN MACHINE LEARNING THROUGH ERASURE
Revised: 2. I NTERPRETABILITY IN MACHINE LEARNING THROUGH ERASURE

Original: Li et al.
Revised: Li et al.

Original: (2016) is a pioneering body of work in the domain of interpretability that highlights the importance of interpreting networks by erasing parts of various layers.
Revised: (2016) is a pioneering body of work in the domain of interpretability that highlights the importance of interpreting networks by erasing parts of various layers.

Original: This Leave-One-Out method is followed by most interpretation algorithms.
Revised: This Leave-One-Out method is followed by most interpretation algorithms.

Original: For a particular word, they calculate importance score as the average of prediction difference due to erasing this word from all test examples.
Revised: For a particular word, they calculate importance score as the average of prediction difference due to erasing this word from all test examples.

Original: Feng et al.
Revised: Feng et al.

Original: (2018) gradually remove unimportant input words so that only the important ones are left at the end.
Revised: (2018) gradually remove unimportant input words so that only the important ones are left at the end.

Original: Barham & Feizi (2019) propose sparse projected gradient descent to generate adversarial examples to improve interpretability.
Revised: Barham & Feizi (2019) propose sparse projected gradient descent to generate adversarial examples to improve interpretability.

Original: Nguyen (2018) looks at different methods of local explanations for labels, which include LIME, random feature deletion and ﬁrst derivative saliency.
Revised: Nguyen (2018) looks at different methods of local explanations for labels, which include LIME, random feature deletion and ﬁrst derivative saliency.

Original: K´ad´ar et al.
Revised: K´ad´ar et al.

Original: (2017) measure salience of a word by removing it and noting the change in prediction.
Revised: (2017) measure salience of a word by removing it and noting the change in prediction.

Original: Jin et al.
Revised: Jin et al.

Original: (2019) mention deleting a particular word to calculate its importance score.
Revised: (2019) mention deleting a particular word to calculate its importance score.

Original: Ren et al.
Revised: Ren et al.

Original: (2019) use word saliency which is the change in the classiﬁer output if a word is set to unknown.
Revised: (2019) use word saliency which is the change in the classiﬁer output if a word is set to unknown.

Original: Carter et al.
Revised: Carter et al.

Original: ﬁnd sufﬁcient input subsets while calculating the feature importance by masking words.
Revised: ﬁnd sufﬁcient input subsets while calculating the feature importance by masking words.

Original: For calculating word score matrices,Xu & Du (2020) propose a method which involves masking words.
Revised: For calculating word score matrices,Xu & Du (2020) propose a method which involves masking words.

Original: We want to highlight the aspect that all the dominant techniques for interpretation use leave-oneout method for calculating word importance.
Revised: We want to highlight the aspect that all the dominant techniques for interpretation use leave-oneout method for calculating word importance.

Original: Since WordsWorth scores provide a reliable way of calculating feature importance, it is possible to directly use them to interpret a model after it has been trained.
Revised: Since WordsWorth scores provide a reliable way of calculating feature importance, it is possible to directly use them to interpret a model after it has been trained.

Original:  G REEDY ALGORITHM FOR BLACK BOX ATTACKS
Revised:  G REEDY ALGORITHM FOR BLACK BOX ATTACKS

Original: The greedy algorithm mentioned in Yang et al.
Revised: The greedy algorithm mentioned in Yang et al.

Original: (2018) consists of two steps: ﬁnding the most important words in a text, and ﬁnding the most distracting replacements for these words, with some constraint.
Revised: (2018) consists of two steps: ﬁnding the most important words in a text, and ﬁnding the most distracting replacements for these words, with some constraint.

Original: For an attack where k features are allowed to be perturbed, the top k important words are picked ﬁrst, and then replaced one by one.
Revised: For an attack where k features are allowed to be perturbed, the top k important words are picked ﬁrst, and then replaced one by one.

Original: In the ﬁrst step, greedy ﬁnds the most important words by calculating importance scores for each word in the input as the difference in prediction probability for original input and for input with the word removed.
Revised: In the ﬁrst step, greedy ﬁnds the most important words by calculating importance scores for each word in the input as the difference in prediction probability for original input and for input with the word removed.

Original: The second step of the algorithm includes ﬁnding suitable replacement for these words.
Revised: The second step of the algorithm includes ﬁnding suitable replacement for these words.

Original: Throughout this paper we will use their greedy algorithm as a baseline for comparison, since it achieves the highest success rate among all black box methods (Hsieh et al., 2019).
Revised: Throughout this paper we will use their greedy algorithm as a baseline for comparison, since it achieves the highest success rate among all black box methods (Hsieh et al., 2019).

Original: Greedy uses the pretrained GloVe embeddings and limits the search in second step to within a prespeciﬁed distance, to preserve semantics.
Revised: Greedy uses the pretrained GloVe embeddings and limits the search in second step to within a prespeciﬁed distance, to preserve semantics.

Original: However, it should be noted that GloVe embeddings do not always provide semantic preserving replacements, and a post-processed form of embeddings would work better, such as the ones used by Kuleshov et al.
Revised: However, it should be noted that GloVe embeddings do not always provide semantic preserving replacements, and a post-processed form of embeddings would work better, such as the ones used by Kuleshov et al.

Original: In our experiments, we use 50-dimensional GloVe embeddings to ﬁnd replacements for important words.
Revised: In our experiments, we use 50-dimensional GloVe embeddings to ﬁnd replacements for important words.

Original: We limit our search to the ten nearest neighbours for each word to preserve semantics.
Revised: We limit our search to the ten nearest neighbours for each word to preserve semantics.

Original:  W ORDS W ORTH SCORES FOR FEATURE IMPORTANCE
Revised:  W ORDS W ORTH SCORES FOR FEATURE IMPORTANCE

Original: For determining importance of individual words in a text document, we propose WordsWorth scores, which are simply the prediction scores of each individual word in the vocabulary, from the trained classiﬁer.
Revised: For determining importance of individual words in a text document, we propose WordsWorth scores, which are simply the prediction scores of each individual word in the vocabulary, from the trained classiﬁer.

Original: Since the CNNs and LSTms on which we experiment have a ﬁxed input size(limited to 200 words throughout the experiments), for calculating these scores, the integer representation of the word (from the tokenizer) is appended with zeros and fed to the classiﬁer.
Revised: Since the CNNs and LSTms on which we experiment have a ﬁxed input size(limited to 200 words throughout the experiments), for calculating these scores, the integer representation of the word (from the tokenizer) is appended with zeros and fed to the classiﬁer.

Original: This is equivalent to evaluating the classiﬁer on a piece of text where the text consists of a single word.
Revised: This is equivalent to evaluating the classiﬁer on a piece of text where the text consists of a single word.

Original: The algorithm for greedy attack using WW scores is given below.
Revised: The algorithm for greedy attack using WW scores is given below.

Original: Algorithm
Revised: Algorithm

Original: Input F , a trained CNN or LSTM
Revised: Input F , a trained CNN or LSTM

Original: Input p the number of classes in data
Revised: Input p the number of classes in data

Original: Input d the size of classiﬁer input
Revised: Input d the size of classiﬁer input

Original: Input V ∈ R d , the training vocabulary having m words
Revised: Input V ∈ R d , the training vocabulary having m words

Original: Output WW ∈ R d ∗ p , Wordsworth scores over the training vocabulary 1: for w ∈ V do 2: deﬁne x = 0 0 , 0 1 , 0 2 , ... 

Revised: Output WW ∈ R d ∗ p , Wordsworth scores over the training vocabulary 1: for w ∈ V do 2: deﬁne x = 0 0 , 0 1 , 0 2 , ...

Original: d − 2 , w 3:
Revised: d − 2 , w 3:

Original: WW ( w ) =
Revised: WW ( w ) =

Original: F ( x ) 4: end for
Revised: F ( x ) 4: end for

Original: Algorithm 2 Step 2: Greedily replace top k words to maximize incorrect class probability
Revised: Algorithm 2 Step 2: Greedily replace top k words to maximize incorrect class probability

Original: Input F , a trained CNN or LSTM
Revised: Input F , a trained CNN or LSTM

Original: Input X ∈ R d , text input to be modiﬁed
Revised: Input X ∈ R d , text input to be modiﬁed

Original: Input k the maximum number of features to be perturbed
Revised: Input k the maximum number of features to be perturbed

Original: Input D ∈ R m ∗ 10 the nearest neighbour dictionary with 10 neighbours for each training vocab word
Revised: Input D ∈ R m ∗ 10 the nearest neighbour dictionary with 10 neighbours for each training vocab word

Original: Input WW ∈ R d ∗ p , Wordsworth scores over the training vocabulary
Revised: Input WW ∈ R d ∗ p , Wordsworth scores over the training vocabulary

Original: Output X (cid:48) , the maliciously modiﬁed input with maximum k words modiﬁed 1: 2: Pick i 1 , i 2 , .., i k
Revised: Output X (cid:48) , the maliciously modiﬁed input with maximum k words modiﬁed 1: 2: Pick i 1 , i 2 , .., i k

Original: such that WW ( X i 1 ) >
Revised: such that WW ( X i 1 ) >

Original: = WW ( X i 2 ) ..
Revised: = WW ( X i 2 ) ..

Original: >
Revised: >

Original: =
Revised: =

Original: WW ( X i k ) 3: Initialize X (cid:48)
Revised: WW ( X i k ) 3: Initialize X (cid:48)

Original: =
Revised: =

Original: X 4: for j ∈ i 1 , i 2 , .., i k do 5: w j = X (cid:48)
Revised: X 4: for j ∈ i 1 , i 2 , .., i k do 5: w j = X (cid:48)

Original: j , D j = 10 nearest neighbour of w j 6: for w ∈ D
Revised: j , D j = 10 nearest neighbour of w j 6: for w ∈ D

Original: j do 7:
Revised: j do 7:

Original: X (cid:48)
Revised: X (cid:48)

Original: w = (cid:26) X (cid:48)
Revised: w = (cid:26) X (cid:48)

Original: j if j (cid:54) =
Revised: j if j (cid:54) =

Original: i w if j = i 8: end for 9:
Revised: i w if j = i 8: end for 9:

Original: X (cid:48)
Revised: X (cid:48)

Original: =
Revised: =

Original: arg X (cid:48)
Revised: arg X (cid:48)

Original: w max |F
Revised: w max |F

Original: ( X (cid:48) w ) − F ( X (cid:48) )
Revised: ( X (cid:48) w ) − F ( X (cid:48) )

Original: | 10: end for
Revised: | 10: end for

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Comparison with two other blackbox attacks: Here we present the performance of del one (Li et al., 2016) and greedy (Yang et al., 2018), along with their modiﬁed versions, where word importance has been computed through WordsWorth scores for the modiﬁed versions.
Revised: Comparison with two other blackbox attacks: Here we present the performance of del one (Li et al., 2016) and greedy (Yang et al., 2018), along with their modiﬁed versions, where word importance has been computed through WordsWorth scores for the modiﬁed versions.

Original: We call the modiﬁed versions as del one ww and greedy ww respectively.
Revised: We call the modiﬁed versions as del one ww and greedy ww respectively.

Original: We also show the AUC score for original data, named as original to serve as a baseline.
Revised: We also show the AUC score for original data, named as original to serve as a baseline.

Original: 5. S ENTIMENT A NALYSIS : IMDB REVIEWS 5.1. D ATASET AND MODEL ARCHITECTURE
Revised: 5. S ENTIMENT A NALYSIS : IMDB REVIEWS 5.1. D ATASET AND MODEL ARCHITECTURE

Original: We use the IMDB dataset (Maas et al., 2011), which consists of 25000 training reviews andtest reviews of variable length.
Revised: We use the IMDB dataset (Maas et al., 2011), which consists of 25000 training reviews andtest reviews of variable length.

Original: Each review in the training set has a positive/negative label attached to it.
Revised: Each review in the training set has a positive/negative label attached to it.

Original: Training vocabulary size is 5000 and we cut each review to 200 words max.
Revised: Training vocabulary size is 5000 and we cut each review to 200 words max.

Original: We use a simple
Revised: We use a simple

Original: CNN as the starting point of our experiments, with 32 dimensional embedding layer, 50 ﬁlters and 100 units in a dense layer.
Revised: CNN as the starting point of our experiments, with 32 dimensional embedding layer, 50 ﬁlters and 100 units in a dense layer.

Original: The input layer uses word2vec embeddings that are learned during training.
Revised: The input layer uses word2vec embeddings that are learned during training.

Original: ReLU activation is used.
Revised: ReLU activation is used.

Original: The network is trained on 25000 training examples.
Revised: The network is trained on 25000 training examples.

Original: We use the
Revised: We use the

Original: Adam optmizer with default learning rate of 0.001 and use early stopping.
Revised: Adam optmizer with default learning rate of 0.001 and use early stopping.

Original: Test accuracy is 88.78%.
Revised: Test accuracy is 88.78%.

Original: We pick 300 examples at random from the test dataset and plot the ROC AUC values versus number of features perturbed for different algorithms.
Revised: We pick 300 examples at random from the test dataset and plot the ROC AUC values versus number of features perturbed for different algorithms.

Original: The results for CNN in ﬁgure 1(left) show that the modiﬁed versions of both algorithms have a performance that is comparable to that of the original versions.
Revised: The results for CNN in ﬁgure 1(left) show that the modiﬁed versions of both algorithms have a performance that is comparable to that of the original versions.

Original: Notice that the distance between greedy and greedy ww is larger than that between del one and del one ww.
Revised: Notice that the distance between greedy and greedy ww is larger than that between del one and del one ww.

Original: This implies that if simply deleting words is the strategy, WordsWorth scores are almost as effective as manually deleting each word one by one and ﬁnding the one that contributes most to the model prediction.
Revised: This implies that if simply deleting words is the strategy, WordsWorth scores are almost as effective as manually deleting each word one by one and ﬁnding the one that contributes most to the model prediction.

Original: 5.1. R UNTIME COMPARISON WITH BASELINES
Revised: 5.1. R UNTIME COMPARISON WITH BASELINES

Original: Notice that WordsWorth scores can be calculated once over the vocabulary learnt during training once the classiﬁer has been trained.
Revised: Notice that WordsWorth scores can be calculated once over the vocabulary learnt during training once the classiﬁer has been trained.

Original: At test time, model evaluation can be replaced by a simple lookup.
Revised: At test time, model evaluation can be replaced by a simple lookup.

Original: Thus, for a 5000 vocabulary size, WW score calculation takes 5000 model prediction.
Revised: Thus, for a 5000 vocabulary size, WW score calculation takes 5000 model prediction.

Original: On the other hand, with 200 word reviews on average, the original baselines(greedy as well as del one) need 5000 evaluations to locate important words just for 25 text examples.
Revised: On the other hand, with 200 word reviews on average, the original baselines(greedy as well as del one) need 5000 evaluations to locate important words just for 25 text examples.

Original: If attacks are carried out in bulk, WW evaluations are essentially free after the ﬁrst 25 reviews.
Revised: If attacks are carried out in bulk, WW evaluations are essentially free after the ﬁrst 25 reviews.

Original: This indicates a considerable slashing of computation time and resources.
Revised: This indicates a considerable slashing of computation time and resources.

Original: Additionally, notice that WW score computations use a sparse input, which could be more suitable for some platforms as compared to the full input.
Revised: Additionally, notice that WW score computations use a sparse input, which could be more suitable for some platforms as compared to the full input.

Original: 5.1. D O GREEDY AND GREEDY WW FIND THE SAME WORDS TO BE IMPORTANT ?
Revised: 5.1. D O GREEDY AND GREEDY WW FIND THE SAME WORDS TO BE IMPORTANT ?

Original: During this experiment, when we compared the top ten words found by greedy and greedy ww for each test example, 7.3 words were same on average.
Revised: During this experiment, when we compared the top ten words found by greedy and greedy ww for each test example, 7.3 words were same on average.

Original: When we looked at the top 5 words, 3.5 were same on average.
Revised: When we looked at the top 5 words, 3.5 were same on average.

Original: This strengthens the idea that both algorithms choose quite similar set of words for each instance.
Revised: This strengthens the idea that both algorithms choose quite similar set of words for each instance.

Original: 5.1. LSTM
Revised: 5.1. LSTM

Original: We repeat the experiment on an LSTM with 100 examples and report the results in ﬁgure 1(right), where similar trends can be observed, with del one ww performing close to del one and greedy ww performing close to greedy.
Revised: We repeat the experiment on an LSTM with 100 examples and report the results in ﬁgure 1(right), where similar trends can be observed, with del one ww performing close to del one and greedy ww performing close to greedy.

Original: However, the difference here is larger as compared to CNN, which could be due to the LSTM learning a more robust representation.
Revised: However, the difference here is larger as compared to CNN, which could be due to the LSTM learning a more robust representation.

Original: 5. S ENTIMENT A NALYSIS : Y ELP R EVIEWS
Revised: 5. S ENTIMENT A NALYSIS : Y ELP R EVIEWS

Original: The Yelp reviews dataset consists of positive and negative reviews.
Revised: The Yelp reviews dataset consists of positive and negative reviews.

Original: We train a CNN with 32 input units, 32 ﬁlters and 64 hidden units with Relu activation.
Revised: We train a CNN with 32 input units, 32 ﬁlters and 64 hidden units with Relu activation.

Original: We use 83200 training example andvalidation examples.
Revised: We use 83200 training example andvalidation examples.

Original: The CNN has 89.96% train accuracy, 93.74% validation accuracy.
Revised: The CNN has 89.96% train accuracy, 93.74% validation accuracy.

Original: We use the Adam optmizer with default learning rate of 0.001 and use early stopping.
Revised: We use the Adam optmizer with default learning rate of 0.001 and use early stopping.

Original: The input layer uses word2vec embeddings that are learned during training.
Revised: The input layer uses word2vec embeddings that are learned during training.

Original: We carry out attacks on 500 random test examples and report results in ﬁgure 2(left).
Revised: We carry out attacks on 500 random test examples and report results in ﬁgure 2(left).

Original: Here we show the accuracy of the classiﬁer on allexamples as the number of perturbed features increases.
Revised: Here we show the accuracy of the classiﬁer on allexamples as the number of perturbed features increases.

Original: Here we have added replace random and delete random as two additional baselines.
Revised: Here we have added replace random and delete random as two additional baselines.

Original: Replace random replaces k features chosen at random, whereas delete random deletes k random features.
Revised: Replace random replaces k features chosen at random, whereas delete random deletes k random features.

Original: 5. T OPIC C LASSIFICATION : AG N EWS
Revised: 5. T OPIC C LASSIFICATION : AG N EWS

Original: The AG news dataset consists of news related to 4 categories.
Revised: The AG news dataset consists of news related to 4 categories.

Original: We train a CNN with 32 input units, 32 ﬁlters and 64 hidden units with ReLU activation.
Revised: We train a CNN with 32 input units, 32 ﬁlters and 64 hidden units with ReLU activation.

Original: It has 96000 training example andvalidation examples.
Revised: It has 96000 training example andvalidation examples.

Original: We train for 2 epochs with 96.4% train accuracy, 94.8% validation accuracy.
Revised: We train for 2 epochs with 96.4% train accuracy, 94.8% validation accuracy.

Original: We use the Adam optmizer with default learning rate of 0.001 and use early stopping.
Revised: We use the Adam optmizer with default learning rate of 0.001 and use early stopping.

Original: The input layer uses word2vec embeddings that are learned during training.
Revised: The input layer uses word2vec embeddings that are learned during training.

Original: We carry out attacks on 500 random test examples and report results in ﬁgure 2(right).
Revised: We carry out attacks on 500 random test examples and report results in ﬁgure 2(right).

Original: Here we show the accuracy of the classiﬁer on all 500 examples as the number of perturbed features increases.
Revised: Here we show the accuracy of the classiﬁer on all 500 examples as the number of perturbed features increases.

Original: The results on this multiclass dataset conﬁrm that WW scores are a good proxy for local word importance.
Revised: The results on this multiclass dataset conﬁrm that WW scores are a good proxy for local word importance.

Original: The attacks here are untargeted, with the objective being to minimize the correct class probability.
Revised: The attacks here are untargeted, with the objective being to minimize the correct class probability.

Original: Targeted attacks can also be similarly launched using these scores.
Revised: Targeted attacks can also be similarly launched using these scores.

Original: 5. A DDITIONAL EXPERIMENTS ON IMDB REVIEWS
Revised: 5. A DDITIONAL EXPERIMENTS ON IMDB REVIEWS

Original: In this section we describe a number of other experiments we ran on the IMDB reviews dataset.
Revised: In this section we describe a number of other experiments we ran on the IMDB reviews dataset.

Original: 5.4. I S GREEDY A LOWER BOUND FOR ATTACK SUCCESS ?
Revised: 5.4. I S GREEDY A LOWER BOUND FOR ATTACK SUCCESS ?

Original: We carried out further experiments by creating a new algorithm that evaluates greedy and greedy ww for each test example and chooses the best result of both.
Revised: We carried out further experiments by creating a new algorithm that evaluates greedy and greedy ww for each test example and chooses the best result of both.

Original: If greedy and greedy ww were ﬁnding different types of vulnerabilities, we would have expected the algorithm to perform better than both.
Revised: If greedy and greedy ww were ﬁnding different types of vulnerabilities, we would have expected the algorithm to perform better than both.

Original: In fact, the algorithm did no better than greedy, and thus greedy appears to be a bound for greedy ww.
Revised: In fact, the algorithm did no better than greedy, and thus greedy appears to be a bound for greedy ww.

Original: Recall that in greedy, feature deletion is followed by feature insertion, so it does not follow directly that evaluation of input with a feature deleted should perform better than evaluation with everything except the feature deleted.
Revised: Recall that in greedy, feature deletion is followed by feature insertion, so it does not follow directly that evaluation of input with a feature deleted should perform better than evaluation with everything except the feature deleted.

Original: We hypothesize that the success of greedy attacks is partially explained by WordsWorth scores.
Revised: We hypothesize that the success of greedy attacks is partially explained by WordsWorth scores.

Original: Most of the times, greedy is just picking the words with the highest global importance and ﬁnding replacements.
Revised: Most of the times, greedy is just picking the words with the highest global importance and ﬁnding replacements.

Original: In some cases it optimizes further, which explains its improved performance over greedy ww.
Revised: In some cases it optimizes further, which explains its improved performance over greedy ww.

Original: We would like to point out that a surprisingly high fraction of successful greedy attacks is explained by our single word scores.
Revised: We would like to point out that a surprisingly high fraction of successful greedy attacks is explained by our single word scores.

Original: This suggests that most of the time, the impact of a word on the prediction is independent of its context.
Revised: This suggests that most of the time, the impact of a word on the prediction is independent of its context.

Original: 5.4. T HE CASE OF SMALL ARCHITECTURES
Revised: 5.4. T HE CASE OF SMALL ARCHITECTURES

Original: To test the algorithm with smaller networks, we train a very small CNN (8 dimensional embedding layer,8 ﬁlters and 16 units in a dense layer) and run our experiments on it.
Revised: To test the algorithm with smaller networks, we train a very small CNN (8 dimensional embedding layer,8 ﬁlters and 16 units in a dense layer) and run our experiments on it.

Original: The test accuracy is the same as that for the larger CNN, but the appears to be holding up better to greedy ww attacks, as shown in ﬁgure 3(left).
Revised: The test accuracy is the same as that for the larger CNN, but the appears to be holding up better to greedy ww attacks, as shown in ﬁgure 3(left).

Original: the results to
Revised: the results to

Original: The Pearson correlation between WordsWorth scores for this model and our main CNN is 0.83.
Revised: The Pearson correlation between WordsWorth scores for this model and our main CNN is 0.83.

Original: The relatively poor performance of the bigger CNN could be due to overﬁtting.
Revised: The relatively poor performance of the bigger CNN could be due to overﬁtting.

Original: Since the task of binary classiﬁcation is rather simple, the smaller network could be learning more robust and meaningful representations.
Revised: Since the task of binary classiﬁcation is rather simple, the smaller network could be learning more robust and meaningful representations.

Original: However, contradictory hypotheses exist for images, such as Oscar Deniz &
Revised: However, contradictory hypotheses exist for images, such as Oscar Deniz &

Original: Bueno (2020).
Revised: Bueno (2020).

Original: Here, we highlight the fact that robustness to attacks, as well as score comparison, could be one interesting way to compare small vs big and deep vs shallow models.
Revised: Here, we highlight the fact that robustness to attacks, as well as score comparison, could be one interesting way to compare small vs big and deep vs shallow models.

Original: 5.4. T RANSFERABILITY AND S ECURITY
Revised: 5.4. T RANSFERABILITY AND S ECURITY

Original: The phenomenon of transferability is well documented in adversarial attacks on deep models, where adversarial examples generated for one trained model are often successful in attacking another model (Papernot et al., 2016).
Revised: The phenomenon of transferability is well documented in adversarial attacks on deep models, where adversarial examples generated for one trained model are often successful in attacking another model (Papernot et al., 2016).

Original: To demonstrate the phenomenon of transferability, we attack an LSTM with greedy and del one, and with greedy ww cnn and del one ww cnn where the WordsWorth scores have been calculated through a CNN, and the second step in adversarial search is evaluated directly on the LSTM.
Revised: To demonstrate the phenomenon of transferability, we attack an LSTM with greedy and del one, and with greedy ww cnn and del one ww cnn where the WordsWorth scores have been calculated through a CNN, and the second step in adversarial search is evaluated directly on the LSTM.

Original: The correlation between the CNN and LSTM WW scores came out to be 0.88.
Revised: The correlation between the CNN and LSTM WW scores came out to be 0.88.

Original: Results of the attacks are shown in ﬁgure 3(right).
Revised: Results of the attacks are shown in ﬁgure 3(right).

Original: There is some drop in performance but still a noticeable degree of success.
Revised: There is some drop in performance but still a noticeable degree of success.

Original: The close alignment of greedy and greedy ww cnn shows that the importance scores calculated through CNN are valid for LSTM too, even though directly using LSTM scores gives better performance.
Revised: The close alignment of greedy and greedy ww cnn shows that the importance scores calculated through CNN are valid for LSTM too, even though directly using LSTM scores gives better performance.

Original: Compare this to ﬁgure 1(right) where LSTM was attacked with WW scores from the LSTM itself.
Revised: Compare this to ﬁgure 1(right) where LSTM was attacked with WW scores from the LSTM itself.

Original: We argue that this close, non-random alignment in ﬁgure 3 (right) explains the phenomenon of transferability in general.
Revised: We argue that this close, non-random alignment in ﬁgure 3 (right) explains the phenomenon of transferability in general.

Original: Features that are important for one architecture are important for another architecture too, when both models have been trained on the same dataset.
Revised: Features that are important for one architecture are important for another architecture too, when both models have been trained on the same dataset.

Original: Additionally, this highlights the aspect that for attacking a black box model, an adversary can train a small model locally and use it to highlight the vulnerable points of a piece of text, while using the black box model to ﬁnd out substitutes, since the latter requires a much fewer number of model evaluations than the former.
Revised: Additionally, this highlights the aspect that for attacking a black box model, an adversary can train a small model locally and use it to highlight the vulnerable points of a piece of text, while using the black box model to ﬁnd out substitutes, since the latter requires a much fewer number of model evaluations than the former.

Original:  I NTERPRETING NEURAL NETWORKS THROUGH W ORDS W ORTH SCORES
Revised:  I NTERPRETING NEURAL NETWORKS THROUGH W ORDS W ORTH SCORES

Original: In this section we show how to use WordsWorth scores for interpreting a model.
Revised: In this section we show how to use WordsWorth scores for interpreting a model.

Original: 6. IMDB R EVIEWS : L OCATING IMPORTANT WORDS
Revised: 6. IMDB R EVIEWS : L OCATING IMPORTANT WORDS

Original: For the IMDB dataset, we computed the WordsWorth scores over our entire vocabulary (limited to 5000 top words) for the CNN as well as the LSTM.
Revised: For the IMDB dataset, we computed the WordsWorth scores over our entire vocabulary (limited to 5000 top words) for the CNN as well as the LSTM.

Original: The top ten important words for the CNN are given in the table 2.
Revised: The top ten important words for the CNN are given in the table 2.

Original: For the CNN, the mean WW score is 0.559 and standard deviation is 0.0530.
Revised: For the CNN, the mean WW score is 0.559 and standard deviation is 0.0530.

Original: In this manner, the model designer can directly ﬁnd the top ranked words associated with each sentiment after training and examine errors in training.
Revised: In this manner, the model designer can directly ﬁnd the top ranked words associated with each sentiment after training and examine errors in training.

Original: We also include a snapshot of the scores for the entire vocabulary for the CNN in ﬁgure 4.
Revised: We also include a snapshot of the scores for the entire vocabulary for the CNN in ﬁgure 4.

Original: 6. AG N EWS
Revised: 6. AG N EWS

Original: We computed the WordsWorth scores over our entire vocabulary (limited to 20000 top words) for the trained CNN.
Revised: We computed the WordsWorth scores over our entire vocabulary (limited to 20000 top words) for the trained CNN.

Original: The top ten important words for each category are given in the table 3.
Revised: The top ten important words for each category are given in the table 3.

Original: 6.2. AG N EWS W ORD IMPORTANCE SCORE DISTRIBUTION
Revised: 6.2. AG N EWS W ORD IMPORTANCE SCORE DISTRIBUTION

Original: We also plot the scores for each class for AG News in ﬁgure 6 and 7.
Revised: We also plot the scores for each class for AG News in ﬁgure 6 and 7.

Original: Notice that different categories have different distributions associated with them.
Revised: Notice that different categories have different distributions associated with them.

Original: This could point to differences in writing style for each category, or to a difference in word distribution.
Revised: This could point to differences in writing style for each category, or to a difference in word distribution.

Original: 6. O VERTRAINING AND WORD IMPORTANCE SCORES
Revised: 6. O VERTRAINING AND WORD IMPORTANCE SCORES

Original: We overtrain a CNN on the IMDB reviews dataset and calculate word importance scores over the training vocabulary after every epoch.
Revised: We overtrain a CNN on the IMDB reviews dataset and calculate word importance scores over the training vocabulary after every epoch.

Original: A few snapshots are in ﬁgure 5.
Revised: A few snapshots are in ﬁgure 5.

Original: Stats related to training are shown in table 1.
Revised: Stats related to training are shown in table 1.

Original: Overtraining has a noticeable effect on individual word scores, with score distribution becoming homogenous with overtraining.
Revised: Overtraining has a noticeable effect on individual word scores, with score distribution becoming homogenous with overtraining.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We consider the problem of quickly ﬁnding important words in a text to perturb in order to maximize the efﬁcacy of black box attacks on deep NLP models, in the context of text classiﬁcation.
Revised: We consider the problem of quickly ﬁnding important words in a text to perturb in order to maximize the efﬁcacy of black box attacks on deep NLP models, in the context of text classiﬁcation.

Original: For this problem we present WordsWorth, a feature ranking algorithm that performs comparably well to the state of the art approaches, particularly when only a small number of feature perturbations are allowed, while being orders of magnitude faster by virtue of being essentially a lookup on training vocabulary.
Revised: For this problem we present WordsWorth, a feature ranking algorithm that performs comparably well to the state of the art approaches, particularly when only a small number of feature perturbations are allowed, while being orders of magnitude faster by virtue of being essentially a lookup on training vocabulary.

Original: Wealsouse these scores as a tool model interpretation, compare different architectures and give a metric for evaluating performance beyond training and validation accuracy.
Revised: Wealsouse these scores as a tool model interpretation, compare different architectures and give a metric for evaluating performance beyond training and validation accuracy.

Original: We also explain the phenomenon of transferability observed in text adversarial attacks and show that black box attacks can yield valuable information about the training dataset.
Revised: We also explain the phenomenon of transferability observed in text adversarial attacks and show that black box attacks can yield valuable information about the training dataset.

Original: All in all, we argue that text generated by humans is a highly compact and informative representation of data and the way neural networks interpret language aligns with human understanding.
Revised: All in all, we argue that text generated by humans is a highly compact and informative representation of data and the way neural networks interpret language aligns with human understanding.

Original: Overall, we provide a method for evaluating importance in parallel with word erasure techniques.
Revised: Overall, we provide a method for evaluating importance in parallel with word erasure techniques.

Original: Combining the two techniques would yield even richer insights into the workings of models.
Revised: Combining the two techniques would yield even richer insights into the workings of models.

Original: Seen another way, WordsWorth attacks uncover a particular kind of vulnerability in deep models.
Revised: Seen another way, WordsWorth attacks uncover a particular kind of vulnerability in deep models.

Original: Our work is the ﬁrst step in designing a rule based algorithm to attack deep models that deal with text, and the next one would be to look at complex interactions.
Revised: Our work is the ﬁrst step in designing a rule based algorithm to attack deep models that deal with text, and the next one would be to look at complex interactions.

Original: By aligning the performance of rule based algorithms with empirical methods currently popular in deep learning, we can improve our understanding of these otherwise blackbox models.
Revised: By aligning the performance of rule based algorithms with empirical methods currently popular in deep learning, we can improve our understanding of these otherwise blackbox models.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: [Table] [Table]
Revised: [Table] [Table]