Original: C HARACTER L EVEL BASED D ETECTION OF DGA D OMAIN N AMES
Revised: C HARACTER L EVEL BASED D ETECTION OF DGA D OMAIN N AMES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classiﬁcation.
Revised: Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classiﬁcation.

Original: Little studies are available that compare the effectiveness of these approaches for character based text classiﬁcation with each other.
Revised: Few studies are available that compare the effectiveness of these approaches for character based text classiﬁcation with each other.

Original: In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm).
Revised: In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm).

Original: Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overﬁtting.
Revised: Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overﬁtting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Malware is software that infects computers in order to perform unauthorized malicious activities.
Revised: Malware is software that infects computers in order to perform unauthorized malicious activities.

Revised: In order to successfully achieve its goals, the malware needs to be able to connect to a command and control (C&C) center.

Revised: To this end, both the controller behind the C&C center (hereafter called botmaster) and the malware on the infected machines can run a Domain Generation Algorithm (DGA) that generates hundreds or even thousands of domains automatically.

Revised: The malware then attempts at resolving each one of these domains with its local DNS server.

Revised: The botmaster will have registered one or a few of these automatically generated domains.

Original: These activities range from stealing information, to exploiting the victims’ computing resources to mine bitcoin.
Revised: For these domains that have been actually registered, the malware will obtain a valid IP address and will be able to communicate with the C&C center. The binary text classiﬁcation task that we address in this paper is: given a domain name string as input, classify it as either malicious, i.e. generated by a DGA, or as benign. Deep neural networks have recently appeared in the literature on DGA detection Woodbridge et al. ; Saxe & Berlin (2017); Yu et al.

Revised: They signiﬁcantly outperform traditional machine learning methods in accuracy, at the price of increasing the complexity of training the model and requiring larger datasets.

Revised: Independent of the work on deep networks for DGA detection, other deep learning approaches for character based text classiﬁcation have recently been proposed, including deep neural network architectures designed for processing and classiﬁcation of tweets (Dhingra et al.

Revised: ; Vosoughi et al.

Revised: as well as

Revised: general natural language text (Zhang et al.

Revised: No systematic study is available that compares the predictive accuracy of all these different character based deep learning architectures, leaving one to wonder which one works best for DGA detection.

Revised: To answer this open question, in this paper we compare the performance of ﬁve different deep learning architectures for character based text classiﬁcation (see Table 1) for the problem of detecting

Revised: DGAs.

Revised: They all rely on character-level embeddings, and they all use a deep learning architecture based on convolutional neural network (CNN) layers, recurrent neural network (RNN) layers, or a combination of both.

Revised: Our most important ﬁnding is that for DGA detection, which can be thought of as classiﬁcation of short character strings, despite of vast differences in the deep network architectures, there is remarkably little difference among the methods in terms of accuracy and false positive rates, while they all comfortably outperform a random forest trained on human engineered features.

Revised: This ﬁnding is of practical value for the design of deep neural network based classiﬁers for short text classiﬁcation in industry and academia: it provides evidence that one can select an architecture that

Revised: [Table]

Revised: is faster to train, without loss of accuracy.

Revised: In the context of DGA detection, optimizing the training time is of particular importance, as the models need to be retrained on a regular basis to stay current with respect to new, emerging malware.

Revised:  B ACKGROUND

Revised: Malware controllers, or botmasters, use malware for all kinds of unauthorized malicious activities.

Revised: These activities range from stealing information, to exploiting the victims’ computing resources to mine bitcoin.

Original: They can also include launching a distributed denial of service attack from the victims’s computers or encrypting the victims hard drive (ransomware).
Revised: They can also include launching a distributed denial of service attack from the victims’s computers or encrypting the victims hard drive (ransomware).

Original: In order to successfully achieve its goals, it is vital that the malware be able to connect to a command and control (C&C) center.
Revised: In order to successfully achieve its goals, it is vital that the malware be able to connect to a command and control (C&C) center.

Original: This communication can serve many purposes.
Revised: This communication can serve many purposes.

Original: The malware can use it to send stolen information (such as passwords or access credentials) to the malware designer/controler (hereafter called botmaster) behind the C&C center, it can use this communication channel to receive instructions or even to update itself to a newer version.
Revised: The malware can use it to send stolen information (such as passwords or access credentials) to the malware designer behind the C&C center, it can use this communication channel to receive instructions or even to update itself to a newer version.

Original: Initially, botmasters established such a communication channel to the command and control center by hard-coding an IP address inside the malware.
Revised: Initially, botmasters established such a communication channel to the C&C center by hard-coding an IP address inside the malware.

Original: This approach has obvious shortcomings: once the malware is reversed engineered, the IP address is discovered and shut down.
Revised: This approach has obvious shortcomings from the botmasters’ perspective: once the malware is reversed engineered, the IP address is discovered and shut down.

Original: Over time, malware designers came up with a much more effective strategy: Domain Generation Algorithms (DGAs).
Revised: Over time, malware designers came up with a much more effective strategy: Domain Generation Algorithms (DGAs).

Original: Domain Generation Algorithms work by having the malware accessing some available source of randomness and inputting it into an algorithm that generates hundreds or even thousands of domains automatically.
Revised: Domain Generation Algorithms work by having the malware accessing some available source of randomness and inputting it into an algorithm that generates hundreds or even thousands of domains automatically.

Original: The malware then attempts at resolving each one of these domains with its local DNS server (a DNS server runs a protocol that translates domain names into IP addresses; it is a vital piece of the Internet).
Revised: The malware then attempts at resolving each one of these domains with its local DNS server (a DNS server runs a protocol that translates domain names into IP addresses; it is a vital piece of the Internet).

Original: The botmaster will have registered one or a few of these automatically generated domains.
Revised: The botmaster will have registered one or a few of these automatically generated domains.

Original: For these domains that have been actually registered, the malware will obtain a valid IP address and will be able to communicate with the C&C center.
Revised: For these domains that have been actually registered, the malware will obtain a valid IP address and will be able to communicate with the C&C center.

Original: For all the other domains that were automatically generated but not registered, the malware obtains a message stating that these domains could not have been resolved and ignores them.
Revised: For all the other domains that were automatically generated but not registered, the malware obtains a message stating that these domains could not have been resolved and ignores them.

Original: DGAs make blacklisting of domains extremely difﬁcult, since by changing the initial randomness (while keeping the same algorithm)
Revised: DGAs make blacklisting of domains extremely difﬁcult, since by changing the initial randomness (while keeping the same algorithm)

Original: the malware can potentially generate completely different domains.
Revised: the malware can potentially generate completely different domains.

Original: This technique has been used by high-proﬁle malware such as Conﬁcker, Stuxnet (the malware designed to attack Iran nuclear facilities) and Flame.
Revised: This technique has been used by high-proﬁle malware such as Conﬁcker, Stuxnet (the malware designed to attack Iran nuclear facilities) and Flame.

Original: Catching domain names generated by malware has become a central topic in information security, leading to a recent interest in detecting
Revised: Catching domain names generated by malware has become a central topic in information security, leading to a recent interest in detecting

Original: DGA domains using machine learning techniques.
Revised: DGA domains using machine learning techniques.

Original: Models that classify domain names as benign or malicious based solely on the domain name string are of particular interest for their generality, as context information beyond the domain name string might be unavailable or expensive to acquire.
Revised: Models that classify domain names as benign or malicious based solely on the domain name string are of particular interest for their generality, as context information beyond the domain name string might be unavailable or expensive to acquire.

Original: [Table]

Original: Traditional machine learning methods for DGA detection based on the domain name string rely on extraction of predeﬁned, human engineered lexical features, see e.g. Antonakakis et al.
Revised: Traditional machine learning methods for DGA detection based on the domain name string rely on extraction of predeﬁned, human engineered lexical features, see e.g. Antonakakis et al.

Original: Schiavoni et al.
Revised: Schiavoni et al.

Original: Whenever human engineered features are used, it is obvious that this opens the door for an adversary to carefully craft its DGA to avoid detection by using the aforementioned features.
Revised: Whenever human engineered features are used, it is obvious that this opens the door for an adversary to carefully craft its DGA to avoid detection by using the aforementioned features.

Original: This makes maintaining such machine learning systems labor intensive.
Revised: This makes maintaining such machine learning systems labor intensive.

Original: Recently proposed deep learning techniques for detecting DGAs learn features automatically, thereby offering the potential to bypass the human effort of feature engineering (Woodbridge et al.
Revised: Recently proposed deep learning techniques for detecting DGAs learn features automatically, thereby offering the potential to bypass the human effort of feature engineering (Woodbridge et al.

Original: ; Saxe & Berlin (2017); Yu et al.
Revised: ; Saxe & Berlin (2017); Yu et al.

Original: In addition these deep learning approaches outperform the traditional machine learning techniques with human engineered features in terms of accuracy and false positive rates.
Revised: In addition these deep learning approaches outperform the traditional machine learning techniques with human engineered features in terms of accuracy and false positive rates.

Original: Independent ofthe work on deep networks for DGA detection, other deep learning approaches forcharacter based text classiﬁcation  haverecently been proposed, including deep neural network architectures designed for processing and classiﬁcation of tweets (Dhingra et al.
Revised: The choice of deep network architecture in these recent works appears fairly arbitrary, and it is unclear whether they would perform better or worse than other deep neural networks approaches for text classiﬁcation that have been introduced recently, in particular character-level methods for processing and classiﬁcation of tweets (Dhingra et al.

Original: ; Vosoughi et al.
Revised: ; Vosoughi et al.

Original: as well as general natural language text (Zhang et al.
Revised: and for general natural language text (Zhang et al.

Original: No systematic study is available that compares the predictive accuracy of all these different character based deep learning architectures, leaving one to wonder which one works best for DGA detection. To answer this open question, inthis paper we compare the performance of ﬁve different deep learning architectures for character based text classiﬁcation (see Table 1) for the  problem  of detecting
Revised: In this paper, we provide the ﬁrst comparative study of all these different methods, showing that, despite of vast differences in the architectures, they can be easily tuned to result in the same predictive accuracy. The problem of adversarial examples, i.e. instances that have intentionally been designed to cause the model to make a mistake, are a well known problem in machine learning. The scenario sketched above — in which a malware designer exploits knowledge about the lexical features used by a random forest to craft his DGA to avoid detection — is a prime example of this.

Original: DGAs.
Revised: Deep neural networks are famously not immune to adversarial examples either, and generative adversarial networks (GANs) can be trained to generate them automatically (see e.g. Goodfellow et al. (2014a;b)).

Original: They all rely on character-level embeddings, and they all use a deep learning architecture based on convolutional neural network (CNN) layers, recurrent neural network (RNN) layers, or a combination of both.
Revised: Speciﬁcally in the context of DGA detection, Anderson et al. (2016) have used a character-based generative adversarial network (GAN) to augment training sets in order to harden other machine learning models (like a random forest) against yet-to-be-observed DGAs.

Original: Our most important ﬁnding is that for  DGA detection, which can be thought of as classiﬁcation of short character strings, there is remarkably little difference among the methods in terms of accuracy and false positive rates, while they all comfortably outperform a random forest trained on human engineered features.
Revised: It is highly unlikely for attackers to use GANs themselves, because DGA algorithms must be light enough to be embedded inside malware code. Furthermore, generating domain names that look like a benign domain is not enough for an effective DGA. Ideally, every domain produced by a DGA must not have been registered yet or must have a low likelihood of being registered already – if a domain produced by a DGA has already been taken, it is useless for the botmaster.

Revised: Combining all these requirements is essential for a serious study of adversarial generated domains and outside the scope of this paper.

Original:  M ETHODS
Revised:  M ETHODS

Original: We compare ﬁve different deep learning methods for short string classiﬁcation, when applied to the problem of DGA detection speciﬁcally.
Revised: We compare ﬁve different deep learning methods for short string classiﬁcation, when applied to the problem of DGA detection speciﬁcally.

Original: For each of the methods, we started from the original proposals as can be found in the references in Table 1 and only made modiﬁcations when they improved the predictive accuracy for the classiﬁcation of domain names.
Revised: For each of the methods, we start from the original proposals as can be found in the references in Table 1 and only make modiﬁcations when they improve the predictive accuracy for the classiﬁcation of domain names.

Original: Below we give an overview of the methods and the adaptations made, alongside with a Keras 1 code snippet for each method.
Revised: Below we give an overview of the methods and the adaptations made. A Keras 1 code snippet for each method is included in Appendix A.

Original: The strings that we give as input to all classiﬁers consist of a second level domain (SLD) and a top level domain (TLD), separated by a dot, as in e.g. wikipedia.org .
Revised: The strings that we give as input to all classiﬁers consist of a second level domain (SLD) and a top level domain (TLD), separated by a dot, as in e.g. wikipedia.org .

Original: Following Woodbridge et al.
Revised: Following Woodbridge et al.

Original: (2016), we set the maximum length at 75 characters, padded with zeros on the left for domains whose length is less than 75 2 .
Revised: (2016), we set the maximum length at 75 characters, padded with zeros on the left for domains whose length is less than 75.

Original: We convert each domain name string to lower case, since domain names are case insensitive, and encode it as an ASCII code sequence of length 128, effectively representing each domain name string as a 75 by 128 matrix.
Revised: We convert each domain name string to lower case, since domain names are case insensitive, and encode it as an ASCII code sequence of length 128, effectively representing each domain name string as a 75 by 128 matrix in which each character corresponds to a column.

Original: 2. RNN BASED A RCHITECTURES
Revised: 3. RNN BASED A RCHITECTURES

Original: Endgame Model Long short-term memory networks (LSTMs), a special kind of recurrent neural networks (RNNs) have recently attracted a lot of attention because of their successful application to problems that involve processing of sequences (Hochreiter & Schmidhuber (1997)).
Revised: Endgame Model Long short-term memory networks (LSTMs), a special kind of recurrent neural networks (RNNs) have recently attracted a lot of attention because of their successful application to problems that involve processing of sequences (Hochreiter & Schmidhuber (1997)).

Original: Since domain names can be thought of as sequences of characters, LSTMs are a natural kind of classiﬁers to apply.
Revised: Since domain names can be thought of as sequences of characters, LSTMs are a natural kind of classiﬁers to apply.

Original: The LSTM network proposed by Woodbridge et al.
Revised: The LSTM network proposed by Woodbridge et al.

Original: (2016) was designed speciﬁcally for DGA detection, so we stay very close to the original model.
Revised: (2016) was designed speciﬁcally for DGA detection, so we stay very close to the original model.

Original: As can be seen in Listing 1, the network is comprised of an embedding layer, an LSTM layer (128 LSTM cells with default Tanh activation), and a single node output layer with sigmoid activation.
Revised: The network is comprised of an embedding layer, an LSTM layer (128 LSTM cells with default Tanh activation), and a single node output layer with sigmoid activation.

Original: Instead of using RMSProp as the optimization algorithm, as was done in Woodbridge et al.
Revised: Instead of using RMSProp as the optimization algorithm, as was done in Woodbridge et al.

Original: , we switched to Adam (Kingma & Ba (2014)) because it resulted in better loss convergence results (see Section 3).
Revised: , we switched to Adam (Kingma & Ba (2014)) because it resulted in better loss convergence results (see Section 4).

Revised: The Endgame model includes dropout, a technique to improve model performance and overcome over-ﬁtting by randomly excluding nodes during training,

Revised: The maximum allowed length for SLDs and TLDs is 63 characters each.

Revised: In practice they are typically shorter.

Revised: The longest domain name string we encountered in our experiments is 73 characters.

Original:  The role of the embedding layer is to learn to represent each character that can occur in a domain name by a 128-dimensional numerical vector.
Revised: This string includes the SLD, the TLD, and the dot that separates them. which serves to break up complex co-adaptations in the network Srivastava et al. This is conﬁned to the training phase; all nodes are active during testing and deployment. The role of the embedding layer is to learn to represent each character that can occur in a domain name by a 128-dimensional numerical vector.

Original: This vector is different from the original 128dimensional ASCII encoding.
Revised: This vector is different from the original 128dimensional ASCII encoding.

Original: The embedding maps semantically similar characters to similar vectors, where the notion of similarity is implicitly derived (learned) based on the classiﬁcation task at hand.
Revised: The embedding maps semantically similar characters to similar vectors, where the notion of similarity is implicitly derived (learned) based on the classiﬁcation task at hand.

Original: As will become clear in the remainder of this section, all ﬁve deep neural network architectures under study start with such an embedding layer.
Revised: As will become clear in the remainder of this section, all ﬁve deep neural network architectures under study start with such an embedding layer.

Original: To allow for a fair comparison, we have made the parameter choices for the embedding layer, such as the dimensionality of the embedding space, identical for all ﬁve models.
Revised: To allow for a fair comparison, we have made the parameter choices for the embedding layer, such as the dimensionality of the embedding space, identical for all ﬁve models.

Original: In addition, for comparison purposes, in Section 3 we also present the results of a “baseline neural network model” consisting only of an embedding layer as its hidden layer.
Revised: In addition, for comparison purposes, in Section 4 we also present the results of a “baseline neural network model” consisting only of an embedding layer as its hidden layer.

Original: The Endgame model presented in Listing 1 includes dropout, a technique to improve model performance and overcome over-ﬁtting by randomly excluding nodes during training, which serves to break up complex co-adaptations in the network Srivastava et al.

Original: This is conﬁned to the training phase; all nodes are active during testing and deployment.

Original: main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) lstm

Original: = LSTM(128, return sequences=False)(embedding) drop = Dropout(0.5)(lstm) output = Dense(1, activation =’sigmoid’)(drop) model =

Original: Model(inputs

Original: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Original: CMU Model Bidirectional RNNs extend regular RNNs by processing the input string in two ways.
Revised: CMU Model Bidirectional RNNs extend regular RNNs by processing the input string in two ways.

Original: In a forward layer, the input sequence is processed from the left to the right, as in a traditional RNN, while in a backward layer, the processing happens from the right to the left.
Revised: In a forward layer, the input sequence is processed from the left to the right, as in a traditional RNN, while in a backward layer, the processing happens from the right to the left.

Original: The output from the forward and the backward layer is then combined and passed on to further layers.
Revised: The output from the forward and the backward layer is then combined and passed on to further layers.

Original: Bidirectional LSTMs for character level text processing have been proposed in Ling et al.
Revised: Bidirectional LSTMs for character level text processing have been proposed in Ling et al.

Original: (2015), and, following up on that, very similar bidirectional GRUs (gated recurrent units) have been applied in a “Tweet2Vec” model for tweet classiﬁcation (predicting hashtags of tweets) by Dhingra et al.
Revised: (2015), and, following up on that, very similar bidirectional GRUs (gated recurrent units) have been applied in a “Tweet2Vec” model for tweet classiﬁcation (predicting hashtags of tweets) by Dhingra et al.

Original: Listingpresents an adaptation of the latter to our problem of DGA detection.
Revised: We use an adaptation of the latter; see Listing 2 in Appendix A.

Original: Including dropout or replacing LSTM by GRUin Listing 2 did not cause a signiﬁcant change in predictive accuracy, although the latter did result in a decrease of training runtime.
Revised: Including dropout or replacing LSTM by GRU did not cause a signiﬁcant change in predictive accuracy, although the latter did result in a decrease of training runtime.

Original: main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) bi lstm =

Original: Bidirectional ( layer=LSTM(64, return sequences=

Original: False), merge mode=’concat’)(embedding) output = Dense(1, activation =’sigmoid’)( bi lstm ) model =

Original: Model(inputs

Original: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 2. CNN BASED A RCHITECTURES
Revised: 3. CNN BASED A RCHITECTURES

Original: NYU Model Convolutional neural networks (CNNs) are known for their ability to process input data with a grid like topology, such as images consisting of a grid of pixels.
Revised: NYU Model Convolutional neural networks (CNNs) are known for their ability to process input data with a grid like topology, such as images consisting of a grid of pixels.

Original: To the best of our knowledge, Zhang et al.
Revised: To the best of our knowledge, Zhang et al.

Original: (2015) were the ﬁrst to apply 1-dimensional or “temporal” CNNs successfully to text classiﬁcation at character level.
Revised: (2015) were the ﬁrst to apply 1-dimensional or “temporal” CNNs successfully to text classiﬁcation at character level.

Original: Their proposed deep network architecture, which is intended to process full-blown natural language text such as news articles or reviews, includes 6 stacked CNN layers, with each subsequent layer consuming the output from the previous layer.
Revised: Their proposed deep network architecture, which is intended to process full-blown natural language text such as news articles or reviews, includes 6 stacked CNN layers, with each subsequent layer consuming the output from the previous layer.

Original: Each CNN layer consists of a set of ﬁlters or kernels that “slide” over the input to the layer in search for patterns.

Original: Next, a pooling step with a predeﬁned pooling size is applied to make the network less sensitive to the exact position where the pattern was detected in the input string (the larger the pooling size, the less sensitive).

Original: During training of the network, each ﬁlter automatically learns which pattern it should look for.

Original: In contrast to natural language text, domain names are very short and they do not have an internal grammatical structure, naturally resulting the original architectures from Zhang et al.
Revised: In contrast to natural language text, domain names are very short and they do not have an internal grammatical structure, naturally resulting the original architectures from Zhang et al.

Original: (2015) to overﬁt on our data.
Revised: (2015) to overﬁt on our data.

Original: We therefore reduced the number of stacked CNN layers to two, and decreased the size and the number of ﬁlters on the CNN layers (see Listing 3).
Revised: We therefore reduced the number of stacked CNN layers to two, and decreased the size and the number of ﬁlters on the CNN layers (see Listing 3).

Original: [-main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) conv
-]
Revised: [-main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) conv
-]

Original: =

Original: [-Conv1D( ﬁlters =128, kernel size =3, padding=’same’, strides =1)(embedding) thresh
-]
Revised: [-Conv1D( ﬁlters =128, kernel size =3, padding=’same’, strides =1)(embedding) thresh
-]

Original: = ThresholdedReLU(1e − 6)(conv1) max pool1 =

Original: MaxPooling1D(pool size=2, padding=’same’)(thresh1) conv2 =

Original: Conv1D( ﬁlters =128, kernel size =2, padding=’same’, strides =1)(max pool1) thresh2 =

Original: ThresholdedReLU(1e − 6)(conv2) max pool2 =

Original: MaxPooling1D(pool size=2, padding=’same’)(thresh2) ﬂatten =

Original: Flatten () (max pool2) fc = Dense(64)( ﬂatten ) thresh fc =

Original: ThresholdedReLU(1e − 6)(fc) drop = Dropout(0.5)

Original: ( thresh fc ) output = Dense(1, activation =’sigmoid’)(drop) model =

Original: Model(inputs

Original: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Original: Listing 3: NYU model with stacked CNN layers, adapted from Zhang et al.

Original: Invincea Model Saxe & Berlin (2017) proposed a CNN based classiﬁer that takes generic short character strings as its input and learns to detect whether they are indicators of malicious behavior.
Revised: Invincea Model Saxe & Berlin (2017) proposed a CNN based classiﬁer that takes generic short character strings as its input and learns to detect whether they are indicators of malicious behavior.

Original: The short character strings can be e.g. URLs, ﬁle paths, or registry keys.
Revised: The short character strings can be e.g. URLs, ﬁle paths, or registry keys.

Original: The fundamental difference between the Invincea model versus the NYU model described above, is that in the Invincea model the CNN layers are parallel instead of stacked, and that pooling always happens over the entire domain name instead of within a small pooling window.
Revised: The fundamental difference between the Invincea model versus the NYU model described above, is that in the Invincea model the CNN layers are parallel instead of stacked, and that pooling always happens over the entire domain name instead of within a small pooling window.

Original: That means that the Invincea model is only detecting the presence or absence of patterns in the domain names, and does not retain any information on where exactly in the domain name string these patterns occur.
Revised: That means that the Invincea model is only detecting the presence or absence of patterns in the domain names, and does not retain any information on where exactly in the domain name string these patterns occur.

Original: Keras code for the

Original:  Invincea model is given in Listing 4. The embedding layer is followed by a convolutional layer with 1024 ﬁlters, namely 256 ﬁlters for each of the sizes 2, 3, 4, and 5.
Revised: In the Invincea model, the embedding layer is followed by a convolutional layer with 1024 ﬁlters, namely 256 ﬁlters for each of the sizes 2, 3, 4, and 5.

Original: Each of these ﬁlters learns to detect the soft presence of an interesting soft n -gram (with n = 2 , 3 , 4 , 5 ).
Revised: Each of these ﬁlters learns to detect the soft presence of an interesting soft n -gram (with n = 2 , 3 , 4 , 5 ).

Original: The output of the convolutional layer is consumed by two dense hidden layers, each with 1024 nodes, before reaching a single node output layer with sigmoid activation.
Revised: The output of the convolutional layer is consumed by two dense hidden layers, each with 1024 nodes, before reaching a single node output layer with sigmoid activation.

Original: Out of all the models that we compared, this one has the most extensive architecture.
Revised: Out of all the models that we compared, this one has the most extensive architecture.

Original: def getconvmodel(self , kernel size , ﬁlters ):

Original: model =

Original: Sequential () model.add( Conv1D( ﬁlters = ﬁlters , input shape =(128, 128), kernel size = kernel size , padding=’same’, activation =’relu ’, strides =1))

Original: model.add(Lambda(lambda x: K.sum(x, axis=1), output shape=( ﬁlters , ))) model.add(Dropout(0.5))

Original: returnmodel

Original: [Figure]

Original: main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) conv1 =

Original: getconvmodel(2, 256)(embedding) conv2 =

Original: getconvmodel(3, 256)(embedding) conv3 = getconvmodel(4, 256)(embedding) conv4 =

Original: getconvmodel(5, 256)(embedding) merged =

Original: Concatenate() ([conv1, conv2, conv3, conv4]) middle = Dense(1024, activation =’relu ’)

Original: (merged) middle = Dropout(0.5)(middle) middle = Dense(1024, activation =’relu ’)

Original: (middle) middle = Dropout(0.5)(middle) output = Dense(1, activation =’sigmoid’)(middle) model =

Original: Model(inputs

Original: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Original: Listing

Original: 2. H YBRID CNN/RNN BASED A RCHITECTURE
Revised: 3. H YBRID CNN/RNN BASED A RCHITECTURE

Original: MIT Model
Revised: MIT Model

Original: The MIT model proposed by Vosoughi et al.
Revised: The MIT model proposed by Vosoughi et al.

Original: (2016) is an extension of the NYU model, where the stacked CNN layers are followed by an LSTM layer.
Revised: (2016) is an extension of the NYU model, where the stacked CNN layers are followed by an LSTM layer.

Original: Similarly as with the NYU model, the use of multiple stacked CNN layers (which worked well for tweets in Vosoughi et al.
Revised: Similarly as with the NYU model, the use of multiple stacked CNN layers (which worked well for tweets in Vosoughi et al.

Original: (2016)) resulted in the models to overﬁt on our data.
Revised: (2016)) resulted in the models to overﬁt on our data.

Original: For this reason, we reduced the MIT model architecture to the minimum that preserves its spirit: one CNN layer followed by one LSTM layer (see Listing 5).
Revised: For this reason, we reduced the MIT model architecture to the minimum that preserves its spirit: one CNN layer followed by one LSTM layer.

Original: main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) conv = Conv1D( ﬁlters =128, kernel size =3, padding=’same’, activation =’relu ’, strides =1)(embedding) max pool = MaxPooling1D(pool size=2, padding=’same’)(conv) encode = LSTM(64, return sequences=False)(max pool) output = Dense(1, activation =’sigmoid’)(encode) model = Model(inputs
Revised: [Figure]

Original: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Original: Listing 5: MIT model with a stacked CNN and LSTM layer, adapted from Vosoughi et al.

Original:  R ESULTS AND L EARNED R EPRESENTATIONS
Revised:  R ESULTS AND L EARNED R EPRESENTATIONS

Original: We trained and evaluated the models on a dataset with 1 million DGA domain names from Bambenek 3 (positive examples) and the top 1 million domain names from Alexa 4 (negative examples).
Revised: We trained and evaluated the models on a dataset with 1 million DGA domain names from Bambenek 3 (positive examples) and the top 1 million domain names from Alexa 4 (negative examples).

Original: Alexa ranks websites basedtheir on  popularity in terms of number of page views and number of unique visitors.
Revised: Alexa ranks websites based on their popularity in terms of number of page views and number of unique visitors.

Original: It only retains the websites’ SLD and TLD, aggregating across any subdomains.
Revised: It only retains the websites’ SLD and TLD, aggregating across any subdomains.

Original: For example, according to Alexa, the ﬁve highest ranked domain names in terms of popularity on 2017-10-26 are google.com , youtube.com , facebook.com baidu.com , and wikipedia.org .
Revised: For example, according to Alexa, the ﬁve highest ranked domain names in terms of popularity on 2017-10-26 are google.com , youtube.com , facebook.com baidu.com , and wikipedia.org .

Original: For our experiments, we assume that the top 1 million domain names in this ranking are benign domain names, although it is possible that the bottom of the ranking may contain some noise.
Revised: For our experiments, we assume that the top 1 million domain names in this ranking are benign domain names, although it is possible that the bottom of the ranking may contain some noise.

Original: In addition to these benign domain names, we collected 1 million DGA domain names from the
Revised: In addition to these benign domain names, we collected 1 million DGA domain names from the

Original: Bambenek Consulting feeds for 3 different days, namely Jun 24, Jul 22, Jul 23, 2017.
Revised: Bambenek Consulting feeds for 3 different days, namely Jun 24, Jul 22, Jul 23, 2017.

Original: These feeds contain DGA domain names from speciﬁc malware families that were observed in real trafﬁc on those days.
Revised: These feeds contain DGA domain names from speciﬁc malware families that were observed in real trafﬁc on those days.

Original: Such domain names can be collected by reverse engineering a known malware family, generating lists of domain names with the reverse engineered malware, and checking which of these domain names also occur in real trafﬁc.
Revised: Such domain names can be collected by reverse engineering a known malware family, generating lists of domain names with the reverse engineered malware, and checking which of these domain names also occur in real trafﬁc.

Original: Note that our goal in this paper is the development of  a neural network classiﬁer that can detect DGAs without the need to reverse engineer malware families.
Revised: Note that our goal in this paper is the development of [Table] a neural network classiﬁer that can detect DGAs without the need to reverse engineer malware families.

Original: An important advantage of such a classiﬁer is that it can also be used against new and previously unknown malware families.
Revised: An important advantage of such a classiﬁer is that it can also be used against new and previously unknown malware families.

Original: We randomly split the data into 80% for training, 10% for validation, and 10% for testing.
Revised: We randomly split the data into 80% for training, 10% for validation [Table] and 10% for testing.

Original: Figure 1 shows the training and validation loss curves for each of the models described in Section 2.
Revised: Figure 1 shows the training and validation loss curves for each of the models described in Section 3.

Original: The displayed epochs indicate where we stopped the training to obtain the models used to produce the ﬁnal results in Table 2 and 3.
Revised: The displayed epochs indicate where we stopped the training to obtain the models used to produce the ﬁnal results in Table 2 and 5.

Original: The training loss is higher than the validation loss in the pictures in
Revised: The training loss is higher than the validation loss in the pictures in

Original: Figure 1 because the loss against the training data is computed in an average way across batches (the batch size is 128) while dropout is being applied, whereas performance on the validation set is determined at the end of each epoch with dropout disabled.
Revised: Figure 1 because the loss against the training data is computed in an average way across batches (the batch size is 100) while dropout is being applied, whereas performance on the validation set is determined at the end of each epoch with dropout disabled.

Original: Figure 1(f) displays the loss curves for training a simple neural network consisting of only an embedding layer (see Listing 6).
Revised: Figure 1(f) displays the loss curves for training a simple neural network consisting of only an embedding layer.

Original: We include the performance of this network in our results as a baseline.
Revised: We include the performance of this network in our results as a baseline.

Original: main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) ﬂatten =

Original: Flatten () (embedding) output = Dense(1, activation =’sigmoid’)( ﬂatten ) model = Model(inputs

Original: =main input, outputs=output) print (model.summary()) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Original: Listing 6: Baseline Model with only Embedding Layer

Original: [Table]

Original: [Table]

Original: [Table] [Table] (10 Epochs) 97.27% The accuracy of each of the trained models when applied to the test data is recorded in Table 2.
Revised: The accuracy of each of the trained models when applied to the test data is recorded in Table 2.

Original: In addition to accuracy, this table includes the true positive rate (TPR) and false positive rate (FPR) for each of the models.
Revised: In addition to accuracy, this table includes the true positive rate (TPR) and false positive rate (FPR) for each of the models.

Original: Recall that TPR = TP/(TP+FN) and FPR = FP/(FP+TN) where TP, FP, TN, and
Revised: Recall that TPR = TP/(TP+FN) and FPR = FP/(FP+TN) where TP, FP, TN, and

Original: FN are the number of true positives, false positives, true negatives, and false negatives respectively.
Revised: FN are the number of true positives, false positives, true negatives, and false negatives respectively.

Original: A low false positive rate is very important in deployed DGA detection systems, because blocking legitimate trafﬁc is highly undesirable.
Revised: A low false positive rate is very important in deployed DGA detection systems, because blocking legitimate trafﬁc is highly undesirable.

Original: All classiﬁers in Table 2 output a probability that a given instance belongs to the positive class, so we can tune a threshold probability at which to consider a prediction positive.
Revised: All classiﬁers in Table 2 output a probability that a given instance belongs to the positive class, so we can tune a threshold probability at which to consider a prediction positive.

Original: For each model, we choose this threshold such that the model trained over the training data has a 0.001 FPR over the validation data.
Revised: For each model, we choose this threshold such that the model trained over the training data has a 0.001 FPR over the validation data.

Original: Then we report the accuracy, TPR and FPR obtained with this classiﬁcation threshold over the test data.
Revised: Then we report the accuracy, TPR and FPR obtained with this classiﬁcation threshold over the test data.

Original: Finally, we also report AUC@1%FPR, which is the integral of the ROC curve from FPR = 0 to FPR = 0.01 on the test data.
Revised: Finally, we also report AUC@1%FPR, which is the integral of the ROC curve from FPR = 0 to FPR = 0.01 on the test data.

Original: For comparison purposes, Table 2 also contains results for a Random Forest (RF) with 100 trees, trained on the following 11 features, extracted from each domain name string (see Yu et al.
Revised: For comparison purposes, Table 2 also contains results for a Random Forest (RF) and a Multilayer Perceptron (MLP) trained on the following 11 features, extracted from each domain name string (see Yu et al.

Original: (2014; 2016)): ent (normalized entropy of characters); nl2 (median of 2-gram);
Revised: (2014; 2016)): ent (normalized entropy of characters); nl2 (median of 2-gram);

Original: nl3 (median of 3-gram); naz (symbol character ratio); hex (hex character ratio); vwl (vowel character ratio); len (domain label length); gni (gini index of characters); cer (classiﬁcation error of characters); tld (top level domain hash); dgt (ﬁrst character digit). 
Revised: nl3 (median of 3-gram); naz (symbol character ratio); hex (hex character ratio); vwl (vowel character ratio); len (domain label length); gni (gini index of characters); cer (classiﬁcation error of characters); tld (top level domain hash); dgt (ﬁrst character digit). The Random Forest consists of 100 trees.

Revised: The MLP has a single hidden layer with 128 nodes; see Listing 7 in Appendix A. Like the deep networks in this paper, this MLP was trained with batch size 100.

Revised: The values of the 11 features are normalized so that they are all on the same scale before presenting them to the MLP.

Original: As expected, the FPR of all classiﬁers is around 0.001.
Revised: As expected, the FPR of all classiﬁers is around 0.001.

Original: There is a clear variation in the TPR that the classiﬁer achieve against that small FPR.
Revised: There is a clear variation in the TPR that the classiﬁers achieve against that small FPR.

Original: While the Random Forest is only able to “catch” 83% of the malicious domain names, all the deep neural network architectures achieve a recall of 9798%.
Revised: While the Random Forest is only able to “catch” 83% of the malicious domain names, all the deep neural network architectures achieve a recall of 9798%.

Original: The baseline neural network consisting of only an embedding layer as its hidden layer clearly performs the worst with a TPR of less than 69%, highlighting that it is advantageous to extend the network architecture with one or more LSTM or CNN layers.
Revised: The baseline neural network consisting of only an embedding layer as its hidden layer clearly performs the worst with a TPR of less than 69%, highlighting that it is advantageous to extend the network architecture with one or more LSTM or CNN layers.

Original: Interestingly, there is little to no variation among the ﬁve deep neural network architectures in terms of TPR.
Revised: Interestingly, there is little to no variation among the ﬁve deep neural network architectures in terms of TPR.

Original: As Table 3 shows there is a substantial distinction among the different models in terms of complexity (number of parameters that have to be learned during the training process) and required training time per epoch.

Original: The platform used for training is an AWS virtual machine with access to multiple GPUs.

Original: Table 4 contains examples of domain names that were randomly selected among those misclassiﬁed by either the Random Forest or by all ﬁve deep neural networks.
Revised: Table 3 contains examples of domain names that were randomly selected among those misclassiﬁed by either the Random Forest or by all ﬁve deep neural networks.

Original: Inspecting the column of the
Revised: Inspecting the column of the

Original: [Table]
Revised: [Table]

Original: benign domain names, i
Revised: benign domain names, i

Original: [Table]
Revised: [Table]

Original: e. the Alexa domain names, it is interesting to note that most of those misclassiﬁed by the deep neural networks (bottom left in the table) come across as gibberish that a human annotator would likely also classify as malicious.
Revised: e. the Alexa domain names, it is interesting to note that most of those misclassiﬁed by the deep neural networks (bottom left in the table) come across as gibberish that a human annotator would likely also classify as malicious.

Original: As also evident from the top right of
Revised: As also evident from the top right of

Original: Table 4, the deep neural networks have become very good at considering such gibberish-looking domain names to be malicious, even though they were never explicitly told to do so (unlike the
Revised: Table 3, the deep neural networks have become very good at considering such gibberish-looking domain names to be malicious, even though they were never explicitly told to do so (unlike the

Original: Random Forest, which explicitly includes a normalized entropy of characters feature).
Revised: Random Forest, which explicitly includes a normalized entropy of characters feature).

Original: The fact that the malicious domain names at the bottom right of Table 4 were missed by the deep neural networks might be due to our deliberate choice to tune the classiﬁcation threshold to achieve a very low FPR.
Revised: The fact that the malicious domain names at the bottom right of Table 3 were missed by the deep neural networks might be due to our deliberate choice to tune the classiﬁcation threshold to achieve a very low FPR.

Original: This makes all the classiﬁers hold back from labeling a domain name as malicious if they are not almost completely certain.
Revised: This makes all the classiﬁers hold back from labeling a domain name as malicious if they are not almost completely certain.

Original: As explained above, a low FPR is very important in deployed DGA detection systems, as blocking legitimate trafﬁc is highly undesirable.
Revised: As explained above, a low FPR is very important in deployed DGA detection systems, as blocking legitimate trafﬁc is highly undesirable.

Original: Note that if a deployed DGA detection system would rely on the Random Forest classiﬁer, it would block all domain names from the ﬁrst row in Table 4, whereas, if it would rely on any of the deep neural network classiﬁers, it would block all domain names from the second row in Table 4.
Revised: Note that if a deployed DGA detection system would rely on the Random Forest classiﬁer, it would block all domain names from the ﬁrst row in Table 3, whereas, if it would rely on any of the deep neural network classiﬁers, it would block all domain names from the second row in Table 3.

Original: The domain names from the ﬁrst column would have been unjustly blocked.
Revised: The domain names from the ﬁrst column would have been unjustly blocked.

Original: For those negatively affected by this, it would be easier to “understand” (and perhaps forgive) the decisions made by the deep neural network classiﬁers, as they are more in line with decisions that a human would make when confronted with these domain name strings.
Revised: For those negatively affected by this, it would be easier to “understand” (and perhaps forgive) the decisions made by the deep neural network classiﬁers, as they are more in line with decisions that a human would make when confronted with these domain name strings.

Revised: The results in Table 2 are for DGA domain names that appeared at the same time in real trafﬁc as the training data used to construct the classiﬁers, namely July 2017.

Revised: To evaluate how well the trained models hold up against DGA domain names that appear at a later point, we collected an additional 100K DGA domain names from the Bambenek Consulting feeds for Dec 6, Dec 9, and

Revised: Dec 10, 2017.

Revised: Table 4 contains results for this “prospective” dataset.

Revised: The only difference between the experimental setup for Table 2 and Table 4 are the 100K DGA domain names used in the test set.

Original: C ONCLUSION
Revised: The training dataset, the validation dataset, and the 100K Alexa domain names used in the test set are kept the same. As can be seen when comparing the results in Table 2 and Table 4, the to be expected drop in TPR is minor. As Table 5 shows there is a substantial distinction among the different models in terms of complexity (number of parameters that have to be learned during the training process) and required training time per epoch. The platform used for training is an AWS virtual machine with access to multiple GPUs.

Revised: Both the number of epochs needed to train a network, and the number of seconds required per epoch, are contributing factors to the overall training runtime.

Revised: The NYU model took less than 8 minutes to train, while the CMU model took as much as 10 hours.

Revised: The last column in Table 5 shows the time needed to classify 200K domain names with an already trained model.

Revised: The ranking of the deep networks in terms of this “scoring time” coincides with the ranking in terms of training time.

Revised: Given that all deep networks achieve a similar accuracy (TPR), the NYU model with it short training and scoring time comes out as the winner.

Original:  DGA detection, i.e. the classiﬁcation task of distinguishing between benign domain names and those generated by malware (Domain Generation Algorithms), has become a central topic in information security.
Revised: [Table] [Table]  C ONCLUSION DGA detection, i.e. the classiﬁcation task of distinguishing between benign domain names and those generated by malware (Domain Generation Algorithms), has become a central topic in information security.

Original: In this paper we have compared ﬁve different deep neural network architectures that perform this classiﬁcation task based purely on the domain name string, given as a raw input signal at character level.
Revised: In this paper we have compared ﬁve different deep neural network architectures that perform this classiﬁcation task based purely on the domain name string, given as a raw input signal at character level.

Original: All ﬁve models, i.e. two RNN based architectures, two CNN based architectures, and one hybrid RNN/CNN architecture perform equally well, catching around 97-98% of malicious domain names against a false positive rate of 0.001.
Revised: All ﬁve models, i.e. two RNN based architectures, two CNN based architectures, and one hybrid RNN/CNN architecture perform equally well, catching around 97-98% of malicious domain names against a false positive rate of 0.001.

Original: This roughly means that for every 970 malicious domain names that the deep networks catch, they ﬂag only one benign domain name erroneously as malicious.
Revised: This roughly means that for every 970 malicious domain names that the deep networks catch, they ﬂag only one benign domain name erroneously as malicious.

Original: A Random Forest based on human deﬁned linguistic features achieves a recall of only 83% against the same 0.001 false positive rate when trained and tested on the same data that was used for the deep networks.
Revised: A Random Forest based on human deﬁned linguistic features achieves a recall of only 83% against the same 0.001 false positive rate when trained and tested on the same data that was used for the deep networks.

Original: The use of a deep neural network that automatically learns features is attractive in a cybersecurity setting because it is a lot harder to craft malware to avoid detection by a system that relies on automatically learned features instead of on human engineered features.
Revised: The use of a deep neural network that automatically learns features is attractive in a cybersecurity setting because it is a lot harder to craft malware to avoid detection by a system that relies on automatically learned features instead of on human engineered features.

Original: An interesting direction for future work is to test the trained deep networks  on domain names generated by new and previously unseen malware families. 
Revised: An interesting direction for future work is to test the trained deep networks more extensively on domain names generated by new and previously unseen malware families. A K ERAS C ODE FOR D EEP N ETWORKS

Revised: main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) lstm

Revised: = LSTM(128, return sequences=False)(embedding) drop = Dropout(0.5)(lstm) output = Dense(1, activation =’sigmoid’)(drop) model =

Revised: Model(inputs

Revised: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Revised: Listing 1: Endgame model with single LSTM layer, adapted from Woodbridge et al.

Revised: (2016) main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) bi lstm =

Revised: Bidirectional ( layer=LSTM(64, return sequences=

Revised: False), merge mode=’concat’)(embedding) output = Dense(1, activation =’sigmoid’)( bi lstm ) model =

Revised: Model(inputs

Revised: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Revised: Listing 2: CMU model with bidirectional LSTM, adapted from Dhingra et al.

Original: {+(2016) main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) conv
+}
Revised: {+(2016) main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) conv
+}

Revised: =

Original: {+Conv1D( ﬁlters =128, kernel size =3, padding=’same’, strides =1)(embedding) thresh
+}
Revised: {+Conv1D( ﬁlters =128, kernel size =3, padding=’same’, strides =1)(embedding) thresh
+}

Revised: = ThresholdedReLU(1e − 6)(conv1) max pool1 =

Revised: MaxPooling1D(pool size=2, padding=’same’)(thresh1) conv2 =

Revised: Conv1D( ﬁlters =128, kernel size =2, padding=’same’, strides =1)(max pool1) thresh2 =

Revised: ThresholdedReLU(1e − 6)(conv2) max pool2 =

Revised: MaxPooling1D(pool size=2, padding=’same’)(thresh2) ﬂatten =

Revised: Flatten () (max pool2) fc = Dense(64)( ﬂatten ) thresh fc =

Revised: ThresholdedReLU(1e − 6)(fc) drop = Dropout(0.5)

Revised: ( thresh fc ) output = Dense(1, activation =’sigmoid’)(drop) model =

Revised: Model(inputs

Revised: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Revised: Listing 3: NYU model with stacked CNN layers, adapted from Zhang et al.

Revised: def getconvmodel(self , kernel size , ﬁlters ): model =

Revised: Sequential () model.add(

Revised: Conv1D( ﬁlters = ﬁlters , input shape =(128, 128), kernel size = kernel size , padding=’same’, activation =’relu ’, strides =1))

Revised: model.add(Lambda(lambda x: K.sum(x, axis=1), output shape=( ﬁlters , ))) model.add(Dropout(0.5)) returnmodel main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding =

Revised: Embedding(input dim=128, output dim=128, input length =75)(main input) conv1 =

Revised: getconvmodel(2, 256)(embedding) conv2 =

Revised: getconvmodel(3, 256)(embedding) conv3 = getconvmodel(4, 256)(embedding) conv4 =

Revised: getconvmodel(5, 256)(embedding) merged =

Revised: Concatenate() ([conv1, conv2, conv

Revised: , conv4]) middle = Dense(1024, activation =’relu ’)

Revised: (merged) middle = Dropout(0.5)(middle) middle = Dense(1024, activation =’relu ’)

Revised: (middle)

Revised: middle =

Revised: (0.5)(middle) output = Dense(1, activation =’

Revised: sigmoid’)(middle) model =

Revised: Model(inputs

Revised: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Revised: Listing 4: Invincea CNN model with parallel CNN layers, adapted from Saxe & Berlin (2017) main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding =

Revised: Embedding(input dim=128, output dim=128, input length =75)(main input) conv = Conv1D( ﬁlters =128, kernel size =3, padding=’same’, activation =’relu ’, strides =1)(embedding) max pool =

Revised: MaxPooling1D(pool size=2, padding=’same’)(conv) encode =

Revised: LSTM(64, return sequences=False)(max pool) output = Dense(1, activation =’sigmoid’)(encode) model =

Revised: Model(inputs

Revised: =main input, outputs=output) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Revised: Listing 5: MIT model with a stacked CNN and LSTM layer, adapted from Vosoughi et al.

Revised: (2016) main input = Input(shape=(75, ) , dtype=’int32 ’, name=’main input’) embedding = Embedding(input dim=128, output dim=128, input length =75)(main input) ﬂatten =

Revised: Flatten () (embedding) output = Dense(1, activation =’sigmoid’)( ﬂatten ) model = Model(inputs

Revised: =main input, outputs=output) print (model.summary()) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’) main input = Input(shape=(11, ) , name=’main input’)

Revised: dense

Revised: = Dense(128, activation =’relu ’)

Revised: (main input) output = Dense(1, activation =’sigmoid’)(dense) model =

Revised: Model(inputs

Revised: =main input, outputs=output) print (model.summary()) model.compile(loss=’ binary crossentropy ’, optimizer=’adam’)

Revised: Listing 7: MLP Model with 128 Nodes Dense Layer