From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Follow-up comments - part 3
comment: 
“We clearly agree that such information would be very informative. However, the page limit for an ICLR conference paper does not allow for reporting such an analysis. We are planning to report our analysis in further detail in a future journal paper.”

I disagree: A fair hyper-parameter comparison is crucial to compare different models and can be briefly described in few sentences. For a fair comparison, the authors should optimize the same hyper-parameters for Hilbert-CNN and baseline models (1D CNN, etc.). They should then describe how (hold-out validation?) and which hyper-parameters they optimized.

“Although we have not tried this yet, we do not see any principled obstacle to using longer sequences: this will just result in larger 2D images.”

The main limitation is memory. How much memory (in GB) does their model require for 0.5, 1.0, 1.5, 2.0 kbp sequences? Can such a model still be trained on a single GPU like Titan X?

“We added this information in Appendix C, to the extent that we have access to the models. “

Appendix C shows hyper-parameters for seq-CNN and bi-directional CNN, but not the number of TRAINABLE parameters. The authors should compare both the number of trainable parameters and memory (see comments part 2) of Hilbert-CNN, seq-CNN, and LSTM. Such a table is important since models of different capacity (trainable parameters) are likely to perform differently.

“While it is indeed interesting to compare Hilbert-CNNs to other CNN architectures on other tasks, we have to leave this to future work.”

The authors should at least compare Hilbert-CNN to a gapped-kmer SVM instead of SVM since it is a stronger baseline and commonly used for biological sequence classification tasks.

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Innovative but insufficiently evaluated
rating: 7: Good paper, accept
review: Dear editors,

the authors addressed all of my comments and clearly improved their manuscript over multiple iterations. I therefore increased my rating from ‘6: Marginally above acceptance threshold’ to ‘7: Good paper, accept’.
Please note that the authors made important edits to their manuscript after the ICLR deadline and could hence not upload their most current version, which you can from https://file.io/WIiEw9. If you decided to publish the manuscript, I hence highly suggest using this (https://file.io/WIiEw9) version.

Best,

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The authors present Hilbert-CNN, a convolutional neural network for DNA sequence classification. Unlike existing methods, their model does not use the raw one-dimensional (1D) DNA sequence as input, but two-dimensional (2D) images obtained by mapping sequences to images using spacing-filling Hilbert-Curves. They further present a model (Hilbert-CNN) that is explicitly designed for Hilbert-transformed DNA sequences. The authors show that their approach can increase classification accuracy and decrease training time when applied to predicting histone-modification marks and splice junctions. 

Major comments
=============
1. The motivation of transforming sequences into images is unclear and claimed benefits are not sufficiently supported by experiments. The essence of deep neural networks is to learn a hierarchy of features from the raw data instead of engineering features manually. Using space filling methods such as Hilbert-curves to transform (DNA) sequences into images can be considered as unnecessary feature-engineering. 

The authors claim that ‘CNNs have proven to be most powerful when operating on multi-dimensional input, such as in image classification’, which is wrong. Sequence-based convolutional and recurrent models have been successfully applied for modeling natural languages (translation, sentiment classification, …), acoustic signals (speech recognition, audio generation), or biological sequences (e.g. predicting various epigenetic marks from DNA as reviewed in Angermueller et al).    

They further claim that their method can ‘better take the spatial features of DNA sequences into account’ and  can better model ‘long-term interactions’ between distant regions. This is not obvious since Hilbert-curves map adjacent sequence characters to pixels that are close to each other as described by the authors, but distant characters to distant pixels. Hence, 2D CNN must be deep enough for modeling interactions between distant image features, in the same way as a 1D CNN.

Transforming sequences to images has several drawbacks. 1) Since the resulting images have a small width and height but many channels, existing 2D CNNs such as ResNet or Inception can not be applied, which also required the authors to design a specific model (Hilbert-CNN). 2) Hilbert-CNN requires more memory due to empty image regions. 3) Due to the high number of channels, convolutional filters have more parameters. 4) The sequence-to-image transformation makes model-interpretability hard, which is in particular important in biology. For example, motifs of the first convolutional layers can not be interpreted as sequence motifs (as described in Angermueller et al) and it is unclear how to analyze the influence of sequence characters using attention or gradient-based methods.

The authors should more clearly motivate their model in the introduction, tone-down the benefit of sequence-to-image transformations, and discuss drawbacks of their model. This requires major changes of introduction and discussion.

2. The authors should more clearly describe which and how they optimized hyper-parameters. The authors should optimize the most important hyper-parameters of their model (learning rate, batch size, weight decay, max vs. average pooling, ELU vs. ReLU, …) and baseline models on a holdout validation set. The authors should also report the validation accuracy for different sequence lengths, k-mer sizes, and space filling functions. Can their model be applied to longer sequences (>= 1kbp) which had been shown to improve performance (e.g. 10.1101/gr.200535.115)? Does Figure 4 show the performance on the training, validation, or test set?

3. It is unclear if the performance gain is due the proposed sequence-to-image transformation, or due to the proposed network architecture (Hilbert-CNN). It is also unclear if Hilbert-CNNs are applicable to DNA sequence classification tasks beyond predicting chromatin states and splice junctions. To address these points, the authors should compare Hilbert-CNN to models of the same capacity (number of parameters) and optimize hyper-parameters (k-mer size, convolutional filter size, learning rate, …) in the same way as they did for Hilbert-CNN. The authors should report the number of parameters of all models (Hilbert-CNN, Seq-CNN, 1D-sequence-CNN (Table 5), and LSTM (Table 6), …) in an additional table. The authors should also compare Hilbert-CNN to the DanQ architecture on predicting epigenetic markers using the same dataset as reported in the DanQ publication (DOI: 10.1093/nar/gkw226). The authors should also compare Hilbert-CNNs to gapped-kmer SVM, a shallow model that had been successfully applied for genomic prediction tasks.

4. The authors should report the AUC and area under precision-recall curve (APR) in additional to accuracy (ACC) in Table 3.

5. It is unclear how training time was measured for baseline models (Seq-CNN, LSTM, …). The authors should use the same early stopping criterion as they used for training Hilber-CNNs. The authors should also report the training time of SVM and gkm-SVM (see comment 3) in Table 3.


Minor comments
=============
1. The authors should avoid uninformative adjectives and clutter throughout the manuscript, for example ‘DNA is often perceived’, ‘Chromatin can assume’,  ‘enlightening’, ‘very’, ‘we first have to realize’, ‘do not mean much individually’, ‘very much like the tensor’, ‘full swing’, ‘in tight communication’, ‘two methods available in the literature’.

The authors should point out in section two that k-mers can be overlapping.

2. Section 2.1: One-hot vectors is not the only way for embedding words. The authors should also mention Glove and word2vec. Similar approaches had been applied to protein sequences (DOI: 10.1371/journal.pone.0141287)

3. The authors should more clearly describe how Hilbert-curves map sequences to images and how images are cropped. What does ‘that is constructed in a recursive manner’ mean? Simply cropping the upper half of Figure 1c would lead to two disjoint sequences. What is the order of Figure 1e?

4. The authors should consistently use ‘channels’ instead of ‘full vector of length’ to denote the dimensionality of image pixels.

5. The authors should use ‘Batch norm’ instead of ‘BN’ in Figure 2 for clarification.

6. Hilber-CNN is similar to ResNet (DOI: 10.1371/journal.pone.0141287), which consists of multiple ‘residual blocks’, where each block is a sequence of ‘residual units’. A ‘computational block’ in Hilbert-CNN contains two parallel ‘residual blocks’ (Figure 3) instead of a sequence of ‘residual units’. The authors should use ‘residual block’ instead of ‘computational block’, and ‘residual units’ as in the original ResNet publication. The authors should also motivate why two residual units/blocks are applied parallely instead of sequentially.

7. Caption table 1: the authors should clarify if ‘Output size’ is ‘height, width, channels’, and explain the notation in ‘Description’ (or refer to the text.)
confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: ICLR.cc/2018/Conference/Program_Chairs
title: ICLR 2018 Conference Acceptance Decision
comment: This paper addresses an important application in genomics, i.e. the prediction of chromatin structure from nucleotide sequences.  The authors develop a novel method for converting the nucleotide sequences to a 2D structure that allows a CNN to detect interactions between distant parts of the sequence.  The reviewers found the paper innovative, interesting and convincing.  Two reviewers gave a 7 and there was one 6.  The 6, however, indicated during rather lengthy discussion that they were willing to raise their scores if their comments were addressed.  Hopefully the authors will address these comments in the camera ready version.  Overall a solid application paper with novel insights and technical innovation.


decision: Accept (Poster)

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Final comments
comment: Thanks for your comprehensive reply.  I understand that there is no longer enough time to address all of my comments for ICLR but I suggest to address them in a future journal submission. However, I insist on addressing the following points before I will increase the rating of your submission:

* Describe in the discussion intended future work.
* Describe evaluation in appendix.
* Add table that compared the number of parameters and memory.
* Change caption of figure 2.

Below my reply to your comments from part 1 to part 4. ‘>>>’ highlight your original sentences.

-------------------------------------------------------------------------------------------------------------------------------------

Part 1
=====

>>> Concluding from your comment, we will, in the Discussion, insert an outlook type of sentence ‘Given that a substantial part of the improvements in performance rates are due to our novel architecture, we plan on investigating the details of how components of the architecture are intertwined with improvements in prediction performance in more detail.’ (as pointed out above, if you agree, we will add this sentence at an appropriate place in the Discussion if this is still possible)

I would highly appreciate if you could add a sentence in the discussion saying that you will investigate in the future which network components in combination with transforming sequences to images account for the observed improvements.

Part 2
=====

>>> As said above, the number of channels is identical for Hilbert-CNN, 1D-CNN (256, as outlined in the above response), which also applies for seq-CNN, because also Nguyen et al. use 4-mer type one-hot encoding. The LSTM approach uses 4-word embedding, which amounts to the equivalent of 256 channels. Again, we will include corresponding explanations at an appropriate place in the final version of the manuscript.

Thanks for clarifying. I was referring to the number of channels of Hilbert-CNN compared with a model trained on natural images with three channels. The number of parameters of Hilbert-CNN (width*height*channels) is also higher compared with a 1D-CNN (width*channels) since it uses 2D instead of 1D kernels. 

Part 3
=====

>>> We did that (and again we agree that this is very informative), please also see our initial comment that we have already run this analysis. However, we indeed were missing the space to describe it within the boundaries of the manuscript, so we restricted ourselves to explaining that we use 90% of the data for training, 5% for testing and 5% for validation.
>>> We can offer to prepare an additional section in the appendix, where we describe what we did.

Please also describe *which* hyper-parameters were optimized for the different models.

>>> We will try to add a sentence regarding that our approach is on a par with other approaches in terms of an increase in memory with respect to raising sequence length, while making use of the memory-vs-learning-time trade off by decreasing batch size might particularly favor our approach.
>>> We are also planning to discuss this in ultimate detail in a future journal version.

Please discuss the memory usage in the discussion. Since it has been shown that prediction performance often scales with the length of the input sequence (e.g. DOI: 10.1101/gr.200535.115), I suggest to investigate in a future journal version if this is also true for Hilbert-CNN. Please also discuss in a future journal version if Hilbert-CNN can be still trained on sequences longer than 1 kbp on a single desktop GPU for a batch size that is not too small for batch-normalization.

>>> Is there a misunderstanding? To our understanding, these tables describe the number of trainable parameters in detail. For example, in the first convolutional layer of seq-CNN, a filter of 7x1 with 60 channels is used, resulting in 7x1x60 parameters per filter. The size of the input sequence then determines the number of parameters. We will try and add a remark pointing this out more clearly.

It it true that the number of parameters can be inferred for seq-CNN and biLSTM from the tables in Appendix C. However, this is very cumbersome. Please add a table that explicitly compares the total number of parameters *and memory* for seq-CNN, biLSTM, and Hilbert-CNN.

>>> We absolutely agree that a comparison with gapped k-mer SVM’s makes a lot of sense. However, we have opted to not do this here and now for two reasons:

I can not follow reason two but I understand that there is no longer enough time to show the results for gapped k-mer SVMs. I highly suggest doing this for a future journal submission, which would clearly strengthen the results.


Part 4
=====
>>> Indeed, we overlooked this -- thanks for pointing out, we will adapt this before the manuscript goes in print, if possible.

Please correct the caption also for ICLR. This should not take longer than a few minutes!

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer2
title: interesting method
rating: 7: Good paper, accept
review: There are functional elements attached on the DNA sequence, such as transcription factors and different kinds of histones as stated in this ms. A hidden assumption is that the binding sites of these functional elements over the genome share some common features. It is therefore biologically interesting to predict if a new DNA sequence could be a binding site. Naturally this is is classification problem where the input is the DNA sequence and the output is whether the give sequence is a binding site.

This ms makes a novel way to transform the DNA sequence into a 3-dimensional tensor which could be easily utilised by CNN for images. The DNA sequence is first made into a a list of 4-mers. Then then each 4-mer is coded as a 4^4=256 dimensional vector. The order of the 4-mers is then coded into a image using Hilbert curve which presumably has nice properties to keep spatial information.

I am not familiar with neural networks and do not comment on the methods but rather from the application point of view. 

First to my best knowledge, it is still controversial if the binding sites of different histones carries special features. I mean it could be possible that the assumption I mentioned in the beginning may not hold for this special application, especially for human data. I feel this method is more suitable for transcription factor motif data. see https://www.nature.com/articles/nbt.3300

Second, the experiments data in 2005 is measured using microarray, which uses probes of 500bp long. But the whole binding site for a nucleosome (or histone complex) is 147bp, which is much shorter than the probe. Nowadays we have more accurate sequencing data for nucleosome (check https://www.ncbi.nlm.nih.gov/pubmed/26411474). I am not sure whether this result will generalised to some other similar dataset. 

Third, the results only list the accuracy, it will be interesting to see the proportion of false negatives.

In general I feel the transformation is quite useful, it nicely reserves the spatial information, also can be seen from the improved results over all datasets. The result, in my opinion, is not sufficient to support the assumption that we could predict the DNA structures solely base on the sequence.


confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer3
title: The idea of converting k-mer representations to 2D images using Hilbert curves is novel in application.
rating: 7: Good paper, accept
review: The authors of this manuscript transformed the k-mer representation of DNA fragments to a 2D image representation using the space-filling Hilbert curves for the classification of chromatin occupancy. In generally, this paper is easy to read. The components of the proposed model mainly include Hilbert curve theory and CNN which are existing technologies. But the authors make their combination useful in applications. Some specific comments are:

1. In page 5, I could not understand the formula d_kink < d_out. d_link ;
2. There may exist some new histone modification data that were captured by the next-generation sequencing (e.g. ChIP-seq) and are more accurate;  
3. It seems that the authors treat it as a two-class problem for each data set. It would be more useful in real applications if all the data sets are combined to form a multi-class problem.

confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: ~Akash_Singh1
title: On reproducing “Convolving DNA using two-dimensional Hilbert curve representations”[1]
comment: Hello authors,

We found this paper really interesting and therefore took this opportunity to fully understand your work by reproducing the results. Please refer to the link below to review the challenges that one can face while using your proposed paper for further innovations.
Link: 

https://drive.google.com/file/d/1ged4-Yxisl8HKltGEvBMf936BZCJ4SQZ/view?usp=sharing

Thanks and regards
Akash Singh
akash.singh@mail.mcgill.ca
(on behalf of team)

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Follow-up comments - part 2
comment: 
“We agree. The empty image regions indeed increase the required memory. However, the net effect is still that the computation is faster for the 2D representation than the 1D representation by approximately a factor of 2, as shown in Table 4; with the relatively small datasets (compared to eg Imagenet), the additional computation due to empty regions is at present not a bottleneck, which justifies our choices made.” 

Although Hilbert-CNN might be faster than 1D-CNNs, they might still require more memory due to empty image regions and the high number of channels in the first convolutional layers. The authors should therefore compare both the number of parameters (see following comment) and memory of Hilbert-CNN, seq-CNN, and LSTM.

“Yes, compared to the 1D sequence CNN this is correct. However, the use of large convolution filters already at an early stage allows for the detection of long-range interactions. The use of computation blocks (which have smaller filters) makes sure that the information found in these early layers does not vanish. Finally, by using pooling layers, we reduce the size of the layers before including a fully connected layer, resulting in a small input for these fully connected layers and thus a strong reduction in the number of parameters (an explanation on this strategy has been added in the paragraph starting with “One way to enable…” at the end of page 1). So overall, the number of parameters in our network is much lower than the current state of the art from Nguyen et al, and thus results in a huge reduction in training time.”

My comment was not about the width and height of convolutional filters to capture spatial dependencies, but the number of channels, which is large due to the one-hot encoding of kmers (4^4=256 for 4-mers; compared to 3 RGB channels). The authors should therefore compare both the number of parameters and the memory of Hilbert-CNN, seq-CNN, and LSTM.

“We agree, and we can now more clearly distinguish between the benefits of the novel elements we are suggesting. We have rewritten the introduction and discussion to provide a more balanced argument for both the representation method and the computational structure.”

I suggest removing ‘some’ in the last section of the discussion.

From: ICLR.cc/2018/Conference/Paper377/Authors
title: Differences between the two approaches
comment: Dear Akash and team,

Thank you for your interest in our paper, we appreciate your effort in reproducing the results.

We thoroughly read your report, and noted some differences between your implementation and ours which are likely to have caused the differences in results. First, the early stopping approach that is used differs: the use of GL0 stops the training process earlier than then the GL2 we used. Additionally, we used a combination of GL2 and No-improvement-in-N-steps. Second, the output is of a different form: instead of using 0/1 values for class prediction, we used one-hot vectors as the output. This generally improves prediction accuracy (see also https://stackoverflow.com/questions/17469835/why-does-one-hot-encoding-improve-machine-learning-performance). Third, we used self-defined droput and normalization which improves efficiency and accuracy. Finally, our k-mer representation is indeed different from the one you are using: the value of a k-mer is based on the occurrence of the subsequence in the dataset (higher occurence - lower number (Bojian: or higher?)). The latter however is probably not causing a significant difference, according to some tests we ran. We will add the missing information in the next version of our manuscript in as far as space allows.

We hope this answers your questions. If you have any further questions, feel free to contact us.

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Follow-up comments - part 4
comment: “We have adapted this .” (The authors should use ‘Batch norm’ instead of ‘BN’ in Figure 2 for clarification.)

The authors are using “Batch normalization” in Figure 3, but not figure 2. They should replace “BN” by “Batch-norm” or “Batch normalization” in figure caption 2.

From: ICLR.cc/2018/Conference/Paper377/Authors
title: Bigger 1D model improves while 2D Hilbert curves are faster and more consistent
comment: Dear reviewer,

Thank you very much for your comment. We feel that you have made an excellent point here regarding the filter-size difference between the 1D-sequence CNN and the Hilbert CNN.  Your suggestion has resulted in further insight into the particular advantages of the components of our approach, which, to our feeling, has led to non-negligible improvements of the manuscript. 

First, one comment on properties: while continuity and clustering properties are trivially preserved by keeping 1D sequence, the third property we are citing is not preserved by 1D sequence. Namely, different, separate subsequences going into the filter, are, on average, farthest apart within the sequence for Hilbert curves. So, beyond considering larger filter sizes, Hilbert curves can still have particular advantages because of mapping distal relationships (and are even optimal in that regard).

To investigate the interplay of the novelties we have been suggesting further, we ran the 1D-sequence CNN now with an identical number of parameters in all layers (e.g. a 7x7 convolution in the first layer of Hilbert-CNN corresponds to a 49x1). In addition, following a suggestion from one of the other reviewers, we computed recall, precision, area under precision-recall curve (AP) and AUC, where precision-recall and ROC curves were computed by filtering the softmax output. The corresponding additional results can be found in the updated version of the document, in tables 3, 4 and 5.  

We find that :
1. Both the 1D sequence input and the 2D image input significantly outperform the current state of the art (which, as we feel, justifies that our paper is discussed at a venue such as the ICLR).
2. We further observe that the standard deviation of the networks across different training runs is much smaller for the 2D (e.g. 7x7) Hilbert curve representation than for the naive 1D (e.g. 49x1) one. 
We find that the 2D representation yields improvement over the 1D sequence representation in terms of recall and precision, and, in terms of AP and AUC, even drastic improvements (raising performance by more than 5% on average over the 49x1 filters).

We conclude that:
1. In terms of basic performance measures such as accuracy, recall, precision, the architecture of our network (which in comparison to prior work avoids large layers to precede the fully connected layers) is the decisive factor.
2. Nevertheless, Hilbert curves enable 
a. More robust predictions, as indicated by the significantly lower variance in accuracy across different runs.
b. Filtering of the output for optimizing in terms of precision-recall - tradeoffs (which the 1D output does not reliably allow to do).

Overall, these findings suggest that there is particular advantage with respect to using 2D representations in the form of space filling (in particular Hilbert) curves, while there is also substantial power in the proposed network structure (which, as a novelty, more decidedly reduces numbers of hidden nodes in layers preceding the fully connected layers). 

We reworked the introduction and discussion to more properly reflect this finding and adapted the title accordingly.

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Follow-up comments - part 4-2
comment: Please correct the caption also for ICLR. This should not take longer than a few minutes!

From: (anonymous)
title: Validation of method
comment: Hi authors,

The usage of a Hilbert space filling curve for chromatin structure classification is not clearly motivated to me. If I understand the method correctly, the goal is to perform some deterministic mapping from the 1D structure to the 2D space filling curve in order to leverage the spatial structure of popular CNN methods. However, since the Hilbert space filling curve does not leverage any biologically motivated information to construct the 2D image, it is unclear how this would help. The Hilbert space filling curve has nice properties such as (continuity, clustering, etc), but the original 1D sequence also possesses all of these properties. It seems simply using a larger filter size on the 1D sequence should achieve similar results since the 1D sequence itself should be able to achieve the continuity, clustering, etc properties of the Hilbert space filling curve. 

In addition, as the below reviewers have mentioned, the success of CNNs are not specific to 2D images, and there is not really a theoretical argument for converting a 1D structure to a 2D structure. Am I missing something here regarding the problem of chromatin structure classification or the representational power of Hilbert space filling curves?

Furthermore, the results section has not fully convinced me that the Hilbert-CNN is more powerful than applying a direct 1D CNN on the sequence. The method compared against in Nguyen et al (2016) only uses a 7 x 1 filter size in the first layer while the Hilbert-CNN uses a 7x7 filter size in the first layer. Perhaps a more convincing comparison would be comparing against a method that uses the same number of parameters (i.e. a 1D method that uses a filter size of 49 x 1 in the first layer) to demonstrate the utility of converting a 1D sequence to a 2D image.

I would be very interested if the Hilbert space filling curve was indeed a superior representation than its 1D counterpart.

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Follow-up comments 
comment: The authors addressed most of my comments. However, I disagree with the authors on the comments below and part 2-4. Once the authors addressed also my remaining comments, I am willing to higher my rating from “6: Marginally above acceptance threshold” to “7: Good paper, accept”.

Follow-up comments
==================
“Concerning the claimed benefits, this is a good point. As pointed out above, we made further investigations into the possible particular benefits of the network and those of Hilbert curves. We found that a major part of the improved prediction accuracy is attributable to the network.”

This should be pointed out more clearly in the discussion. The authors should clearly discuss that they presented a) Hilbert curves for transforming sequences into images, and b) a new network architecture for sequence images. Since the observed improvement in performance is mainly due to the the network architecture (point b), same improvements might be possible by novel architectures for sequences.

“In general, we would argue that using space-filling curves is not feature-engineering, but rather a representation modification, since no features are explicitly defined.”

The same argument could be used for comparing (shallow) kernel-SVMs with deep networks. The point is that a deep network can learn to transform the input into a suitable representation itself. This is why ‘deep learning’ is a form of ‘representation learning’.

From: ICLR.cc/2018/Conference/Paper377/AnonReviewer1
title: Revised review
comment: Dear authors,

thanks for incorporating my final comments. I hence revised my review (see above) and informed the editors to use your most current version (https://file.io/WIiEw9). I suggest to not bold-print your last edits, and describe in Appendix D which hyper-parameters you optimized for seq-CNN and biLSTM.