Paper:
L E A R N I N G T O D E S I G N R N A

Review:
General comment
==============
The authors used policy gradient optimization for generating RNA sequences that fold into a target secondary structure, reporting clear accuracy and runtime improvements over the previous state-of-the-art. The authors used BOHR for optimizing hyper-parameters and present a new dataset for evaluating RNA design methods. The paper is well motivated and mostly clearly written. However, the methodological contributions are limited and I have some important concerns about their evaluation. Overall, I feel it’s a good paper for an ICLR workshop or biological journal if the authors address the outstanding comments.
Major comments
=============
1. The methodological contributions are limited. The authors used existing approaches (policy gradient optimization and BOHR for hyperparameter optimization) but do not report new methods, e.g. for sequence modeling. Performing hyper-parameter optimization is in my eyes not novel, but common practice in the field. It would me more informative if the authors compared reinforcement learning to other approaches for (conditional) sequence generations, e.g. RNNs, autoregressive models, VAEs, or GANs, which have been previously reported for biological sequence generation (e.g. http://arxiv.org/abs/1804.01694).
2. Did the authors split all three datasets (Eterna, Rfam-Taneda, Rfam-learn-test) into train, eval, and test set, trained their method on the training set, optimized hyper-parameters on the eval set, and measured generalization and runtime on the test set? This is not described clearly enough in section 5. I suggest to summarize the number of sequences for each dataset and split in a table.
3. Did the authors also optimize the most important hyperparameters of RL-LS and other methods? Otherwise it is unclear if the performance gain is due to hyperparameter optimization or the method itself.
4. The time measurement (x-axis figure 3) is unclear. Is it the time that methods were given to solve a particular target structure and does figure 3 show the average number of solved structures in the test for a the time shown on the x-axis? 
5. Were all methods compared on the same hardware (section 5; 20 cores; Broadwell E5-2630v4 2.2 GHz CPUs) and can they be parallelized over multiple CPU or GPU cores? This is essential for a fair runtime comparison.
6. The term ‘run’ (“unreliable outcomes in single runs”, section 4) is unclear. Is it a single sample from the model (one rollout), a particular hyperparameter configuration, or training the model once for a single target structure? This must be clarified for understanding the evaluation.
7. How does the accuracy and runtime or LEARNA scale depending on the sequence (structure) length?
8. How sensitive is the model performance depending on the context size k for representing the current state? Did the authors try to encode the entire target structure with, e.g. recurrent models, instead of using a window centered on the current position?
9. The authors should more clearly describe the local optimization step (section 3.1; reward). Were all nucleotides that differ mutated independently, or enumerated exhaustively? The latter would have a high runtime of O(3^d), where d is the number of nucleotides that differ. When do the authors start with the local optimization? 
Minor comments
=============
10. The authors should replace ‘450x’ faster in the abstract by ‘clearly’ faster since the evaluation does not show that LEARNA is 450x faster than all other methods.
11. Does “At its most basic form” (introduction) mean that alternative RNA nucleotides exist? If so, this should be cited.
12. The authors should more clearly motive in the introduction why they created a new dataset.
13. The authors should mention in section 2.1 that the dot-bracket notation is not the only notation for representing RNA structures (https://www.tbi.univie.ac.at/RNA/ViennaRNA/doc/html/rna_structure_notations.html).
14. The authors should define the hamming distance (section 2.1). Do other distance metrics exist?
15. For the Traveling Salesman Problem (section 2.2) should the reward be the *negative* tour length?
16. The authors should more clearly describe the embedding layer (section 4). Are nucleotides one-hot encoded or represented as integers (0, 1  for ‘(‘ and ‘.’)?