Paper:
Semantic parsing is the problem of deriving machine interpretable meaning representations from natural language utterances. Neural models with encoder-decoder architectures have recently achieved substantial improvements over traditional methods. Although neural semantic parsers appear to have relatively high recall using large beam sizes, there is room for improvement with respect to onebest precision. In this work, we propose a generator-reranker architecture for semantic parsing. The generator produces a list of potential candidates and the reranker, which consists of a pre-processing step for the candidates followed by a novel critic network, reranks these candidates based on the similarity between each candidate and the input sentence. We show the advantages of this approach along with how it improves the parsing performance through extensive analysis. We experiment our model on three semantic parsing datasets (GEO, ATIS, and OVERNIGHT). The overall architecture achieves the state-of-the-art results in all three datasets.

Review:
This paper proposes a reranking architecture with a LogicForm-to-NaturalLanguage preprocessing step for semantic parsing. The authors experiment their method on three datasets and get the state of the art results. 
The proposed method is natural. But using neural models to rank (or rerank) is a long-existing technique, regardless of the chosen parametrization of the reranking model. This paper chose BERT. See section-2.6 of this tutorial for more details about using neural models to rank: https://www.microsoft.com/en-us/research/uploads/prod/2017/06/INR-061-Mitra-neuralir-intro.pdf. 
Overall, I think the paper is not ready to publish for the following reasons.
1. The method relies much upon manual designs that seem hard to generalize. 
By converting the logic forms to natural languages, the authors can leverage paraphrase datasets and pre-train the critic as a paraphrase model. However, the way they convert the logic forms is different for each dataset and they have to manually design rules for each logic form. 
2. It is not clear how certain experimental designs were made. 
The authors chose to not rerank if the candidates' scores are too low or high but close. Such choice and associated thresholds seem arbitrary: how were they actually found out? Were they tuned on a development set? How does the method work if the candidate with the highest score is always picked: in the end, this is what the model is supposed to learn, correct? 
Other designs include beam size, whether or not to use a pretrained model, etc. How were such decisions made? Tuned on a development set? 
3. The results are not sound enough. Given the issued pointed out in 1 and 2, I am not sure if the results are really sound as the authors claimed. 
For example, what if the authors don’t use a LogicForm-to-NaturalLanguage conversion? What is the result if we directly learn to match input and logic forms? 
Moreover, the authors better answer questions in 2 so I can gauge if their hyper-parameters were chosen in the principled ways. Once those are answered, a significant test had better be done since the improvement seems small. 
4. Claiming Shaw et al. 2019 in table-3 as ``our methods’’ is wrong. It is clear that Shaw et al. (2019) didn't experiment on OVERNIGHT dataset, but setting up the baseline on a dataset should not be classified as ``our method’’. 
Moreover, I have some comments on the model and experiments. These are not weakness, but I think some work in this direction may help improve the paper. 
1. The model architecture should be better justified. In its current form, the two arguments (input query and output sequence translated from a logic form) are interchangeable. Why so? Why isn’t an asymmetric architecture more natural? How can the authors use a pair of logic forms as negative examples (in figure-2)? Why do the authors use the Quora dataset in particular? 
2. The error analysis might be better to be a bit more quantitative. Its current form doesn’t seem to give insight on how the proposed method really helps. What the authors can do is: you can sample some sentences from the test/development set and count how many comparative words are misused in the original model, among which how many are corrected by reranking.