Paper:
Current work on neural code synthesis consists of increasingly sophisticated architectures being trained on highly simplified domain-specific languages, using uniform sampling across program space of those languages for training. By comparison, program space for a C-like language is vast, and extremely sparsely populated in terms of 'useful' functionalities; this requires a far more intelligent approach to corpus generation for effective training. We use a genetic programming approach using an iteratively retrained discriminator to produce a population suitable as labelled training data for a neural code synthesis architecture. We demonstrate that use of a discriminator-based training corpus generator, trained using only unlabelled problem specifications in classic Programming-by-Example format, greatly improves network performance compared to current uniform sampling techniques.

Review:
# Summary
This paper describes an approach for generating synthetic training corpora for neural program synthesis systems. It
assumes a test set of tasks to be solved, and designates a special "discriminator" network to force a distribution of
I/O examples derived from randomly sampled programs to be as close as possible to the I/O examples supplied for the
desired programs. The system first randomly samples a set of programs, then retains only those that are "sufficiently
close" to the desired I/O examples, and mutates them to further refine this training corpus. The whole approach is
evaluated on a DSL of array-manipulating programs, assuming 40 reasonable test programs and randomly sampled I/O pairs.
# Strengths
I like the core idea of the paper: assuming some desired distribution of I/O examples, construct the training corpus of
random programs in such a way that its associated I/O examples approximate the desired distribution. I can see how it
could force the sampling process to discover useful program snippets for training. If evaluated properly, this work
might get accepted at a future conference.
# Weaknesses
I have some issues with the (a) evaluation, and (b) presentation of the work in its current state.
First, the evaluation process fixes a set of 40 "useful" programs along with their associated I/O pairs, uses these I/O
pairs to build the training corpus as described, and then tests the finally trained synthesizer on the **same** set.
While the test programs themselves are not are not used in corpus building, only their I/O pairs are, this is still
sufficient to significantly bias the training set toward the test set. To properly test the core idea above, I'd expect
_at least_ something like:
- Write down or collect a set of "useful" test programs. Ideally make sure they all use different compositions of the
  language operators but still cover the space of combinatorial operator combinations well.
- Split this set into "dev" and "test" randomly. Use the I/O pairs from the "dev" set to build the training corpus.
  Evaluate the final synthesizer on the test set.
- Ideally also evaluate the sensitivity of this process to the chosen dev/test split.
Second, the evaluation baselines for corpus generation are straightforward: random sampling and genetic programming. The
former is self-evidently strawman. The latter is interesting, but is effectively a different implementation of the same
core idea of the paper, with two changes: (a) hardcoded fitness function instead of a trained discriminator, (b)
tournament selection for population evolution instead of roulette.
In any case, neither of the baselines are truly independent strong alternatives. The authors remarked that they were
"unable re-use baselines from ... literature, as for example ... DeepCoder framework caused by the relative generality
of our programming language for synthesis", which is a confusing statement for two reasons:
1. The programming language considered is a simple DSL of array manipulations, not that different in spirit from the
   DeepCoder DSL, AlgoLisp [Polosukhin & Skidanov, 2018], or the list manipulations DSL of Ellis et al. [2018].
2. Regardless of the DSL, what is being compared are **corpus generation techniques, not synthesis algorithms**.
   DeepCoder is not a baseline for that, its corpus generation is not a contribution.
The most proper baseline would be the work of Shin et al. [2019], which the authors mention in Section 2. It was
evaluated on Karel, which leaves the authors the choice of either (a) additionally evaluate their technique on the
Karel DSL, or (b) adapt the method of Shin et al. by designing some salient variables for the DSL of this paper.
I would prefer (a), which would significantly strengthen the paper, but (b) is also acceptable assuming honest effort
in designing the salient variables to make the baseline strong.
Finally, the presentation of the technical sections is informal and often confusing. Section 3 required several passes
to understand the corpus generation process. The authors should introduce proper formalism for all the involved concepts
such as corpora, train/test programs, their associated I/O pair, both neural networks with their input-output signatures
and architecture, and so on. Most of these concepts have established mathematical notation in the literature which would
make Section 3 much easier to follow. In addition, Sections 3.3-3.5 need to be formalized in algorithmic pseudocode.
# Minor remarks
* Please use \citet and \citep appropriately: citations should be enclosed in parentheticals if they don't participate
  in a sentence.
* In your synthesizer network architecture, do you know the number of program output lines ahead of time?
* In this DSL, how many possibilities are there in total for an output line (including all its arguments)?
  Given the beam size of 1M (which is a lot!), I want to compare it with the total space (#possibilities × #lines).
# References
Ellis, K., Morales, L., Sablé-Meyer, M., Solar-Lezama, A., & Tenenbaum, J. (2018). Learning libraries of subroutines for
neurally–guided bayesian program induction. In Advances in Neural Information Processing Systems (pp. 7805-7815).
Polosukhin, I., & Skidanov, A. (2018). Neural program search: Solving programming tasks from description and examples.
arXiv preprint arXiv:1802.04335.
Shin, R., Kant, N., Gupta, K., Bender, C., Trabucco, B., Singh, R., & Song, D. (2019). Synthetic Datasets for Neural
Program Synthesis. In ICLR.