Paper:
The source code of a program not only serves as a formal description of an executable task, but it also serves to communicate developer intent in a humanreadable form. To facilitate this, developers use meaningful identifier names and natural-language documentation. This makes it possible to successfully apply sequence-modeling approaches, shown to be effective in natural-language processing, to source code. A major advancement in natural-language understanding has been the use of pre-trained token embeddings; BERT and other works have further shown that pre-trained contextual embeddings can be extremely powerful and can be finetuned effectively for a variety of downstream supervised tasks. Inspired by these developments, we present the first attempt to replicate this success on source code. We curate a massive corpus of Python programs from GitHub to pre-train a BERT model, which we call Code Understanding BERT (CuBERT). We also pre-train Word2Vec embeddings on the same dataset. We create a benchmark of five classification tasks and compare finetuned CuBERT against sequence models trained with and without the Word2Vec embeddings. Our results show that CuBERT outperforms the baseline methods by a margin of 2.9-22%. We also show its superiority when finetuned with smaller datasets, and over fewer epochs. We further evaluate CuBERT's effectiveness on a joint classification, localization and repair task involving prediction of two pointers.

Review:
# Summary
This paper presents a BERT-inspired pretraining/finetuning setup for source code tasks. It collects a corpus of
unlabeled Python files for BERT pretraining, designs or adopts 5 tasks on established smaller-scale Python corpora, and
adjusts the BERT model to tokenize and encode source code snippets appropriately.
# Strengths
* The idea of applying the pretraining/finetuning paradigm to program analysis tasks makes sense, and has been
  informally attempted by multiple groups in the community in 2019. This is the first high-quality submission to a
  top-tier ML conference I've seen on the subject, though.
* The authors exercised commendable care and diligence in preparing the training data, adopting BERT to source code
  inputs, and ensuring correctness of the experimental setup. I appreciated all the provided details on tokenization
  (Section 3.3), deduplication (Sections 3.1-3.2), and task setup (Section 3.5). This should become a technical standard
  in the community.
* The paper is written clearly and concisely, and is generally a pleasure to read.
# Weaknesses
I have a gripe with the authors' choice to ignore program structure (e.g. abstract syntax trees) or features (e.g.
types) in their program representation. Without this extra information (easily available from a compiler/interpreter
API) the pipeline is not substantially different from the original NLP pipeline of BERT et al. The main program-related
representation insight comes in tokenization (Section 3.3) and the definition of "sentences". To repeat, I appreciate
the effort the authors put in making tokenization appropriate for BERT processing of source code, but this is a drop in
the bucket compared to the all the other program-related features the work is leaving off the table. Programs are not
natural language.
The argument that source code analysis would "pass on the burden ... to downstream tasks" (Page 3) is odd. First, most
downstream tasks of interest occur in the settings where this analysis is already available: IDEs, code review
assistants, linters, etc. Second, one often needs program analysis to even define downstream tasks in the first place --
for example, determining whether function arguments are swapped required detecting a function call and boundaries of its
arguments, thus parsing the program!
This work obtains (and nicely analyzes) impressive results obtained by applying CuBERT. However, it does not put the
results in context with prior work based on structured program representations. Without this, it is difficult to say
whether the improvement comes from pretraining or from the language model simply learning a better "parsed"
representation of an input program from all the unlabeled corpus. If it's the latter, one might argue that supplying the
model with structured program features explicitly might eliminate much of the need for the unlabeled corpus.
I personally think that there will still be a gap between pretraining and finetuning even with structured program
features simply due to the sheer volume of available data, which, as the authors showed, is crucial for good
generalization of Transformer. However, this still needs to be shown empirically.
The "Function-Docstring Mismatch" task, as presented, seems too easy. If the distractors (negative examples) are truly
chosen at random, most of them are going to use obviously different vocabulary from the original function signature (as
Figure 4 demonstrates). A well designed task would somehow bias the sampling toward subtle distractors such as `get` vs.
`set` docstrings, but this seems challenging.
This also explains why the task is not influenced as much by reduction of training data (Table 3).
The Next Sentence Prediction pretraining task, as adapted for CuBERT, seems too difficult, in contrast. If the paired
sentences (i.e. code lines) are chosen at random, the model would lack most of the context required to make a decision
about the logical relationship between them, such as which variables are defined and available in context, which
functionality is being implemented, etc. I wonder, can the authors experiment with pretraining CuBERT only with the
Masked Language Model task? Will it worsen the results substantially or at all?
# Questions
Section 3.2: "similar files according to the same similarity metric..."
What are these metrics?
What is the fraction of positive/negative examples in the constructed finetuning datasets?
What is the motivation for making Variable Misuse and Wrong Operator/Operand into a simple classification tasks instead
of the original (more useful) correction task?