Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit the abundance of patterns of code. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities. M. Allamanis et al. Research in this area has been dominated by the formal, or logico-deductive, approach. Practitioners of this approach hold that, since software is constructed in mathematically well-defined programming languages, software tools can be conceived in purely formal terms. The design of software tools is to be approached using formal methods of definition, abstraction, and deduction. Properties of tools thus built should be proven using rigorous proof techniques such as induction over discrete structures. This logico-deductive approach has tremendous appeal in programming languages research, as it holds the promise of proving facts and properties of the program. Many elegant and powerful abstractions, definitions, algorithms, and proof techniques have been developed, which have led to important practical tools for program verification, bug finding, and refactoring [24, 42, 45] . It should be emphasized that these are theory-first approaches. Software constructions are viewed primarily as mathematical objects, and when evaluating software tools built using this approach, the elegance and rigor of definitions, abstractions, and formal proofs-ofproperties are of dominant concern. The actual varieties of use of software constructs, in practice, become relevant later, in case studies, that typically accompany presentations in this line of work. Of late, another valuable resource has arisen: the large and growing body of successful, widely used, open source software systems. Open source software systems such as Linux, MySQL, Django, Ant, and OpenEJB have become ubiquitous. These systems publicly expose not just source code but also metadata concerning authorship, bug-fixes, and review processes. The scale of available data is massive: billions of tokens of code and millions of instances of metadata, such as changes, bug-fixes, and code reviews ("big code"). The availability of "big code" suggests a new, data-driven approach to developing software tools: Why not let the statistical distributional properties, estimated over large and representative software corpora, also influence the design of development tools? Thus rather than performing well in the worst case, or in case studies, our tools can perform well in most cases, thus delivering greater advantages in expectation. The appeal of this approach echoes that of earlier work in computer architecture: Amdahl's law [15] , for example, tells us to focus on the common case. This motivates a similar hope for development tools, that tools for software development and program analysis can be improved by focusing on the common cases using a fine-grained estimate of the statistical distribution of code. Essentially, the hope is that analyzing the text of thousands of well-written software projects can uncover patterns that partially characterize software that is reliable, easy to read, and easy to maintain. The promise and power of machine learning rests on its ability to generalize from examples and handle noise. To date, software engineering (SE) and programming languages (PL) research has largely focused on using machine-learning (ML) techniques as black boxes to replace heuristics and find features, sometimes without appreciating the subtleties of the assumptions these techniques make. A key contribution of this survey is to elucidate these assumptions and their consequences. Just as natural language processing (NLP) research changed focus from brittle rule-based expert systems that could not handle the diversity of real-life data to statistical methods [99] , SE/PL should make the same transition, augmenting traditional methods that consider only the formal structure of programs with information about the statistical properties of code. A Survey of Machine Learning for Big Code and Naturalness 81:3 a "bag of words" representation of code, modeling code using sequences, trees, and continuous representations. We describe a wide range of emerging applications, ranging from recommender systems, debugging, program analysis, and program synthesis. The large body of work on semantic parsing [138] is not the focus of this survey, but we include some methods that output code in general-purpose programming languages (rather than carefully crafted domain-specific languages). This review is structured as follows. We first discuss the different characteristics of natural language and source code to motivate the design decisions involved in machine-learning models of code (Section 3). We then introduce a taxonomy of probabilistic models of source code (Section 4). Then we describe the software engineering and programming language applications of probabilistic source code models (Section 5). Finally, we mention a few overlapping research areas (Section 7), and we discuss challenges and interesting future directions (Section 6). Related Reviews and Other Resources. There have been short reviews summarizing the progress and the vision of the research area, from both software engineering [52] and programming languages perspectives [28, 195] . However, none of these articles can be considered extensive literature reviews, which is the purpose of this work. Ernst [57] discusses promising areas of applying natural language processing to software development, including error messages, variable names, code comments, and user questions. Some resources, datasets, and code can be found at http://learnbigcode.github.io/. An online version of the work reviewed here-which we will keep up to date by accepting external contributions-can be found at https://ml4code.github.io. Many aspects of code, such as names, formatting, the lexical order of methods, and so on, have no impact on program semantics. This is precisely why we abstract them in most program analyses. But then, why should statistical properties of code matter at all? To explain this, we recently suggested a hypothesis, called the naturalness hypothesis. The inspiration for the naturalness hypothesis can be traced back to the "literate programming" concept of D. Knuth, which draws from the insight that programming is a form of human communication: "Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. . . " [105] . The naturalness hypothesis, then, holds that The naturalness hypothesis. Software is a form of human communication; software corpora have similar statistical properties to natural language corpora;and these properties can be exploited to build better software engineering tools. The exploitation of the statistics of human communication is a mature and effective technology, with numerous applications [99] . Large corpora of human communication, viz. natural language corpora, have been extensively studied, and highly refined statistical models of these corpora have been used to great effect in speech recognition, translation, error-correction, and so on. The naturalness hypothesis holds that, because coding is an act of communication, one might expect large code corpora to have rich patterns, similar to natural language, thus allowing software engineering tools to exploit probabilistic ML models. The first empirical evidence of this hypothesis, showing that models originally developed for natural language were surprisingly effective for source code, was presented by Hindle et al. [87, 88] . More evidence and numerous applications of this approach have followed, which are the subject of this review. The naturalness hypothesis, then, inspires the goal to apply machine-learning approaches to create probabilistic source code models that learn how developers naturally write and use code. These models can be used to augment existing tools with statistical information and enable new machine-learning-based software engineering tools, such as recommender systems and program 81:4 M. Allamanis et al. analyses. At a high level, statistical methods allow a system to make hypotheses, along with probabilistic confidence values, of what a developer might want to do next or what formal properties might be true of a chunk of code. Probabilistic methods also provide natural ways of learning correspondences between code and other types of documents, such as requirements, blog posts, comments, and so on-such correspondences will always be uncertain, because natural language is ambiguous, and so the quantitative measure of confidence provided by probabilities is especially natural. As we discuss in Section 5, one could go so far as to claim that almost every area of software engineering and programming language research has potential opportunities for exploiting statistical properties. Although the "naturalness hypothesis" may not seem surprising, one should appreciate the root cause of "naturalness." Naturalness of code seems to have a strong connection with the fact that developers prefer to write [5] and read [85] code that is conventional, idiomatic, and familiar, because it helps in understanding and maintaining software systems. Code that takes familiar forms is more transparent, in that its meaning is more readily apparent to an experienced reader. Thus, the naturalness hypothesis leads seamlessly to a "code predictability" notion, suggesting that code artifacts-from simple token sequences to formal verification statements-contain useful recurring and predictable patterns that can be exploited. "Naturalness" and "big code" should be viewed as instances of a more general concept that there is exploitable regularity across humanwritten code that can be "absorbed" and generalized by a learning component that can transfer its knowledge and probabilistically reason about new code. This article reviews the emerging area of machine-learning and statistical natural language processing methods applied to source code. We focus on probabilistic models of code, that is, methods that estimate a distribution over all possible source files. Machine learning in probabilistic models has seen wide application throughout artificial intelligence, including natural language processing, robotics, and computer vision, because of its ability to handle uncertainty and to learn in the face of noisy data. One might reasonably ask why it is necessary to handle uncertainty and noise in software development tools, when in many cases the program to be analyzed is known (there is no uncertainty about what the programmer has written) and is deterministic. In fact, there are several interesting motivations for incorporating probabilistic modeling into machine-learning methods for software development. First, probabilistic methods offer a principled method for handling uncertainty and fusing multiple, possibly ambiguous, sources of information. Second, probabilistic models provide a natural framework for connecting prior knowledge to data-providing a natural framework to design methods based on abstractions of statistical properties of code corpora. In particular, we often wish to infer relationships between source code and natural language text, such as comments, bug reports, requirements documents, documentation, search queries, and so on. Because natural language text is ambiguous, it is useful to quantify uncertainty in the correspondence between code and text. Finally, when predicting program properties, probabilities provide a way to relax strict requirements onsoundness: We can seek unsound methods that predict program properties based on statistical patterns in the code, using probabilities as a way to quantify the method's confidence in its predictions. Programming languages narrow the gap between computers and the human mind: They construct palatable abstractions out of a multitude of minute state transitions. Source code has two audiences and is inherently bimodal: It communicates along two channels, one with humans and one with computers. Humans must understand code to read and write it; computers must be able to execute it. The bimodality of code drives the similarities and differences between it and text. Below, we discuss these similarities and differences with forward pointers to how they have been exploited,
Allamanis et al. REF provide a comprehensive survey of recent advancements in techniques that adopt machine learning for source-code analysis.
A Survey of Machine Learning for Big Code and Naturalness
CSUR
Computer Science