Paper:
Learned representations of source code enable various software developer tools, e.g., to detect bugs or to predict program properties. At the core of code representations often are word embeddings of identifier names in source code, because identifiers account for the majority of source code vocabulary and convey important semantic information. Unfortunately, there currently is no generally accepted way of evaluating the quality of word embeddings of identifiers, and current evaluations are biased toward specific downstream tasks. This paper presents IdBench, the first benchmark for evaluating to what extent word embeddings of identifiers represent semantic relatedness and similarity. The benchmark is based on thousands of ratings gathered by surveying 500 software developers. We use IdBench to evaluate state-of-the-art embedding techniques proposed for natural language, an embedding technique specifically designed for source code, and lexical string distance functions, as these are often used in current developer tools. Our results show that the effectiveness of embeddings varies significantly across different embedding techniques and that the best available embeddings successfully represent semantic relatedness. On the downside, no existing embedding provides a satisfactory representation of semantic similarities, e.g., because embeddings consider identifiers with opposing meanings as similar, which may lead to fatal mistakes in downstream developer tools. IdBench provides a gold standard to guide the development of novel embeddings that address the current limitations.

Review:
The paper introduces a new dataset that includes manually labelled identifier name pairs. The labels determine how much the corresponding embeddings are useful for determining their meaning in context of code, a setting that has sufficient differences from a natural language.
A significant part of the paper is devoted on data cleaning and relating the computed metrics to similar efforts in natural language processing. While there is not much novelty in this part of the paper, it is doing a good job at addressing many possible questions on the validity of their dataset. Another important aspect that is covered by the work is different kinds of similarities of identifier names - similarity corresponding to having the same or similar type in a precise type system or similarity corresponding to being synonyms. Having several of these dimensions would make the results applicable for a wide range of applications of identifier name embeddings.
While not introducing new concepts, this paper is important for the community, because it has the potential to change the way embedding computation is done for “Big Code” problems. Right now, most papers either introduce their own embeddings, or use non-optimal ones like Code2Vec.
The paper also has a surprising finding that even techniques designed for code are in some cases not as good as the FastText embeddings. This is an interesting result, because few other works include this kind of embedding in their experiments. Furthermore, the paper deep dives into the strong and weak sides of several solutions and shows that there is a large opportunity to improve on the existing results.