pwc-archive/papers-with-abstracts huggingface dataset:
Leveraging Class Hierarchy for Code Comprehension
Authors: Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Li, Ray Mooney, Milos Gligoric
Date: 2020-10-13 00:00:00
Proceeding: NeurIPS Workshop CAP 2020 12

Object-oriented programming languages enable a hierarchical class structure, which provides rich contextual information to guide code comprehension and synthesis. In this work, we propose the novel task of generating comments for overriding methods to facilitate code comprehension. To address this task, we formulate a deep learning framework which (1) exploits context from the comments of overridden methods and class names; (2) learns to generate comments in overriding methods that are more specific than those in the overridden methods; and (3) ensures that the generated comments are compatible with comments of overridden methods.
Oasis: ILP-Guided Synthesis of Loop Invariants
Authors: Sahil Bhatia, Saswat Padhi, Nagarajan Natarajan, Rahul Sharma, Prateek Jain
Date: 2020-10-13 00:00:00
Proceeding: NeurIPS Workshop CAP 2020 12

Automated synthesis of inductive invariants is an important problem in software verification. We propose a novel technique that is able to solve complex loop invariant synthesis problems involving large number of variables. We reduce the problem of synthesizing invariants to a set of integer linear programming (ILP) problems.  We instantiate our techniques in the tool Oasis that outperforms state-of-the-art systems on benchmarks from the invariant synthesis track of the Syntax Guided Synthesis competition.
On-the-Fly Adaptation of Source Code Models
Authors: Disha Shrivastava, Hugo Larochelle, Daniel Tarlow
Date: 2020-10-13 00:00:00
Proceeding: NeurIPS Workshop CAP 2020 12

The ability to adapt to unseen, local contexts is an important challenge that successful models of source code must overcome. One of the most popular approaches for the adaptation of such models is dynamic evaluation. With dynamic evaluation, when running a model on an unseen file, the model is updated immediately after having observed each token in that file. In this work, we propose instead to approach this problem in two steps: (a) We select targeted information (\textit{support tokens}) from the given context; (b) We use these support tokens to learn adapted parameters which are then used to predict the target hole. We refer to our proposed framework as Targeted Support Set Adaptation (TSSA).  We consider an evaluation setting that we call \textit{line-level maintenance}, designed to reflect the downstream task of code auto-completion in an IDE. We demonstrate improved performance in experiments on a large scale Java GitHub corpus, compared to other adaptation baselines including dynamic evaluation. Moreover, our analysis shows that, compared to a non-adaptive baseline, our approach improves performance on identifiers and literals by 44% and 19%, respectively.
Measuring few-shot extrapolation with program induction
Authors: Ferran Alet, Javier Lopez-Contreras, Joshua B. Tenenbaum, Tomas Perez, Leslie Pack Kaelbling
Date: 2020-10-13 00:00:00
Proceeding: NeurIPS Workshop CAP 2020 12
Tasks: Meta-Learning, Program induction

Neural networks are capable of learning complex functions, but still have problems generalizing from few examples and beyond their training distribution. Meta-learning provides a paradigm to train networks to learn from few examples, but it has been shown that its most popular benchmarks require very limited generalization capabilities. Program induction lies at the opposite end of the spectrum: programs are capable of extrapolating from very few examples, but we still do not know how to efficiently search for complex programs. We propose a common benchmark for both communities, measuring extrapolation from few examples coming from the execution of small programs. These are obtained by leveraging a C++ interpreter on codes from programming competitions; extracting small sub-codes with their corresponding input-output pairs. Statistical analysis and preliminary human experiments show the potential of this benchmark for enabling progress in few-shot extrapolation.
Automated Generation of Executable Cross-Language Background Knowledge
Authors: Anonymous
Date: 2020-10-13 00:00:00
Tasks: Program Synthesis

Certain   problems   require   specific   semanticknowledge  to  be  solved,  e.g.   the  transformation  “January→1”.Without  the  requireddomain  specific  knowledge,  a  program  synthesizer  can’t  generalize  towards  a  program  thatalso  learns  the  other  months  of  the  year.   Online repositories of code contain many of thesetransformations  and  they  can  be  used  to  improve the semantic strength of a synthesis technique.   By reusing previouslywritten code that contains parts of or the entiretransformation, these domain specific problemscan  now  be  solved  as  well.   This  backgroundknowledge, in the form of method declarations,is however not easily available in a uniform format.   The lack of formalization of this processresults in handcrafted crawling and parsing software  for  each  program  synthesis  technique,  ascan be found in recent works.   Moreover,  when  the  program  space  spans  over  multiple  programminglanguages these techniques have to be adapted bythe synthesis researchers for each separate language.  This work tries to formalize the processof generating executable background knowledgein multiple programming languages that can beused  in  different  synthesis  techniques.Suchframework would lower the amount of time spenton crawling, parsing and runtime environmentswithin future research projects.
Learning Joint Intensity in a Multivariate Poisson Process on Statistical Manifolds
Authors: Simon Luo, Feng Zhou, Lamiae Azizi, Mahito Sugiyama
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: Additive models

We show that generalized additive models (GAMs) can be treated via the log-linear model on a structured sample space, which has a well established information geometric background. Connecting GAMs with multivariate stochastic processes, we present the additive Poisson process (APP), a novel framework that can model the higher-order interaction effects of the intensity functions in stochastic processes using lower dimensional projections. Learning of the model is achieved via convex optimization, thanks to the dually flat statistical manifold generated by the log-linear model.
A Deep Architecture for Log-Linear Models
Authors: Simon Luo, Sally Cripps, Mahito Sugiyama
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: Deep Learning

We present a novel perspective on deep learning architectures using a partial order structure, which is naturally incorporated into the information geometric formulation of the log-linear model. Our formulation provides a different perspective of deep learning by realizing the bias and weights as different layers on our partial order structure. This formulation of the neural network does not require any gradients and can efficiently estimate the parameters using the EM algorithm.
Sample Space Truncation on Boltzmann Machines
Authors: Mahito Sugiyama, Koji Tsuda, Hiroyuki Nakahara
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12

We present a lightweight variant of Boltzmann machines via sample space truncation, called a truncated Boltzmann machine (TBM), which has not been investigated before while can be naturally introduced from the log-linear model viewpoint. TBMs can alleviate the massive computational cost of exact training of Boltzmann machines that requires exponential time evaluation of expected values and the partition function of the model distribution. To analyze the learnability of TBMs, we theoretically provide bias-variance decomposition of the log-linear model using dually flat structure of statistical manifolds.
From em-Projections to Variational Auto-Encoder
Authors: Tian Han, Jun Zhang, Ying Nian Wu
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12

This paper reviews the em-projections in information geometry and the recent understanding of variational auto-encoder, and explains that they share a common formulation as joint minimization of the Kullback-Leibler divergence between two manifolds of probability distributions, and the joint minimization can be implemented by alternating projections or alternating gradient descent.
Generalisation and the Geometry of Class Separability
Authors: Dominic Belcher, Adam Prugel-Bennett, Srinandan Dasmahapatra
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: Deep Learning

Recent results in deep learning show that considering only the capacity of machines does not adequately explain the generalisation performance we can observe. We propose that by considering the geometry of the data we can better explain generalisation achieved in deep learning. In particular we show that in classification the separability of the data can explain how good generalisation can be achieved in high dimensions. Further we show that layers within a CNNs sequentially increase the linear separability of data, and that the information these layers retain or discard can help explain why these models generalise.
Comparing Fisher Information Regularization with Distillation for DNN Quantization
Authors: Prad Kadambi, Karthikeyan Natesan Ramamurthy, Visar Berisha
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: Knowledge Distillation, Quantization

A large body of work addresses deep neural network (DNN) quantization and pruning to mitigate the high computational burden of deploying DNNs. We analyze two prominent classes of methods; the first class uses regularization based on the Fisher Information Matrix (FIM) of parameters, whereas the other uses a student-teacher paradigm, referred to as Knowledge Distillation (KD). The Fisher criterion can be interpreted as regularizing the network by penalizing the approximate KL-divergence (KLD) between the output of the original model and that of the quantized model. The KD approach bypasses the need to estimate the FIM and directly minimizes the KLD between the two models. We place these two approaches in a unified setting, and study their generalization characteristics using their loss landscapes. Using CIFAR-10 and CIFAR-100 datasets, we show that for higher temperatures, distillation produces wider minima in loss landscapes and yields higher accuracy than the Fisher criterion.
Noisy Neural Network Compression for Analog Storage Devices
Authors: Berivan Isik, Kristy Choi, Xin Zheng, H.-S. Philip Wong, Stefano Ermon, Tsachy Weissman, Armin Alaghi
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: Knowledge Distillation, Model Compression, Neural Network Compression

Efficient compression and storage of neural network (NN) parameters is critical for resource-constrained, downstream machine learning applications. Although several methods for NN compression have been developed, there has been considerably less work in the efficient storage of NN weights. While analog storage devices are promising alternatives to digital systems, the fact that they are noisy presents challenges for model compression as slight perturbations of the weights may significantly compromise the network’s overall performance. In this work, we study an analog NVM array fabricated in hardware (Phase Change Memory (PCM)) and develop a variety of robust coding strategies for NN weights that work well in practice. We demonstrate the efficacy of our approach on MNIST and CIFAR-10 datasets for pruning and knowledge distillation.
The Volume of Non-Restricted Boltzmann Machines and Their Double Descent Model Complexity
Authors: Prasad Cheema, Mahito Sugiyama
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12

The double descent risk phenomenon has received much interest in the machine learning and statistics community. Motivated through Rissanen's minimum description length (MDL) principle, and Amari's information geometry, we investigate how a double descent-like behavior may manifest by considering the $\log V$ modeling term - which is the logarithm of the model volume. In particular, the $\log V$ term will be studied for the general class of fully-observed statistical lattice models, of which Boltzmann machines form a subset. Ultimately, it is found that for such models the $\log V$ term can decrease with increasing model dimensionality, at a rate which appears to overwhelm the classically understood $\mathcal{O}(D)$ complexity terms of AIC and BIC. Our analysis aims to deepen the understanding of how the double descent behavior may arise in deep lattice structures, and by extension, why generalization error may not necessarily continue to grow with increasing model dimensionality.
AdaBelief Optimizer: Adapting Stepsizes by theBelief in Observed Gradients
Authors: Juntang Zhuang, Tommy Tang, Sekhar Tatikonda, Nicha C Dvornek, Yifan Ding, Xenophon Papademetris, James S Duncan
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: image-classification, Image Classification, Language Modeling, Language Modelling, Prediction

Optimization is at the core of modern deep learning. We propose AdaBelief optimizer to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction.
Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step.
We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer.
Sniper GMMs: Structured Gaussian mixtures poison ML on large n small p data with high efficacy
Authors: Anonymous
Date: 2020-10-19 00:00:00
Tasks: BIG-bench Machine Learning, Data Poisoning, Privacy Preserving

We propose a method for structured learning of Gaussian mixtures with low KL-divergence from target mixture models that in turn model the raw data. We show that samples from these structured distributions are highly effective and evasive in poisoning training datasets of popular machine learning training pipelines such as neural networks, XGBoost and random forests. Such attacks are especially destructive given the current uptrends towards distributed machine learning with several untrusted client devices that provide their data to servers and cloud service providers for privacy preserving distributed machine learning. In current day and
age of machine learning, Gaussian mixtures are perceived to be an older/classical technique in practice, although they are still actively studied from a theoretical perspective. Therefore it is quite interesting to see that they can be highly effective in performing data poisoning attacks on complex ML pipelines if learned with the right structural constraints.
Local Linear Approximation Algorithm for Neural Network
Authors: Anonymous
Date: 2020-10-19 00:00:00
Tasks: regression

This paper is concerned with estimation of weights and biases in feed forward neural network (FNN). We propose using local linear approximation (LLA) for the activation function, and develop a LLA algorithm to estimate the weights and biases of one hidden layer FNN by iteratively linear regression. We further propose the layerwise optimized adaptive neural network (LOAN), in which we use the LLA to estimate the weights and biases in the LOAN layer by layer adaptively. We compare the performance of the LOAN with the commonly-used procedures in deep learning via analyses of four benchmark data sets. The numerical comparison implies that the proposed LOAN may outperform the existing procedures.
Understanding The Role of Adversarial Regularization in Supervised Learning
Authors: Anonymous
Date: 2020-10-23 00:00:00

Despite numerous attempts sought to provide empirical evidence of adversarial regularization outperforming sole supervision in various inverse problems, the theoretical understanding of such phenomena remains elusive. In this study, we aim to resolve whether adversarial regularization indeed performs better than sole supervision at a fundamental level. To bring this insight into fruition, we study vanishing gradient issue, asymptotic iteration complexity, gradient flow and provable convergence guarantee in the context of sole supervision and adversarial regularization. The key ingredient is a theoretical justification supported by empirical evidence of adversarial acceleration in gradient descent. In addition, motivated by a recently introduced unit-wise capacity based generalization bound, we analyze the generalization error in adversarial framework. Guided by our observation, we cast doubts on the ability of this measure to explain generalization. We therefore leave as open questions to explore new measures that can explain generalization behavior in adversarial learning.
Stochastic Inverse Reinforcement Learning
Authors: Anonymous
Date: 2020-10-23 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The goal of the inverse reinforcement learning (IRL) problem is to recover the reward functions from expert demonstrations. However, the IRL problem like any ill-posed inverse problem suffers the congenital defect that the policy may be optimal for many reward functions, and expert demonstrations may be optimal for many policies. In this work, we generalize the IRL problem to a well-posed expectation optimization problem stochastic inverse reinforcement learning (SIRL) to recover the probability distribution over reward functions. We adopt the Monte Carlo expectation-maximization (MCEM) method to estimate the parameter of the probability distribution as the first solution to the SIRL problem. The solution is succinct, robust, and transferable for a learning task and can generate alternative solutions to the IRL problem. Through our formulation, it is possible to observe the intrinsic property for the IRL problem from a global viewpoint, and our approach achieves a considerable performance on the object world.
Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEs
Authors: Anonymous
Date: 2020-10-23 00:00:00
Tasks: input filtering

Event-based cameras are novel, efficient sensors inspired by the human vision
system, generating an asynchronous, pixel-wise stream of data. Learning from
such data is generally performed through event integration into images. This
requires buffering long sequences and can limit the response time of the inference
system. In this work, we propose to directly use events from a DVS camera, which
produces a stream of intensity changes and their spatial coordinates. This sequence
is used as an input for a novel asynchronous RNN-like architecture, the Input-
filtering Neural ODE (INODE). INODE allows for input signals to be continuously
fed to the network, as done for filtering dynamical systems. INODE learns to
discriminate short event sequences and to perform event-by-event online inference.
We demonstrate our approach on a series of classification tasks, comparing against
a set of LSTM baselines. We show that, independently of the camera resolution,
INODE can outperform the baselines by a large margin on the ASL task and it is
on par with a considerably larger LSTM for the NCALTECH task. Finally, we
show that INODE is accurate even when provided with very few events.
AlphaGAN: Fully Differentiable Architecture Searchfor Generative Adversarial Networks
Authors: Anonymous
Date: 2020-10-23 00:00:00
Tasks: GPU

Generative Adversarial Networks (GANs) are formulated as minimax game problems, whereby generators attempt to approach real data distributions by virtue of adversarial learning against discriminators. In this work, we aim to boost model learning from the perspective of network architectures, by incorporating recent progress on automated architecture search into GANs. To this end, we propose a fully differentiable search framework for generative adversarial networks, dubbed {\em alphaGAN}. The searching process is formalized as solving a bi-level minimax optimization problem, where the outer-level objective aims for seeking a suitable network architecture towards pure Nash Equilibrium conditioned on the network parameters optimized in the inner level. The entire optimization performs a first-order method by alternately optimizing the two-level objective in a fully differentiable manner, enabling architecture search to be completed in an enormous search space. Extensive experiments on CIFAR-10 and STL-10 datasets show that our algorithm can obtain high-performing architectures only with $3$-GPU hours on a single GPU in the search space comprised of approximate $2\times 10^{11}$ possible configurations.
Residual Network Based Direct Synthesis of EM Structures: A Study on One-to-One Transformers
Authors: Anonymous
Date: 2020-10-23 00:00:00

We propose using machine learning models for the direct synthesis of on-chip electromagnetic (EM) passive structures to enable rapid or even automated designs and optimizations of RF/mm-Wave circuits. As a proof of concept, we demonstrate the direct synthesis of a 1:1 transformer on a 45nm silicon on insulator (SOI) process using our proposed neural network model. Using pre-existing transformer s-parameter files and their geometric design training samples, the model efficiently predicts target geometric designs based on desired circuit specification.
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
Authors: Anonymous
Date: 2020-10-23 00:00:00
Tasks: Open-Ended Question Answering

A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size  and the inverse of the target error , deep neural networks learned by (stochastic) gradient descent enjoy nice optimization and generalization guarantees. Very recently, it is shown that under certain margin assumptions on the training data, a polylogarithmic width condition suffices for two-layer ReLU networks to converge and generalize (Ji and Telgarsky, 2020). However, whether deep neural networks can be learned with such a mild over-parameterization is still an open question. In this work, we answer this question affirmatively and establish sharper learning guarantees for deep ReLU networks trained by (stochastic) gradient descent. In specific, under certain assumptions made in previous work, our optimization and generalization guarantees hold with network width polylogarithmic in  and . Our results push the study of over-parameterized deep neural networks towards more practical settings.