pwc-archive/papers-with-abstracts huggingface dataset:
Automatically Learning Feature Crossing from Model Interpretation for Tabular Data
Authors: Zhaocheng Liu, Qiang Liu, Haoli Zhang
Date: 2019-09-25 00:00:00

Automatically feature generation is a major topic of automated machine learning. Among various feature generation approaches, feature crossing, which takes cross-product of sparse features, is a promising way to effectively capture the interactions among categorical features in tabular data. Previous works on feature crossing try to search in the set of all the possible cross feature fields. This is obviously not efficient when the size of original feature fields is large. Meanwhile, some deep learning-based methods combines deep neural networks and various interaction components. However, due to the existing of Deep Neural Networks (DNN), only a few cross features can be explicitly generated by the interaction components. Recently, piece-wise interpretation of DNN has been widely studied, and the piece-wise interpretations are usually inconsistent in different samples. Inspired by this, we give a definition of interpretation inconsistency in DNN, and propose a novel method called CrossGO, which selects useful cross features according to the interpretation inconsistency. The whole process of learning feature crossing can be done via simply training a DNN model and a logistic regression (LR) model. CrossGO can generate compact candidate set of cross feature fields, and promote the efficiency of searching. Extensive experiments have been conducted on several real-world datasets. Cross features generated by CrossGO can empower a simple LR model achieving approximate or even better performances comparing with complex DNN models.
Semi-Supervised Boosting via Self Labelling
Authors: Akul Goyal, Yang Liu
Date: 2019-09-25 00:00:00

Attention to semi-supervised learning grows in machine learning as the price to expertly label data increases. Like most previous works in the area, we focus on improving an algorithm's ability to discover the inherent property of the entire dataset from a few expertly labelled samples. In this paper we introduce Boosting via Self Labelling (BSL), a solution to semi-supervised boosting when there is only limited access to labelled instances. Our goal is to learn a classifier that is trained on a data set that is generated by combining the generalization of different algorithms which have been trained with a limited amount of supervised training samples. Our method builds upon a combination of several different components. First, an inference aided ensemble algorithm developed on a set of weak classifiers will offer the initial noisy labels. Second, an agreement based estimation approach will return the average error rates of the noisy labels. Third and finally, a noise-resistant boosting algorithm will train over the noisy labels and their error rates to describe the underlying structure as closely as possible. We provide both analytical justifications and experimental results to back the performance of our model. Based on several benchmark datasets, our results demonstrate that BSL is able to outperform state-of-the-art semi-supervised methods consistently, achieving over 90% test accuracy with only 10% of the data being labelled.
Depth creates no more spurious local minima in linear networks
Authors: Li Zhang
Date: 2019-09-25 00:00:00

We show that for any convex differentiable loss,  a deep linear network has no spurious local minima as long as it is true for the two layer case.  This reduction greatly simplifies the study on the existence of spurious local minima in deep linear networks. When applied to the quadratic loss, our result immediately implies the powerful result by Kawaguchi (2016). Further, with the recent work by Zhou& Liang (2018), we can remove all the assumptions in (Kawaguchi, 2016).  This property  holds  for  more  general  “multi-tower”  linear  networks  too.  Our  proof builds on the work in (Laurent & von Brecht, 2018) and develops a new perturbation argument to show that any spurious local minimum must have full rank, a structural property which can be useful more generally
Learning by shaking: Computing policy gradients by physical forward-propagation
Authors: Arash Mehrjou, Ashkan Soleymani, Stefan Bauer, Bernhard Schölkopf
Date: 2019-09-25 00:00:00
Tasks: Model-based Reinforcement Learning, Reinforcement Learning (RL)

Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation (shaking) of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.
Consistent Meta-Reinforcement Learning via Model Identification and Experience Relabeling
Authors: Russell Mendonca, Xinyang Geng, Chelsea Finn, Sergey Levine
Date: 2019-09-25 00:00:00
Tasks: Meta Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement learning algorithms can acquire policies for complex tasks automatically, however the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning has enabled agents to leverage prior experience to adapt quickly to new tasks, the performance of these methods depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data due to on-policy training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, even if policies and value functions cannot. These dynamics models can then be used to continue training policies for out-of-distribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.
Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over the Simplex
Authors: Yufei Cui, Wuguannan Yao, Qiao Li, Antoni Chan, Chun Jason Xue
Date: 2019-09-25 00:00:00
Tasks: Adversarial Attack, Bayesian Inference, Decision Making

Estimating the predictive uncertainty of a Bayesian learning model is critical in various decision-making problems, e.g., reinforcement learning, detecting adversarial attack, self-driving car. As the model posterior is almost always intractable, most efforts were made on finding an accurate approximation the true posterior. Even though a decent estimation of the model posterior is obtained, another approximation is required to compute the predictive distribution over the desired output. A common accurate solution is to use Monte Carlo (MC) integration. However, it needs to maintain a large number of samples, evaluate the model repeatedly and average multiple model outputs. In many real-world cases, this is computationally prohibitive. In this work, assuming that the exact posterior or a decent approximation is obtained, we propose a generic framework to approximate the output probability distribution induced by model posterior with a parameterized model and in an amortized fashion. The aim is to approximate the true uncertainty of a specific Bayesian model, meanwhile alleviating the heavy workload of MC integration at testing time. The proposed method is universally applicable to Bayesian classification models that allow for posterior sampling. Theoretically, we show that the idea of amortization incurs no additional costs on approximation performance. Empirical results validate the strong practical performance of our approach.
Trajectory growth through random deep ReLU networks
Authors: Ilan Price, Jared Tanner
Date: 2019-09-25 00:00:00

This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks,  which, among other things, is  one measure of the expressivity of deep networks. We generalise existing  results, providing an alternative, simpler method for lower bounding expected trajectory growth through random networks, for a more general class of weights distributions, including sparsely connected  networks. We illustrate this approach by deriving bounds for sparse-Gaussian, sparse-uniform, and sparse-discrete-valued random nets. We prove that trajectory growth can remain exponential in depth with these new distributions, including their sparse variants, with the sparsity parameter appearing in the base of the exponent.
Ellipsoidal Trust Region Methods for Neural Network Training
Authors: Leonard Adolphs, Jonas Kohler, Aurelien Lucchi
Date: 2019-09-25 00:00:00

We investigate the use of ellipsoidal trust region constraints for second-order optimization of neural networks. This approach can be seen as a higher-order counterpart of adaptive gradient methods, which we here show to be interpretable as first-order trust region methods with ellipsoidal constraints. In particular, we show that the preconditioning matrix used in RMSProp and Adam satisfies the necessary conditions for provable convergence of second-order trust region methods with standard worst-case complexities. Furthermore, we run experiments across different neural architectures and datasets to find that the ellipsoidal constraints constantly outperform their spherical counterpart both in terms of number of backpropagations and asymptotic loss value. Finally, we find comparable performance to state-of-the-art first-order methods in terms of backpropagations, but further advances in hardware are needed to render Newton methods competitive in terms of time.
Defensive Tensorization: Randomized Tensor Parametrization for Robust Neural Networks
Authors: Adrian Bulat, Jean Kossaifi, Sourav Bhattacharya, Yannis Panagakis, Georgios Tzimiropoulos, Nicholas D.  Lane, Maja Pantic
Date: 2019-09-25 00:00:00
Tasks: Adversarial Defense, Audio Classification, image-classification, Image Classification

As deep neural networks become widely adopted for solving most problems in computer vision and audio-understanding, there are rising concerns about their potential vulnerability. In particular, they are very sensitive to adversarial attacks, which manipulate the input to alter models' predictions. Despite large bodies of work to address this issue, the problem remains open.  In this paper, we propose defensive tensorization, a novel adversarial defense technique that leverages a latent high order factorization of the network. Randomization is applied in the latent subspace, therefore resulting in dense reconstructed weights, without the sparsity or perturbations typically induced by the randomization.
Our approach can be easily integrated with any arbitrary neural architecture and combined with techniques like adversarial training. We empirically demonstrate the effectiveness of our approach on standard image classification benchmarks. We further validate the generalizability of our approach across domains and low-precision architectures by considering an audio classification task and binary networks. In all cases, we demonstrate superior performance compared to prior works in the target scenario.
ADAPTIVE GENERATION OF PROGRAMMING PUZZLES
Authors: Ashwin Kalyan, Oleksandr Polozov, Adam Tauman Kalai
Date: 2019-09-25 00:00:00
Tasks: Program Synthesis

AI today is far from being able to write complex programs. What type of problems
would be best for computers to learn to program, and how should such problems
be generated? To answer the first question, we suggest programming puzzles as a
domain for teaching computers programming. A programming puzzle consists of a
short program for a Boolean function f(x) and the goal is, given the source code, to
find an input that makes f return True. Puzzles are objective in that one can easily
test the correctness of a given solution x by seeing whether it satisfies f, unlike the
most common representations for program synthesis: given input-output pairs or an
English problem description, the correctness of a given solution is not determined
and is debatable. To address the second question of automatic puzzle generation,
we suggest a GAN-like generation algorithm called “Troublemaker” which can
generate puzzles targeted at any given puzzle-solver. The main innovation is that it
adapts to one or more given puzzle-solvers: rather than generating a single dataset
of puzzles, Tro
Visual Interpretability Alone Helps Adversarial Robustness
Authors: Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Pin-Yu Chen, Shiyu Chang, Luca Daniel
Date: 2019-09-25 00:00:00
Tasks: Adversarial Robustness

Recent works have empirically shown that there exist   adversarial examples that can be hidden from neural network interpretability, and  interpretability is itself susceptible to adversarial attacks. In this paper, we theoretically show  that with the correct measurement of interpretation, it is actually difficult to hide adversarial examples, as confirmed by experiments on MNIST, CIFAR-10 and Restricted ImageNet. Spurred by that, we develop a novel defensive scheme built only on robust interpretation (without resorting to adversarial loss minimization). We show that our defense achieves similar classification robustness to state-of-the-art robust training methods while attaining higher interpretation robustness under various settings of adversarial attacks.
Linguistic Embeddings as a Common-Sense Knowledge Repository: Challenges and Opportunities
Authors: Nancy Fulda
Date: 2019-09-25 00:00:00
Tasks: Common Sense Reasoning, Machine Translation, Question Answering

Many applications of linguistic embedding models rely on their value as pre-trained inputs for end-to-end tasks such as dialog modeling, machine translation, or question answering. This position paper presents an alternate paradigm: Rather than using learned embeddings as input features, we instead treat them as a common-sense knowledge repository that can be queried via simple mathematical operations within the embedding space. We show how linear offsets can be used to (a) identify an object given its description, (b) discover relations of an object given its label, and (c) map free-form text to a set of action primitives. Our experiments provide a valuable proof of concept that language-informed common sense reasoning, or `reasoning in the linguistic domain', lies within the grasp of the research community. In order to attain this goal, however, we must reconsider the way neural embedding models are typically trained an evaluated. To that end, we also identify three empirically-motivated evaluation metrics for use in the training of future embedding models.
Starfire: Regularization-Free Adversarially-Robust Structured Sparse Training
Authors: Noah Gamboa, Kais Kudrolli, Anand Dhoot, Ardavan Pedram
Date: 2019-09-25 00:00:00

This paper studies structured sparse training of CNNs with a gradual pruning technique that leads to fixed, sparse weight matrices after a set number of epochs. We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization. The proposed training methodology explores several options for structured sparsity.

We study various tradeoffs with respect to pruning duration, learning-rate configuration, and the total length of training.
We show that our method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within a negligible <1% margin of accuracy loss. To make sure that this type of sparse training does not harm the robustness of the network, we also demonstrate how the network behaves in the presence of adversarial attacks.  Our results show that with 70% target sparsity, over 75% top-1 accuracy is achievable.
Pixel Co-Occurence Based Loss Metrics for Super Resolution Texture Recovery
Authors: Ying Da Wang, Pawel Swietojanski, Ryan T Armstrong, Peyman Mostaghimi
Date: 2019-09-25 00:00:00
Tasks: Image Super-Resolution, Super-Resolution

Single Image Super Resolution (SISR) has significantly improved with Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), often achieving order of magnitude better pixelwise accuracies (distortions) and state-of-the-art perceptual accuracy. Due to the stochastic nature of GAN reconstruction and the ill-posed nature of the problem, perceptual accuracy tends to correlate inversely with pixelwise accuracy which is especially detrimental to SISR, where preservation of original content is an objective. GAN stochastics can be guided by intermediate loss functions such as the VGG featurewise loss, but these features are typically derived from biased pre-trained networks. Similarly, measurements of perceptual quality such as the human Mean Opinion Score (MOS) and no-reference measures have issues with pre-trained bias. The spatial relationships between pixel values can be measured without bias using the Grey Level Co-occurence Matrix (GLCM), which was found to match the cardinality and comparative value of the MOS while reducing subjectivity and automating the analytical process. In this work, the GLCM is also directly used as a loss function to guide the generation of perceptually accurate images based on spatial collocation of pixel values. We compare GLCM based loss against scenarios where (1) no intermediate guiding loss function, and (2) the VGG feature function are used. Experimental validation is carried on X-ray images of rock samples, characterised by significant number of high frequency texture features. We find GLCM-based loss to result in images with higher pixelwise accuracy and better perceptual scores.
Towards Principled Objectives for Contrastive Disentanglement
Authors: Anwesa Choudhuri, Ashok Vardhan Makkuva, Ranvir Rana, Sewoong Oh, Girish Chowdhary, Alexander Schwing
Date: 2019-09-25 00:00:00
Tasks: Disentanglement

Unsupervised learning is an important tool that has received a significant amount of attention for decades. Its goal is `unsupervised recovery,' i.e., extracting salient factors/properties  from unlabeled data. Because of the challenges in defining salient properties, recently, `contrastive disentanglement' has gained popularity to discover the additional variations that are enhanced in one dataset relative to another. %In fact, contrastive disentanglement and unsupervised recovery are often combined in that we seek additional variations that exhibit salient factors/properties. 
Existing formulations have devised a variety of losses for this task. However, all present day methods exhibit two major shortcomings: (1) encodings for data that does not exhibit salient factors is not pushed to carry no signal; and (2) introduced losses are often hard to estimate and require additional trainable parameters. We present a new formulation for contrastive disentanglement which avoids both shortcomings by carefully formulating a probabilistic model and by using non-parametric yet easily computable metrics. We show on four challenging datasets that the proposed approach is able to better disentangle salient factors.
Probabilistic modeling the hidden layers of deep neural networks
Authors: Xinjie Lan, Kenneth E. Barner
Date: 2019-09-25 00:00:00
Tasks: valid

In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) cannot satisfy the i.i.d. prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs. Hence, the Gaussian Process cannot correctly explain all the hidden layers of DNNs. Alternatively, we introduce a novel probabilistic representation for the hidden layers of DNNs in two aspects: (i) a hidden layer formulates a Gibbs distribution, in which neurons define the energy function, and (ii) the connection between two adjacent layers can be modeled by a product of experts model. Based on the probabilistic representation, we demonstrate that the entire architecture of DNNs can be explained as a Bayesian hierarchical model. Moreover, the proposed probabilistic representation indicates that DNNs have explicit regularizations defined by the hidden layers serving as prior distributions. Based on the Bayesian explanation for the regularization of DNNs, we propose a novel regularization approach to improve the generalization performance of DNNs. Simulation results validate the proposed theories.
Empirical confidence estimates for classification by deep neural networks
Authors: Chris Finlay, Adam M. Oberman
Date: 2019-09-25 00:00:00
Tasks: Classification

How well can we estimate the probability that the classification predicted by a deep neural network is correct (or in the Top 5)?  It is well-known that the softmax values of the network are not estimates of the probabilities of class labels.  However, there is a misconception that these values are not informative.  We define the notion of implied loss and prove that if an uncertainty measure is an implied loss, then low uncertainty means high probability of correct (or Top-k) classification on the test set.   We demonstrate empirically that these values can be used to measure the confidence that the classification is correct.  Our method is simple to use on existing networks: we proposed confidence measures for Top-k which can be evaluated by binning values on the test set.
Biologically Plausible Neural Networks via Evolutionary Dynamics and Dopaminergic Plasticity
Authors: Sruthi Gorantla, Anand Louis, Christos H. Papadimitriou, Santosh Vempala, Naganand Yadati
Date: 2019-09-25 00:00:00
Proceeding: NeurIPS Workshop Neuro_AI 2019 12

Artificial neural networks (ANNs) lack in biological plausibility, chiefly because backpropagation requires a variant of plasticity (precise changes of the synaptic weights informed by neural events that occur downstream in the neural circuit) that is profoundly incompatible with the current understanding of the animal brain. Here we propose that backpropagation can happen in evolutionary time, instead of lifetime, in what we call neural net evolution (NNE). In NNE the weights of the links of the neural net are sparse linear functions of the animal's genes, where each gene has two alleles, 0 and 1. In each generation, a population is generated at random based on current allele frequencies, and it is tested in the learning task. The relative performance of the two alleles of each gene over the whole population is determined, and the allele frequencies are updated via the standard population genetics equations for the weak selection regime. We prove that, under assumptions, NNE succeeds in learning simple labeling functions with high probability, and with polynomially many generations and individuals per generation. We test the NNE concept, with only one hidden layer, on MNIST with encouraging results. Finally, we explore a further version of biologically plausible ANNs inspired by the recent discovery in animals of dopaminergic plasticity: the increase of the strength of a synapse that fired if dopamine was released soon after the firing.