pwc-archive/papers-with-abstracts huggingface dataset:
On the Effectiveness of Minimal Context Selection for Robust Question Answering
Authors: Anonymous
Date: 2018-10-30 00:00:00
Tasks: Adversarial Robustness, Question Answering, Sentence

Machine learning models for question-answering (QA), where given a question and a passage, the learner must select some span in the passage as an answer, are known to be brittle. By inserting a single nuisance sentence into the passage, an adversary can fool the model into selecting the wrong span. A promising new approach for QA decomposes the task into two stages: (i) select relevant sentences from the passage; and (ii) select a span among those sentences. Intuitively, if the sentence selector excludes the offending sentence, then the downstream span selector will be robust. While recent work has hinted at the potential robustness of two-stage QA, these methods have never, to our knowledge, been explicitly combined with adversarial training. This paper offers a thorough empirical investigation of adversarial robustness, demonstrating that although the two-stage approach lags behind single-stage span selection, adversarial training improves its performance significantly, leading to an improvement of over 22 points in F1 score over the adversarially-trained single-stage model.
Progressive Disentanglement Using Relevant Factor VAE
Authors: Anonymous
Date: 2019-09-04 00:00:00
Proceeding: NeurIPS Workshop DC_S1 2019 12
Tasks: Dimensionality Reduction, Disentanglement


Unsupervised learning of meaningful disentanglement remains an open challenge. This problem has roughly two perpendicular objectives: dimensionality reduction of high-dimensional data (such as images) to a low-dimensional latent space, and enforcing a disentanglement structure on the obtained latent space. It has been shown that improved performance in one can potentially hurt the other i.e. increased disentanglement can reduce reconstruction quality. Previous works have developed various reformulations to better decouple these objectives but there is always still a connection which often requires hyperparameter search / tuning to find the right trade-off.
In this work, we propose a systematic approach that automatically adapts the relative weights of both components to obtain the right trade-off.
Based on the Factor VAE approach, we can adaptively increase or decrease the weight of disentanglement objective as a function of the discriminator performance. This makes the unsupervised learning process insensitive to the initial choice of hyperparameters.
Our approach also enables a learning curriculum that initially places focus on the reconstruction and adaptively shifts emphasis to learning disentanglements.
Disentanglement with Hyperspherical Latent Spaces using Diffusion Variational Autoencoders
Authors: Anonymous
Date: 2019-09-04 00:00:00
Proceeding: NeurIPS Workshop DC_S1 2019 12
Tasks: Disentanglement

A disentangled representation of a data set should be capable of recovering the underlying factors that generated it. One question that arises is whether using Euclidean space for latent variable models can produce a disentangled representation when the underlying generating factors have a certain geometrical structure. Take for example the images of a car seen from different angles. The angle has a periodic structure but a 1-dimensional representation would fail to capture this topology. How can we address this problem? The submissions presented for the first stage of the  NeurIPS2019 Disentanglement Challenge consist of a Diffusion Variational Autoencoder ($\Delta$VAE) with a hyperspherical latent space which can for example recover periodic true factors. The training of the $\Delta$VAE is enhanced by incorporating a modified version of the Evidence Lower Bound (ELBO) for tailoring the encoding capacity of the posterior approximate.
Neural Disentanglement using Mixture Latent Space with Continuous and Discrete Variables
Authors: Anonymous
Date: 2019-09-04 00:00:00
Proceeding: NeurIPS Workshop DC_S1 2019 12
Tasks: Disentanglement

Recent advances in deep learning techniques has shown the usefulness of the deep neural networks in extracting features required to perform the task at hand.
However, these features learnt are in particular helpful only for the initial task. This is due to the fact that the features learnt are very task specific and does not capture the most general and task agnostic features of the input.
In fact the way humans are seen to learn is by disentangling features which task agnostic. This indicates that leaning task agnostic features by disentangling only the most informative features from the input data.
Recently Variational Auto-Encoders (VAEs) have shown to be the de-facto models to capture the latent variables in a generative sense.
As these latent features can be represented as continuous and/or discrete variables, this indicates us to use VAE with a mixture of continuous and discrete variables for the latent space.
We achieve this by performing our experiments using a modified version of joint-vae to learn the disentangled features.
Differentiable Disentanglement Filter: an Application Agnostic Core Concept Discovery Probe
Authors: Anonymous
Date: 2019-09-04 00:00:00
Proceeding: NeurIPS Workshop DC_S1 2019 12
Tasks: Disentanglement, Visual Grounding, Word Embeddings

It has long been speculated that deep neural networks function by discovering a hierarchical set of domain-specific core concepts or patterns, which are further combined to recognize even more elaborate concepts for the classification or other machine learning tasks. Meanwhile disentangling the actual core concepts engrained in the word embeddings (like word2vec or BERT) or deep convolutional image recognition neural networks (like PG-GAN) is difficult and some success there has been achieved only recently. In this paper we propose a novel neural network nonlinearity named Differentiable Disentanglement Filter (DDF) which can be transparently inserted into any existing neural network layer to automatically disentangle the core concepts used by that layer. The DDF probe is inspired by the obscure properties of the hyper-dimensional computing theory. The DDF proof-of-concept implementation is shown to disentangle concepts within the neural 3D scene representation - a task vital for visual grounding of natural language narratives.
Improved Disentanglement through Aggregated Convolutional Feature Maps
Authors: Anonymous
Date: 2019-09-04 00:00:00
Proceeding: NeurIPS Workshop DC_S1 2019 12
Tasks: Disentanglement

We present a simple image preprocessing method for training VAEs leading to improved disentanglement compared to directly using the images. In particular, we propose to use regionally aggregated feature maps extracted from CNNs pretrained on ImageNet. Our method achieves the first rank on 3 of 5 metrics on the challenge’s public leaderboard.
A Preliminary Study of Disentanglement With Insights on the Inadequacy of Metrics
Authors: Anonymous
Date: 2019-09-04 00:00:00
Proceeding: NeurIPS Workshop DC_S1 2019 12
Tasks: Disentanglement, Representation Learning

Disentangled encoding is an important step towards a better representation learning. However, despite the numerous efforts, there still is no clear winner that captures the independent features of the data in an unsupervised fashion. In this work we empirically evaluate the performance of six unsupervised disentanglement approaches on the mpi3d toy dataset curated and released for the NeurIPS 2019 Disentanglement Challenge. The methods investigated in this work are Beta-VAE, Factor-VAE, DIP-I-VAE, DIP-II-VAE, Info-VAE, and Beta-TCVAE. The capacities of all models were progressively increased throughout the training and the hyper-parameters were kept intact across experiments. The methods were evaluated based on five disentanglement metrics, namely, DCI, Factor-VAE, IRS, MIG, and SAP-Score. Within the limitations of this study, the Beta-TCVAE approach was found to outperform its alternatives with respect to the normalized sum of metrics. However, a qualitative study of the encoded latents reveal that there is not a consistent correlation between the reported metrics and the disentanglement potential of the model.
PVAE: Learning Disentangled Representations with Intrinsic Dimension via Approximated L0 Regularization
Authors: Anonymous
Date: 2019-11-15 00:00:00
Proceeding: NeurIPS Workshop DC_S2 2019 12
Tasks: Disentanglement

Many models based on the Variational Autoencoder are proposed to achieve disentangled latent variables in inference. However, most current work is focusing on designing powerful disentangling regularizers, while the given number of dimensions for the latent representation at initialization could severely inﬂuence the disentanglement. Thus, a pruning mechanism is introduced, aiming at automatically seeking for the intrinsic dimension of the data while promoting disentangled representations. The proposed method is validated on MPI3D and MNIST to be advancing state-of-the-art methods in disentanglement, reconstruction, and robustness. The code is provided on the https://github.com/WeyShi/FYP-of-Disentanglement.
Variational Learning with Disentanglement-PyTorch
Authors: Anonymous
Date: 2019-11-15 00:00:00
Proceeding: NeurIPS Workshop DC_S2 2019 12
Tasks: Disentanglement, Scheduling

Unsupervised learning of disentangled representations is an open problem in machine learning. The Disentanglement-PyTorch library is developed to facilitate research, implementation, and testing of new variational algorithms. In this modular library, neural architectures, dimensionality of the latent space, and the training algorithms are fully decoupled, allowing for independent and consistent experiments across variational methods. The library handles the training scheduling, logging, and visualizations of reconstructions and latent space traversals. It also evaluates the encodings based on various disentanglement metrics. The library, so far, includes implementations of the following unsupervised algorithms VAE, Beta-VAE, Factor-VAE, DIP-I-VAE, DIP-II-VAE, Info-VAE, and Beta-TCVAE, as well as conditional approaches such as CVAE and IFCVAE. The library is compatible with the Disentanglement Challenge of NeurIPS 2019, hosted on AICrowd and was used to compete in the first and second stages of the challenge, where it was ranked among the best few participants.
Improved Disentanglement through Learned Aggregation of Convolutional Feature Maps
Authors: Anonymous
Date: 2019-11-15 00:00:00
Proceeding: NeurIPS Workshop DC_S2 2019 12
Tasks: Disentanglement, Inductive Bias, Position

We present and discuss a simple image preprocessing method for learning disentangled latent factors. 
In particular, we utilize the implicit inductive bias contained in features from networks pretrained on the ImageNet database. 
We enhance this bias by explicitly fine-tuning such pretrained networks on tasks useful for the NeurIPS2019 disentanglement challenge, such as angle and position estimation or color classification.
Furthermore, we train a VAE on regionally aggregate feature maps, and discuss its disentanglement performance using metrics proposed in recent literature.
Disentanglement Challenge: From Regularization to Reconstruction
Authors: Anonymous
Date: 2019-11-15 00:00:00
Proceeding: NeurIPS Workshop DC_S2 2019 12
Tasks: Disentanglement

The challenge of learning disentangled representation has recently attracted much attention and boils down to a competition. Various methods based on variational auto-encoder have been proposed to solve this problem, by enforcing the independence between the representation and modifying the regularization term in the variational lower bound. However recent work by Locatello et al. (2018) has demonstrated that the proposed methods are heavily influenced by randomness and the choice of the hyper-parameter. This work is built upon the same framework in Stage 1 (Li et al., 2019), but with different settings; to make it self-contained, we provide this manuscript, which is unavoidably very similar to the report for Stage 1. In detail, in this work, instead of designing a new regularization term, we adopt the FactorVAE but improve the reconstruction performance and increase the capacity of network and the training step. The strategy turns out to be very effective in achieving disentanglement.
$C^\infty$ Smooth Algorithmic Neural Networks for Solving Inverse Problems
Authors: Felix Petersen, Christian Borgelt, Oliver Deussen
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12

Artificial neural networks revolutionized many areas of computer science in recent years since they provide solutions to a number of previously unsolved problems.
On the other hand, for many problems, classic algorithms exist, which typically exceed the accuracy and stability of neural networks.
To combine these two concepts, we present a new kind of neural networks—algorithmic neural networks.
These networks integrate smooth versions of classic algorithms into the topology of neural networks.
Our novel reconstructive adversarial network (RAN) enables solving inverse problems without or with only weak supervision.
Phase Retrieval using Untrained Neural Network Priors
Authors: Gauri Jagatap, Chinmay Hegde
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12
Tasks: Compressive Sensing, Denoising, Retrieval, Super-Resolution

Untrained deep neural networks as image priors have been recently introduced for linear inverse imaging problems such as denoising, super-resolution,  inpainting and compressive sensing with promising performance gains over hand-crafted image priors such as sparsity. Moreover, unlike learned generative priors they do not require any training over large datasets.  In this paper, we consider the problem of solving the non-linear inverse problem of compressive phase retrieval; this involves reconstructing a $d$-dimensional image signal from $n$ magnitude-only measurements, and $n<d$. We model images to lie in the range of an untrained deep generative network with a fixed seed. We further present two approaches for solving this problem: vanilla gradient descent and a projected gradient descent scheme and show superior empirical performance when compared to algorithms that use hand crafted priors.
Generative Inpainting Network Applications on Seismic Image Compression and Non-Uniform Sampling
Authors: Xiaoyang Rebecca Li, Nikolaos Mitsakos, Ping Lu, Yuan Xiao, Xing Zhao
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12
Tasks: Compressive Sensing, Generative Adversarial Network, Image Compression, Survey

The use of deep learning models as priors for compressive sensing tasks presents new potential for inexpensive seismic data acquisition. An appropriately designed Wasserstein generative adversarial network is designed based on a generative adversarial network architecture trained on several historical surveys, capable of learning the statistical properties of the seismic wavelets. The usage of validating and performance testing of compressive sensing are three steps. First, the existence of a sparse representation with different compression rates for seismic surveys is studied. Then, non-uniform samplings are studied, using the proposed methodology. Finally, recommendations for non-uniform seismic survey grid, based on the evaluation of reconstructed seismic images and metrics, is proposed. The primary goal of the proposed deep learning model is to provide the foundations of an optimal design for seismic acquisition, with less loss in imaging quality. Along these lines, a compressive sensing design of a non-uniform grid over an asset in Gulf of Mexico, versus a traditional seismic survey grid which collects data uniformly at every few feet, is suggested, leveraging the proposed method.
Robust One-Bit Recovery via ReLU Generative Networks: Improved Statistical Rate and Global Landscape Analysis
Authors: Shuang Qiu, Xiaohan Wei, Zhuoran Yang
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12
Tasks: compressed sensing

We study the robust one-bit compressed sensing problem whose goal is to design an algorithm that faithfully recovers any sparse target vector
$\theta_0\in\mathbb{R}^d$ \emph{uniformly} from $m$ quantized noisy measurements. Under the assumption that the measurements are sub-Gaussian,  to recover any $k$-sparse $\theta_0$ ($k\ll d$) \emph{uniformly} up to an error $\varepsilon$ with high probability, the best known computationally tractable algorithm requires\footnote{Here, an algorithm is ``computationally tractable'' if it has provable convergence guarantees. The notation $\tilde{\mathcal{O}}(\cdot)$ omits a logarithm factor of $\varepsilon^{-1}$.} $m\geq\tilde{\mathcal{O}}(k\log d/\varepsilon^4)$. In this paper, we consider a new framework for the one-bit sensing problem where the sparsity is implicitly enforced via mapping a low dimensional representation $x_0$ through a known $n$-layer ReLU generative network $G:\mathbb{R}^k\rightarrow\mathbb{R}^d$. Such a framework poses low-dimensional priors on $\theta_0$ without a known basis. We propose to recover the target $G(x_0)$ via an unconstrained empirical risk minimization (ERM) problem under a much weaker \emph{sub-exponential  measurement assumption}.  For such a problem, we establish a joint statistical and computational analysis. In particular, we prove that the ERM estimator in this new framework achieves an improved statistical rate of $m=\tilde{\mathcal{O}} (kn\log d /\epsilon^2)$ recovering any $G(x_0)$ uniformly up to an error $\varepsilon$. Moreover, from the lens of computation, we prove that under proper conditions on the ReLU weights, our proposed empirical risk, despite non-convexity, has no stationary point outside of small neighborhoods around the true representation $x_0$ and its negative multiple. Furthermore, we show that the global minimizer of the empirical risk stays within the neighborhood around $x_0$ rather than its negative multiple. Our analysis sheds some light on the possibility of inverting a deep generative model under partial and quantized measurements, complementing the recent success of using deep generative models for inverse problems.
Learning to Solve Linear Inverse Problems in Imaging with Neumann Networks
Authors: Greg Ongie, Davis Gilton, Rebecca Willett
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12

Recent advances have illustrated that it is often possible to learn to solve linear inverse problems in imaging using training data that can outperform more traditional regularized least squares solutions. Along these lines, we present some extensions of the Neumann network, a recently introduced end-to-end learned architecture inspired by a truncated Neumann series expansion of the solution map to a regularized least squares problem. Here we summarize the Neumann network approach, and show that it has a form compatible with the optimal reconstruction function for a given inverse problem. We also investigate an extension of the Neumann network that incorporates a more sample efficient patch-based regularization approach.
Auto-encoders for compressed sensing
Authors: Pei Peng, Shirin Jalali, Xin Yuan
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12
Tasks: compressed sensing

Compressed sensing is about recovering a structured high-dimensional signal  ${\bf x}\in R^n$ from its under-determined noisy linear measurements ${\bf y}\in R^m$, where $m\ll n$. While the vast majority of the literature in this area is on sparse signals, in recent years, there has been considerable progress on compressed sensing  of signals with structures beyond sparsity. One of the promising approaches in this field is to employ generative models that are based on trained neural networks. 
In this paper, we study the performance of an iterative algorithm  based on projected gradient descent that employs an auto-encoder to define and enforce the source structure. The auto-encoder is defined by a generative function $g:R^k\rightarrow R^n$ and a separate neural network that is trained to function as the inverse of $g$. We prove that, for a generative model $g$ with $\ell_2$ representation error $\delta$,  given roughly $m>40k\log{1\over \delta}$ measurements, such an algorithm converges, even in the presence of additive white Gaussian noise.
Learning to Recover Sparse Signals
Authors: Sichen Zhong, Yue Zhao, Jianshu Chen
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12
Tasks: compressed sensing, reinforcement-learning, Reinforcement Learning (RL)

In compressed sensing, a primary problem to solve is to reconstruct a high dimensional sparse signal from a small number of observations. In this work, we develop a new sparse signal recovery algorithm using reinforcement learning (RL) and Monte CarloTree Search (MCTS). Similarly to orthogonal matching pursuit (OMP), our RL+MCTS algorithm chooses the support of the signal sequentially. The key novelty is that the proposed algorithm learns how to choose the next support as opposed to following a pre-designed rule as in OMP. Empirical results are provided to demonstrate the superior performance of the proposed RL+MCTS algorithm over existing sparse signal recovery algorithms.
Interpretable and robust blind image denoising with bias-free convolutional neural networks
Authors: Zahra Kadkhodaie, Sreyas Mohan, Eero P. Simoncelli, Carlos Fernandez-Granda
Date: 2019-09-14 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2019 12
Tasks: Denoising, Image Denoising

Deep convolutional networks often append additive constant ("bias") terms to their convolution operations, enabling a richer repertoire of functional mappings. Biases are also used to facilitate training, by subtracting mean response over batches of training images (a component of "batch normalization"). Recent state-of-the-art blind denoising methods seem to require these terms for their success. Here, however, we show that bias terms used in most CNNs (additive constants, including those used for batch normalization) interfere with the interpretability of these networks, do not help performance, and in fact prevent generalization of performance to noise levels not including in the training data. In particular, bias-free CNNs (BF-CNNs) are locally linear, and hence amenable to direct analysis with linear-algebraic tools. These analyses provide interpretations of network functionality in terms of projection onto a union of low-dimensional subspaces, connecting the learning-based method to more traditional denoising methodology. Additionally, BF-CNNs generalize robustly, achieving near-state-of-the-art performance at noise levels well beyond the range over which they have been trained.