pwc-archive/papers-with-abstracts huggingface dataset:
Invertible Learned Primal-Dual
Authors: Jevgenija Rudzusika, Buda Bajic, Ozan Öktem, Carola-Bibiane Schönlieb, Christian Etmann
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: GPU, Image Reconstruction

We propose invertible Learned Primal-Dual as a method for tomographic image reconstruction. This is a learned iterative method based on the Learned Primal-Dual neural network architecture, which incorporates ideas from invertible neural networks. The invertibility significantly reduces the GPU memory footprint of the Learned Primal-Dual architecture, thus making it applicable to 3D tomographic reconstruction as demonstrated in the experiments.
Photon-Limited Deblurring using Algorithm Unrolling
Authors: Yash Sanghvi, Abhiram Gnanasambandam, Stanley Chan
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Astronomy, Deblurring, Image Deblurring, Rolling Shutter Correction

Image deblurring in a photon-limited condition is ubiquitous in a variety of low-light applications such as photography, microscopy and astronomy. However, presence of photon shot noise due to low-illumination and/or short exposure time makes the deblurring task substantially more challenging . This paper presents an algorithm unrolling approach for the photon-limited deblurring problem that unrolls a Plug-and-Play algorithm using a fixed-iteration network. By modifying the typical two-variable splitting to a three-variable splitting, our unrolled network is differentiable and can be trained end-to-end. We demonstrate the usage of our algorithm on real photon-limited image data.
Zero-Shot Physics-Guided Deep Learning for Subject-Specific MRI Reconstruction
Authors: Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Mehmet Akcakaya
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Anatomy, Deep Learning, MRI Reconstruction, Self-Supervised Learning, Transfer Learning

Physics-guided deep learning (PG-DL) has emerged as a powerful tool for accelerated MRI reconstruction, while often necessitating a database of fully-sampled measurements for training. Recent self-supervised and unsupervised learning approaches enable training without fully-sampled data. However, a database of undersampled measurements may not be available in many scenarios, especially for scans involving contrast or recently developed sequences, necessitating new methodology for subject-specific PG-DL reconstructions. A main challenge for developing subject-specific PG-DL methods is the large number of parameters, making it prone to over-fitting.  Moreover, database-trained models may not generalize well to unseen measurements that differ in terms of SNR, image contrast, sampling pattern, and anatomy. In this work, we propose a zero-shot self-supervised learning approach to perform subject-specific PG-DL reconstruction to tackle these issues. The proposed approach splits available measurements for each scan into three disjoint sets. Two of these sets are used to enforce data consistency and define loss during training, while the last set is used to establish an early stopping criterion. In the presence of models pre-trained on a database, we show that the proposed approach can be adapted as subject-specific fine-tuning via transfer learning to further improve reconstruction quality.
Greedy Learning for Large-Scale Neural MRI Reconstruction
Authors: Batu Ozturkler, Arda Sahiner, Tolga Ergen, Arjun D Desai, John M. Pauly, Shreyas Vasanawala, Morteza Mardani, Mert Pilanci
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: MRI Reconstruction

Model-based deep learning approaches have recently shown state-of-the-art performance for accelerated MRI reconstruction. These methods unroll iterative proximal gradient descent by alternating between data-consistency and a neural-network based proximal operation. However, they demand several unrolled iterations with sufficiently expressive proximals for high resolution and multi-dimensional imaging (e.g., 3D MRI). This impedes traditional training via backpropagation due to prohibitively intensive memory and compute needed to calculate gradients and store intermediate activations per layer. To address this challenge, we advocate an alternative training method by greedily relaxing the objective. We split the end-to-end network into decoupled network modules, and optimize each network module separately, thereby avoiding the need to compute costly end-to-end gradients. We empirically demonstrate that the proposed greedy learning method requires 6x less memory with no additional computations, while generalizing slightly better than backpropagation.
DARTS for Inverse Problems: a Study on Stability
Authors: Jonas Geiping, Jovita Lukasik, Margret Keuper, Michael Moeller
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: image-classification, Image Classification, Neural Architecture Search

Differentiable architecture search (DARTS) is a widely researched tool for neural architecture search, due to its promising results for image classification. The main benefit of DARTS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efficient architecture search.
In this work, we investigate DARTS in a systematic case study of inverse problems, which allows us to analyze these potential benefits in a controlled manner. 
Although we demonstrate that the success of DARTS can be extended from classification to reconstruction, our experiments yield a fundamental difficulty in the evaluation of DARTS-based methods: The results show a large variance in all test cases and the weight-sharing performance of the architecture found during training does not always reflect its final performance. 
We conclude the necessity to 1) report the results of any DARTS-based methods from several runs along with its underlying performance statistics and 2) show the correlation between the training and final architecture performance.
Return Dispersion as an Estimator of Learning Potential for Prioritized Level Replay
Authors: Iryna Korshunova, Minqi Jiang, Jack Parker-Holder, Tim Rocktäschel, Edward Grefenstette
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Prioritized Level Replay (PLR) has been shown to induce adaptive curricula that improve the sample-efficiency and generalization of reinforcement learning policies in environments featuring multiple tasks or levels. PLR selectively samples training levels weighed by a function of recent temporal-difference errors experienced on each level. We explore the dispersion of returns as an alternative prioritization criterion to address certain issues with TD error scores.
Adversarial Training Blocks Generalization in Neural Policies
Authors: Ezgi Korkmaz
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Deep neural networks have made it possible for reinforcement learning algorithms to learn from raw high dimensional inputs. This jump in the progress has caused deep reinforcement learning algorithms to be deployed in many different fields from financial markets to biomedical applications. While the vulnerability of deep neural networks to imperceptible specifically crafted perturbations has also been inherited by deep reinforcement learning agents, several adversarial training methods have been proposed to overcome this vulnerability. In this paper we focus on state-of-the-art adversarial training algorithms and investigate their robustness to semantically meaningful natural perturbations ranging from changes in brightness to rotation. We conduct several experiments in the OpenAI Atari environments, and find that state-of-the-art adversarially trained neural policies are more sensitive to natural perturbations than vanilla trained agents. We believe our investigation lays out intriguing properties of adversarial training and our observations can help build robust and generalizable neural policies.
Sampling via Controlled Stochastic Dynamical Systems
Authors: Benjamin Zhang, Tuhin Sahai, Youssef Marzouk
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12

We present a general framework for constructing controlled stochastic dynamical systems that exactly sample from a class of probability distributions with Gaussian tails. Given a target distribution and a reference stochastic differential equation (SDE), the Doob $h$-transform produces a controlled stochastic process whose marginal at a finite time $T$ will be equal to the target distribution. Our method constructs a reference linear SDE and uses the eigenfunctions of its associated Markov operator to approximate the Doob $h$-transform. The control is approximated by projecting the ratio between the target density and the reference system’s time $T$ marginal onto the span of a finite set of eigenfunctions. This projection is performed by minimizing the Kullback-Leibler (KL) divergence from the marginal produced by the approximate control to the true target distribution. In practice, the method lacks robustness due to the high sensitivity to the algorithm's parameters.
Beauty in Machine Learning: Purpose and Enlightenment
Authors: Benjamin Bloem-Reddy
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: BIG-bench Machine Learning

Drawing on empirical research and theories of beauty from psychology, and on philosophical investigations into the role of beauty in scientific research, I argue that aesthetic considerations are valuable to machine research in two modes. First, as a statistical and computational criterion for selecting theories (i.e., models), which leads to quantifiable benefits and appears to be largely automatable. These notions of beauty are largely the same as ``objective'' notions of beauty that are closely tied to human sensory stimulation. Second, somewhat less tangibly and less automatable, as desiderata for our models, methods, and scientific literature. These are manifestations of more complicated notions of beauty that are modulated by experience accumulated over time.
Causal Inference, is just Inference: A beautifully simple idea that not everyone accepts
Authors: David Rohde
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Causal Inference

It is often argued that causal inference is a step that follows probabilistic estimation in a two step procedure, with a separate statistical estimation and causal inference step and each step is governed by its own principles.  We have argued to the contrary that  Bayesian decision theory is perfectly adequate to do causal inference in a single step using nothing more than Bayesian conditioning.  If true this formulation greatly simplifies causal inference.  We outline this beautifully simple idea and discuss why some object to it.
The Beauty Everywhere: How Aesthetic Criteria Contribute to the Development of AI
Authors: Paulo Pirozelli, João F. N. B. Cortese
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Diversity, Philosophy

"Beauty" is a highly disputed word in philosophy and art. It also appears frequently in the scientific debates. But what is the role of "beauty" in science, and how can it be useful to AI? In this paper, we argue that scientific progress depends on the diversity of judgments of scientists, something that is only possible by the multiplicity of aspects involved in theoretical evaluations. Particularly important within these criteria are those related to aesthetic considerations, such as simplicity, consistency, broadness, and fertility. We claim that AI should be less focused on accuracy and related metrics, and should aim at integrating epistemic measures related to these aesthetic concepts.
GradML: A Gradient-based Loss for Deep Metric Learning
Authors: Bhavya Vasudeva, Puneesh Deora, Saumik Bhattacharya, Umapada Pal, Sukalpa Chanda
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Metric Learning, Retrieval

Deep metric learning (ML) uses a carefully designed loss function to learn distance metrics for improving the discriminatory ability for tasks like clustering and retrieval. Most loss functions are designed by considering the distance between the embeddings to induce certain properties without exploring how such losses would impact the movement of the said embeddings via their gradients during optimization. In this work, we analyze the gradients of various ML loss functions and propose a gradient-based loss for ML (GradML). Instead of directly formulating the loss, we first formulate the gradients of the loss and use them to derive the loss to be optimized. It has a simple formulation and lowers the computational cost as compared to other methods. We evaluate our approach on three datasets and find that the performance is data-dependent on properties like inter-class variance.
Nonlinear Denoising, Linear Demixing
Authors: Rainer Kelz, Gerhard Widmer
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Denoising

We cast the combinatorial problem of polyphonic piano transcription as a two stage
process. A nonlinear denoising stage maps spectrogram representations of arbitrary
piano music with unknown timbral characteristics onto a canonical spectrogram
representation with known timbral characteristics. A subsequent linear demixing
stage aims to exploit the knowledge about the canonical timbral characteristics.
The idea behind this two stage process is to try to elegantly sidestep any musical
bias inherent in the training dataset that is easily picked up by a single stage,
nonlinear (neural) transcription system (with large capacity). The two stage process
tries not to force the nonlinear system to solve a combinatorial problem, which
is more amenable to being solved by a linear decomposition method that has the
superposition property. Using the simplest setup we could think of, we obtain
(rather mixed (pun intended)) results on a standard polyphonic piano transcription
dataset — the two stage process still suffers from generalization problems after the
first stage, which the second stage is unable to compensate.
Test accuracy is not all you need: Examining neural network behavior at the classification boundary
Authors: Tiffany Joyce Vlaar
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: All

We evaluate the classification of both human volunteers and various neural network models on a set of GAN-generated images that reflect the transition from one MNIST class to another. We find that models that obtain the same test accuracy on the standard MNIST test data set exhibit different behavior on these images. Further, we find that although the number of misclassified images decreases with test accuracy, the spread in predictions over multiple runs on images that are difficult to classify (for humans) also decreases with test accuracy. We conclude that test accuracy is an insufficient metric to capture the behavior of a network on images that lie along the boundary between classes.
GOPHER: Categorical probabilistic forecasting withgraph structure via local continuous-time dynamics
Authors: Ke Alexander Wang, Danielle C. Maddix, Bernie Wang
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Inductive Bias

We consider the problem of probabilistic forecasting over categories with graph structure, where the dynamics at a vertex depends on its local connectivity structure. We present GOPHER, a method that combines the inductive bias of graph neural networks with neural ODEs to capture the intrinsic local continuous-time dynamics of our probabilistic forecasts. We study the benefits of these two inductive biases by comparing against baseline models that help disentangle the benefits of each. We find that capturing the graph structure is crucial for accurate in-domain probabilistic predictions and more sample efficient models. Surprisingly, our experiments demonstrate that the continuous time evolution inductive bias brings little to no benefit despite reflecting the true probability dynamics.
Unit-level surprise in neural networks
Authors: Cian Eastwood, Ian Mason, Chris Williams
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12

To adapt to changes in real-world data distributions, neural networks must update their parameters. We argue that unit-level surprise should be useful for: (i) determining which few parameters should update to adapt quickly; and (ii) learning a modularization such that few modules need be adapted to transfer. We empirically validate (i) in simple settings and reflect on the challenges and opportunities of realizing both (i) and (ii) in more general settings.
Language Models as Recommender Systems: Evaluations and Limitations
Authors: Yuhui Zhang, Hao Ding, Zeren Shui, Yifei Ma, James Zou, Anoop Deoras, Hao Wang
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Movie Recommendation, Recommendation Systems, Session-Based Recommendations, World Knowledge

Pre-trained language models (PLMs) such as BERT and GPT learn general text representations and encode extensive world knowledge; thus, they can be efficiently and accurately adapted to various downstream tasks. In this work, we propose to leverage these powerful PLMs as recommender systems and use prompts to reformulate the session-based recommendation task to a multi-token cloze task. We evaluate the proposed method on a movie recommendation dataset in zero-shot and fine-tuned settings where no or limited training data are available. In the zero-shot setting: we find that PLMs outperform the random recommendation baseline by a large margin; in the meantime, we observe strong linguistic bias when using PLMs as recommenders. In the fine-tuned setting: such bias is reduced with available training data; however, PLMs tend to under-perform traditional recommender system baselines such as GRU4Rec. Our observations demonstrate the current challenges of multi-token inference and shed light on future works in this novel direction.
CDF Normalization for Controlling the Distribution of Hidden Nodes
Authors: Mike Van Ness, Madeleine Udell
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12

Batch Normalizaiton (BN) is a normalization method for deep neural networks that has been shown to accelerate training.  While the effectiveness of BN is undisputed, the explanation of its effectiveness is still being studied.  The original BN paper attributes the success of BN to reducing internal covariate shift, so we take this a step further and explicitly enforce a Gaussian distribution on hidden layer activations.  This approach proves to be ineffective, demonstrating that reducing internal covariate shift in and of itself does not help training.
A multivariate extension to the Exponentially-modified Gaussian distribution
Authors: Sandhya Prabhakaran
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Translation

The exponentially-modified Gaussian (EMG) distribution is a convolution sum of a univariate Gaussian and an exponential distribution. This has been used to model univariate skewed data such as chromatographic peaks' shape, cell population dynamics from single-cell data and reaction times in neuropsychology. Currently, the EMG is only available in its univariate form. In this work, we propose a multivariate extension to the EMG, called $\textit{mvEMG}$, by using an affine transformation involving rotation, translation and shearing to accommodate for the three moments (mean, variance and skew). We derive statistical properties for mvEMG. Although we demonstrate its performance in synthetic data compared with the multivariate skew normal distribution, we are unable to show its practical applicability, mainly due to lack of efficient sampling strategies and a viable real-world dataset.
Continual Learning with Memory Cascades
Authors: David Kappel, Franscesco Negri, Christian Tetzlaff
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12
Tasks: Continual Learning, Permuted-MNIST

Continual learning poses an important challenge to machine learning models. Kirkpatrick et al. introduced a model that combats forgetting during continual learning by using a Bayesian prior to transfer knowledge between task switches. This approach showed promising results but the algorithm was given access to the time points when tasks were switched. Using a model of stochastic learning dynamics we show that this model is very closely related to the previously developed cascade model to combat catastrophic forgetting. This general formulation allows us to use the model also for online learning where no knowledge about task switching times is given to the network. Also it allows us to use deeper hierarchies of Bayesian priors. We evaluate this model on the permuted MNIST task. We demonstrate improved task performance during task switching, but find that online learning is still significantly worse when task switching times are unknown to the network.
The Curse of Depth in Kernel Regime
Authors: Soufiane Hayou, Arnaud Doucet, Judith Rousseau
Date: 2021-09-22 00:00:00
Proceeding: NeurIPS Workshop ICBINB 2021 12

Recent work by Jacot et al. (2018) has shown that training a neural network of any kind with gradient descent is strongly related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Empirical results in (Lee et al., 2019) demonstrated high performance of a linearized version of training using the so-called NTK regime. In this paper, we show that the large depth limit of this regime is unexpectedly trivial, and we fully characterize the convergence rate to this trivial regime.
Learning Background Invariance Improves Generalization and Robustness in Self-Supervised Learning on ImageNet and Beyond
Authors: Chaitanya Ryali, David J. Schwab, Ari S. Morcos
Date: 2021-09-28 00:00:00
Proceeding: NeurIPS Workshop ImageNet_PPF 2021 12
Tasks: Data Augmentation, Saliency Detection, Self-Supervised Learning, Unsupervised Saliency Detection

Recent progress in self-supervised learning has demonstrated promising results in multiple visual tasks. An important ingredient in high-performing self-supervised methods is the use of data augmentation by training models to place different augmented views of the same image nearby in embedding space. However, commonly used augmentation pipelines treat images holistically, ignoring the semantic relevance of parts of an image—e.g. a subject vs. a background—which can lead to the learning of spurious correlations. Our work addresses this problem by investigating a class of simple, yet highly effective “background augmentations", which encourage models to focus on semantically-relevant content by discouraging them from focusing on image backgrounds. Through a systematic, comprehensive investigation, we show that background augmentations lead to improved generalization with substantial improvements ($\sim$1-2% on ImageNet) in performance across a spectrum of state-of-the-art self-supervised methods (MoCo-v2, BYOL, SwAV) on a variety of tasks, even enabling performance on par with the supervised baseline. We also find improved label efficiency with even larger performance improvements in limited-labels settings (up to 4.2%). Further, we find improved training efficiency, attaining a benchmark accuracy of 74.4%, outperforming many recent self-supervised learning methods trained for 800-1000 epochs, in only 100 epochs. Importantly, we also demonstrate that background augmentations boost generalization and robustness to a number of out-of-distribution settings, including ImageNet-9, natural adversarial examples, adversarial attacks, ImageNet-Renditions and ImageNet ReaL. We also make progress in completely unsupervised saliency detection, in the process of generating saliency masks that we use for background augmentations.