pwc-archive/papers-with-abstracts huggingface dataset:
Layer-Parallel Training of Residual Networks with Auxiliary Variables
Authors: Qi Sun, Hexin Dong, Zewei Chen, Weizhen Dian, Jiacheng Sun, Yitong Sun, Zhenguo Li, Bin Dong
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Data Augmentation

Backpropagation algorithm is indispensable for training modern residual networks (ResNets) and usually tends to be time-consuming due to its inherent algorithmic lockings. Auxiliary-variable methods, e.g., the penalty and augmented Lagrangian (AL) methods, have attracted much interest lately due to their ability to exploit layer5 wise parallelism. However, we find that large communication overhead and lacking data augmentation are two key challenges of these approaches, which may lead to low speedup and accuracy drop. Inspired by the continuous-time formulation of ResNets, we propose a novel serial-parallel hybrid (SPH) training strategy to enable the use of data augmentation during training, together with downsampling (DS) filters to reduce the communication cost. This strategy first trains the network by solving a succession of independent sub-problems in parallel and then improve the trained network through a full serial forward-backward propagation of data. We validate our methods on modern ResNets across benchmark datasets, achieving speedup over the backpropagation while maintaining comparable accuracy.
Sparse Gaussian Processes for Stochastic Differential Equations
Authors: Prakhar Verma, Vincent Adam, Arno Solin
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Gaussian Processes, Variational Inference

We frame the problem of learning stochastic differential equations (SDEs) from noisy observations as an inference problem and aim to maximize the marginal likelihood of the observations in a joint model of the latent paths and the noisy observations. As this problem is intractable, we derive an approximate (variational) inference algorithm and propose a novel parameterization of the approximate distribution over paths using a sparse Markovian Gaussian process. The approximation is efficient in storage and computation, allowing the usage of well-established optimizing algorithms such as natural gradient descent. We demonstrate the capability of the proposed method on the Ornstein-Uhlenbeck process.
Actor-Critic Algorithm for High-dimensional PDEs
Authors: Xiaohan Zhang
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: BIG-bench Machine Learning, Vocal Bursts Intensity Prediction

We develop a machine learning model to effectively solve high-dimensional nonlinear parabolic partial differential equations (PDE). We use Feynman-Kac formula to reformulate PDE into the equivalent stochastic control problem governed by a Backward Stochastic Differential Equation (BSDE) system. Our model is designed to maximally exploit the Markovian property of the BSDE system and utilizes an Actor-Critic network architecture, which is novel in the high dimensional PDE literature. We show that our algorithm design leads to a significant speedup with  higher accuracy level compared to other neural network solvers. Our model advances the state-of-the-art machine learning PDE solvers in a few aspects: 1) the trainable parameters are reduced by $N$ times, where $N$ is the number of steps to discretize the PDE in time, 2) the model convergence rate is an order of magnitude faster, 3) our model has fewer tuning hyperparameters. We demonstrate the performance improvements by solving six equations including Hamilton-Jacobian-Bellman equation, Allen-Cahn equation and Black-Scholes equation, all with dimensions on the order of 100. Those equations in high dimensions have wide applications in control theory, material science and Quantitative finance.
Stability analysis of Strange Attractors using Attractor Networks
Authors: Anonymous
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Understanding the behavior of nonlinear differential equations is an extremely difficult problem. This problem is compounded by the frequent chaotic behavior demonstrated by high-dimensional dynamical systems. A subset of these systems, namely strange attractors, are of particular interest due to their sensitive dependence on initial conditions. Analytical reports on the stability of these attractors rely heavily on the ability to solve the underlying equations. In this work, an attractor network is used for the identification of different regions and the parameters resulting in that particular region, in well-defined strange attractors. The network takes in the initial configuration of the system, stores the pattern of neuronal firing as a state vector, and predicts the behavior of the attractor.
A Lyapunov Condition for Training ODEs
Authors: Anonymous
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

 Control theory is widely used in the study of differential equations to obtain desired behavior from underlying dynamics. We propose a novel method for training ordinary differential equations by using a control-theoretic Lyapunov Condition for stability. This method avoids rolling-out ODE's during training and thus saves the cost of back propagating through a solver or using the adjoint method. We validate our approach experimentally and verify that it has similar performance to ODEs trained by backpropagating through rollouts.
Scaling physics-informed neural networks to large domains by using domain decomposition
Authors: Ben Moseley, Andrew Markham, Tarje Nissen-Meyer
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Recently, physics-informed neural networks (PINNs) have offered a powerful new paradigm for solving forward and inverse problems relating to differential equations. Whilst promising, a key limitation to date is that PINNs struggle to accurately solve problems with large domains and/or multi-scale solutions, which is crucial for their real-world application. In this work we propose a new approach called finite basis physics-informed neural networks (FBPINNs). FBPINNs combine PINNs with domain decomposition and separate subdomain normalisation to address the issues related to scaling PINNs to large domains, namely the increasing complexity of the underlying optimisation problem and the spectral bias of neural networks. Our experiments show that FBPINNs are more effective than PINNs in solving problems with large domains and/or multi-scale solutions, potentially paving the way to the application of PINNs on large, real-world problems.
PANOM: Automatic Hyper-parameter Tuning for Inverse Problems
Authors: Tianci Liu, Quan Zhang, Qi Lei
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Bilevel Optimization

Automated hyper-parameter tuning for unsupervised learning, including inverse problems, remains a long-standing open problem due to the lack of validation data. In this work, we design an automatic tuning criterion for inverse problems and formulate it as a bilevel optimization task. We demonstrate the efficiency of our tuning scheme on various inverse problems and different test and out-of-distribution image samples at no expense of performance drops.
ReFIn: A Refinement Approach for Video Frame Interpolation
Authors: Saikat Dutta, Anurag Mittal
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Optical Flow Estimation, Video Enhancement, Video Frame Interpolation

Video Frame Interpolation is an important video enhancement problem which aims to generate one or multiple frames between consecutive frames in video. Optical flow-based frame interpolation approaches estimate intermediate optical flow from interpolated frame to input frames and warped frames are fused to generate interpolated frame. However, intermediate flow estimates can itself be erroneous leading to inaccurate interpolation results. In this work, we improve an flow-based intrtpolation algorithm, Super-SloMo by residual refinement. Specifically, we feed intermediate flowmaps, visibility map, warped input frames and intermediate interpolation estimate to a refinement network to predict a frame residual. We have also experimented with different architecture choices to be used in different modules to further improve the results. We found out that GridNet with four pyramid levels achieves the best results whereas UNet++ performs moderately well with significantly less number of parameters.
Learning Structured Sparse Matrices for Signal Recovery via Unrolled Optimization
Authors: Jonathan Sauder, Martin Genzel, Peter Jung
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: compressed sensing, Rolling Shutter Correction

Countless signal processing applications include the reconstruction of an unknown signal from very few indirect linear measurements. Because the measurement operator is commonly constrained by the hardware or the physics of the observation process, finding measurement matrices that enable accurate signal recovery poses a challenging discrete optimization task. Meanwhile, recent advances in the field of machine learning have highlighted the effectiveness of gradient-based
optimization methods applied to large computational graphs such as those arising naturally when unrolling iterative algorithms for signal recovery. However, it has remained unclear how to leverage this technique when the set of admissible measurement matrices is both discrete and sparse. In this paper, we tackle this problem and propose an efficient and flexible method for learning structured sparse measurement matrices. Our approach uses unrolled optimization in conjunction with Gumbel reparametrizations. We empirically demonstrate the effectiveness of our method in two prototypical compressed sensing situations.
Physics-Based Learned Diffuser for Single-shot 3D Imaging
Authors: Eric Markley, Fanglin Linda Liu, Michael Kellman, Nick Antipa, Laura Waller
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: compressed sensing

A diffuser in the Fourier space of an imaging system can encode 3D fluorescence intensity information in a single-shot 2D measurement, which is then recovered by a compressed sensing algorithm. Typically, the diffusers used in such systems are either off-the-shelf, heuristically designed, or merit function driven. In this work we use a differentiable forward model of single-shot 3D microscopy
in conjunction with an invertible and differentiable reconstruction algorithm, ISTA-Net+, to jointly optimize both the diffuser surface shape and the reconstruction parameters. By choosing a differentiable and invertible reconstruction method, we enable the use of memory-efficient backpropagation to trade off storage with a reasonable increase in compute time, in order to fit an unrolled network containing a large-scale 3D volume into a single GPU’s memory. We validate our method on 2D and 3D single-shot imaging, where our learned diffuser demonstrates improved reconstruction quality compared to previous heuristic designs.
Robust Compressed Sensing MR Imaging with Deep Generative Priors
Authors: Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alex Dimakis, Jonathan Tamir
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: compressed sensing

The CSGM framework (Bora-Jalal-Price-Dimakis'17) has shown that deep
generative priors can be powerful tools for solving inverse problems.
However, to date this framework has been empirically successful only on
certain datasets (for example, human faces and MNIST digits), and it
is known to perform poorly on out-of-distribution samples. In this
paper, we present the first successful application of the CSGM
framework on clinical MRI data. We train a generative prior on brain
scans from the fastMRI dataset, and show that posterior sampling via
Langevin dynamics achieves high quality reconstructions. Furthermore,
our experiments and theory show that posterior sampling is robust to
changes in the ground-truth distribution and measurement process.
Photoacoustic imaging with conditional priors from normalizing flows
Authors: Rafael Orozco, Ali Siahkoohi, Gabrio Rizzuti, Tristan van Leeuwen, Felix Johan Herrmann
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12

For many ill-posed inverse problems, such as photoacoustic imaging, the uncertainty of the solution is highly affected by measurement noise and data incompleteness (due, for example, to limited aperture). For these problems, the choice of prior information is a crucial aspect of a computationally effective solution scheme. We propose a regularization scheme for photoacoustic imaging that leverages prior information learned by a generative network. We train conditional normalizing flows on pairs of photoacoustic sources (the unknowns of the problem) and the associated data in order to exploit the posterior distribution of the solution. The learned prior is combined with physics-based optimization (enforced by partial differential equations), according to the deep prior framework, in order to achieve robustness with respect to out-of-distribution data. Numerical evidence suggests the superiority of this approach with respect to non-conditional deep priors, and the ability to retrieve features of the unknowns that are typically challenging for limited-view photoacoustics.
Matching Plug-and-Play Algorithms to the Denoiser
Authors: Saurav K Shastri, Rizwan Ahmad, Christopher Metzler, Philip Schniter
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12

To solve inverse problems, plug-and-play (PnP) methods have been developed that replace the proximal step in a convex optimization algorithm with a call to an application-specific denoiser, often implemented using a deep neural network (DNN). Although such methods have been successful, they can be improved. For example, denoisers are usually designed/trained to remove white Gaussian noise, but the denoiser input error in PnP algorithms is usually far from white or Gaussian. Approximate message passing (AMP) methods provide white and Gaussian denoiser input error, but only when the forward operator is a large random matrix. In this work, we design a PnP method using the generalized expectation consistent (GEC) approximation algorithm that offers predictable error statistics at each iteration, from which DNN denoisers can be effectively trained.
Multi-Task Accelerated MR Reconstruction Schemes for Jointly Training Multiple Contrasts
Authors: Victoria Liu, Kanghyun Ryu, Cagan Alkan, John M. Pauly, Shreyas Vasanawala
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Diagnostic, MRI Reconstruction, Multi-Task Learning

Model-based accelerated MRI reconstruction methods leverage large datasets to reconstruct diagnostic-quality images from undersampled k-space. These networks require matching training and test time distributions to achieve high quality reconstructions. However, there is inherent variability in MR datasets, including different contrasts, orientations, anatomies, and institution-specific protocols. The current paradigm is to train separate models for each dataset. However, this is a demanding process and cannot exploit information that may be shared amongst datasets. To address this issue, we propose multi-task learning (MTL) schemes that can jointly reconstruct multiple datasets. We test multiple MTL architectures and weighted loss functions against single task learning (STL) baselines. Our quantitative and qualitative results suggest that MTL can outperform STL across a range of dataset ratios for two knee contrasts.
Learning Lipschitz-Controlled Activation Functions in Neural Networks for Plug-and-Play Image Reconstruction Methods
Authors: Pakshal Bohra, Alexis Goujon, Dimitris Perdios, Sébastien Emery, Michael Unser
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Denoising, Image Reconstruction

Ill-posed linear inverse problems are frequently encountered in image reconstruction tasks. Image reconstruction methods that combine the Plug-and-Play (PnP) priors framework with convolutional neural network (CNN) based denoisers have shown impressive performances. However, it is non-trivial to guarantee the convergence of such algorithms, which is necessary for sensitive applications such as medical imaging. It has been shown that PnP algorithms converge when deployed with a certain class of averaged denoising operators. While such averaged operators can be built from 1-Lipschitz CNNs, imposing such a constraint on CNNs usually leads to a severe drop in performance. To mitigate this effect, we propose the use of deep spline neural networks which benefit from learnable piecewise-linear spline activation functions. We introduce "slope normalization" to control the Lipschitz constant of these activation functions. We show that averaged denoising operators built from 1-Lipschitz deep spline networks consistently outperform those built from 1-Lipschitz ReLU networks.
Deep subspace learning for efficient reconstruction of spatiotemporal imaging data
Authors: Christopher Michael Sandino, Frank Ong, Siddharth Srinivasan Iyer, Adam Bush, Shreyas Vasanawala
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Image Reconstruction

Model-based deep learning approaches, such as unrolled neural networks, have been shown to be effective tools for efficiently solving inverse problems. However, the memory costs of training unrolled networks remain high, especially when the target data is high-resolution and high-dimensional. This often requires trade-offs in either network depth to reduce model size, or data resolution to reduce data size. To address this, we propose DL-Subspace - a novel unrolled network architecture which reduces memory usage by solving for a compact, low-dimensional representation of the target instead of the target itself. DL-Subspace is applied to accelerated magnetic resonance image reconstruction, demonstrating up to 4$\times$ higher memory efficiency and 4$\times$ faster inference speed while maintaining similar image quality as conventional unrolled networks.
Efficient posterior inference & generalization in physics-based Bayesian inference with conditional GANs
Authors: Deep Ray, Dhruv V Patel, Harisankar Ramaswamy, Assad Oberai
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Bayesian Inference, Generative Adversarial Network

In this work, we propose a conditional generative adversarial network (cGAN) to sample from the posterior of physics-based Bayesian inference problems. We utilize a U-Net architecture for the generator and inject the latent variable using conditional instance normalization. We solve the inverse heat conduction problem and demonstrate how the proposed strategy effectively quantifies the uncertainty in the inferred field.  We also show that the structure of the generator promotes generalizability due to the local-nature of the learned inverse map.
Near-Exact Recovery for Sparse-View CT via Data-Driven Methods
Authors: Martin Genzel, Ingo Gühring, Jan Macdonald, Maximilian März
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Image Reconstruction

This work presents an empirical study on the design and training of iterative neural networks for image reconstruction from tomographic measurements with unknown geometry. It is based on insights gained during our participation in the recent AAPM DL-Sparse-View CT challenge and a further analysis of our winning submission (team name: robust-and-stable) subsequent to the competition period. The goal of the challenge was to identify the state of the art in sparse-view CT with data-driven techniques, thereby addressing a fundamental research question: Can neural-network-based solvers produce near-perfect reconstructions for noise-free data? We answer this in the affirmative by demonstrating that an iterative end-to-end scheme enables the computation of near-perfect solutions on the test set. Remarkably, the fanbeam geometry of the used forward model is completely inferred through a data-driven geometric calibration step.
PLUGIn-CS: A simple algorithm for compressive sensing with generative prior
Authors: Babhru Joshi, Xiaowei Li, Yaniv Plan, Ozgur Yilmaz
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Compressive Sensing

We consider the problem of recovering an unknown latent code vector under a known generative model from compressive measurements. For a $d$-layer deep generative network $\mathcal{G}:\mathbb{R}^{n_0}\rightarrow \mathbb{R}^{n_d}$ with ReLU activation functions and compressive measurement matrix $\Phi \in \mathbb{R}^{m\times n_d}$, let the observation be $\Phi\mathcal{G}(x)+\epsilon$ where $\epsilon$ is noise. We introduce a simple novel algorithm, Partially Linearized Update for Generative Inversion in Compressive Sensing (PLUGIn-CS), to estimate $x$ (and thus $\mathcal{G}(x)$). We prove that, when sensing matrix and weights are Gaussian, if layer widths $n_i \gtrsim 5^i n_0$ and number of measurements $m \gtrsim 2^dn_0$ (both up to log factors), then the algorithm converges geometrically to a (small) neighbourhood of $x$ with high probability. Note the inequality on layer widths allows $n_i>n_{i+1}$ when $i\geq 1$ and thus allows the network to have some contractive layers. After a sufficient number of iterations, the estimation errors for both $x$ and $\mathcal{G}(x)$ are at most in the order of $\sqrt{4^dn_0/m} \|\epsilon\|$. Numerical experiments on synthetic data and real data are provided to validate our theoretical results and to illustrate that the algorithm can effectively recover images from compressive measurements.
SSFD: Self-Supervised Feature Distance as an MR Image Reconstruction Quality Metric
Authors: Philip M Adamson, Beliz Gunel, Jeffrey Dominic, Arjun D Desai, Daniel Spielman, Shreyas Vasanawala, John M. Pauly, Akshay Chaudhari
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Image Reconstruction, MRI Reconstruction, Self-Supervised Learning, SSIM

 Evaluation of accelerated magnetic resonance imaging (MRI) reconstruction methods is imperfect due to the discordance between quantitative image quality metrics and radiologist-perceived image quality. Self-supervised learning (SSL) has become a popular pre-training tool due to its ability to capture generalizable and domain-specific feature representations of the underlying data for downstream tasks. In this study, we use SSL to extract image-level feature representations of MR images, and use those features to compute a self-supervised feature distance (SSFD) metric to assess MR image reconstruction quality. We demonstrate preliminary results showing the superiority of SSFD to common image quality metrics such as PSNR and SSIM, its robustness to image perturbations, and its ability to capture both pixel-level and global image quality information.
Bayesian Inference in Physics-Based Nonlinear Flame Models
Authors: Maximilian L. Croci, Ushnish Sengupta, Matthew P Juniper
Date: 2021-10-19 00:00:00
Proceeding: NeurIPS Workshop Deep_Invers 2021 12
Tasks: Bayesian Inference

This study uses a Bayesian machine learning method to infer the parameters of a physics-based model of a bluff-body-stabilised flame in real-time. An ensemble of neural networks is trained on a library of simulated flame fronts with known parameters, generated using a level-set solver, LSGEN2D. The ensemble learns a surrogate of the approximate Bayesian posterior of the parameters given the observations, from which the flame can be re-simulated beyond the observation window of the experiment. The method is general: once trained, the ensemble can be used to infer the parameters from any bluff-body-stabilised  flame as long as the flame is qualitatively similar and the parameters lie within the ranges in the training library. Amortized inference takes milliseconds, which is fast enough to work in real-time.