pwc-archive/papers-with-abstracts huggingface dataset:
Numerical Reasoning over Legal Contracts via Relational Database
Authors: Jiani Huang, Ziyang Li, Ilias Fountalis, Mayur Naik
Date: 2021-10-03 00:00:00
Proceeding: NeurIPS Workshop DBAI 2021 12

Numerical reasoning over text requires deep integration between the semantic understanding of the natural language context and the mathematical calculation of the symbolic terms. However, existing approaches are limited in their ability to incorporate domain-specific knowledge and express mathematical formulas over data structures.  Delegating logic reasoning to a relational database is a promising approach to enhance the reasoning complexity. We study the problem of distilling natural language text into a relational database with numerical data structure and querying this database to obtain desired answers. Specifically, given a legal contract and a set of date-related questions in natural language, we utilize pre-trained neural network models to create a relational database to retrieve and generate the target dates. We evaluate our method on the CUAD dataset and demonstrate that our approach has high correct answer coverage and reduces a significant amount of incorrect results even without any labels.
Performance-Guaranteed ODE Solvers with Complexity-Informed Neural Networks
Authors: Feng Zhao, Xiang Chen, Jun Wang, Zuoqiang Shi, Shao-Lun Huang
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Traditionally, we provide technical parameters for ODE solvers, such as the order, the stepsize and the local error threshold. However, there is no for performance metrics that users care about, such as the time consumption and the global error. In this paper, we provide such a user-oriented by using neural networks to fit the complex relationship between the technical parameters and performance metrics. The form of the neural network is carefully designed to incorporate the prior knowledge from time complexity analysis of ODE solvers, which has better performance than purely data-driven approaches. We test our strategy on some parametrized ODE problems, and experimental results show that the fitted model can achieve high accuracy, thus providing error for fixed methods and time for adaptive stepsize methods.
Statistical Numerical PDE : Fast Rate, Neural Scaling Law and When it’s Optimal
Authors: Yiping Lu, Haoxuan Chen, Jianfeng Lu, Lexing Ying, Jose Blanchet
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs). To simplify the problem, we focus on a prototype elliptic PDE: the Schr\"odinger equation on a hypercube with zero Dirichlet boundary condition, which is applied in quantum-mechanical systems. We establish upper and lower bounds for both methods, which improve upon concurrently developed upper bounds for this problem via a fast rate generalization bound. We discover that the current Deep Ritz Method is sub-optimal and propose a modified version of it. We also prove that PINN and the modified version of DRM can achieve minimax optimal bounds over Sobolev spaces. Empirically, following recent work which has shown that the deep model accuracy will improve with growing training sets according to a power law, we supply computational experiments to show similar-behavior of dimension dependent power law for deep PDE solvers.
Expressive Power of Randomized Signature
Authors: Christa Cuchiero, Lukas Gonon, Lyudmila Grigoryeva, Juan-Pablo Ortega, Josef Teichmann
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: compressed sensing, LEMMA, Transfer Learning

We consider the question whether the time evolution of controlled differential equations on general state spaces can be arbitrarily well approximated by (regularized) regressions on features generated themselves through randomly chosen dynamical systems of moderately high dimension. On the one hand this is motivated by paradigms of reservoir computing, on the other hand by ideas from rough path theory and compressed sensing. Appropriately interpreted this yields provable approximation and generalization results for generic dynamical systems by regressions on states of random, otherwise untrained dynamical systems, which usually are approximated by recurrent or LSTM networks. The results have important implications for transfer learning and energy efficiency of training.

We apply methods from rough path theory, convenient analysis, non-commutative algebra and the Johnson-Lindenstrauss Lemma to prove the approximation results.
Spectral PINNs: Fast Uncertainty Propagation with Physics-Informed Neural Networks
Authors: Björn Lütjens, Catherine H Crawford, Mark Veillette, Dava Newman
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Physics-informed neural networks (PINNs) promise to significantly speed up partial differential equation (PDE) solvers. However, most PINNs can only solve deterministic PDEs. Here, we consider \textit{stochastic} PDEs that contain partially unknown parameters. We aim to quickly quantify the impact of uncertain parameters onto the solution of a PDE - that is - we want to perform fast uncertainty propagation. Classical uncertainty propagation methods such as Monte Carlo sampling, stochastic Galerkin, collocation, or discrete projection methods become computationally too expensive with an increasing number of stochastic parameters. For example, the well-known spectral or polynomial chaos expansions achieve to separate the spatiotemporal and probabilistic domains and offer theoretical guarantees and fast computation of stochastic summaries (e.g., mean), but can be computationally expensive to form. Our Spectral-PINNs approximate the underlying spectral coefficients with a neural network and reduce the computational cost of the spectral expansion while maintaining guarantees. We derive the method for partial differential equations, discuss runtime, demonstrate initial results on the convection-diffusion equation, and provide steps towards convergence guarantees.
Empirics on the expressiveness of Randomized Signature
Authors: Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Time Series, Time Series Analysis

Time series analysis is a widespread task in Natural Sciences, Social Sciences and Engineering. A fundamental problem is finding an expressive yet efficient-to-compute representation of the input time series to use as a starting point to perform arbitrary downstream tasks. In this paper, we build upon recent work using the signature of a path as a feature map and investigate a computationally efficient technique to approximate these features based on linear random projections. We present several theoretical results to justify our approach, we analyze and showcase its empirical performance on the task of learning a mapping between the input controls of a Stochastic Differential Equation (SDE) and its corresponding solution. Our results show that the representational power of the proposed random features allows to efficiently learn the aforementioned mapping.
Enhancing the trainability and expressivity of deep MLPs with globally orthogonal initialization
Authors: Hanwen Wang, Isabelle Crawford-Eng, Paris Perdikaris
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Multilayer Perceptrons (MLPs) defines a fundamental model class that forms the backbone of many modern deep learning architectures. 
Despite their universality guarantees, practical training via stochastic gradient descent often struggles to attain theoretical error bounds due to issues including (but not limited to) frequency bias, vanishing gradients, and stiff gradient flows. In this work we postulate that many of such issues find origins in the initialization of the network's parameters. While the initialization schemes proposed by Glorot {\it et al.} and He {\it et al.} have become the de-facto choices among practitioners, their goal to preserve the variance of forward- and backward-propagated signals is mainly achieved by assumptions on linearity, while the presence of nonlinear activation functions may partially destroy these efforts. Here, we revisit the initialization of MLPs from a dynamical systems viewpoint to explore why and how under these classical scheme, the MLP could still fail even at the beginning. Drawing inspiration from classical numerical methods for differential equations that leverage orthogonal feature representations, we propose a novel initialization scheme that promotes orthogonality in the features of the last hidden layer, ultimately leading to more diverse and localized features. Our results demonstrate that network initialization alone can be sufficient in mitigating frequency bias and yields competitive results for high-frequency function approximation and image regression tasks, without any additional modifications to the network architecture or activation functions.
Gotta Go Fast with Score-Based Generative Models
Authors: Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, Ioannis Mitliagkas
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Denoising

Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data to noise and generate data by reversing it. Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. In this work, we aim to accelerate this process by devising a more efficient SDE solver. Our solver requires only two score function evaluations per step, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.
Accelerated PDEs for Construction and Theoretical Analysis of an SGD Extension
Authors: Yuxin Sun, Dong Lao, Ganesh Sundaramoorthi, Anthony Yezzi
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: image-classification, Image Classification

We introduce a recently developed framework PDE Acceleration, which is a variational approach to accelerated optimization with partial differential equations (PDE), in the context of optimization of deep networks. We derive the PDE evolution equations for optimization of general loss functions using this variational approach.  We propose discretizations of these PDE based on numerical PDE discretizations, and establish a mapping between these discretizations and stochastic gradient descent (SGD). We show that our framework can give rise to new PDEs that can be mapped to new optimization algorithms, and thus theoretical insights from the PDE domain can be used to analyze optimization algorithms. We show an example by introducing a new PDE with diffusion that naturally arises from the viscosity solution, which translates to a novel extension of SGD.  We analytically analyze the stability and convergence using Von-Neumann analysis. We apply the proposed extension to optimization of convolutional neural networks (CNNs). We empirically validate the theory and evaluate our new extension on image classification showing empirical improvement over SGD.
Shape-Tailored Deep Neural Networks With PDEs
Authors: Naeemullah Khan, Angira Sharma, Philip Torr, Ganesh Sundaramoorthi
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

We present Shape-Tailored Deep Neural Networks (ST-DNN). ST-DNN are deep networks  formulated through the use of partial differential equations (PDE) to be defined on arbitrarily shaped regions. This is natural for problems in computer vision such as segmentation, where descriptors should describe regions (e.g., of objects) that have diverse shape. We formulate ST-DNNs through the Poisson PDE, which can be used to generalize convolution to arbitrary regions. We stack multiple PDE layers to generalize a deep CNN to arbitrarily shaped regions. We show that ST-DNN are provably covariant to translations and rotations and robust to domain deformations, which are important properties for computer vision tasks. We show proof-of-concept empirical validation.
Long-time prediction of nonlinear parametrized dynamical systems by deep learning-based ROMs
Authors: Stefania Fresca, Federico Fatone, Andrea Manzoni
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Dimensionality Reduction

Deep learning-based reduced order models (DL-ROMs) have been recently proposed to overcome common limitations shared by conventional ROMs - built, e.g., through proper orthogonal decomposition (POD) - when applied to nonlinear time-dependent parametrized PDEs. Although extremely efficient at testing time, when evaluating the PDE solution for any new testing-parameter instance, DL-ROMs require an expensive training stage. To avoid this latter, a prior dimensionality reduction through POD, and a multi-fidelity pretraining stage, are introduced, yielding the  POD-DL-ROM framework, which allows to solve time-dependent PDEs even faster than in real-time. Equipped with LSTM networks, the resulting POD-LSTM-ROMs better grasp the time evolution of the PDE system, ultimately allowing long-term prediction of complex systems’ evolution, with respect to the training window, for unseen input parameter values.
Data-driven Taylor-Galerkin finite-element scheme for convection problems
Authors: Luciano DROZDA, Pavanakumar Mohanamuraly, Yuval Realpe, Corentin Lapeyre, Amir Adler, Guillaume Daviller, Thierry Poinsot
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Graph Neural Network

High-fidelity large-eddy simulations (LES) of high Reynolds number flows are essential to design low-carbon footprint energy conversion devices. The two-level Taylor-Galerkin (TTGC) finite-element  method (FEM) has remained the workhorse of modern industrial-scale combustion LES. In this work, we propose an improved FEM termed ML-TTGC that introduces locally tunable parameters in the TTGC scheme, whose values are provided by a graph neural network (GNN). We show that ML-TTGC outperforms TTGC in solving the convection problem in both irregular and regular meshes over a wide-range of initial conditions. We train the GNN using parameter values that (i) minimize a weighted loss function of the dispersion and dissipation error and (ii) enforce them to be numerically stable. As a result no additional ad-hoc dissipation is necessary for numerical stability or to damp spurious waves amortizing the additional cost of running the GNN.
NeurInt-Learning Interpolation by Neural ODEs
Authors: Avinandan Bose, Aniket Das, Yatin Dandi, Piyush Rai
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Image Generation

 A range of applications require learning image generation models whose latent space effectively captures the high-level factors of variation in the data distribution, which can be judged by its ability to interpolate between images smoothly. However, most generative models mapping a fixed prior to the generated images lead to interpolation trajectories lacking smoothness and images of reduced quality. We propose a novel generative model that learns a flexible non-parametric prior over interpolation trajectories, conditioned on a pair of source and target images. Instead of relying on deterministic interpolation methods like linear or spherical interpolation in latent space, we devise a framework that learns a distribution of trajectories between two given images using Latent Second-Order Neural Ordinary Differential Equations. Through a hybrid combination of reconstruction and adversarial losses, the generator is trained to map the sampled points from these trajectories to sequences of realistic images of improved quality that smoothly transition from the source to the target image.
Learning Implicit PDE Integration with Linear Implicit Layers
Authors: Marcel Nonnenmacher, David S. Greenberg
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Neural networks can learn local interactions to faithfully reproduce large-scale dynamics in important physical systems. Trained on PDE integrations or noisy observations, these emulators can assimilate data, tune parameters and learn sub-grid process representations. However, implicit integration schemes cannot be expressed as local feedforward computations. We therefore introduce linear implicit layers (LILs), which learn and solve linear systems with locally computed coefficients. LILs use diagonal dominance to ensure parallel solver convergence and support efficient backward mode differentiation. As a challenging test case, we train emulators on semi-implicit integration of 2D shallow-water equations with closed boundaries. LIL networks learned compact representations of the local interactions controlling the 30.000 degrees of freedom of this discretized system of PDEs. This enabled accurate and stable LIL-based emulation over many time steps where feedforward networks failed.
Fitting Regularized Population Dynamics with Neural Differential Equations
Authors: David Calhas, Rui Henriques
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Neural differential equations (neural DEs) are yet to see success in its application as interpretable autoencoders/descriptors, where they directly model a population of signals with the learned vector field. In this manuscript, we show that there is a threshold to which these models capture the dynamics of a population of signals produced under the same monitoring protocol. This threshold is computed by taking the derivative at each time point and analyzing the variance of its dynamics. In addition, we show that this can be tackled by projecting a highly-variant population to a lower dynamically variant space, where the model is able to capture dynamics, and similarly project the modelled signal back to the original space.
Investigating the Role of Overparameterization While Solving the Pendulum with DeepONets
Authors: Pulkit Gopalani, Anirbit Mukherjee
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

Machine learning methods have made substantial advances in various aspects of physics. In particular multiple deep-learning methods have emerged as efficient ways of numerically solving differential equations arising commonly in physics. DeepONets [1] are one of the most prominent ideas in this theme which entails an optimization over a space of inner-products of neural nets. In this work we study the training dynamics of DeepONets for solving the pendulum to bring to light some intriguing properties of it. We demonstrate that contrary to usual expectations, test error here has its first local minima at the interpolation threshold i.e when model size $\approx$ training data size. Secondly, as opposed to the average end-point error, the best test error over iterations has better dependence on model size, as in it shows only a very mild double-descent. Lastly, we show evidence that triple-descent [2] is unlikely to occur for DeepONets.  

[1] Lu Lu, Pengzhan Jin, and George Em Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. 2020. arXiv:1910.03193 [cs.LG]
[2] Ben Adlam and Jeffrey Pennington. The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. 2020. arXiv:2008.06786 [stat.ML].
Non Vanishing Gradients for Arbitrarily Deep Neural Networks: a Hamiltonian System Approach
Authors: Clara Galimberti, Luca Furieri, Liang Xu, Giancarlo Ferrari-Trecate
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Sensitivity

Deep Neural Networks (DNNs) training  can be difficult due to vanishing or exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stems from the discretization of continuous-time Hamiltonian systems. Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth. This is obtained by proving that, using a semi-implicit Euler discretization scheme, the backward sensitivity matrices involved in gradient computations are symplectic.
Neural ODE Processes: A Short Summary
Authors: Alexander Luke Ian Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, Pietro Lio
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Time Series, Time Series Analysis

Neural Ordinary Differential Equations (NODEs) use a neural network to model the instantaneous rate of change in the state of a system. However, despite their apparent suitability for dynamics-governed time-series,  NODEs present a few disadvantages.  First, they are unable to adapt to incoming data-points, a fundamental requirement for real-time applications imposed by the natural direction of time.  Second, time-series are often composed of a sparse set of measurements, which could be explained by many possible underlying dynamics.  NODEs do not capture this uncertainty.  To this end, we introduce Neural ODE Processes (NDPs), a new class of stochastic processes determined by a distribution over Neural ODEs.  By maintaining an adaptive data-dependent distribution over the underlying ODE, we show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.  At the same time, we demonstrate that NDPs scale up to challenging high-dimensional time-series with unknown latent dynamics such as rotating MNIST digits. Code is available online at https://github.com/crisbodnar/ndp.
On Second Order Behaviour in Augmented Neural ODEs: A Short Summary
Authors: Alexander Luke Ian Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, Pietro Lio
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12

In Norcliffe et al.[13], we discussed and systematically analysed how Neural ODEs (NODEs) can learn higher-order order dynamics. In particular, we focused on second-order dynamic behaviour and analysed Augmented NODEs (ANODEs), showing that they can learn second-order dynamics with only a few augmented dimensions, but are unable to correctly model the velocity (first derivative).  In response, we proposed Second Order NODEs (SONODEs), that build on top of ANODEs, but explicitly take into account the second-order physics-based inductive biases. These biases, besides making them more efficient and noise-robust when modelling second-order dynamics, make them more interpretable than ANODEs, therefore more suitable in many real-world scientific modelling applications.
A neural multilevel method for high-dimensional parametric PDEs
Authors: Cosmas Heiß, Ingo Gühring, Martin Eigel
Date: 2021-09-27 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: Uncertainty Quantification, Vocal Bursts Intensity Prediction

In scientific machine learning, neural networks recently have become a popular tool for learning the solutions of differential equations.
However, practical results often conflict the existing theoretical predictions in that observed convergence stagnates early. A substantial improvement can be achieved by the presented multilevel scheme which decomposes the considered problem into easier to train sub-problems, resulting in a sequence of neural networks. The efficacy of the approach is demonstrated for high-dimensional parametric elliptic PDEs that are common benchmark problems in uncertainty quantification. Moreover, a theoretical analysis of the expressivity of the developed neural networks is devised.