pwc-archive/papers-with-abstracts huggingface dataset:
Continual Density Ratio Estimation (CDRE): A new method for evaluating generative models in continual learning
Authors: Yu Chen, Song Liu, Tom Diethe, Peter Flach
Date: 2019-09-25 00:00:00
Tasks: Continual Learning, Density Ratio Estimation

We propose a new method Continual Density Ratio Estimation (CDRE), which can estimate density ratios between a target distribution of real samples and a distribution of samples generated by a model while the model is changing over time and the data of the target distribution is not available after a certain time point. This method perfectly fits the setting of continual learning, in which one model is supposed to learn different tasks sequentially and the most crucial restriction is that model has none or very limited access to the data of all learned tasks. Through CDRE, we can evaluate generative models in continual learning using f-divergences. To the best of our knowledge, there is no existing method that can evaluate generative models under the setting of continual learning without storing real samples from the target distribution.
Prune or quantize? Strategy for Pareto-optimally low-cost and accurate CNN
Authors: Kengo Nakata, Daisuke Miyashita, Asuka Maki, Fumihiko Tachibana, Shinichi Sasaki, Jun Deguchi
Date: 2019-09-25 00:00:00
Tasks: CPU, GPU, Quantization

Pruning and quantization are typical approaches to reduce the computational cost of CNN inference. Although the idea to combine them together seems natural, it is being unexpectedly difficult to figure out the resultant effect of the combination unless measuring the performance on a certain hardware which a user is going to use. This is because the benefits of pruning and quantization strongly depend on the hardware architecture where the model is executed. For example, a CPU-like architecture without any parallelization may fully exploit the reduction of computations by unstructured pruning for speeding up, but a GPU-like massive parallel architecture would not. Besides, there have been emerging proposals of novel hardware architectures such as one supporting variable bit precision quantization. From an engineering viewpoint, optimization for each hardware architecture is useful and important in practice, but this is quite a brute-force approach. Therefore, in this paper, we first propose hardware-agnostic metric to measure the computational cost. And using the metric, we demonstrate that Pareto-optimal performance, where the best accuracy is obtained at a given computational cost, is achieved when a slim model with smaller number of parameters is quantized moderately rather than a fat model with huge number of parameters is quantized to extremely low bit precision such as binary or ternary. Furthermore, we empirically found the possible quantitative relation between the proposed metric and the signal to noise ratio during SGD training, by which the information obtained during SGD training provides the optimal policy of quantization and pruning. We show the Pareto frontier is improved by 4 times in post-training quantization scenario based on these findings. These findings are available not only to improve the Pareto frontier for accuracy vs. computational cost, but also give us some new insights on deep neural network.
Concise Multi-head Attention Models
Authors: Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, Sanjiv Kumar
Date: 2019-09-25 00:00:00

Attention based Transformer architecture has enabled significant advances in the field of natural language processing. In addition to new pre-training techniques, recent improvements crucially rely on working with a relatively larger embedding dimension for tokens. This leads to models that are prohibitively large to be employed in the downstream tasks. In this paper we identify one of the important factors contributing to the large embedding size requirement. In particular, our analysis highlights that the scaling between the number of heads and the size of each head in the existing architectures gives rise to this limitation, which we further validate with our  experiments. As a solution, we propose a new way to set the projection size in attention heads that allows us to train models with a relatively smaller embedding dimension, without sacrificing the performance.
GraphMix: Regularized Training of Graph Neural Networks for Semi-Supervised Learning
Authors: Vikas Verma, Meng Qu, Alex Lamb, Yoshua Bengio, Juho Kannala, Jian Tang
Date: 2019-09-25 00:00:00
Tasks: Graph Neural Network

We present GraphMix, a regularization technique for Graph Neural Network based semi-supervised object classification, leveraging the recent advances in the regularization of classical deep neural networks. Specifically, we propose a unified approach in which we train a fully-connected network jointly with the graph neural network via parameter sharing, interpolation-based regularization and self-predicted-targets. Our proposed method is architecture agnostic in the sense that it can be applied to any variant of graph neural networks which applies a parametric transformation to the features of the graph nodes. Despite its simplicity, with GraphMix we can consistently improve results and achieve or closely match state-of-the-art performance using even simpler architectures such as Graph Convolutional Networks, across three established graph benchmarks: Cora, Citeseer and Pubmed citation network datasets, as well as three newly proposed datasets : Cora-Full, Co-author-CS and Co-author-Physics.
Measuring causal influence with back-to-back regression: the linear case
Authors: Jean-Remi King, Francois Charton, Maxime Oquab, David Lopez-Paz
Date: 2019-09-25 00:00:00
Tasks: Causal Identification, regression

Identifying causes from observations can be particularly challenging when i) potential factors are difficult to manipulate individually and ii) observations are complex and multi-dimensional. To address this issue, we introduce “Back-to-Back” regression (B2B), a method designed to efficiently measure, from a set of co-varying factors, the causal influences that most plausibly account for multidimensional observations. After proving the consistency of B2B and its links to other linear approaches, we show that our method outperforms least-squares regression and cross-decomposition techniques (e.g. canonical correlation analysis and partial least squares) on causal identification. Finally, we apply B2B to neuroimaging recordings of 102 subjects reading word sequences. The results show that the early and late brain representations, caused by low- and high-level word features respectively, are more reliably detected with B2B than with other standard techniques.
Longitudinal Enrichment of Imaging Biomarker Representations for Improved Alzheimer's Disease Diagnosis
Authors: Saad Elbeleidy, Lyujian Lu, L. Zoe Baker, Hua Wang, Feiping Nie
Date: 2019-09-25 00:00:00

Longitudinal data is often available inconsistently across individuals resulting in ignoring of additionally available data. Alzheimer's Disease (AD) is a progressive disease that affects over 5 million patients in the US alone, and is the 6th leading cause of death. Early detection of AD can significantly improve or extend a patient's life so it is critical to use all available information about patients. 
We propose an unsupervised method to learn a consistent representation by utilizing inconsistent data through minimizing the ratio of $p$-Order Principal Components Analysis (PCA) and Locality Preserving Projections (LPP). Our method's representation can outperform the use of consistent data alone and does not require the use of complex tensor-specific approaches. We run experiments on patient data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), which consists of inconsistent data, to predict patients' diagnosis.
Neural Dynamical Systems
Authors: Viraj Mehta, Ian Char, Willie Neiswanger, Youngseog Chung, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, Jeff Schneider
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12

We introduce Neural Dynamical Systems (NDS), a method of learning dynamical models which incorporates prior knowledge in the form of systems of ordinary differential equations. NDS uses neural models to estimate free parameters of the system, predicts residual terms, and numerically integrates over time to predict future states. It also natively handles irregularly sampled data and implicitly learns values of interpretable system parameters. We find that NDS learns dynamics with higher accuracy and fewer samples than a variety of deep learning methods that do not incorporate the prior knowledge. We demonstrate these advantages first on synthetic dynamical systems and then on real data captured from deuterium shots from a nuclear fusion reactor.
Wavelet-Powered Neural Networks for Turbulence
Authors: Arvind T. Mohan, Daniel Livescu, Michael Chertkov
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12
Tasks: GPU

One of the fundamental driving phenomena for applications in engineering, earth sciences and climate is fluid turbulence. Modeling these flows and explaining its associated spatio-temporal phenomena are notoriously difficult tasks. Navier-Stokes (NS) equations describe all the details of the fluid motions, but require accounting for unfeasibly many degrees of freedom in the regime of developed turbulence. Model reduction and surrogate modeling of turbulence is a general methodology aiming to circumvent this curse of dimensionality. Originally driven by phenomenological considerations,  multiple attempts to model-reduce NS equations got a new boost recently with Deep Learning (DL), trained on the ground truth data, e.g. extracted from high-fidelity Direct Numerical Simulations (DNS). However, early attempts of building NNs to model turbulence has also revealed its lack of interpretability as the most significant shortcoming. In this paper we address the key challenge of devising reduced but, at least partially, interpretable model. We take advantage of the balance between strong mathematical foundations and the physical interpretability of wavelet theory to build a spatio-temporally reduced dynamical map which fuses wavelet based spatial decomposition with spatio-temporal modeling based on Convolutional Long Short Term Memory (C-LSTM) architecture.  It is shown that the wavelet-based NN makes progress in scaling to large flows, by reducing computational costs and GPU memory requirements.
Embedding Hard Physical Constraints in Convolutional Neural Networks for 3D Turbulence
Authors: Arvind T. Mohan, Nicholas Lubbers, Daniel Livescu, Michael Chertkov
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12
Tasks: Deep Learning, Dimensionality Reduction, Inductive Bias

Deep learning approaches have shown much promise for physical sciences, especially in dimensionality reduction and compression of large datasets. A major issue in deep learning of large-scale phenomena, like fluid turbulence, is the lack of physical guarantees.  In this work, we propose a general framework to directly embed the notion of incompressible fluids into Convolutional Neural Networks, for coarse-graining of turbulence. These \textbf{physics-embedded neural networks} leverage interpretable strategies from numerical methods and computational fluid dynamics to enforce physical laws and boundary conditions by taking advantage the mathematical properties of the underlying equations. We demonstrate results on 3D fully-developed turbulence, showing that the \textit{physics-aware inductive bias} drastically improves local conservation of mass, without sacrificing performance according to several other metrics characterizing the fluid flow.
Nonlinear Differential Equations with external forcing
Authors: Paul Pukite
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12

Key equatorial climate phenomena such as QBO and ENSO have never been adequately explained as deterministic processes. This in spite of recent research showing growing evidence of predictable behavior. This study applies the fundamental Laplace tidal equations with simplifying assumptions along the equator — i.e. no Coriolis force and a small angle approximation. The solutions to the partial differential equations are highly non-linear related to Navier-Stokes and only search approaches can be used to fit to the data.
Learning-Based Strong Solutions to Forward and Inverse Problems in PDEs
Authors: Leah Bar, Nir Sochen
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12

We introduce a novel neural network-based partial differential equations solver for forward and inverse problems. The solver is grid free, mesh free and shape free, and the solution is approximated by a neural network. 
We employ an unsupervised approach such that the input to the network is a points set in an arbitrary domain, and the output is the
set of the corresponding function values.  The network is trained to minimize deviations of the learned function from the strong PDE solution and satisfy the boundary conditions. 
The resulting solution in turn is an explicit smooth differentiable function with a known analytical form. 
 
Unlike other numerical methods such as finite differences and finite elements, the derivatives of the desired function can be analytically calculated to any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is a unified formulation of both forward and inverse problems
where the optimized loss function consists of few elements: fidelity terms of L2 and L infinity norms that unlike previous methods promote a strong solution. Robust boundary conditions constraints and additional regularizers are included as well. This setting is flexible in the sense that regularizers can be tailored to specific 
problems. We demonstrate our method on several free shape 2D second order systems with application to Electrical Impedance Tomography (EIT).
Differential Equations as a Model Prior for Deep Learning and its Applications in Robotics
Authors: Michael Lutter, Jan Peters
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12

For many decades, much of the scientific knowledge of physics and engineering has been expressed via differential equations. These differential equations describe the underlying phenomena and the relations between different interpretable quantities. Therefore, differential equations are a promising approach to incorporate prior knowledge in machine learning models to obtain robust and interpretable models. In this paper, we summarize a straight forward approach to incorporate deep networks in differential equations to solve first-order non-linear differential equations by minimising the residual end-to-end. We describe the deep differential network that computes the functional value and smooth Jacobians in closed form. Afterwards, we demonstrate that the deep network Jacobians approximate the symbolic Jacboian and apply the proposed approach two robotics applications. These applications use differential equations as model prior for deep networks to learn physically plausible models and optimal feedback control.
Port-Hamiltonian Gradient Flows
Authors: Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12

In this paper we present a general framework for continuous--time gradient descent, often referred to as gradient flow. We extend Hamiltonian gradient flows, which ascribe mechanical dynamics to neural network parameters and constitute a natural continuous-time alternative to discrete momentum-based gradient descent approaches. The proposed Port-Hamiltonian Gradient Flow (PHGF)  casts neural network training into a system--theoretic framework: a fictitious physical system is coupled to the neural network by setting the loss function as an energy term of the system. As autonomous port--Hamiltonian systems naturally tend to dissipate energy towards one of its minima by construction, solving the system simultaneously trains the neural network. We show that general PHGFs are compatible with both continuous-time data--stream optimization, where the optimizer processes a continuous stream of data, as well as standard fixed-step optimization. In continuous-time, PHGFs allow for the embedding of black--box adaptive--step ODE solvers and are able to stick to the energy manifold, thus avoiding divergence due to large learning rates. In fixed-step optimization, on the other hand, PGHFs open the door to novel fixed-step approaches based on symplectic discretizations of the Port--Hamiltonian with similar memory footprint and computational complexity as momentum optimizers.
Neural Ordinary Differential Equation Value Networks for Parametrized Action Spaces
Authors: Stefano Massaroli, Michael Poli, Sanzhar Bakhtiyarov, Atsushi Yamashita, Hajime Asama, Jinkyoo Park
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12
Tasks: Hierarchical Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Action spaces equipped with parameter sets are a common occurrence in reinforcement learning applications. Solutions to problems of this class have been developed under different frameworks, such as parametrized action Markov decision processes (PAMDP) or hierarchical reinforcement learning (HRL). These approaches often require extensions or modifications to standard existing algorithms developed on standard MDPs. For this reason they can be unwieldy and, particularly in the case of HRL, computationally inefficient. We propose adopting a different parametrization scheme for state--action value networks based on neural ordinary differential equations (NODEs) as a scalable, plug--and--play approach for parametrized action spaces. NODEs value networks do not require extensive modification to existing algorithms nor the adoption of HRL methods. Our solution can directly be integrated into existing training algorithms and opens up new opportunities in single-agent and multi-agent settings with tight precision constraints on the action parameters such as robotics.
Generating Control Policies for Autonomous Vehicles Using Neural ODEs
Authors: Houston Lucas, Richard Kelley
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12
Tasks: Autonomous Vehicles

The problem of robot control often requires solving a system of
ordinary differential equations (ODEs).  Traditionally this has been
accomplished by using iterative ODE solvers.  These solvers start with
an initial guess, which is iteratively improved to converge to a
correct solution.  However, traditional solvers can be slow and do not
combine well with other systems since they are not differentiable.  In
response, some researchers have proposed using neural networks in an
end-to-end system that directly maps perceptual inputs to control
actions.  Because of their differentiablity, end-to-end approaches can
be composed with other modules more readily than traditional ODE
solvers.  However the end-to-end approach no longer carries the
guarantee that the solution obeys the required dynamics.
We propose a framework for using Neural ODE to
combine the flexibility of the end-to-end approach with the guarantees
of traditional solvers.  In our approach a neural network is used to
provide the initial guess to a differentiable ODE solver.  The ODE
solver then yields a solution trajectory. We use this trajectory to
improve the guesses of the neural network. This
framework allows the neural network to learn initial guesses that are
close to the correct solution, improving overall system performance
while ensuring that dynamics constraints are always satisfied. We
demonstrate the utility of this framework in the case of robot
control, where we use it to solve a family of boundary value problems
that are essential for steering an autonomous vehicle to a goal state.
Nano-Material Configuration Design with Deep Surrogate Langevin Dynamics
Authors: Thanh V. Nguyen, Youssef Mroueh, Samuel Hoffman, Payel Das, Pierre Dognin, Giuseppe Romano, Chinmay Hegde
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12
Tasks: Bayesian Optimization

We consider the problem of optimizing by sampling under multiple black-box constraints in nano-material design. We leverage the posterior regularization framework and show that the constraint satisfaction problem can be formulated as sampling from a Gibbs distribution. The main challenges come from the black-box nature of the constraints obtained by solving complex and expensive PDEs. To circumvent these issues, we introduce Surrogate-based Constrained Langevin dynamics for black-box sampling. We devise two approaches for learning  surrogate gradients of the black-box functions: first, by using zero-order gradients approximations; and second, by approximating the Langevin gradients  with deep neural networks. We prove the convergence of both approaches when the target distribution is $\log$-concave and smooth.  We also show the effectiveness of our approaches over Bayesian optimization in designing optimal nano-porous material configurations that achieve low thermal conductivity and reasonable mechanical stability.
Enforcing Physical Constraints in CNNs through Differentiable PDE Layer
Authors: Chiyu "Max" Jiang, Karthik Kashinath, Prabhat, Philip Marcus
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12
Tasks: Generative Adversarial Network

Recent studies at the intersection of physics and deep learning have illustrated successes in the application of deep neural networks to partially or fully replace costly physics simulations. Enforcing physical constraints to solutions generated by neural networks remains a challenge, yet it is essential to the accuracy and trustworthiness of such model predictions. Many systems in the physical sciences are governed by Partial Differential Equations (PDEs). Enforcing these as hard constraints, we show, are inefficient in conventional frameworks due to the high dimensionality of the generated fields. To this end, we propose the use of a novel differentiable spectral projection layer for neural networks that efficiently enforces spatial PDE constraints using spectral methods, yet is fully differentiable, allowing for its use as a layer within Convolutional Neural Networks (CNNs) during end-to-end training. We show that its computational cost is cheaper than a single convolution layer. We apply it to an important class of physical systems - incompressible turbulent flows, where the divergence-free PDE constraint is required. We train a 3D Conditional Generative Adversarial Network (CGAN) for turbulent flow superresolution efficiently, while guaranteeing the spatial PDE constraint of zero divergence. Furthermore, our empirical results show that the model produces realistic flow statistics when trained with hard constraints imposed via the proposed novel differentiable spectral projection layer, as compared to soft constrained and unconstrained counterparts.
Differentiable Physics Simulation
Authors: Junbang Liang, Ming C. Lin
Date: 2020-02-26 00:00:00
Proceeding: ICLR Workshop DeepDiffEq 2019 12

Differentiable physics simulation is a powerful family of new techniques that applies gradient-based methods to learning and control of physical systems. It enables optimization for control, and can also be integrated into neural network frameworks for performing complex tasks. We believe that differentiable physics simulation should be a key component for neural networks to bridge the gap between training performance and the generality to previously unseen real-world inputs. However, realizing a practical differentiable simulation is still challenging because of its high dimensionality and fragmented computation flow. In this paper, we motivate the importance of differentiable physics simulation, describe its current challenges, introduce state-of-the-art approaches, and discuss potential improvements and future directions.