pwc-archive/papers-with-abstracts huggingface dataset:
Hidden Markov models are recurrent neural networks: A disease progression modeling application
Authors: Matthew Baucum, Anahita Khojandi, Theodore Papamarkou
Date: 2020-09-28 00:00:00
Tasks: parameter estimation

Hidden Markov models (HMMs) are commonly used for disease progression modeling when the true state of a patient is not fully known. Since HMMs may have multiple local optima, performance can be improved by incorporating additional patient covariates to inform parameter estimation. To allow for this, we formulate a special case of recurrent neural networks (RNNs), which we name hidden Markov recurrent neural networks (HMRNNs), and prove that each HMRNN has the same likelihood function as a corresponding discrete-observation HMM. As a neural network, the HMRNN can also be combined with any other predictive neural networks that take patient covariate information as input. We first show that parameter estimates from HMRNNs are numerically close to those obtained from HMMs via the Baum-Welch algorithm, thus empirically validating their theoretical equivalence. We then demonstrate how the HMRNN can be combined with other neural networks to improve parameter estimation and prediction, using an Alzheimer's disease dataset. The HMRNN yields parameter estimates that improve disease forecasting performance and offer a novel clinical interpretation compared with a standard HMM.
Enhanced First and Zeroth Order Variance Reduced Algorithms for Min-Max Optimization
Authors: Tengyu Xu, Zhe Wang, Yingbin Liang, H. Vincent Poor
Date: 2020-09-28 00:00:00

Min-max optimization captures many important machine learning problems such as robust adversarial learning and inverse reinforcement learning, and nonconvex-strongly-concave min-max optimization has been an active line of research. Specifically, a novel variance reduction algorithm SREDA was proposed recently by (Luo et al. 2020) to solve such a problem, and was shown to achieve the optimal complexity dependence on the required accuracy level $\epsilon$. Despite the superior theoretical performance, the convergence guarantee of SREDA requires stringent initialization accuracy and an $\epsilon$-dependent stepsize for controlling the per-iteration progress, so that SREDA can run very slowly in practice. This paper develops a novel analytical framework that guarantees the SREDA's optimal complexity performance for a much enhanced algorithm SREDA-Boost, which has less restrictive initialization requirement and an accuracy-independent (and much bigger) stepsize. Hence, SREDA-Boost runs substantially faster in experiments than SREDA. We further apply SREDA-Boost to propose a zeroth-order variance reduction algorithm named ZO-SREDA-Boost for the scenario that has access only to the information about function values not gradients, and show that ZO-SREDA-Boost outperforms the best known complexity dependence on $\epsilon$. This is the first study that applies the variance reduction technique to zeroth-order algorithm for min-max optimization problems.
Revisiting Graph Neural Networks for Link Prediction
Authors: Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, Long Jin
Date: 2020-09-28 00:00:00
Tasks: Classification, Graph Classification, Link Prediction, Node Classification, Prediction

Graph neural networks (GNNs) have achieved great success in recent years. Three most common applications include node classification, link prediction, and graph classification. While there is rich literature on node classification and graph classification, GNNs for link prediction is relatively less studied and less understood. Two representative classes of methods exist: GAE and SEAL. GAE (Graph Autoencoder) first uses a GNN to learn node embeddings for all nodes, and then aggregates the embeddings of the source and target nodes as their link representation. SEAL extracts a subgraph around the source and target nodes, labels the nodes in the subgraph, and then uses a GNN to learn a link representation from the labeled subgraph. In this paper, we thoroughly discuss the differences between these two classes of methods, and conclude that simply aggregating \textit{node} embeddings does not lead to effective \textit{link} representations, while learning from \textit{properly labeled subgraphs} around links provides highly expressive and generalizable link representations. Experiments on the recent large-scale OGB link prediction datasets show that SEAL has up to 195\% performance gains over GAE methods, achieving new state-of-the-art results on 3 out of 4 datasets.
On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks
Authors: Ben Lengerich, Eric Xing, Rich Caruana
Date: 2020-09-28 00:00:00

We examine Dropout through the perspective of interactions. Given $N$ variables, there are $\mathcal{O}(N^2)$ possible pairwise interactions, $\mathcal{O}(N^3)$ possible 3-way interactions, i.e. $\mathcal{O}(N^k)$ possible interactions of $k$ variables. Conversely, the probability of an interaction of $k$ variables surviving Dropout at rate $p$ is $\mathcal{O}((1-p)^k)$. In this paper, we show that these rates cancel, and as a result, Dropout selectively regularizes against learning higher-order interactions. We prove this new perspective analytically for Input Dropout and empirically for Activation Dropout. This perspective on Dropout has several practical implications: (1) higher Dropout rates should be used when we need stronger regularization against spurious high-order interactions, (2) caution must be used when interpreting Dropout-based feature saliency measures, and (3) networks trained with Input Dropout are biased estimators, even with infinite data. We also compare Dropout to regularization via weight decay and early stopping and find that it is difficult to obtain the same regularization against high-order interactions with these methods.
Multi-Source Unsupervised Hyperparameter Optimization
Authors: Masahiro Nomura, Yuta Saito
Date: 2020-09-28 00:00:00
Tasks: Hyperparameter Optimization

How can we conduct efficient hyperparameter optimization for a completely new task? In this work, we consider a novel setting, where we search for the optimal hyperparameters for a target task of interest using only unlabeled target task and ‘somewhat relevant’ source task datasets. In this setting, it is essential to estimate the ground-truth target task objective using only the available information. We propose estimators to unbiasedly approximate the ground-truth with a desirable variance property. Building on these estimators, we provide a general and tractable hyperparameter optimization procedure for our setting. The experimental evaluations demonstrate that the proposed framework broadens the applications of automated hyperparameter optimization.
PanRep: Universal node embeddings for heterogeneous graphs
Authors: Vassilis N. Ioannidis, Da Zheng, George Karypis
Date: 2020-09-28 00:00:00
Tasks: Graph Neural Network, Link Prediction, Node Classification

Learning unsupervised node embeddings facilitates several downstream tasks such as node classification and link prediction. A node embedding is universal if it is designed to be used by and benefit various downstream tasks. This work introduces PanRep, a graph neural network (GNN) model, for unsupervised learning of universal node representations for heterogenous graphs. PanRep consists of a GNN encoder that obtains node embeddings and four decoders, each capturing different topological and node feature properties. Abiding to these properties the novel unsupervised framework learns universal embeddings applicable to different downstream tasks. PanRep can be furthered fine-tuned to account for possible limited labels. In this operational setting PanRep is considered as a pretrained model for extracting node embeddings of heterogenous graph data. PanRep outperforms all unsupervised and certain supervised methods in node classification and link prediction, especially when the labeled data for the supervised methods is small. PanRep-FT (with fine-tuning) outperforms all other supervised approaches, which corroborates the merits of pretraining models. Finally, we apply PanRep-FT for discovering novel drugs for Covid-19. We showcase the advantage of universal embeddings in drug repurposing and identify several drugs used in clinical trials as possible drug candidates.
Rewriter-Evaluator Framework for Neural Machine Translation
Authors: Yangming Li, Kaisheng Yao
Date: 2020-09-28 00:00:00
Tasks: Decoder, Machine Translation, NMT, Sentence, Translation

Encoder-decoder architecture has been widely used in neural machine translation (NMT). A few methods have been proposed to improve it with multiple passes of decoding. However, their full potential is limited by a lack of appropriate termination policy. To address this issue, we present a novel framework, Rewriter-Evaluator. It consists of a rewriter and an evaluator. Translating a source sentence involves multiple passes. At every pass, the rewriter produces a new translation to improve the past translation and the evaluator estimates the translation quality to decide whether to terminate the rewriting process. We also propose a prioritized gradient descent (PGD) method that facilitates training the rewriter and the evaluator jointly. Though incurring multiple passes of decoding, Rewriter-Evaluator with the proposed PGD method can be trained with similar time to that of training encoder-decoder models. We apply the proposed framework to improve the general NMT models (e.g., Transformer). We conduct extensive experiments on two translation tasks, Chinese-English and English-German, and show that the proposed framework notably improves the performances of NMT models and significantly outperforms previous baselines.
Neural Time-Dependent Partial Differential Equation
Authors: Yihao Hu, Tong Zhao, Zhiliang Xu, Lizhen Lin
Date: 2020-09-28 00:00:00

Partial differential equations (PDEs) play a crucial role in studying a vast number of problems in science and engineering. Numerically solving nonlinear and/or high-dimensional PDEs is frequently a challenging task. Inspired by the traditional finite difference and finite elements methods and emerging advancements in machine learning, we propose a  sequence-to-sequence learning (Seq2Seq) framework called Neural-PDE, which allows one to automatically learn governing rules of any time-dependent PDE system from existing data by using a bidirectional LSTM encoder, and predict the solutions in next $n$ time steps. One critical feature of our proposed framework is that the Neural-PDE is able to  simultaneously learn and simulate all variables of interest in a PDE system. We test the Neural-PDE by a range of  examples, from one-dimensional PDEs to a multi-dimensional and nonlinear complex fluids model. The results show that the Neural-PDE is capable of learning the initial conditions, boundary conditions and differential operators defining the initial-boundary-value problem of a PDE system without the knowledge of the specific form of the PDE system. In our experiments, the Neural-PDE can efficiently extract the dynamics within 20 epochs training and produce accurate predictions. Furthermore, unlike the traditional machine learning approaches for learning PDEs, such as CNN and MLP, which require great quantity of parameters for model precision, the Neural-PDE shares parameters among all time steps, and thus considerably reduces computational complexity and leads to a fast learning algorithm.
Learn Robust Features via Orthogonal Multi-Path
Authors: Kun Fang, Xiaolin Huang, Yingwen Wu, Tao Li, Jie Yang
Date: 2020-09-28 00:00:00

	It is now widely known that by adversarial attacks, clean images with invisible perturbations can fool deep neural networks.
	To defend adversarial attacks, we design a block containing multiple paths to learn robust features and the parameters of these paths are  required to be orthogonal with each other. 
	The so-called Orthogonal Multi-Path (OMP) block could be posed in any layer of a neural network. 
	Via forward learning and backward correction, one OMP block makes the neural networks learn features that are appropriate for all the paths and hence are expected to be robust. With careful design and thorough experiments on e.g., the positions of imposing orthogonality constraint, and the trade-off between the variety and accuracy, 
	the robustness of the neural networks is significantly improved. 
	For example, under white-box PGD attack with $l_\infty$ bound ${8}/{255}$ (this is a fierce attack that can make the accuracy of many vanilla neural networks drop to nearly $10\%$ on CIFAR10), VGG16 with the proposed OMP block could keep over $50\%$ accuracy. For black-box attacks, neural networks equipped with an OMP block have accuracy over $80\%$. The performance under both white-box and black-box attacks is much better than the existing state-of-the-art adversarial defenders.
No Spurious Local Minima: on the Optimization Landscapes of Wide and Deep Neural Networks
Authors: Johannes Lederer
Date: 2020-09-28 00:00:00
Tasks: Deep Learning

Empirical studies suggest that wide neural networks are comparably easy to optimize, but mathematical support for this observation is scarce. In this paper, we analyze the optimization landscapes of deep learning with wide networks. We prove especially that constraint and unconstraint empirical-risk minimization over such networks has no spurious local minima. Hence, our theories substantiate the common belief that increasing network widths not only improves the expressiveness of deep-learning pipelines but also facilitates their optimizations.
Secure Network Release with Link Privacy
Authors: Carl Yang, Haonan Wang, Ke Zhang, Lichao Sun
Date: 2020-09-28 00:00:00
Tasks: Graph Generation

Many data mining and analytical tasks rely on the abstraction of networks (graphs) to summarize relational structures among individuals (nodes). Since relational data are often sensitive, we aim to seek effective approaches to release utility-preserved yet privacy-protected structured data. In this paper, we leverage the differential privacy (DP) framework, to formulate and enforce rigorous privacy constraints on deep graph generation models, with a focus on edge-DP to guarantee individual link privacy. In particular, we enforce edge-DP by injecting Gaussian noise to the gradients of a link reconstruction based graph generation model, and ensure data utility by improving structure learning with structure-oriented graph comparison. Extensive experiments on two real-world network datasets show that our proposed DPGGAN model is able to generate networks with effectively preserved global structure and rigorously protected individual link privacy.
Proper Measure for Adversarial Robustness
Authors: Hyeongji Kim, Ketil Malde
Date: 2020-09-28 00:00:00
Tasks: Adversarial Robustness

This paper analyzes the problems of adversarial accuracy and adversarial training. We argue that standard adversarial accuracy fails to properly measure the robustness of classifiers. Its definition has a tradeoff with standard accuracy even when we neglect generalization. In order to handle the problems of the standard adversarial accuracy, we introduce a new measure for the robustness of classifiers called genuine adversarial accuracy. It can measure the adversarial robustness of classifiers without trading off accuracy on clean data and accuracy on the adversarially perturbed samples. In addition, it does not favor a model with invariance-based adversarial examples, samples whose predicted classes are unchanged even if the perceptual classes are changed. We prove that a single nearest neighbor (1-NN) classifier is the most robust classifier according to genuine adversarial accuracy for given data and a norm-based distance metric when the class for each data point is unique. Based on this result, we suggest that using poor distance metrics might be one factor for the tradeoff between test accuracy and $l_p$ norm-based test adversarial robustness.
Energy-based Models for Earth Observation Applications
Authors: Javiera Castillo Navarro, Bertrand Le Saux, Alexandre Boulch, Sébastien Lefèvre
Date: 2021-02-26 00:00:00
Proceeding: ICLR Workshop EBM 2021 5
Tasks: Classification, Earth Observation, Image Generation

The large amount of data, available thanks to the recent sensors, have made possible the use of deep learning for Earth Observation.
Yet, actual approaches tend to tackle one problem at a time, e.g. classification or image generation. We propose a new framework for Earth Observation images processing which learns an energy-based model to estimate the underlying distribution, possibly estimated using non-annotated images.
On the varied image types of the EuroSAT benchmark, we show this model obtains classification results on par with state-of-the-art and moreover allows to tackle a high range of high-potential applications, from image synthesis to high performance semi-supervised learning.
Consistent Assignment for Representation Learning
Authors: Thalles Santos Silva, Adín Ramírez Rivera
Date: 2021-02-26 00:00:00
Proceeding: ICLR Workshop EBM 2021 5
Tasks: Clustering, Contrastive Learning, Deep Clustering, Representation Learning

We introduce Consistent Assignment for Representation Learning (CARL).  An unsupervised learning method to learn visual representations by combining contrastive learning with deep clustering. By viewing contrastive learning from a clustering perspective, CARL learns unsupervised representations by learning a set of general prototypes that serve as energy anchors to enforce different views of a given image to be assigned to the same prototype. Unlike contemporary work on contrastive learning with deep clustering, CARL proposes to learn the set of general prototypes in an online fashion, using gradient descent without the necessity of performing offline clustering or using non-differentiable algorithms to solve the cluster assignment problem. CARL achieves comparable results with current state-of-the-art methods in the CIFAR-10, -100, and STL10 datasets.
No Conditional Models for me: Training Joint EBMs on Mixed Continuous and Discrete Data
Authors: Jacob Kelly, Will Sussman Grathwohl
Date: 2021-02-26 00:00:00
Proceeding: ICLR Workshop EBM 2021 5

We propose energy-based models of the joint distribution of data and supervision. While challenging to work with, this approach gives flexibility when designing energy functions and easy parameterization for structured supervision. Further, these models naturally allow training on partially observed data and predictions conditioned on any subset of the modeled variables. We identify and address the main difficulty in working with these models: sampling from the joint distribution of data and supervision. We build upon recent developments in discrete MCMC sampling and apply them alongside continuous MCMC techniques developed for energy-based models. We present experimental results demonstrating that our proposed approach can successfully train joint energy-based models on high-dimensional data with structured supervision capable of both accurate prediction and conditional sampling.
Graph Energy-based Model for Molecular Graph Generation
Authors: Ryuichiro Hataya, Hideki Nakayama, Kazuki Yoshizoe
Date: 2021-02-26 00:00:00
Proceeding: ICLR Workshop EBM 2021 5
Tasks: Graph Generation, model, Molecular Graph Generation

We present Graph Energy-based Model (GEM), an energy-based model for molecular graph generation. GEM uses dequantization and gradient symmetrization to incorporate generation by stochastic gradient Langevin dynamics for graph representation that is discrete and includes symmetric constraint. Experimental results show that \gem can comparably design compounds as other deep generative approaches.
Connecting convex energy-based inference and optimal transport for domain adaptation
Authors: Arip Asadulaev
Date: 2021-02-26 00:00:00
Proceeding: ICLR Workshop EBM 2021 5
Tasks: Benchmarking, Domain Adaptation, Semi-supervised Domain Adaptation

The connection of optimal transport and neural networks finds a rich application in machine learning problems. In this paper, we propose a simple algorithm for the mutual improvement of optimal transport and energy-based models for the semi-supervised domain adaptation. Having target and source domain samples we use convex energy-based inference to create a new domain that is class-wise cyclical monotone to the target domain and its samples contains features from the source domain. Mapping from target to such domain can be solved by optimal transport much more successfully. We study the performance of our approach by benchmarking it on a range of optimal transport methods and showed that in our settings optimal transport can achieve much higher results.
Should EBMs model the energy or the score?
Authors: Tim Salimans, Jonathan Ho
Date: 2021-02-26 00:00:00
Proceeding: ICLR Workshop EBM 2021 5
Tasks: Denoising, model

Recent progress in training unnormalized models through denoising score matching with Langevin dynamics (SMLD) and denoising diffusion probabilistic modeling (DDPM) has made unnormalized models a competitive model class for generative modeling. Unlike earlier work on energy based models, these recent works construct generative models by directly parameterizing the score function of the model density, rather than the density itself. Such unconstrained score models are not guaranteed to output a conservative vector field, meaning they do not correspond to the gradient of any function, unlike constrained score models that are obtained through explicitly differentiating a parameterized energy function. Explicit energy based models thus seem to have a theoretical advantage, but empirical evidence currently points to unconstrained score models performing better in practice. Here we compare both methods for modeling the score of the data distribution, finding that constrained score models, i.e. energy based models, can perform just as well as unconstrained models when using a comparable model structure.
Sanity Check for Persistence Diagrams
Authors: Chen Cai
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Graph Classification, Topological Data Analysis

Recently many efforts have been made to incorporate persistence diagrams, one of major tools in topological data analysis (TDA), into machine learning pipelines. To better understand the power and limitation of persistence diagrams, we carry out a range of experiments on both graph and shape data, aiming to decouple and inspect the effects of different factors involved. To this end, we propose a sanity check for persistence diagrams to delineate critical values and pairings of critical values (structure of persistence module). For graph classification tasks, we note that while persistence pairing yields consistent improvement over various benchmark datasets, it appears that for various filtration functions tested, most discriminative power comes from critical values. For shape segmentation and classification, however, we note that persistence pairing shows significant power on most of the benchmark datasets, and improves over summaries based on merely critical values. Our results help provide insights on when persistence diagram based summaries could be more suitable.