pwc-archive/papers-with-abstracts huggingface dataset:
Semi-supervised regression with skewed data via adversarially forcing the distribution of predicted values
Authors: Dae-Woong Jeong, Kiyoung Kim, ChangYoung Park, Sehui Han, Woohyung Lim
Date: 2021-01-01 00:00:00
Tasks: Drug Discovery, regression

Advances in scientific fields including drug discovery or material design are accompanied by numerous trials and errors. However, generally only representative experimental results are reported. Because of this reporting bias, the distribution of labeled result data can deviate from their true distribution. A regression model can be erroneous if it is built on these skewed data. In this work, we propose a new approach to improve the accuracy of regression models that are trained using a skewed dataset. The method forces the regression outputs to follow the true distribution; the forcing algorithm regularizes the regression results while keeping the information of the training data. We assume the existence of enough unlabeled data that follow the true distribution, and that the true distribution can be roughly estimated from domain knowledge or a few samples. During training neural networks to generate a regression model, an adversarial network is used to force the distribution of predicted values to follow the estimated ‘true’ distribution. We evaluated the proposed approach on four real-world datasets (pLogP, Diamond, House, Elevators). In all four datasets, the proposed approach reduced the root mean squared error of the regression by around 55 percent to 75 percent compared to regression models without adjustment of the distribution.
TEAC: Intergrating Trust Region and Max Entropy Actor Critic for Continuous Control
Authors: Hongyu Zang, Xin Li, Li Zhang, Peiyao Zhao, Mingzhong Wang
Date: 2021-01-01 00:00:00
Tasks: continuous-control, Continuous Control, MuJoCo, Reinforcement Learning (RL)

Trust region methods and maximum entropy methods are two state-of-the-art branches used in reinforcement learning (RL) for the benefits of stability and exploration in continuous environments, respectively. This paper proposes to integrate both branches in a unified framework, thus benefiting from both sides. We first transform the original RL objective to a constraint optimization problem and then proposes trust entropy actor critic (TEAC), an off-policy algorithm to learn stable and sufficiently explored policies for continuous states and actions. TEAC trains the critic by minimizing the refined Bellman error and updates the actor by minimizing KL-divergence loss derived from the closed-form solution to the Lagrangian. 
We prove that the policy evaluation and policy improvement in TEAC is guaranteed to converge.
Extensive experiments on the tasks in the MuJoCo environment show that TEAC outperforms state-of-the-art solutions in terms of efficiency and effectiveness.
Don't stack layers in graph neural networks, wire them randomly
Authors: Diego Valsesia, Giulia Fracastoro, Enrico Magli
Date: 2021-01-01 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Attribute, Benchmarking

Graph neural networks have become a staple in problems addressing learning and analysis of data defined over graphs. However, several results suggest an inherent difficulty in extracting better performance by increasing the number of layers. Besides the classic vanishing gradient issues, recent works attribute this to a phenomenon peculiar to the extraction of node features in graph-based tasks, i.e., the need to consider multiple neighborhood sizes at the same time and adaptively tune them. In this paper, we investigate the recently proposed randomly wired architectures in the context of graph neural networks. Instead of building deeper networks by stacking many layers, we prove that employing a randomly-wired architecture can be a more effective way to increase the capacity of the network and obtain richer representations. We show that such architectures behave like an ensemble of paths, which are able to merge contributions from receptive fields of varied size. Moreover, these receptive fields can also be modulated to be wider or narrower through the trainable weights over the paths. We also provide extensive experimental evidence of the superior performance of randomly wired architectures over three tasks and five graph convolution definitions, using a recent benchmarking framework that addresses the reliability of previous testing methodologies.
Improving Neural Network Accuracy and Calibration Under Distributional Shift with Prior Augmented Data
Authors: Jeffrey Ryan Willette, Juho Lee, Sung Ju Hwang
Date: 2021-01-01 00:00:00

Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. However, neural networks are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. The problem of overconfidence becomes especially apparent in cases where the test-time data distribution differs from that which was seen during training. We propose a solution to this problem by seeking out regions in arbitrary feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the Bayesian prior on the distribution of the labels. Our method results in a better calibrated network and is agnostic to the underlying model structure, so it can be applied to any neural network which produces a probability density as an output. We demonstrate the effectiveness of our method and validate its performance on both classification and regression problems by applying it to the training of recent state-of-the-art neural network models.
Black-Box Adversarial Attacks on Graph Neural Networks as An Influence Maximization Problem
Authors: Jiaqi Ma, Junwei Deng, Qiaozhu Mei
Date: 2021-01-01 00:00:00
Tasks: Adversarial Attack

Graph neural networks (GNNs) have attracted increasing interests. With broad deployments of GNNs in real-world applications, there is an urgent need for understanding the robustness of GNNs under adversarial attacks, especially in realistic setups. In this work, we study the problem of attacking GNNs in a restricted black-box setup, by perturbing the features of a small set of nodes,  with no access to model parameters and model predictions. Our formal analysis draws a connection between this type of attacks and an influence maximization problem on the graph. This connection not only enhances our understanding on the problem of adversarial attack on GNNs, but also allows us to propose a group of effective black-box attack strategies. Our experiments verify that the proposed strategies significantly degrade the performance of three popular GNN models and outperform baseline adversarial attack strategies.
DyHCN: Dynamic Hypergraph Convolutional Networks
Authors: Nan Yin, Zhigang Luo, Wenjie Wang, Fuli Feng, Xiang Zhang
Date: 2021-01-01 00:00:00

Hypergraph Convolutional Network (HCN) has become a default choice for capturing high-order relations among nodes, \emph{i.e., } encoding the structure of a hypergraph. However, existing HCN models ignore the dynamic evolution of hypergraphs in the real-world scenarios, \emph{i.e., } nodes and hyperedges in a hypergraph change dynamically over time. To capture the evolution of high-order relations and facilitate relevant analytic tasks, we formulate dynamic hypergraph and devise the Dynamic Hypergraph Convolution Networks (DyHCN). In general, DyHCN consists of a Hypergraph Convolution (HC) to encode the hypergraph structure at a time point and a Temporal Evolution module (TE) to capture the varying of the relations. The HC is delicately designed by equipping inner attention and outer attention, which adaptively aggregate nodes' features to hyperedge and estimate the importance of each hyperedge connected to the centroid node, respectively. Extensive experiments on the Tiigo and Stocktwits datasets show that DyHCN achieves superior performance over existing methods, which implies the effectiveness of capturing the property of dynamic hypergraphs by HC and TE modules.
Sobolev Training for the Neural Network Solutions of PDEs
Authors: Hwijae Son, Jin Woo Jang, Woo Jin Han, Hyung Ju Hwang
Date: 2021-01-01 00:00:00

Approximating the numerical solutions of partial differential equations (PDEs) using neural networks is a promising application of deep learning. The smooth architecture of a fully connected neural network is appropriate for finding the solutions of PDEs; the corresponding loss function can also be intuitively designed and guarantees the convergence for various kinds of PDEs. However, the rate of convergence has been considered as a weakness of this approach. This paper introduces a novel loss function for the training of neural networks to find the solutions of PDEs, making the training substantially efficient. Inspired by the recent studies that incorporate derivative information for the training of neural networks, we develop a loss function that guides a neural network to reduce the error in the corresponding Sobolev space. Surprisingly, a simple modification of the loss function can make the training process similar to Sobolev Training although solving PDEs with neural networks is not a fully supervised learning task. We provide several theoretical justifications for such an approach for the viscous Burgers equation and the kinetic Fokker--Planck equation. We also present several simulation results, which show that compared with the traditional $L^2$ loss function, the proposed loss function guides the neural network to a significantly faster convergence. Moreover, we provide the empirical evidence that shows that the proposed loss function, together with the iterative sampling techniques, performs better in solving high dimensional PDEs.
Manifold-aware Training: Increase Adversarial Robustness with Feature Clustering
Authors: Ting-An Yen, Chun-Shien Lu, Pau-Choo Chung
Date: 2021-01-01 00:00:00
Tasks: Adversarial Robustness, Clustering

The problem of defending against adversarial attacks has attracted increasing attention in recent years. While various types of defense methods ($\textit{e.g.}$, adversarial training, detection and rejection, and recovery) were proven empirically to bring robustness to the network, their weakness was shown by later works. Inspired by the observation from the distribution properties of the features extracted by the CNNs in the feature space and their link to robustness, this work designs a novel training process called Manifold-Aware Training (MAT), which forces CNNs to learn compact features to increase robustness.  The effectiveness of the proposed method is evaluated via comparisons with existing defense mechanisms, $\textit{i.e.}$, the TRADES algorithm, which has been recognized as a representative state-of-the-art technology, and the MMC method, which also aims to learn compact features. Further verification is also conducted using the attack adaptive to our method. Experimental results show that MAT-trained CNNs exhibit significantly higher performance than state-of-the-art robustness.
Sensory Resilience based on Synesthesia
Authors: Eric Platon, Tom Sonoda
Date: 2021-01-01 00:00:00
Tasks: Emotion Recognition

Situated cognition depends on accessing environmental state through sensors. Engineering and cost constraints usually lead to limited “pathways” where, for example, a vision sub-system only includes a camera and the software to deal with it. This traditional and rational design style entails any hardware defect on the pathway causes the system to grind to a halt until repair. We propose a “sensoriplexer” as drop-in neural component architecture to address this issue, under the common scenario of multiple sensors availability. This component architecture learns to mix and relate pathways, such that an agent facing failure in a sensory sub-system can degrade gracefully and coherently by relying on its other sub- systems. The architecture is inspired by the concept of synesthesia, and relies on statistical coupling between sensor signals. We show the benefit and limitation of the architecture on a simple shape recognition and a more complex emotion recognition scenarios.
Learning to Solve Nonlinear Partial Differential Equation Systems To Accelerate MOSFET Simulation
Authors: Seungcheol Han, Jonghyun Choi, Sung-Min Hong
Date: 2021-01-01 00:00:00

Semiconductor device simulation uses numerical analysis, where a set of coupled nonlinear partial differential equations is solved with the iterative Newton-Raphson method. Since an appropriate initial guess to start the Newton-Raphson method is not available, a solution of practical importance with desired boundary conditions cannot be trivially achieved. Instead, several solutions with intermediate boundary conditions should be calculated to address the nonlinearity and introducing intermediate boundary conditions significantly increases the computation time. In order to accelerate the semiconductor device simulation, we propose to use a neural network to learn an approximate solution for desired boundary conditions. With an initial solution sufficiently close to the final one by a convolutional neural network, computational cost to calculate several unnecessary solutions is significantly reduced. Specifically, a convolutional neural network for MOSFET (Metal-Oxide-Semiconductor Field-Effect Transistor), the most widely used semiconductor device, is trained in a supervised manner to compute the initial solution. Particularly, we propose to consider device grids with varying size and spacing and derive a compact expression of the solution based upon the electrostatic potential. Finally, we empirically show that the proposed method accelerates the simulation by more than 12 times.
Transformer-QL: A Step Towards Making Transformer Network Quadratically Large
Authors: Suvadeep Hajra
Date: 2021-01-01 00:00:00
Tasks: Language Modeling, Language Modelling

Transformer networks have shown outstanding performance on many natural language processing tasks. However the context length (the number of previous tokens on which the output states depend) of a Transformer network grows at best linearly with the memory and computational power used. This limitation prevents a transformer network to have very long context in a resource limited application. In this work, we propose a class of transformer networks, namely Transformer-QL (\bf{Q}uadratically \bf{L}arge), in which, the context length can grow at best quadratically with the memory and computational power used. We have empirically evaluated a Transformer-QL model in three long range language modeling datasets. The results show that Transformer-QL can provide significant improvements over other state of the art networks.
Embedding a random graph via GNN: mean-field inference theory and RL applications to NP-Hard multi-robot/machine scheduling
Authors: Hyunwook Kang, SEUNGWOO SCHIN, James Morrison, Jinkyoo Park
Date: 2021-01-01 00:00:00
Tasks: Scheduling

We develop a theory for embedding a random graph using graph neural networks (GNN) and illustrate its capability to solve NP-hard scheduling problems. We apply the theory to address the challenge of developing a near-optimal learning algorithm to solve the NP-hard problem of scheduling multiple robots/machines with time-varying rewards. In particular, we consider a class of reward collection problems called Multi-Robot Reward Collection (MRRC). Such MRRC problems well model ride-sharing, pickup-and-delivery, and a variety of related problems. We consider the classic identical parallel machine scheduling problem (IPMS) in the Appendix.  

For the theory, we first observe that MRRC system state can be represented as an extension of probabilistic graphical models (PGMs), which we refer to as random PGMs. We then develop a mean-field inference method for random PGMs. 
We prove that a simple modification of a typical GNN embedding is sufficient to embed a random graph even when the edge presence probabilities are interdependent.

Our theory enables a two-step hierarchical inference for precise and transferable Q-function estimation for MRRC and IPMS. For scalable computation, we show that the transferability of Q-function estimation enables us to design a polynomial-time algorithm with 1-1/e optimality bound. 
Experimental results on solving NP-hard MRRC problems (and IMPS in the Appendix) highlight the near-optimality and transferability of the proposed methods.
ARMOURED: Adversarially Robust MOdels using Unlabeled data by REgularizing Diversity
Authors: Kangkang Lu, Cuong Manh Nguyen, Xun Xu, Kiran Chari, Yu Jing Goh, Chuan-Sheng Foo
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Diversity, MULTI-VIEW LEARNING, Point Processes

Adversarial attacks pose a major challenge for modern deep neural networks. Recent advancements show that adversarially robust generalization requires a huge amount of labeled data for training. If annotation becomes a burden, can unlabeled data help bridge the gap? In this paper, we propose ARMOURED, an adversarially robust training method based on semi-supervised learning that consists of two components. The first component applies multi-view learning to simultaneously optimize multiple independent networks and utilizes unlabeled data to enforce labeling consistency. The second component reduces adversarial transferability among the networks via diversity regularizers inspired by determinantal point processes and entropy maximization. Notably, ARMOURED does not rely on generating adversarial samples during training. We demonstrate the robustness of ARMOURED on CIFAR-10 and SVHN datasets against state-of-the-art benchmarks in both the adversarial robust training and the semi-supervised training domains. Experimental results show that under projected gradient descent attacks with bounded $\ell_{\infty}$ norm, ARMOURED achieves substantial gains in accuracy, while maintaining high accuracy on clean samples.
Dynamically locating multiple speakers based on the time-frequency domain
Authors: Hodaya Hammer, Shlomo Chazan, Jacob Goldberger, Sharon Gannot
Date: 2021-01-01 00:00:00

In this study we present a deep neural network-based online multi-speaker localisation algorithm based on a multi-microphone array. A fully convolutional network is trained with instantaneous spatial features to estimate the direction of arrival for each time-frequency  bin. The high resolution classification enables the network to accurately and simultaneously localize and track multiple speakers, both static and dynamic. Elaborated experimental study using simulated and real-life recordings in static and dynamic scenarios, demonstrates that the proposed algorithm significantly outperforms both classic and recent deep-learning-based algorithms.
Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling
Authors: Yang Zhao, Jianwen Xie, Ping Li
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Image Generation, Image-to-Image Translation, Translation, Unsupervised Image-To-Image Translation

Energy-based models (EBMs) for generative modeling parametrize a single net and can be directly trained by maximum likelihood estimation. Despite the simplicity and tractability, current approaches are either unstable to learn or difficult to synthesize diverse and high-fidelity images. We propose to train EBM via a multistage coarse-to-fine expanding and sampling strategy, namely CF-EBM. For the purpose of improving the learning procedure, we construct an effective net architecture and advocate applying smooth activations. The resulting approach is shown to be computationally efficient and achieves the best performance on image generation amongst EBMs and the spectral normalization GAN. Furthermore, we provide a recipe for being the first successful EBM to synthesize $256\times256$-pixel images. In the end, we effortlessly generalize CF-EBM to the one-sided unsupervised image-to-image translation and beat baseline methods with the model size reduced by $1000\times$ and the training budget by $9\times$. In parallel, we present a gradient-based discriminative saliency method to explicitly interpret the translation dynamics which align with human behavior.
An information-theoretic framework for learning models of instance-independent label noise
Authors: Xia Huang, Kai Fong Ernest Chong
Date: 2021-01-01 00:00:00

Given a dataset $\mathcal{D}$ with label noise, how do we learn its underlying noise model? If we assume that the label noise is instance-independent, then the noise model can be represented by a noise transition matrix $Q_{\mathcal{D}}$. Recent work has shown that even without further information about any instances with correct labels, or further assumptions on the distribution of the label noise, it is still possible to estimate $Q_{\mathcal{D}}$ while simultaneously learning a classifier from $\mathcal{D}$. However, this presupposes that a good estimate of $Q_{\mathcal{D}}$ requires an accurate classifier. In this paper, we show that high classification accuracy is actually not required for estimating $Q_{\mathcal{D}}$ well. We shall introduce an information-theoretic-based framework for estimating $Q_{\mathcal{D}}$ solely from $\mathcal{D}$ (without additional information or assumptions). At the heart of our framework is a discriminator that predicts whether an input dataset has maximum Shannon entropy, which shall be used on multiple new datasets $\hat{\mathcal{D}}$ synthesized from $\mathcal{D}$ via the insertion of additional label noise. We prove that our estimator for $Q_{\mathcal{D}}$ is statistically consistent, in terms of dataset size, and the number of intermediate datasets $\hat{\mathcal{D}}$ synthesized from $\mathcal{D}$. As a concrete realization of our framework, we shall incorporate local intrinsic dimensionality (LID) into the discriminator, and we show experimentally that with our LID-based discriminator, the estimation error for $Q_{\mathcal{D}}$ can be significantly reduced. We achieved average Kullback--Leibler loss reduction from $0.27$ to $0.17$ for $40\%$ anchor-like samples removal when evaluated on the CIFAR10 with symmetric noise. Although no clean subset of $\mathcal{D}$ is required for our framework to work, we show that our framework can also take advantage of clean data to improve upon existing estimation methods.
Multi-Prize Lottery Ticket Hypothesis: Finding Generalizable and Efficient Binary Subnetworks in a Randomly Weighted Neural Network
Authors: James Diffenderfer, Bhavya Kailkhura
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Quantization

Recently, \cite{frankle2018lottery} demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network. However, finding these high performing trainable subnetworks is expensive, requiring iterative process of training and pruning weights. 
In this paper, we propose (and prove) a stronger \emph{Multi-Prize Lottery Ticket Hypothesis}:

A sufficiently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3).

These multi-prize tickets enjoy a number of desirable properties including drastically reduced memory size, faster test-time inference, and lower power consumption compared to their dense and full-precision counterparts. Furthermore, we propose an algorithm for finding multi-prize tickets and test it by performing a series of experiments on CIFAR-10 and ImageNet datasets. Empirical results indicate that as models grow deeper and wider, untrained multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to their significantly larger and full-precision counterparts that have been weight-trained. With minimal hyperparameter tuning, our binary weight multi-prize tickets outperform current state-of-the-art in binary neural networks.