pwc-archive/papers-with-abstracts huggingface dataset:
Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
Authors: Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, Florent Krzakala
Date: 2020-03-02 00:00:00

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly. Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization. In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression. We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant. We disentangle the variances stemming from the sampling of the dataset, from the additive noise corrupting the labels, and from the initialization of the weights. Following up on Geiger et al. 2019, we first show that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization. We then quantify how they are suppressed by ensemble averaging the outputs of K independently initialized estimators. When K is sent to infinity, the test error remains constant beyond the interpolation threshold. We further compare the effects of overparametrizing, ensembling and regularizing. Finally, we present numerical experiments on classic deep learning setups to show that our results hold qualitatively in realistic lazy learning scenarios.
Learning from Positive and Unlabeled Data by Identifying the Annotation Process
Authors: Naji Shajarisales, Peter Spirtes, Kun Zhang
Date: 2020-03-02 00:00:00
Tasks: Binary Classification, General Classification

In binary classification, Learning from Positive and Unlabeled data (LePU) is semi-supervised learning but with labeled elements from only one class. Most of the research on LePU relies on some form of independence between the selection process of annotated examples and the features of the annotated class, known as the Selected Completely At Random (SCAR) assumption. Yet the annotation process is an important part of the data collection, and in many cases it naturally depends on certain features of the data (e.g., the intensity of an image and the size of the object to be detected in the image). Without any constraints on the model for the annotation process, classification results in the LePU problem will be highly non-unique. So proper, flexible constraints are needed. In this work we incorporate more flexible and realistic models for the annotation process than SCAR, and more importantly, offer a solution for the challenging LePU problem. On the theory side, we establish the identifiability of the properties of the annotation process and the classification function, in light of the considered constraints on the data-generating process. We also propose an inference algorithm to learn the parameters of the model, with successful experimental results on both simulated and real data. We also propose a novel real-world dataset forLePU, as a benchmark dataset for future studies.
Gaussian Process Policy Optimization
Authors: Ashish Rao, Bidipta Sarkar, Tejas Narayanan
Date: 2020-03-02 00:00:00
Tasks: MuJoCo, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We propose a novel actor-critic, model-free reinforcement learning algorithm which employs a Bayesian method of parameter space exploration to solve environments. A Gaussian process is used to learn the expected return of a policy given the policy's parameters. The system is trained by updating the parameters using gradient descent on a new surrogate loss function consisting of the Proximal Policy Optimization 'Clipped' loss function and a bonus term representing the expected improvement acquisition function given by the Gaussian process. This new method is shown to be comparable to and at times empirically outperform current algorithms on environments that simulate robotic locomotion using the MuJoCo physics engine.
Predictive Coding for Locally-Linear Control
Authors: Rui Shu, Tung Nguyen, Yin-Lam Chow, Tuan Pham, Khoat Than, Mohammad Ghavamzadeh, Stefano Ermon, Hung H. Bui
Date: 2020-03-02 00:00:00
Proceeding: ICML 2020 1
Tasks: Decision Making, Decoder

High-dimensional observations and unknown dynamics are major challenges when applying optimal control to many real-world decision making tasks. The Learning Controllable Embedding (LCE) framework addresses these challenges by embedding the observations into a lower dimensional latent space, estimating the latent dynamics, and then performing control directly in the latent space. To ensure the learned latent dynamics are predictive of next-observations, all existing LCE approaches decode back into the observation space and explicitly perform next-observation prediction---a challenging high-dimensional task that furthermore introduces a large number of nuisance parameters (i.e., the decoder) which are discarded during control. In this paper, we propose a novel information-theoretic LCE approach and show theoretically that explicit next-observation prediction can be replaced with predictive coding. We then use predictive coding to develop a decoder-free LCE model whose latent dynamics are amenable to locally-linear control. Extensive experiments on benchmark tasks show that our model reliably learns a controllable latent space that leads to superior performance when compared with state-of-the-art LCE baselines.
On the Global Convergence of Training Deep Linear ResNets
Authors: Difan Zou, Philip M. Long, Quanquan Gu
Date: 2020-03-02 00:00:00
Proceeding: ICLR 2020 1

We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets). We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss. Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets. Compared with the global convergence result of GD for training standard deep linear networks (Du & Hu 2019), our condition on the neural network width is sharper by a factor of $O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance matrix of the training data. We further propose a modified identity input and output transformations, and show that a $(d+k)$-wide neural network is sufficient to guarantee the global convergence of GD/SGD, where $d,k$ are the input and output dimensions respectively.
Generating EEG features from Acoustic features
Authors: Gautam Krishna, Co Tran, Mason Carnahan, Yan Han, Ahmed H. Tewfik
Date: 2020-02-29 00:00:00
Tasks: EEG, Electroencephalogram (EEG), Generative Adversarial Network, regression, Speech Synthesis

In this paper we demonstrate predicting electroencephalograpgy (EEG) features from acoustic features using recurrent neural network (RNN) based regression model and generative adversarial network (GAN). We predict various types of EEG features from acoustic features. We compare our results with the previously studied problem on speech synthesis using EEG and our results demonstrate that EEG features can be generated from acoustic features with lower root mean square error (RMSE), normalized RMSE values compared to generating acoustic features from EEG features (ie: speech synthesis using EEG) when tested using the same data sets.
Cell Mechanics Based Computational Classification of Red Blood Cells Via Machine Intelligence Applied to Morpho-Rheological Markers
Authors: Yan Ge, Philipp Rosendahl, Claudio Durán, Nicole Töpfner, Sara Ciucci, Jochen Guck, Carlo Vittorio Cannistraci
Date: 2020-03-02 00:00:00
Tasks: General Classification

Despite fluorescent cell-labelling being widely employed in biomedical studies, some of its drawbacks are inevitable, with unsuitable fluorescent probes or probes inducing a functional change being the main limitations. Consequently, the demand for and development of label-free methodologies to classify cells is strong and its impact on precision medicine is relevant. Towards this end, high-throughput techniques for cell mechanical phenotyping have been proposed to get a multidimensional biophysical characterization of single cells. With this motivation, our goal here is to investigate the extent to which an unsupervised machine learning methodology, which is applied exclusively on morpho-rheological markers obtained by real-time deformability and fluorescence cytometry (RT-FDC), can address the difficult task of providing label-free discrimination of reticulocytes from mature red blood cells. We focused on this problem, since the characterization of reticulocytes (their percentage and cellular features) in the blood is vital in multiple human disease conditions, especially bone-marrow disorders such as anemia and leukemia. Our approach reports promising label-free results in the classification of reticulocytes from mature red blood cells, and it represents a step forward in the development of high-throughput morpho-rheological-based methodologies for the computational categorization of single cells. Besides, our methodology can be an alternative but also a complementary method to integrate with existing cell-labelling techniques.
Deep Learning in Mining Biological Data
Authors: Mufti Mahmud, M. Shamim Kaiser, Amir Hussain
Date: 2020-02-28 00:00:00
Tasks: Deep Learning

Recent technological advancements in data acquisition tools allowed life scientists to acquire multimodal data from different biological application domains. Broadly categorized in three types (i.e., sequences, images, and signals), these data are huge in amount and complex in nature. Mining such an enormous amount of data for pattern recognition is a big challenge and requires sophisticated data-intensive machine learning techniques. Artificial neural network-based learning systems are well known for their pattern recognition capabilities and lately their deep architectures - known as deep learning (DL) - have been successfully applied to solve many complex pattern recognition problems. Highlighting the role of DL in recognizing patterns in biological data, this article provides - applications of DL to biological sequences, images, and signals data; overview of open access sources of these data; description of open source DL tools applicable on these data; and comparison of these tools from qualitative and quantitative perspectives. At the end, it outlines some open research challenges in mining biological data and puts forward a number of possible future perspectives.
A Time-dependent SIR model for COVID-19 with Undetectable Infected Persons
Authors: Yi-Cheng Chen, Ping-En Lu, Cheng-Shang Chang, Tzu-Hsuan Liu
Date: 2020-02-28 00:00:00
Tasks: Time Series Analysis

In this paper, we conduct mathematical and numerical analyses to address the following crucial questions for COVID-19: (Q1) Is it possible to contain COVID-19? (Q2) When will be the peak and the end of the epidemic? (Q3) How do the asymptomatic infections affect the spread of disease? (Q4) What is the ratio of the population that needs to be infected to achieve herd immunity? (Q5) How effective are the social distancing approaches? (Q6) What is the ratio of the population infected in the long run? For (Q1) and (Q2), we propose a time-dependent susceptible-infected-recovered (SIR) model that tracks 2 time series: (i) the transmission rate at time t and (ii) the recovering rate at time t. Such an approach is more adaptive than traditional static SIR models and more robust than direct estimation methods. Using the data provided by China, we show that the one-day prediction errors for the numbers of confirmed cases are almost in 3%, and the total number of confirmed cases is precisely predicted. Also, the turning point, defined as the day that the transmission rate is less than the recovering rate can be accurately predicted. After that day, the basic reproduction number $R_0$ is less than 1. For (Q3), we extend our SIR model by considering 2 types of infected persons: detectable and undetectable infected persons. Whether there is an outbreak in such a model is characterized by the spectral radius of a 2 by 2 matrix that is closely related to $R_0$. For (Q4), we show that herd immunity can be achieved after at least 1-1/$R_0$ fraction of individuals being infected. For (Q5) and (Q6), we analyze the independent cascade (IC) model for disease propagation in a configuration random graph. By relating the propagation probabilities in the IC model to the transmission rates and recovering rates in the SIR model, we show 2 approaches of social distancing that can lead to a reduction of $R_0$.
Optimal Feature Manipulation Attacks Against Linear Regression
Authors: Fuwei Li, Lifeng Lai, Shuguang Cui
Date: 2020-02-29 00:00:00
Tasks: regression

In this paper, we investigate how to manipulate the coefficients obtained via linear regression by adding carefully designed poisoning data points to the dataset or modify the original data points. Given the energy budget, we first provide the closed-form solution of the optimal poisoning data point when our target is modifying one designated regression coefficient. We then extend the analysis to the more challenging scenario where the attacker aims to change one particular regression coefficient while making others to be changed as small as possible. For this scenario, we introduce a semidefinite relaxation method to design the best attack scheme. Finally, we study a more powerful adversary who can perform a rank-one modification on the feature matrix. We propose an alternating optimization method to find the optimal rank-one modification matrix. Numerical examples are provided to illustrate the analytical results obtained in this paper.
Comparison of Distal Teacher Learning with Numerical and Analytical Methods to Solve Inverse Kinematics for Rigid-Body Mechanisms
Authors: Tim von Oehsen, Alexander Fabisch, Shivesh Kumar, Frank Kirchner
Date: 2020-02-29 00:00:00

Several publications are concerned with learning inverse kinematics, however, their evaluation is often limited and none of the proposed methods is of practical relevance for rigid-body kinematics with a known forward model. We argue that for rigid-body kinematics one of the first proposed machine learning (ML) solutions to inverse kinematics -- distal teaching (DT) -- is actually good enough when combined with differentiable programming libraries and we provide an extensive evaluation and comparison to analytical and numerical solutions. In particular, we analyze solve rate, accuracy, sample efficiency and scalability. Further, we study how DT handles joint limits, singularities, unreachable poses, trajectories and provide a comparison of execution times. The three approaches are evaluated on three different rigid body mechanisms with varying complexity. With enough training data and relaxed precision requirements, DT has a better solve rate and is faster than state-of-the-art numerical solvers for a 15-DoF mechanism. DT is not affected by singularities while numerical solutions are vulnerable to them. In all other cases numerical solutions are usually better. Analytical solutions outperform the other approaches by far if they are available.
Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning
Authors: Yu Kobayashi, Hideaki Iiduka
Date: 2020-02-29 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification, Stochastic Optimization, text-classification, Text Classification

This paper proposes a conjugate-gradient-based Adam algorithm blending Adam with nonlinear conjugate gradient methods and shows its convergence analysis. Numerical experiments on text classification and image classification show that the proposed algorithm can train deep neural network models in fewer epochs than the existing adaptive stochastic optimization algorithms can.
Tightly Robust Optimization via Empirical Domain Reduction
Authors: Akihiro Yabe, Takanori Maehara
Date: 2020-02-29 00:00:00
Tasks: Decision Making

Data-driven decision-making is performed by solving a parameterized optimization problem, and the optimal decision is given by an optimal solution for unknown true parameters. We often need a solution that satisfies true constraints even though these are unknown. Robust optimization is employed to obtain such a solution, where the uncertainty of the parameter is represented by an ellipsoid, and the scale of robustness is controlled by a coefficient. In this study, we propose an algorithm to determine the scale such that the solution has a good objective value and satisfies the true constraints with a given confidence probability. Under some regularity conditions, the scale obtained by our algorithm is asymptotically $O(1/\sqrt{n})$, whereas the scale obtained by a standard approach is $O(\sqrt{d/n})$. This means that our algorithm is less affected by the dimensionality of the parameters.
Quantum Computing Assisted Deep Learning for Fault Detection and Diagnosis in Industrial Process Systems
Authors: Akshay Ajagekar, Fengqi You
Date: 2020-02-29 00:00:00
Tasks: Deep Learning, Fault Detection, Fault Diagnosis

Quantum computing (QC) and deep learning techniques have attracted widespread attention in the recent years. This paper proposes QC-based deep learning methods for fault diagnosis that exploit their unique capabilities to overcome the computational challenges faced by conventional data-driven approaches performed on classical computers. Deep belief networks are integrated into the proposed fault diagnosis model and are used to extract features at different levels for normal and faulty process operations. The QC-based fault diagnosis model uses a quantum computing assisted generative training process followed by discriminative training to address the shortcomings of classical algorithms. To demonstrate its applicability and efficiency, the proposed fault diagnosis method is applied to process monitoring of continuous stirred tank reactor (CSTR) and Tennessee Eastman (TE) process. The proposed QC-based deep learning approach enjoys superior fault detection and diagnosis performance with obtained average fault detection rates of 79.2% and 99.39% for CSTR and TE process, respectively.
Online Binary Space Partitioning Forests
Authors: Xuhui Fan, Bin Li, Scott A. Sisson
Date: 2020-02-29 00:00:00
Tasks: General Classification, regression

The Binary Space Partitioning-Tree~(BSP-Tree) process was recently proposed as an efficient strategy for space partitioning tasks. Because it uses more than one dimension to partition the space, the BSP-Tree Process is more efficient and flexible than conventional axis-aligned cutting strategies. However, due to its batch learning setting, it is not well suited to large-scale classification and regression problems. In this paper, we develop an online BSP-Forest framework to address this limitation. With the arrival of new data, the resulting online algorithm can simultaneously expand the space coverage and refine the partition structure, with guaranteed universal consistency for both classification and regression problems. The effectiveness and competitive performance of the online BSP-Forest is verified via simulations on real-world datasets.
Dimension-free convergence rates for gradient Langevin dynamics in RKHS
Authors: Boris Muzellec, Kanji Sato, Mathurin Massias, Taiji Suzuki
Date: 2020-02-29 00:00:00

Gradient Langevin dynamics (GLD) and stochastic GLD (SGLD) have attracted considerable attention lately, as a way to provide convergence guarantees in a non-convex setting. However, the known rates grow exponentially with the dimension of the space. In this work, we provide a convergence analysis of GLD and SGLD when the optimization space is an infinite dimensional Hilbert space. More precisely, we derive non-asymptotic, dimension-free convergence rates for GLD/SGLD when performing regularized non-convex optimization in a reproducing kernel Hilbert space. Amongst others, the convergence analysis relies on the properties of a stochastic differential equation, its discrete time Galerkin approximation and the geometric ergodicity of the associated Markov chains.
Differentiating through the Fréchet Mean
Authors: Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, Christopher De Sa
Date: 2020-02-29 00:00:00
Proceeding: ICML 2020 1
Tasks: Representation Learning

Recent advances in deep representation learning on Riemannian manifolds extend classical deep learning operations to better capture the geometry of the manifold. One possible extension is the Fr\'echet mean, the generalization of the Euclidean mean; however, it has been difficult to apply because it lacks a closed form with an easily computable derivative. In this paper, we show how to differentiate through the Fr\'echet mean for arbitrary Riemannian manifolds. Then, focusing on hyperbolic space, we derive explicit gradient expressions and a fast, accurate, and hyperparameter-free Fr\'echet mean solver. This fully integrates the Fr\'echet mean into the hyperbolic neural network pipeline. To demonstrate this integration, we present two case studies. First, we apply our Fr\'echet mean to the existing Hyperbolic Graph Convolutional Network, replacing its projected aggregation to obtain state-of-the-art results on datasets with high hyperbolicity. Second, to demonstrate the Fr\'echet mean's capacity to generalize Euclidean neural network operations, we develop a hyperbolic batch normalization method that gives an improvement parallel to the one observed in the Euclidean setting.
Robust Robotic Pouring using Audition and Haptics
Authors: Hongzhuo Liang, Chuangchuang Zhou, Shuang Li, Xiaojian Ma, Norman Hendrich, Timo Gerkmann, Fuchun Sun, Marcus Stoffel, Jianwei Zhang
Date: 2020-02-29 00:00:00

Robust and accurate estimation of liquid height lies as an essential part of pouring tasks for service robots. However, vision-based methods often fail in occluded conditions while audio-based methods cannot work well in a noisy environment. We instead propose a multimodal pouring network (MP-Net) that is able to robustly predict liquid height by conditioning on both audition and haptics input. MP-Net is trained on a self-collected multimodal pouring dataset. This dataset contains 300 robot pouring recordings with audio and force/torque measurements for three types of target containers. We also augment the audio data by inserting robot noise. We evaluated MP-Net on our collected dataset and a wide variety of robot experiments. Both network training results and robot experiments demonstrate that MP-Net is robust against noise and changes to the task and environment. Moreover, we further combine the predicted height and force data to estimate the shape of the target container.