pwc-archive/papers-with-abstracts huggingface dataset:
Learning to Learn Kernels with Variational Random Features
Authors: Xiantong Zhen, Haoliang Sun, Ying-Jun Du, Jun Xu, Yilong Yin, Ling Shao, Cees Snoek
Date: 2020-06-11 00:00:00
Proceeding: ICML 2020 1
Tasks: Few-Shot Learning, Meta-Learning, Variational Inference

In this work, we introduce kernels with random Fourier features in the meta-learning framework to leverage their strong few-shot learning ability. We propose meta variational random features (MetaVRF) to learn adaptive kernels for the base-learner, which is developed in a latent variable model by treating the random feature basis as the latent variable. We formulate the optimization of MetaVRF as a variational inference problem by deriving an evidence lower bound under the meta-learning framework. To incorporate shared knowledge from related tasks, we propose a context inference of the posterior, which is established by an LSTM architecture. The LSTM-based inference network can effectively integrate the context information of previous tasks with task-specific information, generating informative and adaptive features. The learned MetaVRF can produce kernels of high representational power with a relatively low spectral sampling rate and also enables fast adaptation to new tasks. Experimental results on a variety of few-shot regression and classification tasks demonstrate that MetaVRF delivers much better, or at least competitive, performance compared to existing meta-learning alternatives.
Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously
Authors: Karim Lounici, Katia Meziani, Benjamin Riu
Date: 2020-06-11 00:00:00
Tasks: feature selection

Generalization is a central problem in Machine Learning. Most prediction methods require careful calibration of hyperparameters carried out on a hold-out \textit{validation} dataset to achieve generalization. The main goal of this paper is to present a novel approach based on a new measure of risk that allows us to develop novel fully automatic procedures for generalization. We illustrate the pertinence of this new framework in the regression problem. The main advantages of this new approach are: (i) it can simultaneously train the model and perform regularization in a single run of a gradient-based optimizer on all available data without any previous hyperparameter tuning; (ii) this framework can tackle several additional objectives simultaneously (correlation, sparsity,...) $via$ the introduction of regularization parameters. Noticeably, our approach transforms hyperparameter tuning as well as feature selection (a combinatorial discrete optimization problem) into a continuous optimization problem that is solvable via classical gradient-based methods ; (iii) the computational complexity of our methods is $O(npK)$ where $n,p,K$ denote respectively the number of observations, features and iterations of the gradient descent algorithm. We observe in our experiments a significantly smaller runtime for our methods as compared to benchmark methods for equivalent prediction score. Our procedures are implemented in PyTorch (code is available for replication).
End-to-end Sinkhorn Autoencoder with Noise Generator
Authors: Kamil Deja, Jan Dubiński, Piotr Nowak, Sandro Wenzel, Tomasz Trzciński
Date: 2020-06-11 00:00:00
Tasks: Astronomy

In this work, we propose a novel end-to-end sinkhorn autoencoder with noise generator for efficient data collection simulation. Simulating processes that aim at collecting experimental data is crucial for multiple real-life applications, including nuclear medicine, astronomy and high energy physics. Contemporary methods, such as Monte Carlo algorithms, provide high-fidelity results at a price of high computational cost. Multiple attempts are taken to reduce this burden, e.g. using generative approaches based on Generative Adversarial Networks or Variational Autoencoders. Although such methods are much faster, they are often unstable in training and do not allow sampling from an entire data distribution. To address these shortcomings, we introduce a novel method dubbed end-to-end Sinkhorn Autoencoder, that leverages sinkhorn algorithm to explicitly align distribution of encoded real data examples and generated noise. More precisely, we extend autoencoder architecture by adding a deterministic neural network trained to map noise from a known distribution onto autoencoder latent space representing data distribution. We optimise the entire model jointly. Our method outperforms competing approaches on a challenging dataset of simulation data from Zero Degree Calorimeters of ALICE experiment in LHC. as well as standard benchmarks, such as MNIST and CelebA.
On the asymptotics of wide networks with polynomial activations
Authors: Kyle Aitken, Guy Gur-Ari
Date: 2020-06-11 00:00:00

We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit. The results that follow from this conjecture include tight bounds on the behavior of wide networks during stochastic gradient descent, and a derivation of their finite-width dynamics. We prove the conjecture for deep networks with polynomial activation functions, greatly extending the validity of these results. Finally, we point out a difference in the asymptotic behavior of networks with analytic (and non-linear) activation functions and those with piecewise-linear activations such as ReLU.
Training Generative Adversarial Networks with Limited Data
Authors: Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, Timo Aila
Date: 2020-06-11 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: 10-shot image generation, Conditional Image Generation, Image Generation

Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes. The approach does not require changes to loss functions or network architectures, and is applicable both when training from scratch and when fine-tuning an existing GAN on another dataset. We demonstrate, on several datasets, that good results are now possible using only a few thousand training images, often matching StyleGAN2 results with an order of magnitude fewer images. We expect this to open up new application domains for GANs. We also find that the widely used CIFAR-10 is, in fact, a limited data benchmark, and improve the record FID from 5.59 to 2.42.
Clustering Residential Electricity Consumption Data to Create Archetypes that Capture Household Behaviour in South Africa
Authors: Wiebke Toussaint, Deshendran Moodley
Date: 2020-06-11 00:00:00
Tasks: Clustering, Time Series Analysis, Time Series Clustering

Clustering is frequently used in the energy domain to identify dominant electricity consumption patterns of households, which can be used to construct customer archetypes for long term energy planning. Selecting a useful set of clusters however requires extensive experimentation and domain knowledge. While internal clustering validation measures are well established in the electricity domain, they are limited for selecting useful clusters. Based on an application case study in South Africa, we present an approach for formalising implicit expert knowledge as external evaluation measures to create customer archetypes that capture variability in residential electricity consumption behaviour. By combining internal and external validation measures in a structured manner, we were able to evaluate clustering structures based on the utility they present for our application. We validate the selected clusters in a use case where we successfully reconstruct customer archetypes previously developed by experts. Our approach shows promise for transparent and repeatable cluster ranking and selection by data scientists, even if they have limited domain knowledge.
A General Framework for Analyzing Stochastic Dynamics in Learning Algorithms
Authors: Chi-Ning Chou, Juspreet Singh Sandhu, Mien Brabeeba Wang, Tiancheng Yu
Date: 2020-06-11 00:00:00

One of the challenges in analyzing learning algorithms is the circular entanglement between the objective value and the stochastic noise. This is also known as the "chicken and egg" phenomenon and traditionally, there is no principled way to tackle this issue. People solve the problem by utilizing the special structure of the dynamic, and hence the analysis would be difficult to generalize. In this work, we present a streamlined three-step recipe to tackle the "chicken and egg" problem and give a general framework for analyzing stochastic dynamics in learning algorithms. Our framework composes standard techniques from probability theory, such as stopping time and martingale concentration. We demonstrate the power and flexibility of our framework by giving a unifying analysis for three very different learning problems with the last iterate and the strong uniform high probability convergence guarantee. The problems are stochastic gradient descent for strongly convex functions, streaming principal component analysis, and linear bandit with stochastic gradient descent updates. We either improve or match the state-of-the-art bounds on all three dynamics.
STONNE: A Detailed Architectural Simulator for Flexible Neural Network Accelerators
Authors: Francisco Muñoz-Martínez, José L. Abellán, Manuel E. Acacio, Tushar Krishna
Date: 2020-06-10 00:00:00

The design of specialized architectures for accelerating the inference procedure of Deep Neural Networks (DNNs) is a booming area of research nowadays. First-generation rigid proposals have been rapidly replaced by more advanced flexible accelerator architectures able to efficiently support a variety of layer types and dimensions. As the complexity of the designs grows, it is more and more appealing for researchers to have cycle-accurate simulation tools at their disposal to allow for fast and accurate design-space exploration, and rapid quantification of the efficacy of architectural enhancements during the early stages of a design. To this end, we present STONNE (Simulation TOol of Neural Network Engines), a cycle-accurate, highly-modular and highly-extensible simulation framework that enables end-to-end evaluation of flexible accelerator architectures running complete contemporary DNN models. We use STONNE to model the recently proposed MAERI architecture and show how it can closely approach the performance results of the publicly available BSV-coded MAERI implementation. Then, we conduct a comprehensive evaluation and demonstrate that the folding strategy implemented for MAERI results in very low compute unit utilization (25% on average across 5 DNN models) which in the end translates into poor performance.
Simulating Tariff Impact in Electrical Energy Consumption Profiles with Conditional Variational Autoencoders
Authors: Margaux Brégère, Ricardo J. Bessa
Date: 2020-06-10 00:00:00
Tasks: Additive models, Clustering

The implementation of efficient demand response (DR) programs for household electricity consumption would benefit from data-driven methods capable of simulating the impact of different tariffs schemes. This paper proposes a novel method based on conditional variational autoencoders (CVAE) to generate, from an electricity tariff profile combined with exogenous weather and calendar variables, daily consumption profiles of consumers segmented in different clusters. First, a large set of consumers is gathered into clusters according to their consumption behavior and price-responsiveness. The clustering method is based on a causality model that measures the effect of a specific tariff on the consumption level. Then, daily electrical energy consumption profiles are generated for each cluster with CVAE. This non-parametric approach is compared to a semi-parametric data generator based on generalized additive models and that uses prior knowledge of energy consumption. Experiments in a publicly available data set show that, the proposed method presents comparable performance to the semi-parametric one when it comes to generating the average value of the original data. The main contribution from this new method is the capacity to reproduce rebound and side effects in the generated consumption profiles. Indeed, the application of a special electricity tariff over a time window may also affect consumption outside this time window. Another contribution is that the clustering approach segments consumers according to their daily consumption profile and elasticity to tariff changes. These two results combined are very relevant for an ex-ante testing of future DR policies by system operators, retailers and energy regulators.
Combining the band-limited parameterization and Semi-Lagrangian Runge--Kutta integration for efficient PDE-constrained LDDMM
Authors: Monica Hernandez
Date: 2020-06-10 00:00:00
Tasks: Anatomy, Computational Efficiency

The family of PDE-constrained LDDMM methods is emerging as a particularly interesting approach for physically meaningful diffeomorphic transformations. The original combination of Gauss--Newton--Krylov optimization and Runge--Kutta integration, shows excellent numerical accuracy and fast convergence rate. However, its most significant limitation is the huge computational complexity, hindering its extensive use in Computational Anatomy applied studies. This limitation has been treated independently by the problem formulation in the space of band-limited vector fields and Semi-Lagrangian integration. The purpose of this work is to combine both in three variants of band-limited PDE-constrained LDDMM for further increasing their computational efficiency. The accuracy of the resulting methods is evaluated extensively. For all the variants, the proposed combined approach shows a significant increment of the computational efficiency. In addition, the variant based on the deformation state equation is positioned consistently as the best performing method across all the evaluation frameworks in terms of accuracy and efficiency.
Speech Denoising Convolutional Neural Network trained with Deep Feature Losses.
Authors: Francois G. Germain, Qifeng Chen, Vladlen Koltun
Date: 2018-06-27 00:00:00
Proceeding: Interspeech 2018 6
Tasks: Audio Tagging, Denoising, Speech Denoising, Speech Enhancement

We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly. Given input audio containing speech corrupted by an additive background signal, the system aims to produce a processed signal that contains only the speech content. Recent approaches have shown promising results using various deep network architectures. In this paper, we propose to train a fully-convolutional context aggregation network using a deep feature loss. That loss is based on comparing the internal feature activations in a different network, trained for acoustic environment detection and domestic audio tagging. Our approach outperforms the state-of-the-art in objective speech quality metrics and in large-scale perceptual experiments with human listeners. It also outperforms an identical network trained using traditional regression losses. The advantage of the new approach is particularly pronounced for the hardest data with the most intrusive background noise, for which denoising is most needed and most challenging.
Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents
Authors: Dookun Park, Hao Yuan, Dongmin Kim, Yinglei Zhang, Matsoukas Spyros, Young-Bum Kim, Ruhi Sarikaya, Edward Guo, Yuan Ling, Kevin Quinn, Pham Hung, Benjamin Yao, Sungjin Lee
Date: 2020-05-29 00:00:00

Measuring user satisfaction level is a challenging task, and a critical component in developing large-scale conversational agent systems serving the needs of real users. An widely used approach to tackle this is to collect human annotation data and use them for evaluation or modeling. Human annotation based approaches are easier to control, but hard to scale. A novel alternative approach is to collect user's direct feedback via a feedback elicitation system embedded to the conversational agent system, and use the collected user feedback to train a machine-learned model for generalization. User feedback is the best proxy for user satisfaction, but is not available for some ineligible intents and certain situations. Thus, these two types of approaches are complementary to each other. In this work, we tackle the user satisfaction assessment problem with a hybrid approach that fuses explicit user feedback, user satisfaction predictions inferred by two machine-learned models, one trained on user feedback data and the other human annotation data. The hybrid approach is based on a waterfall policy, and the experimental results with Amazon Alexa's large-scale datasets show significant improvements in inferring user satisfaction. A detailed hybrid architecture, an in-depth analysis on user feedback data, and an algorithm that generates data sets to properly simulate the live traffic are presented in this paper.
Coronavirus Contact Tracing: Evaluating The Potential Of Using Bluetooth Received Signal Strength For Proximity Detection
Authors: Douglas J. Leith, Stephen Farrell
Date: 2020-05-19 00:00:00

We report on measurements of Bluetooth Low Energy (LE) received signal strength taken on mobile handsets in a variety of common, real-world settings. We note that a key difficulty is obtaining the ground truth as to when people are in close proximity to one another. Knowledge of this ground truth is important for accurately evaluating the accuracy with which contact events are detected by Bluetooth LE. We approach this by adopting a scenario-based approach. In summary, we find that the Bluetooth LE received signal strength can vary substantially depending on the relative orientation of handsets, on absorption by the human body, reflection/absorption of radio signals in buildings and trains. Indeed we observe that the received signal strength need not decrease with increasing distance. This suggests that the development of accurate methods for proximity detection based on Bluetooth LE received signal strength is likely to be challenging. Our measurements also suggest that combining use of Bluetooth LE contact tracing apps with adoption of new social protocols may yield benefits but this requires further investigation. For example, placing phones on the table during meetings is likely to simplify proximity detection using received signal strength. Similarly, carrying handbags with phones placed close to the outside surface. In locations where the complexity of signal propagation makes proximity detection using received signal strength problematic entry/exit from the location might instead be logged in an app by e.g. scanning a time-varying QR code or the like.
Deep Learning based Diagnosis of COVID-19 usingChest CT-scan Images
Authors: Talha Anwar, Seemab Zakir
Date: 2020-05-20 00:00:00
Proceeding: techrxiv 2020 5
Tasks: COVID-19 Diagnosis, Deep Learning, Diagnostic

The Coronavirus disease (COVID-19) is an infectious disease that primarily affects lungs. This virus has spread in almost every continent. Countries are racing to slow down the spread by testing and treating patients. To diagnose the infected people, reverse transcription-polymerase chain reaction (RT-PCR) test is used. Because of colossal demand; PCR kits are under shortage, and to overcome this; radiographic techniques such as X-rays and CT-scan can be used for diagnostic purpose. In this paper, deep learning technology is used to diagnose COVID-19 in subjects through chest CT-scan. EfficientNet deep learning architecture is used for timely and accurate detection of coronavirus with an accuracy 0.897, F1 score 0.896, and AUC 0.895. Three different learning rate strategies are used, such as reducing the learning rate when model performance stops increasing (reduce on plateau), cyclic learning rate, and constant learning rate. Reduce on plateau strategy achieved F1-score of 0.9, cyclic learning rate and constant learning rate resulted in F1-score of 0.86 and 0.82, respectively. Implementation is available at github.com/talhaanwarch/Corona\_Virus/tree/master/CT\_scan
Monotone operator equilibrium networks
Authors: Ezra Winston, J. Zico Kolter
Date: 2020-06-15 00:00:00
Proceeding: NeurIPS 2020 12

Implicit-depth models such as Deep Equilibrium Networks have recently been shown to match or exceed the performance of traditional deep networks while being much more memory efficient. However, these models suffer from unstable convergence to a solution and lack guarantees that a solution exists. On the other hand, Neural ODEs, another class of implicit-depth models, do guarantee existence of a unique solution but perform poorly compared with traditional networks. In this paper, we develop a new class of implicit-depth model based on the theory of monotone operators, the Monotone Operator Equilibrium Network (monDEQ). We show the close connection between finding the equilibrium point of an implicit network and solving a form of monotone operator splitting problem, which admits efficient solvers with guaranteed, stable convergence. We then develop a parameterization of the network which ensures that all operators remain monotone, which guarantees the existence of a unique equilibrium point. Finally, we show how to instantiate several versions of these models, and implement the resulting iterative solvers, for structured linear operators such as multi-scale convolutions. The resulting models vastly outperform the Neural ODE-based models while also being more computationally efficient. Code is available at http://github.com/locuslab/monotone_op_net.
Coherent Reconstruction of Multiple Humans from a Single Image
Authors: Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, Kostas Daniilidis
Date: 2020-06-15 00:00:00
Proceeding: CVPR 2020 6
Tasks: 3D Depth Estimation, 3D Human Reconstruction, 3D Pose Estimation, 3D Reconstruction, Instance Segmentation, Pose Estimation, Semantic Segmentation

In this work, we address the problem of multi-person 3D pose estimation from a single image. A typical regression approach in the top-down setting of this problem would first detect all humans and then reconstruct each one of them independently. However, this type of prediction suffers from incoherent results, e.g., interpenetration and inconsistent depth ordering between the people in the scene. Our goal is to train a single network that learns to avoid these problems and generate a coherent 3D reconstruction of all the humans in the scene. To this end, a key design choice is the incorporation of the SMPL parametric body model in our top-down framework, which enables the use of two novel losses. First, a distance field-based collision loss penalizes interpenetration among the reconstructed people. Second, a depth ordering-aware loss reasons about occlusions and promotes a depth ordering of people that leads to a rendering which is consistent with the annotated instance segmentation. This provides depth supervision signals to the network, even if the image has no explicit 3D annotations. The experiments show that our approach outperforms previous methods on standard 3D pose benchmarks, while our proposed losses enable more coherent reconstruction in natural images. The project website with videos, results, and code can be found at: https://jiangwenpl.github.io/multiperson