pwc-archive/papers-with-abstracts huggingface dataset:
On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks
Authors: Ramy E. Ali, Jinhyun So, A. Salman Avestimehr
Date: 2020-11-11 00:00:00
Tasks: Privacy Preserving

Outsourcing deep neural networks (DNNs) inference tasks to an untrusted cloud raises data privacy and integrity concerns. While there are many techniques to ensure privacy and integrity for polynomial-based computations, DNNs involve non-polynomial computations. To address these challenges, several privacy-preserving and verifiable inference techniques have been proposed based on replacing the non-polynomial activation functions such as the rectified linear unit (ReLU) function with polynomial activation functions. Such techniques usually require polynomials with integer coefficients or polynomials over finite fields. Motivated by such requirements, several works proposed replacing the ReLU function with the square function. In this work, we empirically show that the square function is not the best degree-2 polynomial that can replace the ReLU function even when restricting the polynomials to have integer coefficients. We instead propose a degree-2 polynomial activation function with a first order term and empirically show that it can lead to much better models. Our experiments on the CIFAR and Tiny ImageNet datasets on various architectures such as VGG-16 show that our proposed function improves the test accuracy by up to 10.4% compared to the square function.
Proximal Policy Optimization via Enhanced Exploration Efficiency
Authors: Junwei Zhang, Zhenghao Zhang, Shuai Han, Shuai Lü
Date: 2020-11-11 00:00:00
Tasks: continuous-control, Continuous Control, Deep Reinforcement Learning, MuJoCo, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Proximal policy optimization (PPO) algorithm is a deep reinforcement learning algorithm with outstanding performance, especially in continuous control tasks. But the performance of this method is still affected by its exploration ability. For classical reinforcement learning, there are some schemes that make exploration more full and balanced with data exploitation, but they can't be applied in complex environments due to the complexity of algorithm. Based on continuous control tasks with dense reward, this paper analyzes the assumption of the original Gaussian action exploration mechanism in PPO algorithm, and clarifies the influence of exploration ability on performance. Afterward, aiming at the problem of exploration, an exploration enhancement mechanism based on uncertainty estimation is designed in this paper. Then, we apply exploration enhancement theory to PPO algorithm and propose the proximal policy optimization algorithm with intrinsic exploration module (IEM-PPO) which can be used in complex environments. In the experimental parts, we evaluate our method on multiple tasks of MuJoCo physical simulator, and compare IEM-PPO algorithm with curiosity driven exploration algorithm (ICM-PPO) and original algorithm (PPO). The experimental results demonstrate that IEM-PPO algorithm needs longer training time, but performs better in terms of sample efficiency and cumulative reward, and has stability and robustness.
Energy consumption forecasting using a stacked nonparametric Bayesian approach
Authors: Dilusha Weeraddana, Nguyen Lu Dang Khoa, Lachlan O Neil, Weihong Wang, Chen Cai
Date: 2020-11-11 00:00:00
Tasks: Time Series, Time Series Analysis, Time Series Forecasting

In this paper, the process of forecasting household energy consumption is studied within the framework of the nonparametric Gaussian Process (GP), using multiple short time series data. As we begin to use smart meter data to paint a clearer picture of residential electricity use, it becomes increasingly apparent that we must also construct a detailed picture and understanding of consumer's complex relationship with gas consumption. Both electricity and gas consumption patterns are highly dependent on various factors, and the intricate interplay of these factors is sophisticated. Moreover, since typical gas consumption data is low granularity with very few time points, naive application of conventional time-series forecasting techniques can lead to severe over-fitting. Given these considerations, we construct a stacked GP method where the predictive posteriors of each GP applied to each task are used in the prior and likelihood of the next level GP. We apply our model to a real-world dataset to forecast energy consumption in Australian households across several states. We compare intuitively appealing results against other commonly used machine learning techniques. Overall, the results indicate that the proposed stacked GP model outperforms other forecasting techniques that we tested, especially when we have a multiple short time-series instances.
Probability-Density-Based Deep Learning Paradigm for the Fuzzy Design of Functional Metastructures
Authors: Ying-Tao Luo, Peng-Qi Li, Dong-Ting Li, Yu-Gui Peng, Zhi-Guo Geng, Shu-Huan Xie, Yong Li, Andrea Alu, Jie Zhu, Xue-Feng Zhu
Date: 2020-11-11 00:00:00

In quantum mechanics, a norm squared wave function can be interpreted as the probability density that describes the likelihood of a particle to be measured in a given position or momentum. This statistical property is at the core of the fuzzy structure of microcosmos. Recently, hybrid neural structures raised intense attention, resulting in various intelligent systems with far-reaching influence. Here, we propose a probability-density-based deep learning paradigm for the fuzzy design of functional meta-structures. In contrast to other inverse design methods, our probability-density-based neural network can efficiently evaluate and accurately capture all plausible meta-structures in a high-dimensional parameter space. Local maxima in probability density distribution correspond to the most likely candidates to meet the desired performances. We verify this universally adaptive approach in but not limited to acoustics by designing multiple meta-structures for each targeted transmission spectrum, with experiments unequivocally demonstrating the effectiveness and generalization of the inverse design.
A Quantum-Inspired Probabilistic Model for the Inverse Design of Meta-Structures
Authors: Yingtao Luo, XueFeng Zhu
Date: 2020-11-11 00:00:00
Tasks: Position, Probabilistic Deep Learning

In quantum mechanics, a norm squared wave function can be interpreted as the probability density that describes the likelihood of a particle to be measured in a given position or momentum. This statistical property is at the core of the microcosmos. Meanwhile, machine learning inverse design of materials raised intensive attention, resulting in various intelligent systems for matter engineering. Here, inspired by quantum theory, we propose a probabilistic deep learning paradigm for the inverse design of functional meta-structures. Our probability-density-based neural network (PDN) can accurately capture all plausible meta-structures to meet the desired performances. Local maxima in probability density distribution correspond to the most likely candidates. We verify this approach by designing multiple meta-structures for each targeted transmission spectrum to enrich design choices.
Two-dimensional Bhattacharyya bound linear discriminant analysis with its applications
Authors: Yan-Ru Guo, Yan-Qin Bai, Chun-Na Li, Lan Bai, Yuan-Hai Shao
Date: 2020-11-11 00:00:00
Tasks: Image Reconstruction, Vocal Bursts Valence Prediction

Recently proposed L2-norm linear discriminant analysis criterion via the Bhattacharyya error bound estimation (L2BLDA) is an effective improvement of linear discriminant analysis (LDA) for feature extraction. However, L2BLDA is only proposed to cope with vector input samples. When facing with two-dimensional (2D) inputs, such as images, it will lose some useful information, since it does not consider intrinsic structure of images. In this paper, we extend L2BLDA to a two-dimensional Bhattacharyya bound linear discriminant analysis (2DBLDA). 2DBLDA maximizes the matrix-based between-class distance which is measured by the weighted pairwise distances of class means and meanwhile minimizes the matrix-based within-class distance. The weighting constant between the between-class and within-class terms is determined by the involved data that makes the proposed 2DBLDA adaptive. In addition, the criterion of 2DBLDA is equivalent to optimizing an upper bound of the Bhattacharyya error. The construction of 2DBLDA makes it avoid the small sample size problem while also possess robustness, and can be solved through a simple standard eigenvalue decomposition problem. The experimental results on image recognition and face image reconstruction demonstrate the effectiveness of the proposed methods.
Automatic Open-World Reliability Assessment
Authors: Mohsen Jafarzadeh, Touqeer Ahmad, Akshay Raj Dhamija, Chunchun Li, Steve Cruz, Terrance E. Boult
Date: 2020-11-11 00:00:00
Tasks: image-classification, Image Classification

Image classification in the open-world must handle out-of-distribution (OOD) images. Systems should ideally reject OOD images, or they will map atop of known classes and reduce reliability. Using open-set classifiers that can reject OOD inputs can help. However, optimal accuracy of open-set classifiers depend on the frequency of OOD data. Thus, for either standard or open-set classifiers, it is important to be able to determine when the world changes and increasing OOD inputs will result in reduced system reliability. However, during operations, we cannot directly assess accuracy as there are no labels. Thus, the reliability assessment of these classifiers must be done by human operators, made more complex because networks are not 100% accurate, so some failures are to be expected. To automate this process, herein, we formalize the open-world recognition reliability problem and propose multiple automatic reliability assessment policies to address this new problem using only the distribution of reported scores/probability data. The distributional algorithms can be applied to both classic classifiers with SoftMax as well as the open-world Extreme Value Machine (EVM) to provide automated reliability assessment. We show that all of the new algorithms significantly outperform detection using the mean of SoftMax.
Morphological Disambiguation from Stemming Data
Authors: Antoine Nzeyimana
Date: 2020-11-11 00:00:00
Proceeding: COLING 2020 8
Tasks: Feature Engineering, Morphological Analysis, Morphological Disambiguation

Morphological analysis and disambiguation is an important task and a crucial preprocessing step in natural language processing of morphologically rich languages. Kinyarwanda, a morphologically rich language, currently lacks tools for automated morphological analysis. While linguistically curated finite state tools can be easily developed for morphological analysis, the morphological richness of the language allows many ambiguous analyses to be produced, requiring effective disambiguation. In this paper, we propose learning to morphologically disambiguate Kinyarwanda verbal forms from a new stemming dataset collected through crowd-sourcing. Using feature engineering and a feed-forward neural network based classifier, we achieve about 89% non-contextualized disambiguation accuracy. Our experiments reveal that inflectional properties of stems and morpheme association rules are the most discriminative features for disambiguation.
Unsupervised Learning of Dense Visual Representations
Authors: Pedro O. Pinheiro, Amjad Almahairi, Ryan Y. Benmalek, Florian Golemo, Aaron Courville
Date: 2020-11-11 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Contrastive Learning, Data Augmentation, Representation Learning, Self-Supervised Learning

Contrastive self-supervised learning has emerged as a promising approach to unsupervised visual representation learning. In general, these methods learn global (image-level) representations that are invariant to different views (i.e., compositions of data augmentation) of the same image. However, many visual understanding tasks require dense (pixel-level) representations. In this paper, we propose View-Agnostic Dense Representation (VADeR) for unsupervised learning of dense representations. VADeR learns pixelwise representations by forcing local features to remain constant over different viewing conditions. Specifically, this is achieved through pixel-level contrastive learning: matching features (that is, features that describes the same location of the scene on different views) should be close in an embedding space, while non-matching features should be apart. VADeR provides a natural representation for dense prediction tasks and transfers well to downstream tasks. Our method outperforms ImageNet supervised pretraining (and strong unsupervised baselines) in multiple dense prediction tasks.
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
Authors: Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, Kim Hazelwood
Date: 2020-11-11 00:00:00
Tasks: Deep Learning, GPU

The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models. Meanwhile, when training state-of-the-art personal recommendation models, which consume the highest number of compute cycles at our large-scale datacenters, the use of GPUs came with various challenges due to having both compute-intensive and memory-intensive components. GPU performance and efficiency of these recommendation models are largely affected by model architecture configurations such as dense and sparse features, MLP dimensions. Furthermore, these models often contain large embedding tables that do not fit into limited GPU memory. The goal of this paper is to explain the intricacies of using GPUs for training recommendation models, factors affecting hardware efficiency at scale, and learnings from a new scale-up GPU server design, Zion.
Robust and flexible learning of a high-dimensional classification rule using auxiliary outcomes
Authors: Muxuan Liang, Jaeyoung Park, Qing Lu, Xiang Zhong
Date: 2020-11-11 00:00:00
Tasks: Classification, General Classification, Multi-Task Learning, Transfer Learning

Correlated outcomes are common in many practical problems. In some settings, one outcome is of particular interest, and others are auxiliary. To leverage information shared by all the outcomes, traditional multi-task learning (MTL) minimizes an averaged loss function over all the outcomes, which may lead to biased estimation for the target outcome, especially when the MTL model is mis-specified. In this work, based on a decomposition of estimation bias into two types, within-subspace and against-subspace, we develop a robust transfer learning approach to estimating a high-dimensional linear decision rule for the outcome of interest with the presence of auxiliary outcomes. The proposed method includes an MTL step using all outcomes to gain efficiency, and a subsequent calibration step using only the outcome of interest to correct both types of biases. We show that the final estimator can achieve a lower estimation error than the one using only the single outcome of interest. Simulations and real data analysis are conducted to justify the superiority of the proposed method.
Dense U-net for super-resolution with shuffle pooling layer
Authors: Zhengyang Lu, Ying Chen
Date: 2020-11-11 00:00:00
Tasks: Image Super-Resolution, SSIM, Super-Resolution

Recent researches have achieved great progress on single image super-resolution(SISR) due to the development of deep learning in the field of computer vision. In these method, the high resolution input image is down-scaled to low resolution space using a single filter, commonly max-pooling, before feature extraction. This means that the feature extraction is performed in biased filtered feature space. We demonstrate that this is sub-optimal and causes information loss. In this work, we proposed a state-of-the-art convolutional neural network method called Dense U-net with shuffle pooling. To achieve this, a modified U-net with dense blocks, called dense U-net, is proposed for SISR. Then, a new pooling strategy called shuffle pooling is designed, which is aimed to replace the dense U-Net for down-scale operation. By doing so, we effectively replace the handcrafted filter in the SISR pipeline with more lossy down-sampling filters specifically trained for each feature map, whilst also reducing the information loss of the overall SISR operation. In addition, a mix loss function, which combined with Mean Square Error(MSE), Structural Similarity Index(SSIM) and Mean Gradient Error (MGE), comes up to reduce the perception loss and high-level information loss. Our proposed method achieves superior accuracy over previous state-of-the-art on the three benchmark datasets: SET14, BSD300, ICDAR2003. Code is available online.
A novel method for Causal Structure Discovery from EHR data, a demonstration on type-2 diabetes mellitus
Authors: Xinpeng Shen, Sisi Ma, Prashanthi Vemuri, M. Regina Castro, Pedro J. Caraballo, Gyorgy J. Simon
Date: 2020-11-11 00:00:00
Tasks: Prognosis

Introduction: The discovery of causal mechanisms underlying diseases enables better diagnosis, prognosis and treatment selection. Clinical trials have been the gold standard for determining causality, but they are resource intensive, sometimes infeasible or unethical. Electronic Health Records (EHR) contain a wealth of real-world data that holds promise for the discovery of disease mechanisms, yet the existing causal structure discovery (CSD) methods fall short on leveraging them due to the special characteristics of the EHR data. We propose a new data transformation method and a novel CSD algorithm to overcome the challenges posed by these characteristics. Materials and methods: We demonstrated the proposed methods on an application to type-2 diabetes mellitus. We used a large EHR data set from Mayo Clinic to internally evaluate the proposed transformation and CSD methods and used another large data set from an independent health system, Fairview Health Services, as external validation. We compared the performance of our proposed method to Fast Greedy Equivalence Search (FGES), a state-of-the-art CSD method in terms of correctness, stability and completeness. We tested the generalizability of the proposed algorithm through external validation. Results and conclusions: The proposed method improved over the existing methods by successfully incorporating study design considerations, was robust in face of unreliable EHR timestamps and inferred causal effect directions more correctly and reliably. The proposed data transformation successfully improved the clinical correctness of the discovered graph and the consistency of edge orientation across bootstrap samples. It resulted in superior accuracy, stability, and completeness.
ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery
Authors: Jeremy Irvin, Hao Sheng, Neel Ramachandran, Sonja Johnson-Yu, Sharon Zhou, Kyle Story, Rose Rustowicz, Cooper Elsworth, Kemen Austin, Andrew Y. Ng
Date: 2020-11-11 00:00:00
Tasks: General Classification, Management

Characterizing the processes leading to deforestation is critical to the development and implementation of targeted forest conservation and management policies. In this work, we develop a deep learning model called ForestNet to classify the drivers of primary forest loss in Indonesia, a country with one of the highest deforestation rates in the world. Using satellite imagery, ForestNet identifies the direct drivers of deforestation in forest loss patches of any size. We curate a dataset of Landsat 8 satellite images of known forest loss events paired with driver annotations from expert interpreters. We use the dataset to train and validate the models and demonstrate that ForestNet substantially outperforms other standard driver classification approaches. In order to support future research on automated approaches to deforestation driver classification, the dataset curated in this study is publicly available at https://stanfordmlgroup.github.io/projects/forestnet .
A deep-learning classifier for cardiac arrhythmias
Authors: Carla Sofia Carvalho
Date: 2020-11-11 00:00:00
Tasks: Deep Learning

We report on a method that classifies heart beats according to a set of 13 classes, including cardiac arrhythmias. The method localises the QRS peak complex to define each heart beat and uses a neural network to infer the patterns characteristic of each heart beat class. The best performing neural network contains six one-dimensional convolutional layers and four dense layers, with the kernel sizes being multiples of the characteristic scale of the problem, thus resulting a computationally fast and physically motivated neural network. For the same number of heart beat classes, our method yields better results with a considerably smaller neural network than previously published methods, which renders our method competitive for deployment in an internet-of-things solution.
Incorporating Causal Effects into Deep Learning Predictions on EHR Data
Authors: Jia Li, HaoYu Yang, Xiaowei Jia, Vipin Kumar, Michael Steinbach, Gyorgy Simon
Date: 2020-11-11 00:00:00
Tasks: Causal Inference, Deep Learning

Electronic Health Records (EHR) data analysis plays a crucial role in healthcare system quality. Because of its highly complex underlying causality and limited observable nature, causal inference on EHR is quite challenging. Deep Learning (DL) achieved great success among the advanced machine learning methodologies. Nevertheless, it is still obstructed by the inappropriately assumed causal conditions. This work proposed a novel method to quantify clinically well-defined causal effects as a generalized estimation vector that is simply utilizable for causal models. We incorporated it into DL models to achieve better predictive performance and result interpretation. Furthermore, we also proved the existence of causal information blink spots that regular DL models cannot reach.
Deep Sound Change: Deep and Iterative Learning, Convolutional Neural Networks, and Language Change
Authors: Gašper Beguš
Date: 2020-11-10 00:00:00

This paper proposes a framework for modeling sound change that combines deep learning and iterative learning. Acquisition and transmission of speech is modeled by training generations of Generative Adversarial Networks (GANs) on unannotated raw speech data. The paper argues that several properties of sound change emerge from the proposed architecture. GANs (Goodfellow et al. 2014 arXiv:1406.2661, Donahue et al. 2019 arXiv:1705.07904) are uniquely appropriate for modeling language change because the networks are trained on raw unsupervised acoustic data, contain no language-specific features and, as argued in Begu\v{s} (2020 arXiv:2006.03965), encode phonetic and phonological representations in their latent space and generate linguistically informative innovative data. The first generation of networks is trained on the relevant sequences in human speech from TIMIT. The subsequent generations are not trained on TIMIT, but on generated outputs from the previous generation and thus start learning from each other in an iterative learning task. The initial allophonic distribution is progressively being lost with each generation, likely due to pressures from the global distribution of aspiration in the training data. The networks show signs of a gradual shift in phonetic targets characteristic of a gradual phonetic sound change. At endpoints, the outputs superficially resemble a phonological change -- rule loss.