pwc-archive/papers-with-abstracts huggingface dataset:
If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
Authors: Arnu Pretorius, Elan van Biljon, Benjamin van Niekerk, Ryan Eloff, Matthew Reynard, Steve James, Benjamin Rosman, Herman Kamper, Steve Kroon
Date: 2019-10-13 00:00:00

Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter? We conduct a large-scale controlled experiment, and perform a statistical analysis of over $12000$ trained networks. We find that (1) trainable networks show no statistically significant difference in performance over a wide range of non-critical initialisations; (2) for initialisations that show a statistically significant difference, the net effect on performance is small; (3) only extreme initialisations (very small or very large) perform worse than criticality. These findings also apply to standard ReLU networks of moderate depth as a special case of zero dropout. Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour.
Beyond 5G: Leveraging Cell Free TDD Massive MIMO using Cascaded Deep learning
Authors: Navaneet Athreya, Vishnu Raj, Sheetal Kalyani
Date: 2019-10-13 00:00:00

This paper deals with the calibration of Time Division Duplexing (TDD) reciprocity in an Orthogonal Frequency Division Multiplexing (OFDM) based Cell Free Massive MIMO system where the responses of the (Radio Frequency) RF chains render the end to end channel non-reciprocal, even though the physical wireless channel is reciprocal. We further address the non-availability of the uplink channel estimates at locations other than pilot subcarriers and propose a single-shot solution to estimate the downlink channel at all subcarriers from the uplink channel at selected pilot subcarriers. We propose a cascade of two Deep Neural Networks (DNN) to achieve the objective. The proposed method is easily scalable and removes the need for relative reciprocity calibration based on the cooperation of antennas, which usually introduces dependency in Cell Free Massive MIMO systems.
Contour Sparse Representation with SDD Features for Object Recognition
Authors: Zhenzhou Wang
Date: 2019-10-13 00:00:00
Tasks: Clustering, Gesture Recognition, Image Segmentation, Object, Object Detection, Object Recognition, Segmentation, Semantic Segmentation

Slope difference distribution (SDD) is computed for the one-dimensional curve. It is not only robust to calculate the partitioning point to separate the curve logically, but also robust to calculate the clustering center of each part of the separated curve. SDD has been proposed for image segmentation and it outperforms all existing image segmentation methods. For verification purpose, we have made the Matlab codes of comparing SDD method with existing image segmentation methods freely available at Matlab Central. The contour of the object is similar to the histogram of the image. Thus, feature detection by SDD from the contour of the object is also feasible. In this letter, SDD features are defined and they form the sparse representation of the object contour. The reference model of each object is built based on the SDD features and then model matching is used for on line object recognition. The experimental results are very encouraging. For the gesture recognition, SDD achieved 100% accuracy for two public datasets: the NUS dataset and the near-infrared dataset. For the object recognition, SDD achieved 100% accuracy for the Kimia 99 dataset.
A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels
Authors: Devraj Mandal, Shrisha Bharadwaj, Soma Biswas
Date: 2019-10-13 00:00:00

The major driving force behind the immense success of deep learning models is the availability of large datasets along with their clean labels. Unfortunately, this is very difficult to obtain, which has motivated research on the training of deep models in the presence of label noise and ways to avoid over-fitting on the noisy labels. In this work, we build upon the seminal work in this area, Co-teaching and propose a simple, yet efficient approach termed mCT-S2R (modified co-teaching with self-supervision and re-labeling) for this task. First, to deal with significant amount of noise in the labels, we propose to use self-supervision to generate robust features without using any labels. Next, using a parallel network architecture, an estimate of the clean labeled portion of the data is obtained. Finally, using this data, a portion of the estimated noisy labeled portion is re-labeled, before resuming the network training with the augmented data. Extensive experiments on three standard datasets show the effectiveness of the proposed framework.
Generalization Bounds for Neural Networks via Approximate Description Length
Authors: Amit Daniely, Elad Granot
Date: 2019-10-13 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Generalization Bounds

We investigate the sample complexity of networks with bounds on the magnitude of its weights. In particular, we consider the class \[ H=\left\{W_t\circ\rho\circ \ldots\circ\rho\circ W_{1} :W_1,\ldots,W_{t-1}\in M_{d, d}, W_t\in M_{1,d}\right\} \] where the spectral norm of each $W_i$ is bounded by $O(1)$, the Frobenius norm is bounded by $R$, and $\rho$ is the sigmoid function $\frac{e^x}{1+e^x}$ or the smoothened ReLU function $ \ln (1+e^x)$. We show that for any depth $t$, if the inputs are in $[-1,1]^d$, the sample complexity of $H$ is $\tilde O\left(\frac{dR^2}{\epsilon^2}\right)$. This bound is optimal up to log-factors, and substantially improves over the previous state of the art of $\tilde O\left(\frac{d^2R^2}{\epsilon^2}\right)$. We furthermore show that this bound remains valid if instead of considering the magnitude of the $W_i$'s, we consider the magnitude of $W_i - W_i^0$, where $W_i^0$ are some reference matrices, with spectral norm of $O(1)$. By taking the $W_i^0$ to be the matrices at the onset of the training process, we get sample complexity bounds that are sub-linear in the number of parameters, in many typical regimes of parameters. To establish our results we develop a new technique to analyze the sample complexity of families $H$ of predictors. We start by defining a new notion of a randomized approximate description of functions $f:X\to\mathbb{R}^d$. We then show that if there is a way to approximately describe functions in a class $H$ using $d$ bits, then $d/\epsilon^2$ examples suffices to guarantee uniform convergence. Namely, that the empirical loss of all the functions in the class is $\epsilon$-close to the true loss. Finally, we develop a set of tools for calculating the approximate description length of classes of functions that can be presented as a composition of linear function classes and non-linear functions.
Bayesian Neural Decoding Using A Diversity-Encouraging Latent Representation Learning Method
Authors: Tian Chen, Lingge Li, Gabriel Elias, Norbert Fortin, Babak Shahbaba
Date: 2019-10-13 00:00:00
Tasks: Diversity, Point Processes, Representation Learning

It is well established that temporal organization is critical to memory, and that the ability to temporally organize information is fundamental to many perceptual, cognitive, and motor processes. While our understanding of how the brain processes the spatial context of memories has advanced considerably, our understanding of their temporal organization lags far behind. In this paper, we propose a new approach for elucidating the neural basis of complex behaviors and temporal organization of memories. More specifically, we focus on neural decoding - the prediction of behavioral or experimental conditions based on observed neural data. In general, this is a challenging classification problem, which is of immense interest in neuroscience. Our goal is to develop a new framework that not only improves the overall accuracy of decoding, but also provides a clear latent representation of the decoding process. To accomplish this, our approach uses a Variational Auto-encoder (VAE) model with a diversity-encouraging prior based on determinantal point processes (DPP) to improve latent representation learning by avoiding redundancy in the latent space. We apply our method to data collected from a novel rat experiment that involves presenting repeated sequences of odors at a single port and testing the rats' ability to identify each odor. We show that our method leads to substantially higher accuracy rate for neural decoding and allows to discover novel biological phenomena by providing a clear latent representation of the decoding process.
Radiomic Feature Stability Analysis based on Probabilistic Segmentations
Authors: Christoph Haarburger, Justus Schock, Daniel Truhn, Philippe Weitz, Gustav Mueller-Franzes, Leon Weninger, Dorit Merhof
Date: 2019-10-13 00:00:00
Tasks: feature selection, Segmentation

Identifying image features that are robust with respect to segmentation variability and domain shift is a tough challenge in radiomics. So far, this problem has mainly been tackled in test-retest analyses. In this work we analyze radiomics feature stability based on probabilistic segmentations. Based on a public lung cancer dataset, we generate an arbitrary number of plausible segmentations using a Probabilistic U-Net. From these segmentations, we extract a high number of plausible feature vectors for each lung tumor and analyze feature variance with respect to the segmentations. Our results suggest that there are groups of radiomic features that are more (e.g. statistics features) and less (e.g. gray-level size zone matrix features) robust against segmentation variability. Finally, we demonstrate that segmentation variance impacts the performance of a prognostic lung cancer survival model and propose a new and potentially more robust radiomics feature selection workflow.
Analyzing User Activities Using Vector Space Model in Online Social Networks
Authors: Dhrubasish Sarkar, Premananda Jana
Date: 2019-10-13 00:00:00

The increasing popularity of internet, wireless technologies and mobile devices has led to the birth of mass connectivity and online interaction through Online Social Networks (OSNs) and similar environments. OSN reflects a social structure consist of a set of individuals and different types of ties like connections, relationships, interactions etc among them and helps its users to connect with their friends and common interest groups, share views and to pass information. Now days the users choose OSN sites as a most preferred place for sharing their updates, different views, posting photographs and would like to make it available for others for viewing, rating and making comments. The current paper aims to explore and analyze the association between the objects (like photographs, posts etc) and its viewers (friends, acquaintances etc) for a given user and to find activity relationship among them by using the TF-IDF scheme of Vector Space Model. After vectorization the vector data has been presented through a weighted graph with various properties.
Hardware/Software Codesign for Training/Testing Multiple Neural Networks on Multiple FPGAs
Authors: Brosnan Yuen
Date: 2019-10-13 00:00:00

Most neural network designs for FPGAs are inflexible. In this paper, we propose a flexible VHDL structure that would allow any neural network to be implemented on multiple FPGAs. Moreover, the VHDL structure allows for testing as well as training multiple neural networks. The VHDL design consists of multiple processor groups. There are two types of processor groups: Mini Vector Machine Processor Group and Activation Processor Group. Each processor group consists of individual Mini Vector Machines and Activation Processor. The Mini Vector Machines apply vector operations to the data, while the Activation Processors apply activation functions to the data. A ring buffer was implemented to connect the various processor groups.
eCNN: A Block-Based and Highly-Parallel CNN Accelerator for Edge Inference
Authors: Chao-Tsung Huang, Yu-Chun Ding, Huan-Ching Wang, Chi-Wen Weng, Kai-Ping Lin, Li-Wei Wang, Li-De Chen
Date: 2019-10-13 00:00:00
Tasks: 4k, Denoising, Object Recognition, Style Transfer, Super-Resolution

Convolutional neural networks (CNNs) have recently demonstrated superior quality for computational imaging applications. Therefore, they have great potential to revolutionize the image pipelines on cameras and displays. However, it is difficult for conventional CNN accelerators to support ultra-high-resolution videos at the edge due to their considerable DRAM bandwidth and power consumption. Therefore, finding a further memory- and computation-efficient microarchitecture is crucial to speed up this coming revolution. In this paper, we approach this goal by considering the inference flow, network model, instruction set, and processor design jointly to optimize hardware performance and image quality. We apply a block-based inference flow which can eliminate all the DRAM bandwidth for feature maps and accordingly propose a hardware-oriented network model, ERNet, to optimize image quality based on hardware constraints. Then we devise a coarse-grained instruction set architecture, FBISA, to support power-hungry convolution by massive parallelism. Finally,we implement an embedded processor---eCNN---which accommodates to ERNet and FBISA with a flexible processing architecture. Layout results show that it can support high-quality ERNets for super-resolution and denoising at up to 4K Ultra-HD 30 fps while using only DDR-400 and consuming 6.94W on average. By comparison, the state-of-the-art Diffy uses dual-channel DDR3-2133 and consumes 54.3W to support lower-quality VDSR at Full HD 30 fps. Lastly, we will also present application examples of high-performance style transfer and object recognition to demonstrate the flexibility of eCNN.
An Image Segmentation Model Based on a Variational Formulation
Authors: Carlos M. Paniagua Mejia
Date: 2019-10-13 00:00:00
Tasks: Image Segmentation, Semantic Segmentation

Starting from a variational formulation, we present a model for image segmentation that employs both region statistics and edge information. This combination allows for improved flexibility, making the proposed model suitable to process a wider class of images than purely region-based and edge-based models. We perform several simulations with real images that attest to the versatility of the model. We also show another set of experiments on images with certain pathologies that suggest opportunities for improvement.
Optic-Net: A Novel Convolutional Neural Network for Diagnosis of Retinal Diseases from Optical Tomography Images
Authors: Sharif Amit Kamran, Sourajit Saha, Ali Shihab Sabbir, Alireza Tavakkoli
Date: 2019-10-13 00:00:00
Tasks: Retinal OCT Disease Classification

Diagnosing different retinal diseases from Spectral Domain Optical Coherence Tomography (SD-OCT) images is a challenging task. Different automated approaches such as image processing, machine learning and deep learning algorithms have been used for early detection and diagnosis of retinal diseases. Unfortunately, these are prone to error and computational inefficiency, which requires further intervention from human experts. In this paper, we propose a novel convolution neural network architecture to successfully distinguish between different degeneration of retinal layers and their underlying causes. The proposed novel architecture outperforms other classification models while addressing the issue of gradient explosion. Our approach reaches near perfect accuracy of 99.8% and 100% for two separately available Retinal SD-OCT data-set respectively. Additionally, our architecture predicts retinal diseases in real time while outperforming human diagnosticians.
Extracting Incentives from Black-Box Decisions
Authors: Yonadav Shavit, William S. Moses
Date: 2019-10-13 00:00:00
Tasks: Reinforcement Learning

An algorithmic decision-maker incentivizes people to act in certain ways to receive better decisions. These incentives can dramatically influence subjects' behaviors and lives, and it is important that both decision-makers and decision-recipients have clarity on which actions are incentivized by the chosen model. While for linear functions, the changes a subject is incentivized to make may be clear, we prove that for many non-linear functions (e.g. neural networks, random forests), classical methods for interpreting the behavior of models (e.g. input gradients) provide poor advice to individuals on which actions they should take. In this work, we propose a mathematical framework for understanding algorithmic incentives as the challenge of solving a Markov Decision Process, where the state includes the set of input features, and the reward is a function of the model's output. We can then leverage the many toolkits for solving MDPs (e.g. tree-based planning, reinforcement learning) to identify the optimal actions each individual is incentivized to take to improve their decision under a given model. We demonstrate the utility of our method by estimating the maximally-incentivized actions in two real-world settings: a recidivism risk predictor we train using ProPublica's COMPAS dataset, and an online credit scoring tool published by the Fair Isaac Corporation (FICO).
How are attributes expressed in face DCNNs?
Authors: Prithviraj Dhar, Ankan Bansal, Carlos D. Castillo, Joshua Gleason, P. Jonathon Phillips, Rama Chellappa
Date: 2019-10-12 00:00:00
Tasks: Attribute

As deep networks become increasingly accurate at recognizing faces, it is vital to understand how these networks process faces. While these networks are solely trained to recognize identities, they also contain face related information such as sex, age, and pose of the face. The networks are not trained to learn these attributes. We introduce expressivity as a measure of how much a feature vector informs us about an attribute, where a feature vector can be from internal or final layers of a network. Expressivity is computed by a second neural network whose inputs are features and attributes. The output of the second neural network approximates the mutual information between feature vectors and an attribute. We investigate the expressivity for two different deep convolutional neural network (DCNN) architectures: a Resnet-101 and an Inception Resnet v2. In the final fully connected layer of the networks, we found the order of expressivity for facial attributes to be Age > Sex > Yaw. Additionally, we studied the changes in the encoding of facial attributes over training iterations. We found that as training progresses, expressivities of yaw, sex, and age decrease. Our technique can be a tool for investigating the sources of bias in a network and a step towards explaining the network's identity decisions.
Thompson Sampling in Non-Episodic Restless Bandits
Authors: Young Hun Jung, Marc Abeille, Ambuj Tewari
Date: 2019-10-12 00:00:00
Tasks: Open-Ended Question Answering, Thompson Sampling

Restless bandit problems assume time-varying reward distributions of the arms, which adds flexibility to the model but makes the analysis more challenging. We study learning algorithms over the unknown reward distributions and prove a sub-linear, $O(\sqrt{T}\log T)$, regret bound for a variant of Thompson sampling. Our analysis applies in the infinite time horizon setting, resolving the open question raised by Jung and Tewari (2019) whose analysis is limited to the episodic case. We adopt their policy mapping framework, which allows our algorithm to be efficient and simultaneously keeps the regret meaningful. Our algorithm adapts the TSDE algorithm of Ouyang et al. (2017) in a non-trivial manner to account for the special structure of restless bandits. We test our algorithm on a simulated dynamic channel access problem with several policy mappings, and the empirical regrets agree with the theoretical bound regardless of the choice of the policy mapping.
Model Fusion via Optimal Transport
Authors: Sidak Pal Singh, Martin Jaggi
Date: 2019-10-12 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Continual Learning, model, Model Compression, One-shot model fusion, Transfer Learning

Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual predictions, this approach is often rendered infeasible by given resource constraints in terms of memory and computation, which grow linearly with the number of models. We present a layer-wise model fusion algorithm for neural networks that utilizes optimal transport to (soft-) align neurons across the models before averaging their associated parameters. We show that this can successfully yield "one-shot" knowledge transfer (i.e, without requiring any retraining) between neural networks trained on heterogeneous non-i.i.d. data. In both i.i.d. and non-i.i.d. settings , we illustrate that our approach significantly outperforms vanilla averaging, as well as how it can serve as an efficient replacement for the ensemble with moderate fine-tuning, for standard convolutional networks (like VGG11), residual networks (like ResNet18), and multi-layer perceptrons on CIFAR10, CIFAR100, and MNIST. Finally, our approach also provides a principled way to combine the parameters of neural networks with different widths, and we explore its application for model compression. The code is available at the following link, https://github.com/sidak/otfusion.
Interventional Experiment Design for Causal Structure Learning
Authors: AmirEmad Ghassami, Saber Salehkaleybar, Negar Kiyavash
Date: 2019-10-12 00:00:00

It is known that from purely observational data, a causal DAG is identifiable only up to its Markov equivalence class, and for many ground truth DAGs, the direction of a large portion of the edges will be remained unidentified. The golden standard for learning the causal DAG beyond Markov equivalence is to perform a sequence of interventions in the system and use the data gathered from the interventional distributions. We consider a setup in which given a budget $k$, we design $k$ interventions non-adaptively. We cast the problem of finding the best intervention target set as an optimization problem which aims to maximize the number of edges whose directions are identified due to the performed interventions. First, we consider the case that the underlying causal structure is a tree. For this case, we propose an efficient exact algorithm for the worst-case gain setup, as well as an approximate algorithm for the average gain setup. We then show that the proposed approach for the average gain setup can be extended to the case of general causal structures. In this case, besides the design of interventions, calculating the objective function is also challenging. We propose an efficient exact calculator as well as two estimators for this task. We evaluate the proposed methods using synthetic as well as real data.