pwc-archive/papers-with-abstracts huggingface dataset:
Compound Density Networks
Authors: Agustinus Kristiadi, Asja Fischer
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: scoring rule

Despite the huge success of deep neural networks (NNs), finding good mechanisms for quantifying their prediction uncertainty is still an open problem. It was recently shown, that using an ensemble of NNs trained with a proper scoring rule leads to results competitive to those of Bayesian NNs. This ensemble method can be understood as finite mixture model with uniform mixing weights. We build on this mixture model approach and increase its flexibility by replacing the fixed mixing weights by an adaptive, input-dependent distribution (specifying the probability of each component) represented by an NN, and by considering uncountably many mixture components. The resulting model can be seen as the continuous counterpart to mixture density networks and is therefore referred to as compound density network. We empirically show that the proposed model results in better uncertainty estimates and is more robust to adversarial examples than previous approaches.
N-Ary Quantization for CNN Model Compression and Inference Acceleration
Authors: Günther Schindler, Wolfgang Roth, Franz Pernkopf, Holger Fröning
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Clustering, Model Compression, Quantization

The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance. However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks. In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs. Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties. The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression. For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account. We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet.
Holographic and other Point Set Distances for Machine Learning
Authors: Lukas Balles, Thomas Fischbacher
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: BIG-bench Machine Learning, object-detection, Object Detection

We introduce an analytic distance function for moderately sized point sets of known cardinality that is shown to have very desirable properties, both as a loss function as well as a regularizer for machine learning applications. We compare our novel construction to other point set distance functions and show proof of concept experiments for training neural networks end-to-end on point set prediction tasks such as object detection.
Pixel Redrawn For A Robust Adversarial Defense
Authors: Jiacang Ho, Dae-Ki Kang
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Adversarial Defense

Recently, an adversarial example becomes a serious problem to be aware of because it can fool trained neural networks easily.
To prevent the issue, many researchers have proposed several defense techniques such as adversarial training, input transformation, stochastic activation pruning, etc.
In this paper, we propose a novel defense technique, Pixel Redrawn (PR) method, which redraws every pixel of training images to convert them into distorted images.
The motivation for our PR method is from the observation that the adversarial attacks have redrawn some pixels of the original image with the known parameters of the trained neural network.
Mimicking these attacks, our PR method redraws the image without any knowledge of the trained neural network.
This method can be similar to the adversarial training method but our PR method can be used to prevent future attacks.
Experimental results on several benchmark datasets indicate our PR method not only relieves the over-fitting issue when we train neural networks with a large number of epochs, but it also boosts the robustness of the neural network.
HIGHLY EFFICIENT 8-BIT LOW PRECISION INFERENCE OF CONVOLUTIONAL NEURAL NETWORKS
Authors: Haihao Shen, Jiong Gong, Xiaoli Liu, Guoming Zhang, Ge Jin, and Eric Lin
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: image-classification, Image Classification, Image Segmentation, object-detection, Object Detection, Semantic Segmentation, Super-Resolution

High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including 1) channel-wise scale factors of weights, especially for depthwise convolution, 2) Winograd convolution, and 3) topology-wise 8-bit support. We experiment the techniques on top of a widely-used deep learning framework. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution. We show that the inference throughput
and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6%1to no loss in accuracy from FP32 baseline. We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks. All the code and models will be publicly available soon.
Geometry aware convolutional filters for omnidirectional images representation
Authors: Renata Khasanova, Pascal Frossard
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Autonomous Vehicles, image-classification, Image Classification

Due to their wide field of view, omnidirectional cameras are frequently used by autonomous vehicles, drones and robots for navigation and other computer vision tasks. The images captured by such cameras, are often analysed and classified with techniques designed for planar images that unfortunately fail to properly handle the native geometry of such images. That results in suboptimal performance, and lack of truly meaningful visual features. In this paper we aim at improving popular deep convolutional neural networks so that they can properly take into account the specific properties of omnidirectional data. In particular we propose an algorithm that adapts convolutional layers, which often serve as a core building block of a CNN, to the properties of omnidirectional images. Thus, our filters have a shape and size that adapts with the location on the omnidirectional image. We show that our method is not limited to spherical surfaces and is able to incorporate the knowledge about any kind of omnidirectional  geometry inside the deep learning network. As depicted by our experiments, our method outperforms the existing deep neural network techniques for omnidirectional image classification and compression tasks.
A CASE STUDY ON OPTIMAL DEEP LEARNING MODEL FOR UAVS
Authors: Chandan Kumar, Subrahmanyam Vaddi, Aishwarya Sarkar
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Autonomous Vehicles, Deep Learning

Over the passage of time Unmanned Autonomous Vehicles (UAVs), especially
Autonomous flying drones grabbed a lot of attention in Artificial Intelligence.
Since electronic technology is getting smaller, cheaper and more efficient, huge
advancement in the study of UAVs has been observed recently. From monitoring
floods, discerning the spread of algae in water bodies to detecting forest trail, their
application is far and wide. Our work is mainly focused on autonomous flying
drones where we establish a case study towards efficiency, robustness and accuracy
of UAVs where we showed our results well supported through experiments.
We provide details of the software and hardware architecture used in the study. We
further discuss about our implementation algorithms and present experiments that
provide a comparison between three different state-of-the-art algorithms namely
TrailNet, InceptionResnet and MobileNet in terms of accuracy, robustness, power
consumption and inference time. In our study, we have shown that MobileNet has
produced better results with very less computational requirement and power consumption.
We have also reported the challenges we have faced during our work
as well as a brief discussion on our future work to improve safety features and
performance.
DEEP-TRIM: REVISITING L1 REGULARIZATION FOR CONNECTION PRUNING OF DEEP NETWORK
Authors: Chih-Kuan Yeh, Ian E. H. Yen, Hong-You Chen, Chun-Pei Yang, Shou-De Lin, Pradeep Ravikumar
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones. The compression of DNN models has therefore become an active area of research recently, with \emph{connection pruning} emerging as one of the most successful strategies. A very natural approach is to prune connections of DNNs via $\ell_1$ regularization, but recent empirical investigations have suggested that this does not work as well in the context of DNN compression. In this work, we revisit this simple strategy and analyze it rigorously, to show that: (a) any \emph{stationary point} of an $\ell_1$-regularized layerwise-pruning objective has its number of non-zero elements bounded by the number of penalized prediction logits, regardless of the strength of the regularization; (b) successful pruning highly relies on an accurate optimization solver, and there is a trade-off between compression speed and distortion of prediction accuracy, controlled by the strength of regularization. Our theoretical results thus suggest that $\ell_1$ pruning could be successful provided we use an accurate optimization solver. We corroborate this in our experiments, where we show that simple $\ell_1$ regularization with an Adamax-L1(cumulative) solver gives pruning ratio competitive to the state-of-the-art.
INTERPRETABLE CONVOLUTIONAL FILTER PRUNING
Authors: Zhuwei Qin, Fuxun Yu, ChenChen Liu, Xiang Chen
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

The sophisticated structure of Convolutional Neural Network (CNN) allows for
outstanding performance, but at the cost of intensive computation. As significant
redundancies inevitably present in such a structure, many works have been proposed
to prune the convolutional filters for computation cost reduction. Although
extremely effective, most works are based only on quantitative characteristics of
the convolutional filters, and highly overlook the qualitative interpretation of individual
filter’s specific functionality. In this work, we interpreted the functionality
and redundancy of the convolutional filters from different perspectives, and proposed
a functionality-oriented filter pruning method. With extensive experiment
results, we proved the convolutional filters’ qualitative significance regardless of
magnitude, demonstrated significant neural network redundancy due to repetitive
filter functions, and analyzed the filter functionality defection under inappropriate
retraining process. Such an interpretable pruning approach not only offers outstanding
computation cost optimization over previous filter pruning methods, but
also interprets filter pruning process.
Learning Internal Dense But External Sparse Structures of Deep Neural Network
Authors: Yiqun Duan
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Recent years have witnessed two seemingly opposite developments of deep convolutional neural networks (CNNs). On one hand, increasing the density of CNNs by adding cross-layer connections achieve higher accuracy. On the other hand, creating sparsity structures through regularization and pruning methods enjoys lower computational costs. In this paper, we bridge these two by proposing a new network structure with locally dense yet externally sparse connections. This new structure uses dense modules, as basic building blocks and then sparsely connects these modules via a novel algorithm during the training process. Experimental results demonstrate that the locally dense yet externally sparse structure could acquire competitive performance on benchmark tasks (CIFAR10, CIFAR100, and ImageNet) while keeping the network structure slim.
Understand the dynamics of GANs via Primal-Dual Optimization
Authors: Songtao Lu, Rahul Singh, Xiangyi Chen, Yongxin Chen, Mingyi Hong
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Generative Adversarial Network, Multi-Task Learning

Generative adversarial network (GAN) is one of the best known unsupervised learning techniques these days due to its superior ability to learn data distributions. In spite of its great success in applications, GAN is known to be notoriously hard to train. The tremendous amount of time it takes to run the training algorithm and its sensitivity to hyper-parameter tuning have been haunting researchers in this area. To resolve these issues, we need to first understand how GANs work. Herein, we take a step toward this direction by examining the dynamics of GANs. We relate a large class of GANs including the Wasserstein GANs to max-min optimization problems with the coupling term being linear over the discriminator. By developing new primal-dual optimization tools, we show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate. The same framework also applies to multi-task learning and distributional robust learning problems. We verify our analysis on numerical examples with both synthetic and real data sets. We hope our analysis shed light on future studies on the theoretical properties of relevant machine learning problems.
Do Language Models Have Common Sense?
Authors: Trieu H. Trinh, Quoc V. Le
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Common Sense Reasoning, Language Modeling, Language Modelling

It has been argued that current machine learning models do not have commonsense, and therefore must be hard-coded with prior knowledge (Marcus, 2018). Here we show surprising evidence that language models can already learn to capture certain common sense knowledge. Our key observation is that a language model can compute the probability of any statement, and this probability can be used to evaluate the truthfulness of that statement.  On the Winograd Schema Challenge (Levesque et al., 2011), language models are 11% higher in accuracy than previous state-of-the-art supervised methods. Language models can also be fine-tuned for the task of Mining Commonsense Knowledge on ConceptNet to achieve an F1 score of 0.912 and 0.824, outperforming previous best results (Jastrzebskiet al., 2018).  Further analysis demonstrates that language models can discover unique features of Winograd Schema contexts that decide the correct answers without explicit supervision.
Prototypical Examples in Deep Learning: Metrics, Characteristics, and Utility
Authors: Nicholas Carlini, Ulfar Erlingsson, Nicolas Papernot
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Adversarial Robustness, Deep Learning

Machine learning (ML) research has investigated prototypes: examples that are representative of the behavior to be learned. We systematically evaluate five methods for identifying prototypes, both ones previously introduced as well as new ones we propose, finding all of them to provide meaningful but different interpretations. Through a human study, we confirm that all five metrics are well matched to human intuition. Examining cases where the metrics disagree offers an informative perspective on the properties of data and algorithms used in learning, with implications for data-corpus construction, efficiency, adversarial robustness, interpretability, and other ML aspects. In particular, we confirm that the "train on hard" curriculum approach can improve accuracy on many datasets and tasks, but that it is strictly worse when there are many mislabeled or ambiguous examples.
Learning From the Experience of Others: Approximate Empirical Bayes in Neural Networks
Authors: Han Zhao, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Geoff Gordon
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Representation Learning

Learning deep neural networks could be understood as the combination of representation learning and learning halfspaces. While most previous work aims to diversify representation learning by data augmentations and regularizations, we explore the opposite direction through the lens of empirical Bayes method. Specifically, we propose a matrix-variate normal prior whose covariance matrix has a Kronecker product structure to capture the correlations in learning different neurons through backpropagation. The prior encourages neurons to learn from the experience of others, hence it provides an effective regularization when training large networks on small datasets. To optimize the model, we design an efficient block coordinate descent algorithm with analytic solutions. Empirically, we show that the proposed method helps the network converge to better local optima that also generalize better, and we verify the effectiveness of the approach on both multiclass classification and multitask regression problems with various network structures.
Pearl: Prototype lEArning via Rule Lists
Authors: Tianfan Fu*, Tian Gao*, Cao Xiao*, Tengfei Ma*, Jimeng Sun
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Prediction

Deep neural networks have demonstrated promising prediction and classification performance on many healthcare applications. However, the interpretability of those models are often lacking. On the other hand, classical interpretable models such as rule lists or decision trees do not lead to the same level of accuracy as deep neural networks and can often be too complex to interpret (due to the potentially large depth of rule lists). In this work, we present PEARL,  Prototype lEArning via Rule Lists, which iteratively uses rule lists to guide a neural network to learn representative data prototypes. The resulting prototype neural network provides  accurate prediction, and the prediction can be easily explained by  prototype and its guiding rule lists. Thanks to the prediction power of neural networks, the rule lists from				 prototypes are more concise and hence provide better interpretability. On two real-world electronic healthcare records (EHR) datasets, PEARL consistently outperforms all baselines across both datasets, especially achieving performance improvement over conventional rule learning by up to 28% and over prototype learning by up to 3%. Experimental results also show the resulting interpretation of PEARL is  simpler than the standard rule learning.
Optimal margin Distribution Network
Authors: Shen-Huan Lv, Lu Wang, Zhi-Hua Zhou
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN
model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.
An Information-Theoretic Metric of Transferability for Task Transfer Learning
Authors: Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Amir R. Zamir, Leonidas J. Guibas
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: General Classification, Scene Understanding, Transfer Learning

An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems. Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature. This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies. Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classification, such as recognition tasks for 3D indoor-scene understanding.
K For The Price Of 1: Parameter Efficient Multi-task And Transfer Learning
Authors: Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, Andrew Howard
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Multi-Task Learning, Transfer Learning

We introduce a novel method that enables parameter-efficient transfer and multitask learning. The basic approach is to allow a model patch - a small set of parameters - to specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases allows a network to learn a completely different embedding that could be used for different tasks (such as converting an SSD detection model into a 1000-class classification model while reusing 98% of parameters of the feature extractor). Similarly, we show that re-learning the existing low-parameter layers (such as depth-wise convolutions) also improves accuracy significantly. Our approach allows both simultaneous (multi-task) learning as well as sequential transfer learning wherein we adapt pretrained networks to solve new problems. For multi-task learning, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task-based performance.