pwc-archive/papers-with-abstracts huggingface dataset:
SeCSeq: Semantic Coding for Sequence-to-Sequence based Extreme Multi-label Classification
Authors: Wei-Cheng Chang, Hsiang-Fu Yu, Inderjit S. Dhillon, Yiming Yang
Date: 2018-11-13 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Extreme Multi-Label Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

Extreme multi-label classification (XMC) aims at assigning to an instance the most relevant subset of labels from a colossal label set. There has been some success in formulating the multi-label problem as sequence-to-sequence (Seq2Seq) learning, where the positive class labels of each input instance are used as the corresponding output sequence. Seq2Seq methods, nonetheless, have not yet been scalable to the XMC setting due to the softmax bottleneck. In this paper, we propose a semantic coding framework, namely SeCSeq, for a Seq2Seq approach to the XMC problem. To circumvent the softmax bottleneck, SeCSeq compresses labels into sequences of semantic-aware compact codes, on which Seq2Seq models are trained. For inference, the generated semantic codes are then decompressed into sequences of positive labels using ensemble techniques. Preliminary experiments on XMC benchmark datasets show that SeCSeq is competitive with the state-of-the-art while requiring significantly fewer model parameters.
Linear Backprop in non-linear networks
Authors: Mehrdad Yazdani
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

Backprop is the primary learning algorithm used in many machine learning algorithms. In practice, however, Backprop in deep neural networks is a highly sensitive learning algorithm and successful learning depends on numerous conditions and constraints. One set of constraints is to avoid weights that lead to saturated units. The motivation for avoiding unit saturation is that gradients vanish and as a result learning comes to a halt. Careful weight initialization and re-scaling schemes such as batch normalization ensure that input activity to the neuron is within the linear regime where gradients are not vanished and can flow. Here we investigate backpropagating error terms only linearly. That is, we ignore the saturation that arise by ensuring gradients always flow. We refer to this learning rule as Linear Backprop since in the backward pass the network appears to be linear. In addition to ensuring persistent gradient flow, Linear Backprop is also favorable when computation is expensive since gradients are never computed. Our early results suggest that learning with Linear Backprop is competitive with Backprop and saves expensive gradient computations.
Learning Compact Networks via Adaptive Network Regularization
Authors: Sivaramakrishnan Sankarapandian, Anil Kag, Rachel Manzelli, Brian Kulis
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

Deep neural networks typically feature a fixed architecture, where the number of units per layer are treated as hyperparameters and tuned during training.  Recently, strategies for training adaptive neural networks without a fixed architecture have seen renewed interest.  In this paper, we employ a simple regularizer on the number of hidden units in the networks, which we refer to as adaptive network regularization (ANR). This method places a penalty on the number of hidden units per layer, designed to encourage compactness and flexibility of the network architecture. This penalty acts as the sole tuning parameter over the network size, increasing simplicity during training. We describe a training strategy that grows the number of units during training, and show on several benchmark datasets that our model yields architectures that are smaller than those obtained when tuning the number of hidden units on a standard fixed architecture. Along with smaller architectures, we show on multiple datasets that our algorithm performs comparable to or better than fixed architectures learned via grid-searching over the hyperparameters. We motivate this model using small-variance asymptotics---a Bayesian neural network with a Poisson number of units per layer becomes our model in the small-variance limit.
EnergyNet: Energy-Efficient Dynamic Inference
Authors: Yue Wang, Tan Nguyen, Yang Zhao, Zhangyang Wang, Yingyan Lin, Richard Baraniuk
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

The prohibitive energy cost of running high-performance Convolutional Neural Networks (CNNs) has been limiting their deployment on resource-constrained platforms including mobile and wearable devices. We propose a CNN for energy-aware dynamic routing, called the EnergyNet, that achieves adaptive-complexity inference based on the inputs, leading to an overall reduction of run time energy cost without noticeably losing (or even improving) accuracy. That is achieved by proposing an energy loss that captures both computational and data movement costs. We combine it with the accuracy-oriented loss, and learn a dynamic routing policy for skipping certain layers in the networks, that optimizes the hybrid loss.  Our empirical results demonstrate that, compared to the baseline CNNs, EnergyNetcan trim down the energy cost up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining the same testing accuracies.  It is further encouraging to observe that the energy awareness might serve as a training regularization and can even improve prediction accuracy:  our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Differentiable Training for Hardware Efficient LightNNs
Authors: Ruizhou Ding, Zeye Liu, Ting-Wu Chin, Diana Marculescu, R.D. (Shawn) Blanton
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Quantization

To reduce runtime and resource utilization of Deep Neural Networks (DNNs) on customized hardware, LightNN has been proposed by constraining the weights of DNNs to be a sum of a limited number (denoted as $k\in\{1,2\}$) of powers of 2. LightNNs can therefore can replace the multiplication between activations and weights with a shift operation or two shifts and an add operation. To provide a more continuous Pareto-optimal curve of accuracy and runtime so that hardware designers can have more flexible options of DNN configurations, one can customize the $k$ for each convolutional filter. In this paper, we formulate the selection of $k$ to be differentiable, and train the model weights and per-filter $k$ in an end-to-end fashion. Since flexible-$k$ LightNNs (FLightNNs) fully utilize the hardware resources on Field Programmable Gate Arrays (FPGAs), our experimental results show that FLightNN can achieve 2$\times$ speedup on FPGAs when compared to LightNN-2, with only 0.1\% accuracy degradation. In addition, compared to a 4-bit fixed-point quantization, FLightNN can achieve slightly higher accuracy and 1.4$\times$ speedup, due to its lightweight shift operations.
Dynamic parameter reallocation improves trainability of deep convolutional networks
Authors: Hesham Mostafa, Xin Wang
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Network Pruning

Network pruning has emerged as a powerful technique for reducing the size of deep neural networks. Pruning uncovers high-performance subnetworks by taking a trained dense network and gradually removing unimportant connections. Recently, alternative techniques have emerged for training sparse networks directly without having to train a large dense model beforehand, thereby achieving small memory footprints during both training and inference.These techniques are based on dynamic reallocation of non-zero parameters during training. Thus, they are in effect executing a training-time search for the optimal subnetwork. We investigate a most recent one of these techniques and conduct additional experiments to elucidate its behavior in training sparse deep convolutional networks. Dynamic parameter reallocation converges early during training to a highly trainable subnetwork. We show that neither the structure, nor the initialization of the discovered high-performance subnetwork is sufficient to explain its good performance. Rather, it is the dynamics of parameter reallocation that are responsible for successful learning. Dynamic parameter reallocation thus improves the trainability of deep convolutional networks, playing a similar role as overparameterization, without incurring the memory and computational cost of the latter.
Succinct Source Coding of Deep Neural Networks
Authors: Sourya Basu, Lav R. Varshney
Date: 2018-10-15 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

Deep neural networks have shown incredible performance for inference tasks in a variety of domains. Unfortunately, most current deep networks are enormous cloud-based structures that require significant storage space, which limits scaling of deep learning as a service (DLaaS) and use for on-device augmented intelligence.  This paper finds algorithms that directly use lossless compressed representations of deep feedforward networks (with synaptic weights drawn from discrete sets), to perform inference without full decompression. The basic insight that allows less rate than naive approaches is the recognition that the bipartite graph layers of feedforward networks have a kind of permutation invariance to the labeling of nodes, in terms of inferential operation and that the inference operation depends locally on the edges directly connected to it. We also provide experimental results of our approach on the MNIST dataset.
Compact Neural Network Solutions to Laplace's Equation in a Nanofluidic Device
Authors: Martin Magill, Faisal Z. Qureshi, Hendrick W. de Haan
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: GPU

We explore the use of neural networks to solve the Laplace equation in a two-dimensional geometry. Specifically, we study a PDE problem that models the electric potential inside the slit-well nanofluidic device. Such devices are typically used to separate polymer mixtures by molecular size. Processes like these are commonly studied using GPU-accelerated coarse-grained particle simulations, for which GPU memory is a bottleneck. We compare the memory required to represent the field using neural networks to that needed to store solutions obtained using the finite element method. We find that even simple fully-connected neural networks can achieve accuracy to memory consumption ratios comparable to the good finite element solutions. These preliminary results demonstrate an industrial application that would benefit greatly from compact neural network representation techniques.
Block-wise Intermediate Representation Training for Model Compression
Authors: Animesh Koratana*, Daniel Kang*, Peter Bailis, Matei Zahaira
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Knowledge Distillation, model, Model Compression

Knowledge distillation (KD) is a popular method for reducing the computational
overhead of deep network inference, in which the output of a teacher model is used
to train a smaller, faster student model. Hint training (i.e., FitNets) extends KD by
regressing a student model’s intermediate representation (IR) to a teacher model’s IR.
In this work, we introduce bLock-wise Intermediate representation Training (LIT), a
novel model compression technique that extends the use of IRs in deep network com-
pression, outperforming KD and hint training. LIT has two key ideas: 1) LIT trains a
student of the same width (but shallower depth) as the teacher by directly comparing
the IRs, and 2) LIT uses the IR from the previous block in the teacher model as an
input to the current student block during training, avoiding unstable IRs in the student
network. We show that LIT provides substantial reductions in network depth without
loss in accuracy — for example, LIT can compress a ResNeXt-110 to a ResNeXt-20
(5.5×) on CIFAR10 and a VDCNN-29 to a VDCNN-9 (3.2×) on Amazon Reviews,
outperforming KD and hint training in network size for a given accuracy. Finally, we
show that LIT can effectively compress GAN generators, which are not supported
in the KD framework because GANs output pixels as opposed to probabilities.
Recurrent Convolutions: A Model Compression Point of View
Authors: Zhendong Zhang, Cheolkon Jung
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Model Compression, Rolling Shutter Correction

Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals. We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks. RC reduces the redundancy across layers and is complementary to most existing model compression approaches. However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.  This reduces the value of RC for model compression. In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps. We provide insights on why this works. Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression.
Pruning at a Glance: A Structured Class-Blind Pruning Technique for Model Compression
Authors: Abdullah Salama, Oleksiy Ostapenko, Moin Nabi, Tassilo Klein
Date: 2018-10-20 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Model Compression

High performance of deep learning models typically comes at cost of considerable model size and computation time. These factors limit applicability for deployment on memory and battery constraint devices such as mobile phones or embedded systems. In this work we propose a novel pruning technique that eliminates entire filters and neurons according to their relative L1-norm as compared to the rest of the network, yielding more compression and decreased redundancy in the parameters. The resulting network is non-sparse, however, much more compact and requires no special infrastructure for its deployment. We prove the viability of our method by achieving 97.4%, 47.8% and 53% compression of LeNet-5, ResNet-56 and ResNet-110 respectively, exceeding state-of-the-art compression results reported on ResNet without losing any performance compared to the baseline. Our approach does not only exhibit good performance, but is also easy to implement on many architectures.
Efficient Inference on Deep Neural Networks by Dynamic Representations and Decision Gates
Authors: Mohammad Saeed Shafiee, Mohammad Javad Shafiee, Alexander Wong
Date: 2018-10-16 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

 The current trade-off between depth and computational cost makes it difficult to adopt deep neural networks for many industrial applications, especially when computing power is limited. Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difficult samples, a large number of samples can be well discriminated via much shallower embeddings. In this study, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths.  The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy. Experimental results show that leveraging the proposed d-gate modules led to a ~38% speed-up and ~39% FLOPS reduction on ResNet-101 and ~46% speed-up and $\sim$36\% FLOPS reduction on DenseNet-201 trained on the CIFAR10 dataset with only ~2% drop in accuracy.
On Learning Wire-Length Efficient Neural Networks
Authors: Christopher Blake, Luyu Wang, Giuseppe Castiglione, Christopher Srinivasa, Marcus Brubaker
Date: 2018-10-19 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

Pruning neural networks for wiring length efficiency is considered. Three techniques are proposed and experimentally tested: distance-based regularization, nested-rank pruning, and layer-by-layer bipartite matching. The first two algorithms are used in the training and pruning phases, respectively, and the third is used in the arranging neurons phase. Experiments show that distance-based regularization with weight based pruning tends to perform the best, with or without layer-by-layer bipartite matching. These results suggest that these techniques may be useful in creating neural networks for implementation in widely deployed specialized circuits.
PositNN: Tapered Precision Deep Learning Inference for the Edge
Authors: Hamed F. Langroudi, Zachariah Carmichael, John L. Gustafson, Dhireesha Kudithipudi
Date: 2018-10-20 00:00:00
Tasks: Deep Learning

The performance of neural networks, especially the currently popular form of deep neural networks, is often limited by the underlying hardware. Computations in deep neural networks are expensive, have large memory footprint, and are power hungry. Conventional reduced-precision numerical formats, such as fixed-point and floating point, cannot accurately represent deep neural network parameters with a nonlinear distribution and small dynamic range. Recently proposed posit numerical format with tapered precision represents small values more accurately than the other formats. In this work, we propose an ultra-low precision deep neural network, PositNN, that uses posits during inference. The efficacy of PositNN is demonstrated on a deep neural network architecture with two datasets (MNIST, Fashion MNIST and Cifar-10), where an 8-bit PositNN outperforms other {5-8}-bit low-precision neural networks and a 32-bit floating point baseline network.
Low-bit quantization and quantization-aware training for small-footprint keyword spotting
Authors: Yuriy Mishchenko, Yusuf Goren, Ming Sun, Chris Beauchene, Spyros Matsoukas, Oleg Rybakov, Shiv Naga Prasad Vitaladevuni
Date: 2018-10-19 00:00:00
Tasks: Keyword Spotting, Quantization, Small-Footprint Keyword Spotting

We investigate low-bit quantization to reduce computational cost of deep neural network (DNN) based keyword spotting (KWS). We propose approaches to further reduce quantization bits via integrating quantization into keyword spotting model training, which we refer to as quantization-aware training. Our experimental results on large dataset indicate that quantization-aware training can recover performance models quantized to lower bits representations. By combining quantization-aware training and weight matrix factorization, we are able to significantly reduce model size and computation for small-footprint keyword spotting, while maintaining performance.
A Flexible, Extensible Software Framework for Neural Net Compression
Authors: Yerlan Idelbayev, Miguel Carreira-Perpinan
Date: 2018-10-20 00:00:00
Tasks: Quantization

We propose a software framework based on ideas of the Learning-Compression algorithm , that allows one to compress any neural network by different compression mechanisms (pruning, quantization, low-rank, etc.). By design, the learning of the neural net (handled by SGD) is decoupled from the compression of its parameters (handled by a signal compression function), so that the framework can be easily extended to handle different combinations of neural net and compression type. In addition, it has other advantages, such as easy integration with deep learning frameworks, efficient training time, competitive practical performance in the loss-compression tradeoff, and reasonable convergence guarantees. Our toolkit is written in Python and Pytorch and we plan to make it available by the workshop time, and eventually open it for contributions from the community.
Scalable Recommender Systemsthrough Recursive Evidence Chains
Authors: Elias Tragas, Calvin Luo, Maxime Yvez, Kevin Luk, David Duvenaud
Date: 2018-10-20 00:00:00
Tasks: Matrix Completion, Recommendation Systems

Recommender systems can be formulated as a matrix completion problem, where the goal is to estimate missing ratings.
A popular matrix completion algorithm is matrix factorization, where ratings are predicted from combining learned user and item parameter vectors.
As dataset sizes increase many matrix factorization models suffer from slow rates of convergence, linear parameter scaling and a need to be retrained when new users or items are added.
We develop a novel approach to generate user and item vectors on demand from the ratings matrix itself and a fixed pool of parameters.
In our approach, each vector is generated using chains of evidence that link them to a small set of learned prototypical user and item latent vectors.
We demonstrate that our approach has a number of desirable scaling properties, such as having a constant rate of convergence to a competitive RMSE, requiring the optimization of only a constant number of parameters.
Training Structured Efficient Convolutional Layers
Authors: Gavin Gray, Elliot Crowley, Amos Storkey
Date: 2018-10-20 00:00:00
Tasks: Computational Efficiency

Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting. We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods. A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.
Effective, Fast, and Memory-Efficient Compressed Multi-function Convolutional Neural Networks with Compact Inception-V4
Authors: Luna Zhang
Date: 2018-10-20 00:00:00

Google's Inception-V4 using an activation function RELU is a very deep convolutional neural network (CNN) that consists of 4 Inception-A blocks, 7 Inception-B blocks, and 3 Inception-C blocks. To improve classification performance, reduce training and testing times, and reduce power consumption and memory usage (model size), a new "Compressed Multi-function Inception-V4" (CMIV4) using different activation functions is created by using k Inception-A blocks, m Inception-B blocks, and n Inception-C blocks where k = {1, 2, 3, or 4}, m = {1, 2, 3, 4, 5, 6, or 7}, n = {1, 2, or 3}, and (k+m+n)<14. For performance analysis, two datasets for two different applications (classifying brain MRI images into one of the four stages of Alzheimer's disease and using a sample of CIFAR-10 data) are used to compare three CMIV4 architectures with Inception-V4 in terms of F1-score, training and testing times (related to power consumption), and memory usage (model size). Overall, simulations show that the new CMIV4 can outperform both the commonly used single-function CNN with Inception-V4 and multi-function CNNs with Inception-V4. In the future, other compressed multi-function CNNs, such as compressed multi-function ResNets and compressed multi-function DenseNets with a reduced number of convolutional blocks using different activation functions, will be developed to increase classification accuracy, reduce training and testing times, reduce computational power, and reduce memory usage (model size) for industrial applications in IoT, big data mining, green computing, etc.