pwc-archive/papers-with-abstracts huggingface dataset:
Efﬁcient Neural Network Compression via Transfer Learning for Industrial Optical Inspection
Authors: Seunghyeon Kim, Yung-Kyun Noh, Frank C. Park
Date: 2018-10-20 00:00:00
Tasks: Neural Network Compression, Transfer Learning

In this paper, we investigate learning the deep neural networks for automated optical inspection in industrial manufacturing. Our preliminary result has shown the stunning performance improvement by transfer learning from the completely dissimilar source domain: ImageNet. Further study for demystifying this improvement shows that the transfer learning produces a highly compressible network, which was not the case for the network learned from scratch. The experimental result shows that there is a negligible accuracy drop in the network learned by transfer learning until it is compressed to 1/128 reduction of the number of convolution ﬁlters. This result is contrary to the compression without transfer learning which loses more than 5% accuracy at the same compression rate.
Compact representations and pruning in residual networks
Authors: Fereshteh Lagzi, Tonio Ball, Joschka Boedecker
Date: 2018-10-20 00:00:00

We show that residual networks encode their input signals in the transient dynamics of the neurons in each layer. These representations are similar for inputs from the same class, and distinct for inputs from different classes. Based on the neural transient dynamics, we provide a sufficient criterion to determine the depth of such networks during training. This criterion is based on the convergence of the neural dynamics in the last two successive layers of the residual block. This method compresses the depth of the network and removes unnecessary deep layers.
Compression of Deep Neural Networks by combining pruning and low rank decomposition
Authors: Saurabh Goyal, Anamitra R Choudhury, Vivek Sharma, Yogish Sabharwal, Ashish Verma
Date: 2018-10-20 00:00:00
Tasks: Model Compression

Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on the cloud. 
Prior work has considered reduction in the size of the models, through compression techniques like weight pruning, filter pruning, etc. or through low-rank decomposition of the convolution layers.

In this paper, we demonstrate the use of multiple techniques to achieve not only higher model compression but also reduce the compute resources required during inferencing. We do filter pruning followed by low-rank decomposition using Tucker decomposition for model compression.

We show that our approach achieves upto 57\% higher model compression when compared to either Tucker Decomposition or Filter pruning alone  at similar accuracy for GoogleNet. Also, it reduces the Flops by upto 48\% thereby making the inferencing faster.
Embedded Deep Learning for Face Detection and Emotion Recognition with Intel© Movidius (TM) Neural Compute Stick
Authors: E. Di Nardo, A. Petrosino, V. Santopietro
Date: 2018-10-19 00:00:00
Tasks: Deep Learning, Emotion Recognition, Face Detection

Nowadays deep learning is one of the main topics in almost every field. It helped to get amazing results in a great number of tasks. The main problem is that this kind of learning and consequently neural networks, that can be defined deep, are resource intensive. They need specialized hardware to perform a computation in a reasonable time. Unfortunately, it is not sufficient to make deep learning "usable" in real life. Many tasks are mandatory to be as much as possible real-time. So it is needed to optimize many components such as code, algorithms, numeric accuracy and hardware, to make them "efficient and usable". All these optimizations can help us to produce incredibly accurate and fast learning models.
Compact and Computationally Efficient Representations of Deep Neural Networks
Authors: Simon Wiedemann, Klaus-Robert Mueller, Wojciech Samek
Date: 2018-10-19 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018

At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources. One common approach to reduce the complexity of these operations is to prune and/or quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy value is low, as measured relative to the maximum likelihood estimate of the probability mass distribution of it's elements. In order to efficiently exploit such matrices one usually relies on, inter alia, sparse matrix representations. However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements in the matrix, and can therefore not efficiently represent the entire set of matrices that exhibit low entropy statistics (thus, the entire set of compressed neural network weight matrices). In this work we address this issue and present new efficient representations for matrices with low entropy statistics. We show that the proposed formats can not only be regarded as a generalization of sparse formats, but are also more energy and time efficient under practically relevant assumptions. For instance, we experimentally show that we are able to attain up to x16 compression ratios, x1.7 speed ups and x20 energy savings when we convert the weight matrices of state-of-the-art networks such as AlexNet, VGG-16, ResNet152 and DenseNet into the new representations.
Towards Transparent Neural Network Acceleration
Authors: Nicolas Weber, Mathias Niepert, Felipe Huici
Date: 2018-10-19 00:00:00
Tasks: Deep Learning, object-detection, Object Detection

Deep learning has found numerous applications thanks to its versatility and accuracy on pattern recognition problems such as visual object detection. Learning and inference in deep neural networks, however, are memory and compute intensive and so improving efficiency is one of the major challenges for frameworks such as PyTorch, Tensorflow, and Caffe. While the efficiency problem can be partially addressed with specialized hardware and its corresponding proprietary libraries, we believe that neural network acceleration should be transparent to the user and should support all hardware platforms and deep learning libraries. 

To this end, we introduce a transparent middleware layer for neural network acceleration. The system is built around a compiler for deep learning, allowing one to combine device-specific libraries and custom optimizations while supporting numerous hardware devices. In contrast to other projects, we explicitly target the optimization of both prediction and training of neural networks. We present the current development status and some preliminary but encouraging results: on a standard x86 server, using CPUs our system achieves a 11.8x speed-up for inference and a 8.0x for batched-prediction (128); on GPUs we achieve a 1.7x and 2.3x speed-up respectively.
Efficient Computation of Quantized Neural Networks by {−1, +1} Encoding Decomposition
Authors: Qigong Sun, Fanhua Shang, Xiufang Li, Kang Yang, Peizhuo Lv, Licheng Jiao
Date: 2018-10-08 00:00:00
Tasks: image-classification, Image Classification, Model Compression, object-detection, Object Detection

Deep neural networks require extensive computing resources, and can not be efficiently applied to embedded devices such as mobile phones, which seriously limits their applicability. To address this problem, we propose a novel encoding scheme by using {-1,+1} to decompose quantized neural networks (QNNs) into multi-branch binary networks, which can be efficiently implemented by bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving. Our method can achieve at most ~59 speedup and ~32 memory saving over its full-precision counterparts. Therefore, users can easily achieve different encoding precisions arbitrarily according to their requirements and hardware resources. Our mechanism is very suitable for the use of FPGA and ASIC in terms of data storage and computation, which provides a feasible idea for smart chips. We validate the effectiveness of our method on both large-scale image classification (e.g., ImageNet) and object detection tasks.
Chopout: A Simple Way to Train Variable Sized Neural Networks at Once
Authors: Tatsuya Shirakawa
Date: 2018-10-16 00:00:00

 Large deep neural networks require huge memory to run and their running speed is sometimes too slow for real applications. Therefore network size reduction with keeping accuracy is crucial for practical applications. We present a novel neural network operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible. Chopout is easy to implement and integrate into most type of existing neural networks. Furthermore it enables to reduce size of networks and latent representations even after training just by truncating layers. We show its effectiveness through several experiments.
Hallucinations in neural machine translation
Authors: Anonymous
Date: 2018-10-30 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Data Augmentation, Decoder, Hallucination, Machine Translation, NMT, Translation

Neural machine translation (NMT) systems have reached state of the art performance in translating text and widely deployed.  Yet little is understood about how these systems function or break.  Here we show that NMT systems are susceptible to producing highly pathological translations that are completely untethered from the source material, which we term hallucinations.  Such pathological translations are problematic because they are are deeply disturbing of user trust and easy to find.  We describe a method t generate hallucinations and show that many common variations of the NMT architecture are susceptible to them. We study a variety of approaches to reduce the frequency of hallucinations, including data augmentation, dynamical systems and regularization techniques and show that data augmentation significantly reduces hallucination frequency. Finally, we analyze networks that produce hallucinations and show signatures of hallucinations in the attention matrix and in the stability measures of the decoder.
A Deep Generative Acoustic Model for Compositional Automatic Speech Recognition
Authors: Anonymous
Date: 2018-10-23 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, speech-recognition, Speech Recognition, text-to-speech, Text to Speech

Inspired by the recent successes of deep generative models for Text-To-Speech (TTS) such as WaveNet (van den Oord et al., 2016) and Tacotron (Wang et al., 2017), this article proposes the use of a deep generative model tailored for Automatic Speech Recognition (ASR) as the primary acoustic model (AM) for an overall recognition system with a separate language model (LM). Two dimensions of depth are considered: (1) the use of mixture density networks, both autoregressive and non-autoregressive, to generate density functions capable of modeling acoustic input sequences with much more powerful conditioning than the first-generation generative models for ASR, Gaussian Mixture Models / Hidden Markov Models (GMM/HMMs), and (2) the use of standard LSTMs, in the spirit of the original tandem approach, to produce discriminative feature vectors for generative modeling. Combining mixture density networks and deep discriminative features leads to a novel dual-stack LSTM architecture directly related to the RNN Transducer (Graves, 2012), but with the explicit functional form of a density, and combining naturally with a separate language model, using Bayes rule. The generative models discussed here are compared experimentally in terms of log-likelihoods and frame accuracies.
Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks
Authors: Anonymous
Date: 2018-10-31 00:00:00
Proceeding: NIPS Workshop IRASL 2018

In this paper, we introduce a novel method to interpret recurrent neural networks (RNNs), particularly long short-term memory networks (LSTMs) at the cellular level. We propose a systematic pipeline for interpreting individual hidden state dynamics within the network using response characterization methods. The ranked contribution of individual cells to the network's output is computed by analyzing a set of interpretable metrics of their decoupled step and sinusoidal responses. As a result, our method is able to uniquely identify neurons with insightful dynamics, quantify relationships between dynamical properties and test accuracy through ablation analysis, and interpret the impact of network capacity on a network's dynamical distribution. Finally, we demonstrate generalizability and scalability of our method by evaluating a series of different benchmark sequential datasets.
Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization
Authors: Anonymous
Date: 2018-10-30 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Data Augmentation, Disentanglement, Speech Synthesis, text-to-speech, Text to Speech

To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) models that can synthesize clean speech for all speakers, it is essential to learn disentangled representations which can independently control the speaker identity and background noise in generated signals. However, learning such representations can be challenging, due to the lack of labels describing the recording conditions of each training example, and the fact that  speakers and recording conditions are often correlated, e.g. since users often make many recordings using the same equipment. This paper proposes three components to address this problem by: (1) formulating a conditional generative model with factorized latent variables, (2) using data augmentation to add noise that is not correlated with speaker identity and whose label is known during training, and (3) using adversarial factorization to improve disentanglement. Experimental results demonstrate that the proposed method can disentangle speaker and noise attributes even if they are correlated in the training data, and can be used to consistently synthesize clean speech for all speakers. Ablation studies verify the importance of each proposed component.
Neuron Activation Profiles for Interpreting Convolutional Speech Recognition Models
Authors: Anonymous
Date: 2018-10-26 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Clustering, speech-recognition, Speech Recognition

The increasing complexity of deep Artificial Neural Networks (ANNs) allows to solve complex tasks in various applications. This comes with less understanding of decision processes in ANNs. Therefore, introspection techniques have been proposed to interpret how the network accomplishes its task. Those methods mostly visualize their results in the input domain and often only process single samples. For images, highlighting important features or creating inputs which activate certain neurons is intuitively interpretable. The same introspection for speech is much harder to interpret. In this paper, we propose an alternative method which analyzes neuron activations for whole data sets. Its generality allows application to complex data like speech. We introduce time-independent Neuron Activation Profiles (NAPs) as characteristic network responses to certain groups of inputs. By clustering those time-independent NAPs, we reveal that layers are specific to certain groups. We demonstrate our method for a fully-convolutional speech recognizer. There, we investigate whether phonemes are implicitly learned as an intermediate representation for predicting graphemes. We show that our method reveals, which layers encode phonemes and graphemes and that similarities between phonetic categories are reflected in the clustering of time-independent NAPs.
The divergences minimized by non-saturating GANs
Authors: Anonymous
Date: 2018-10-24 00:00:00
Tasks: Generative Adversarial Network

Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator update. The original variant is theoretically easier to study, but the alternative variant frequently performs better and is recommended for use in practice. The alternative generator update is often regarded as a simple modification to deal with optimization issues, and it appears to be a common misconception that the two variants minimize the same divergence. In this short note we derive the divergences approximately minimized by the original and alternative variants of GAN and f-GAN training. This highlights important differences between the two variants. For example, we show that the alternative variant of KL-GAN training actually minimizes the reverse KL divergence, and that the alternative variant of conventional GAN training minimizes a "softened" version of the reverse KL. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.
Visualizing Music Transformer
Authors: Anonymous
Date: 2018-10-23 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: ARC, Descriptive, Inductive Bias

Like language, music can be represented as a sequence of discrete symbols that form a hierarchical syntax, with notes being roughly like characters and motifs of notes like words.  Unlike text however, music relies heavily on repetition on multiple timescales to build structure and meaning. The Music Transformer has shown compelling results in generating music with structure (Huang et al., 2018).  In this paper, we introduce a tool for visualizing self-attention on polyphonic music with an interactive pianoroll.  We use music transformer as both a descriptive tool and a generative model.  For the former, we use it to analyze existing music to see if the resulting self-attention structure corroborates with the musical structure known from music theory.  For the latter, we inspect the model's self-attention during generation, in order to understand how past notes affect future ones. We also compare and contrast the attention structure of regular attention to that of relative attention (Shaw et al., 2018, Huang et al., 2018), and examine its impact on the resulting generated music.  For example, for the JSB Chorales dataset, a model trained with relative attention is more consistent in attending to all the voices in the preceding timestep and the chords before, and at cadences to the beginning of a phrase, allowing it to create an arc.  We hope that our analyses will offer more evidence for relative self-attention as a powerful inductive bias for modeling music.  We invite the reader to explore our video animations of music attention and to interact with the visualizations at https://storage.googleapis.com/nips-workshop-visualization/index.html.
Sound event classification using ontology-based neural networks
Authors: Anonymous
Date: 2018-10-23 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Classification

State of the art sound event classification relies in neural networks to learn the associations between class labels and audio recordings within a dataset. These datasets typically define an ontology to create a structure that relates these sound classes with more abstract super classes. Hence, the ontology serves as a source of domain knowledge representation of sounds. However, the ontology information is rarely considered, and specially under explored to model neural network architectures.
We propose two ontology-based neural network architectures for sound event classification. We defined a framework to design simple network architectures that preserve an ontological structure. The networks are trained and evaluated using two of the most common sound event classification datasets. Results show an improvement in classification performance demonstrating the benefits of including the ontological information.
Out-of-Order Decoding for Robust Neural Machine Translation
Authors: Anonymous
Date: 2018-10-23 00:00:00
Tasks: Decoder, Machine Translation, Position, Translation

Autoregressive recurrent neural decoders that generate sequences of tokens one-by-one and left-to-right are the workhorse of modern machine translation. In this work, we propose a new decoder architecture that can generate natural language sequences in an arbitrary order. Along with generating tokens from a given vocabulary, our model additionally learns to select the optimal position for each produced token. The proposed decoder architecture is fully compatible with the seq2seq framework and can be used as a drop-in replacement of any classical decoder. We demonstrate the performance of our new decoder on the IWSLT machine translation task as well as inspect and interpret the learned decoding patterns by analyzing how the model selects new positions for each subsequent token.
Bridging HMMs and RNNs through Architectural Transformations
Authors: Anonymous
Date: 2018-10-22 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Language Modeling, Language Modelling

A distinct commonality between HMMs and RNNs is that they both learn hidden representations for sequential data. In addition, it has been noted that the backward computation of the Baum-Welch algorithm for HMMs is a special case of the back-propagation algorithm used for neural networks (Eisner (2016)). Do these observations suggest that, despite their many apparent differences, HMMs are a special case of RNNs? In this paper, we show that that is indeed the case, and investigate a series of architectural transformations between HMMs and RNNs, both through theoretical derivations and empirical hybridization. In particular, we investigate three key design factors—independence assumptions between the hidden states and the observation, the placement of softmaxes, and the use of non-linearities—in order to pin down their empirical effects. We present a comprehensive empirical study to provide insights into the interplay between expressivity and interpretability in this model family with respect to language modeling and parts-of-speech induction.
Making question answering more robust through relevant context selection
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Question Answering, Retrieval, Sentence

Existing neural question answering (QA) models are required to reason over and draw complicated inferences from a long context for most large-scale QA datasets. However, if we view QA as a combined retrieval and reasoning task, we can assume the existence of a minimal context which is necessary and sufficient to answer a given question. Recent work has shown that a sentence selector module that selects a shorter context and feeds it to the downstream QA model achieves performance comparable to a QA model trained on full context, while also being more interpretable. Recent work has also shown that most state-of-the-art QA models break when adversarially generated sentences are appended to the context. While humans are immune to such distractor sentences, QA models get easily misled into selecting answers from these sentences. We hypothesize that the sentence selector module can filter out extraneous context, thereby allowing the downstream QA model to focus and reason over the parts of the context that are relevant to the question. In this paper, we show that the sentence selector itself is susceptible to adversarial inputs. However, we demonstrate that a pipeline consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context. Thus, we provide evidence towards a modular approach for question answering that is more robust and interpretable.