pwc-archive/papers-with-abstracts huggingface dataset:
Understanding Recurrent Neural State Using Memory Signatures
Authors: Skanda Koppula, Khe Chai Sim, Kean Chin
Date: 2018-02-11 00:00:00

We demonstrate a network visualization technique to analyze the recurrent
state inside the LSTMs/GRUs used commonly in language and acoustic models.
Interpreting intermediate state and network activations inside end-to-end
models remains an open challenge. Our method allows users to understand exactly
how much and what history is encoded inside recurrent state in grapheme
sequence models. Our procedure trains multiple decoders that predict prior
input history. Compiling results from these decoders, a user can obtain a
signature of the recurrent kernel that characterizes its memory behavior. We
demonstrate this method's usefulness in revealing information divergence in the
bases of recurrent factorized kernels, visualizing the character-level
differences between the memory of n-gram and recurrent language models, and
extracting knowledge of history encoded in the layers of grapheme-based
end-to-end ASR networks.
Topology Adaptive Graph Convolutional Networks
Authors: Jian Du, Shanghang Zhang, Guanhang Wu, Jose M. F.  Moura, Soummya Kar
Date: 2017-10-28 00:00:00
Proceeding: ICLR 2018 1

Spectral graph convolutional neural networks (CNNs) require approximation to
the convolution to alleviate the computational complexity, resulting in
performance loss. This paper proposes the topology adaptive graph convolutional
network (TAGCN), a novel graph convolutional network defined in the vertex
domain. We provide a systematic way to design a set of fixed-size learnable
filters to perform convolutions on graphs. The topologies of these filters are
adaptive to the topology of the graph when they scan the graph to perform
convolution. The TAGCN not only inherits the properties of convolutions in CNN
for grid-structured data, but it is also consistent with convolution as defined
in graph signal processing. Since no approximation to the convolution is
needed, TAGCN exhibits better performance than existing spectral CNNs on a
number of data sets and is also computationally simpler than other recent
methods.
Large-Scale Validation of Hypothesis Generation Systems via Candidate Ranking
Authors: Justin Sybrandt, Michael Shtutman, Ilya Safro
Date: 2018-02-11 00:00:00
Tasks: Topic Models

The first step of many research projects is to define and rank a short list
of candidates for study. In the modern rapidity of scientific progress, some
turn to automated hypothesis generation (HG) systems to aid this process. These
systems can identify implicit or overlooked connections within a large
scientific corpus, and while their importance grows alongside the pace of
science, they lack thorough validation. Without any standard numerical
evaluation method, many validate general-purpose HG systems by rediscovering a
handful of historical findings, and some wishing to be more thorough may run
laboratory experiments based on automatic suggestions. These methods are
expensive, time consuming, and cannot scale. Thus, we present a numerical
evaluation framework for the purpose of validating HG systems that leverages
thousands of validation hypotheses. This method evaluates a HG system by its
ability to rank hypotheses by plausibility; a process reminiscent of human
candidate selection. Because HG systems do not produce a ranking criteria,
specifically those that produce topic models, we additionally present novel
metrics to quantify the plausibility of hypotheses given topic model system
output. Finally, we demonstrate that our proposed validation method aligns with
real-world research goals by deploying our method within Moliere, our recent
topic-driven HG system, in order to automatically generate a set of candidate
genes related to HIV-associated neurodegenerative disease (HAND). By performing
laboratory experiments based on this candidate set, we discover a new
connection between HAND and Dead Box RNA Helicase 3 (DDX3). Reproducibility:
code, validation data, and results can be found at
sybrandt.com/2018/validation.
Influence-Directed Explanations for Deep Convolutional Networks
Authors: Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, Linyi Li
Date: 2018-02-11 00:00:00
Proceeding: ICLR 2018 1

We study the problem of explaining a rich class of behavioral properties of
deep neural networks. Distinctively, our influence-directed explanations
approach this problem by peering inside the network to identify neurons with
high influence on a quantity and distribution of interest, using an
axiomatically-justified influence measure, and then providing an interpretation
for the concepts these neurons represent. We evaluate our approach by
demonstrating a number of its unique capabilities on convolutional neural
networks trained on ImageNet. Our evaluation demonstrates that
influence-directed explanations (1) identify influential concepts that
generalize across instances, (2) can be used to extract the "essence" of what
the network learned about a class, and (3) isolate individual features the
network uses to make decisions and distinguish related classes.
A General and Adaptive Robust Loss Function
Authors: Jonathan T. Barron
Date: 2017-01-11 00:00:00
Proceeding: CVPR 2019 6
Tasks: Clustering, Depth Estimation, Image Generation, Unsupervised Monocular Depth Estimation

We present a generalization of the Cauchy/Lorentzian, Geman-McClure,
Welsch/Leclerc, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and L2
loss functions. By introducing robustness as a continuous parameter, our loss
function allows algorithms built around robust loss minimization to be
generalized, which improves performance on basic vision tasks such as
registration and clustering. Interpreting our loss as the negative log of a
univariate density yields a general probability distribution that includes
normal and Cauchy distributions as special cases. This probabilistic
interpretation enables the training of neural networks in which the robustness
of the loss automatically adapts itself during training, which improves
performance on learning-based tasks such as generative image synthesis and
unsupervised monocular depth estimation, without requiring any manual parameter
tuning.
Learning Deep Convolutional Networks for Demosaicing
Authors: Nai-Sheng Syu, Yu-Sheng Chen, Yung-Yu Chuang
Date: 2018-02-11 00:00:00
Tasks: Demosaicking, Denoising

This paper presents a comprehensive study of applying the convolutional
neural network (CNN) to solving the demosaicing problem. The paper presents two
CNN models that learn end-to-end mappings between the mosaic samples and the
original image patches with full information. In the case the Bayer color
filter array (CFA) is used, an evaluation with ten competitive methods on
popular benchmarks confirms that the data-driven, automatically learned
features by the CNN models are very effective. Experiments show that the
proposed CNN models can perform equally well in both the sRGB space and the
linear space. It is also demonstrated that the CNN model can perform joint
denoising and demosaicing. The CNN model is very flexible and can be easily
adopted for demosaicing with any CFA design. We train CNN models for
demosaicing with three different CFAs and obtain better results than existing
methods. With the great flexibility to be coupled with any CFA, we present the
first data-driven joint optimization of the CFA design and the demosaicing
method using CNN. Experiments show that the combination of the automatically
discovered CFA pattern and the automatically devised demosaicing method
significantly outperforms the current best demosaicing results. Visual
comparisons confirm that the proposed methods reduce more visual artifacts than
existing methods. Finally, we show that the CNN model is also effective for the
more general demosaicing problem with spatially varying exposure and color and
can be used for taking images of higher dynamic ranges with a single shot. The
proposed models and the thorough experiments together demonstrate that CNN is
an effective and versatile tool for solving the demosaicing problem.
Convex Formulations for Fair Principal Component Analysis
Authors: Matt Olfat, Anil Aswani
Date: 2018-02-11 00:00:00
Tasks: Clustering, Dimensionality Reduction, Fairness

Though there is a growing body of literature on fairness for supervised
learning, the problem of incorporating fairness into unsupervised learning has
been less well-studied. This paper studies fairness in the context of principal
component analysis (PCA). We first present a definition of fairness for
dimensionality reduction, and our definition can be interpreted as saying that
a reduction is fair if information about a protected class (e.g., race or
gender) cannot be inferred from the dimensionality-reduced data points. Next,
we develop convex optimization formulations that can improve the fairness (with
respect to our definition) of PCA and kernel PCA. These formulations are
semidefinite programs (SDP's), and we demonstrate the effectiveness of our
formulations using several datasets. We conclude by showing how our approach
can be used to perform a fair (with respect to age) clustering of health data
that may be used to set health insurance rates.
Learning Intrinsic Sparse Structures within Long Short-Term Memory
Authors: Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li
Date: 2017-09-15 00:00:00
Proceeding: ICLR 2018 1
Tasks: Language Modeling, Language Modelling, Model Compression, Question Answering

Model compression is significant for the wide adoption of Recurrent Neural
Networks (RNNs) in both user devices possessing limited resources and business
clusters requiring quick responses to large-scale service requests. This work
aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the
sizes of basic structures within LSTM units, including input updates, gates,
hidden states, cell states and outputs. Independently reducing the sizes of
basic structures can result in inconsistent dimensions among them, and
consequently, end up with invalid LSTM units. To overcome the problem, we
propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS
will simultaneously decrease the sizes of all basic structures by one and
thereby always maintain the dimension consistency. By learning ISS within LSTM
units, the obtained LSTMs remain regular while having much smaller basic
structures. Based on group Lasso regularization, our method achieves 10.59x
speedup without losing any perplexity of a language modeling of Penn TreeBank
dataset. It is also successfully evaluated through a compact model with only
2.69M weights for machine Question Answering of SQuAD dataset. Our approach is
successfully extended to non- LSTM RNNs, like Recurrent Highway Networks
(RHNs). Our source code is publicly available at
https://github.com/wenwei202/iss-rnns
On the Latent Space of Wasserstein Auto-Encoders
Authors: Paul K. Rubenstein, Bernhard Schoelkopf, Ilya Tolstikhin
Date: 2018-02-11 00:00:00
Tasks: Disentanglement, Representation Learning

We study the role of latent space dimensionality in Wasserstein auto-encoders
(WAEs). Through experimentation on synthetic and real datasets, we argue that
random encoders should be preferred over deterministic encoders. We highlight
the potential of WAEs for representation learning with promising results on a
benchmark disentanglement task.
Multi-set Canonical Correlation Analysis simply explained
Authors: Lucas C. Parra
Date: 2018-02-11 00:00:00

There are a multitude of methods to perform multi-set correlated component
analysis (MCCA), including some that require iterative solutions. The methods
differ on the criterion they optimize and the constraints placed on the
solutions. This note focuses perhaps on the simplest version, which can be
solved in a single step as the eigenvectors of matrix ${\bf D}^{-1} {\bf R}$.
Here ${\bf R}$ is the covariance matrix of the concatenated data, and ${\bf D}$
is its block-diagonal. This note shows that this solution maximizes inter-set
correlation (ISC) without further constraints. It also relates the solution to
a two step procedure, which first whitens each dataset using PCA, and then
performs an additional PCA on the concatenated and whitened data. Both these
solutions are known, although a clear derivation and simple implementation are
hard to find. This short note aims to remedy this.
Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces
Authors: Gellért Weisz, Paweł Budzianowski, Pei-Hao Su, Milica Gašić
Date: 2018-02-11 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Spoken Dialogue Systems

In spoken dialogue systems, we aim to deploy artificial intelligence to build
automated dialogue agents that can converse with humans. A part of this effort
is the policy optimisation task, which attempts to find a policy describing how
to respond to humans, in the form of a function taking the current state of the
dialogue and returning the response of the system. In this paper, we
investigate deep reinforcement learning approaches to solve this problem.
Particular attention is given to actor-critic methods, off-policy reinforcement
learning with experience replay, and various methods aimed at reducing the bias
and variance of estimators. When combined, these methods result in the
previously proposed ACER algorithm that gave competitive results in gaming
environments. These environments however are fully observable and have a
relatively small action set so in this paper we examine the application of ACER
to dialogue policy optimisation. We show that this method beats the current
state-of-the-art in deep learning approaches for spoken dialogue systems. This
not only leads to a more sample efficient algorithm that can train faster, but
also allows us to apply the algorithm in more difficult environments than
before. We thus experiment with learning in a very large action space, which
has two orders of magnitude more actions than previously considered. We find
that ACER trains significantly faster than the current state-of-the-art.
FD-MobileNet: Improved MobileNet with a Fast Downsampling Strategy
Authors: Zheng Qin, Zhaoning Zhang, Xiaotao Chen, Yuxing Peng
Date: 2018-02-11 00:00:00

We present Fast-Downsampling MobileNet (FD-MobileNet), an efficient and
accurate network for very limited computational budgets (e.g., 10-140 MFLOPs).
Our key idea is applying an aggressive downsampling strategy to MobileNet
framework. In FD-MobileNet, we perform 32$\times$ downsampling within 12
layers, only half the layers in the original MobileNet. This design brings
three advantages: (i) It remarkably reduces the computational cost. (ii) It
increases the information capacity and achieves significant performance
improvements. (iii) It is engineering-friendly and provides fast actual
inference speed. Experiments on ILSVRC 2012 and PASCAL VOC 2007 datasets
demonstrate that FD-MobileNet consistently outperforms MobileNet and achieves
comparable results with ShuffleNet under different computational budgets, for
instance, surpassing MobileNet by 5.5% on the ILSVRC 2012 top-1 accuracy and
3.6% on the VOC 2007 mAP under a complexity of 12 MFLOPs. On an ARM-based
device, FD-MobileNet achieves 1.11$\times$ inference speedup over MobileNet and
1.82$\times$ over ShuffleNet under the same complexity.
A Variance Maximization Criterion for Active Learning
Authors: Yazhou Yang, Marco Loog
Date: 2017-06-23 00:00:00
Tasks: Active Learning, Informativeness

Active learning aims to train a classifier as fast as possible with as few
labels as possible. The core element in virtually any active learning strategy
is the criterion that measures the usefulness of the unlabeled data based on
which new points to be labeled are picked. We propose a novel approach which we
refer to as maximizing variance for active learning or MVAL for short. MVAL
measures the value of unlabeled instances by evaluating the rate of change of
output variables caused by changes in the next sample to be queried and its
potential labelling. In a sense, this criterion measures how unstable the
classifier's output is for the unlabeled data points under perturbations of the
training data. MVAL maintains, what we refer to as, retraining information
matrices to keep track of these output scores and exploits two kinds of
variance to measure the informativeness and representativeness, respectively.
By fusing these variances, MVAL is able to select the instances which are both
informative and representative. We employ our technique both in combination
with logistic regression and support vector machines and demonstrate that MVAL
achieves state-of-the-art performance in experiments on a large number of
standard benchmark datasets.
A Generative Model for Dynamic Networks with Applications
Authors: Shubham Gupta, Gaurav Sharma, Ambedkar Dukkipati
Date: 2018-02-11 00:00:00
Tasks: Community Detection, Link Prediction

Networks observed in real world like social networks, collaboration networks
etc., exhibit temporal dynamics, i.e. nodes and edges appear and/or disappear
over time. In this paper, we propose a generative, latent space based,
statistical model for such networks (called dynamic networks). We consider the
case where the number of nodes is fixed, but the presence of edges can vary
over time. Our model allows the number of communities in the network to be
different at different time steps. We use a neural network based methodology to
perform approximate inference in the proposed model and its simplified version.
Experiments done on synthetic and real world networks for the task of community
detection and link prediction demonstrate the utility and effectiveness of our
model as compared to other similar existing approaches.
Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
Authors: Jian Cheng, Peisong Wang, Gang Li, Qinghao Hu, Hanqing Lu
Date: 2018-02-03 00:00:00
Tasks: Network Pruning, Quantization

Deep neural networks have evolved remarkably over the past few years and they
are currently the fundamental tools of many intelligent systems. At the same
time, the computational complexity and resource consumption of these networks
also continue to increase. This will pose a significant challenge to the
deployment of such networks, especially in real-time applications or on
resource-limited devices. Thus, network acceleration has become a hot topic
within the deep learning community. As for hardware implementation of deep
neural networks, a batch of accelerators based on FPGA/ASIC have been proposed
in recent years. In this paper, we provide a comprehensive survey of recent
advances in network acceleration, compression and accelerator design from both
algorithm and hardware points of view. Specifically, we provide a thorough
analysis of each of the following topics: network pruning, low-rank
approximation, network quantization, teacher-student networks, compact network
design and hardware accelerators. Finally, we will introduce and discuss a few
possible future directions.
Binary Classification from Positive-Confidence Data
Authors: Takashi Ishida, Gang Niu, Masashi Sugiyama
Date: 2017-10-19 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Binary Classification, Classification, Clustering, General Classification, One-Class Classification

Can we learn a binary classifier from only positive data, without any
negative data or unlabeled data? We show that if one can equip positive data
with confidence (positive-confidence), one can successfully learn a binary
classifier, which we name positive-confidence (Pconf) classification. Our work
is related to one-class classification which is aimed at "describing" the
positive class by clustering-related methods, but one-class classification does
not have the ability to tune hyper-parameters and their aim is not on
"discriminating" positive and negative classes. For the Pconf classification
problem, we provide a simple empirical risk minimization framework that is
model-independent and optimization-independent. We theoretically establish the
consistency and an estimation error bound, and demonstrate the usefulness of
the proposed method for training deep neural networks through experiments.
Lightweight Classification of IoT Malware based on Image Recognition
Authors: Jiawei Su, Danilo Vasconcellos Vargas, Sanjiva Prasad, Daniele Sgandurra, Yaokai Feng, Kouichi Sakurai
Date: 2018-02-11 00:00:00
Tasks: Classification, General Classification

The Internet of Things (IoT) is an extension of the traditional Internet,
which allows a very large number of smart devices, such as home appliances,
network cameras, sensors and controllers to connect to one another to share
information and improve user experiences. Current IoT devices are typically
micro-computers for domain-specific computations rather than traditional
functionspecific embedded devices. Therefore, many existing attacks, targeted
at traditional computers connected to the Internet, may also be directed at IoT
devices. For example, DDoS attacks have become very common in IoT environments,
as these environments currently lack basic security monitoring and protection
mechanisms, as shown by the recent Mirai and Brickerbot IoT botnets. In this
paper, we propose a novel light-weight approach for detecting DDos malware in
IoT environments.We firstly extract one-channel gray-scale images converted
from binaries, and then utilize a lightweight convolutional neural network for
classifying IoT malware families. The experimental results show that the
proposed system can achieve 94.0% accuracy for the classification of goodware
and DDoS malware, and 81.8% accuracy for the classification of goodware and two
main malware families.
The Need for Speed of AI Applications: Performance Comparison of Native vs. Browser-based Algorithm Implementations
Authors: Bernd Malle, Nicola Giuliani, Peter Kieseberg, Andreas Holzinger
Date: 2018-02-11 00:00:00
Tasks: Computational Efficiency

AI applications pose increasing demands on performance, so it is not
surprising that the era of client-side distributed software is becoming
important. On top of many AI applications already using mobile hardware, and
even browsers for computationally demanding AI applications, we are already
witnessing the emergence of client-side (federated) machine learning
algorithms, driven by the interests of large corporations and startups alike.
Apart from mathematical and algorithmic concerns, this trend especially demands
new levels of computational efficiency from client environments. Consequently,
this paper deals with the question of state-of-the-art performance by
presenting a comparison study between native code and different browser-based
implementations: JavaScript, ASM.js as well as WebAssembly on a representative
mix of algorithms. Our results show that current efforts in runtime
optimization push the boundaries well towards (and even beyond) native binary
performance. We analyze the results obtained and speculate on the reasons
behind some surprises, rounding the paper off by outlining future possibilities
as well as some of our own research efforts.
Formal Ontology Learning from English IS-A Sentences
Authors: Sourish Dasgupta, Ankur Padia, Gaurav Maheshwari, Priyansh Trivedi, Jens Lehmann
Date: 2018-02-11 00:00:00

Ontology learning (OL) is the process of automatically generating an
ontological knowledge base from a plain text document. In this paper, we
propose a new ontology learning approach and tool, called DLOL, which generates
a knowledge base in the description logic (DL) SHOQ(D) from a collection of
factual non-negative IS-A sentences in English. We provide extensive
experimental results on the accuracy of DLOL, giving experimental comparisons
to three state-of-the-art existing OL tools, namely Text2Onto, FRED, and LExO.
Here, we use the standard OL accuracy measure, called lexical accuracy, and a
novel OL accuracy measure, called instance-based inference model. In our
experimental results, DLOL turns out to be about 21% and 46%, respectively,
better than the best of the other three approaches.