pwc-archive/papers-with-abstracts huggingface dataset:
Unsupervised Learning of Neural Networks to Explain Neural Networks
Authors: Quanshi Zhang, Yu Yang, Yuchen Liu, Ying Nian Wu, Song-Chun Zhu
Date: 2018-05-18 00:00:00
Tasks: Disentanglement, Object

This paper presents an unsupervised method to learn a neural network, namely
an explainer, to interpret a pre-trained convolutional neural network (CNN),
i.e., explaining knowledge representations hidden in middle conv-layers of the
CNN. Given feature maps of a certain conv-layer of the CNN, the explainer
performs like an auto-encoder, which first disentangles the feature maps into
object-part features and then inverts object-part features back to features of
higher conv-layers of the CNN. More specifically, the explainer contains
interpretable conv-layers, where each filter disentangles the representation of
a specific object part from chaotic input feature maps. As a paraphrase of CNN
features, the disentangled representations of object parts help people
understand the logic inside the CNN. We also learn the explainer to use
object-part features to reconstruct features of higher CNN layers, in order to
minimize loss of information during the feature disentanglement. More
crucially, we learn the explainer via network distillation without using any
annotations of sample labels, object parts, or textures for supervision. We
have applied our method to different types of CNNs for evaluation, and
explainers have significantly boosted the interpretability of CNN features.
A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training
Authors: Xin Chen, Hua Zhou, Yuxiang Gao, Yu Zhu
Date: 2018-02-07 00:00:00
Tasks: GPU, Scheduling

Large scale deep Convolution Neural Networks (CNNs) increasingly demands the
computing power. It is key for researchers to own a great powerful computing
platform to leverage deep learning (DL) advancing.On the other hand, as the
commonly-used accelerator, the commodity GPUs cards of new generations are more
and more expensive. Consequently, it is of importance to design an affordable
distributed heterogeneous system that provides powerful computational capacity
and develop a well-suited software that efficiently utilizes its computational
capacity. In this paper, we present our co-design distributed system including
a peta-scale GPU cluster, called "Manoa". Based on properties and topology of
Manoa, we first propose job server framework and implement it, named
"MiMatrix". The central node of MiMatrix, referred to as the job server,
undertakes all of controlling, scheduling and monitoring, and I/O tasks without
weight data transfer for AllReduce processing in each iteration. Therefore,
MiMatrix intrinsically solves the bandwidth bottleneck of central node in
parameter server framework that is widely used in distributed DL tasks.
Meanwhile, we also propose a new AllReduce algorithm, GPUDirect RDMA-Aware
AllReduce~(GDRAA), in which both computation and handshake message are O(1) and
the number of synchronization is two in each iteration that is a theoretical
minimum number. Owe to the dedicated co-design distributed system, MiMatrix
efficiently makes use of the Manoa's computational capacity and bandwidth. We
benchmark Manoa Resnet50 and Resenet101 on Imagenet-1K dataset. Some of results
have demonstrated state-of-the-art.
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
Authors: Yu-An Chung, Wei-Hung Weng, Schrasing Tong, James Glass
Date: 2018-05-18 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Cross-Lingual Word Embeddings, cross-modal alignment, speech-recognition, Speech Recognition, Speech-to-Text, Speech-to-Text Translation, Translation, Word Embeddings

Recent research has shown that word embedding spaces learned from text
corpora of different languages can be aligned without any parallel data
supervision. Inspired by the success in unsupervised cross-lingual word
embeddings, in this paper we target learning a cross-modal alignment between
the embedding spaces of speech and text learned from corpora of their
respective modalities in an unsupervised fashion. The proposed framework learns
the individual speech and text embedding spaces, and attempts to align the two
spaces via adversarial training, followed by a refinement procedure. We show
how our framework could be used to perform spoken word classification and
translation, and the results on these two tasks demonstrate that the
performance of our unsupervised alignment approach is comparable to its
supervised counterpart. Our framework is especially useful for developing
automatic speech recognition (ASR) and speech-to-text translation systems for
low- or zero-resource languages, which have little parallel audio-text data for
training modern supervised ASR and speech-to-text translation models, but
account for the majority of the languages spoken across the world.
Drug Selection via Joint Push and Learning to Rank
Authors: Yicheng He, Junfeng Liu, Xia Ning
Date: 2018-01-23 00:00:00
Tasks: Learning-To-Rank

Selecting the right drugs for the right patients is a primary goal of
precision medicine. In this manuscript, we consider the problem of cancer drug
selection in a learning-to-rank framework. We have formulated the cancer drug
selection problem as to accurately predicting 1). the ranking positions of
sensitive drugs and 2). the ranking orders among sensitive drugs in cancer cell
lines based on their responses to cancer drugs. We have developed a new
learning-to-rank method, denoted as pLETORg , that predicts drug ranking
structures in each cell line via using drug latent vectors and cell line latent
vectors. The pLETORg method learns such latent vectors through explicitly
enforcing that, in the drug ranking list of each cell line, the sensitive drugs
are pushed above insensitive drugs, and meanwhile the ranking orders among
sensitive drugs are correct. Genomics information on cell lines is leveraged in
learning the latent vectors. Our experimental results on a benchmark cell
line-drug response dataset demonstrate that the new pLETORg significantly
outperforms the state-of-the-art method in prioritizing new sensitive drugs.
Fast Kernel Approximations for Latent Force Models and Convolved Multiple-Output Gaussian processes
Authors: Cristian Guarnizo, Mauricio A. Álvarez
Date: 2018-05-18 00:00:00
Tasks: Gaussian Processes

A latent force model is a Gaussian process with a covariance function inspired by a differential operator. Such covariance function is obtained by performing convolution integrals between Green's functions associated to the differential operators, and covariance functions associated to latent functions. In the classical formulation of latent force models, the covariance functions are obtained analytically by solving a double integral, leading to expressions that involve numerical solutions of different types of error functions. In consequence, the covariance matrix calculation is considerably expensive, because it requires the evaluation of one or more of these error functions. In this paper, we use random Fourier features to approximate the solution of these double integrals obtaining simpler analytical expressions for such covariance functions. We show experimental results using ordinary differential operators and provide an extension to build general kernel functions for convolved multiple output Gaussian processes.
PG-TS: Improved Thompson Sampling for Logistic Contextual Bandits
Authors: Bianca Dumitrascu, Karen Feng, Barbara E. Engelhardt
Date: 2018-05-18 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Multi-Armed Bandits, Thompson Sampling

We address the problem of regret minimization in logistic contextual bandits,
where a learner decides among sequential actions or arms given their respective
contexts to maximize binary rewards. Using a fast inference procedure with
Polya-Gamma distributed augmentation variables, we propose an improved version
of Thompson Sampling, a Bayesian formulation of contextual bandits with
near-optimal performance. Our approach, Polya-Gamma augmented Thompson Sampling
(PG-TS), achieves state-of-the-art performance on simulated and real data.
PG-TS explores the action space efficiently and exploits high-reward arms,
quickly converging to solutions of low regret. Its explicit estimation of the
posterior distribution of the context feature covariance leads to substantial
empirical gains over approximate approaches. PG-TS is the first approach to
demonstrate the benefits of Polya-Gamma augmentation in bandits and to propose
an efficient Gibbs sampler for approximating the analytically unsolvable
integral of logistic contextual bandits.
Adversarial Structure Matching for Structured Prediction Tasks
Authors: Jyh-Jing Hwang, Tsung-Wei Ke, Jianbo Shi, Stella X. Yu
Date: 2018-05-18 00:00:00
Proceeding: CVPR 2019 6
Tasks: Depth Estimation, image-classification, Image Classification, Monocular Depth Estimation, Prediction, Semantic Segmentation, Structured Prediction

Pixel-wise losses, e.g., cross-entropy or L2, have been widely used in structured prediction tasks as a spatial extension of generic image classification or regression. However, its i.i.d. assumption neglects the structural regularity present in natural images. Various attempts have been made to incorporate structural reasoning mostly through structure priors in a cooperative way where co-occurring patterns are encouraged. We, on the other hand, approach this problem from an opposing angle and propose a new framework, Adversarial Structure Matching (ASM), for training such structured prediction networks via an adversarial process, in which we train a structure analyzer that provides the supervisory signals, the ASM loss. The structure analyzer is trained to maximize the ASM loss, or to emphasize recurring multi-scale hard negative structural mistakes among co-occurring patterns. On the contrary, the structured prediction network is trained to reduce those mistakes and is thus enabled to distinguish fine-grained structures. As a result, training structured prediction networks using ASM reduces contextual confusion among objects and improves boundary localization. We demonstrate that our ASM outperforms pixel-wise IID loss or structural prior GAN loss on three different structured prediction tasks: semantic segmentation, monocular depth estimation, and surface normal prediction.
Fisher Efficient Inference of Intractable Models
Authors: Song Liu, Takafumi Kanamori, Wittawat Jitkrittum, Yu Chen
Date: 2018-05-18 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Density Ratio Estimation

Maximum Likelihood Estimators (MLE) has many good properties. For example, the asymptotic variance of MLE solution attains equality of the asymptotic Cram{\'e}r-Rao lower bound (efficiency bound), which is the minimum possible variance for an unbiased estimator. However, obtaining such MLE solution requires calculating the likelihood function which may not be tractable due to the normalization term of the density model. In this paper, we derive a Discriminative Likelihood Estimator (DLE) from the Kullback-Leibler divergence minimization criterion implemented via density ratio estimation and a Stein operator. We study the problem of model inference using DLE. We prove its consistency and show that the asymptotic variance of its solution can attain the equality of the efficiency bound under mild regularity conditions. We also propose a dual formulation of DLE which can be easily optimized. Numerical studies validate our asymptotic theorems and we give an example where DLE successfully estimates an intractable model constructed using a pre-trained deep neural network.
Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks
Authors: Yingzhou Li, Xiuyuan Cheng, Jianfeng Lu
Date: 2018-05-18 00:00:00

Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-Net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-Net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-Net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increase the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
Authors: Arash Vahdat, Evgeny Andriyash, William G. Macready
Date: 2018-05-18 00:00:00
Proceeding: NeurIPS 2018 12

Boltzmann machines are powerful distributions that have been shown to be an
effective prior over binary latent variables in variational autoencoders
(VAEs). However, previous methods for training discrete VAEs have used the
evidence lower bound and not the tighter importance-weighted bound. We propose
two approaches for relaxing Boltzmann machines to continuous distributions that
permit training with importance-weighted bounds. These relaxations are based on
generalized overlapping transformations and the Gaussian integral trick.
Experiments on the MNIST and OMNIGLOT datasets show that these relaxations
outperform previous discrete VAEs with Boltzmann priors. An implementation
which reproduces these results is available at
https://github.com/QuadrantAI/dvae .
Modeling Global Dynamics from Local Snapshots with Deep Generative Neural Networks
Authors: Scott Gigante, David van Dijk, Kevin Moon, Alexander Strzalkowski, Guy Wolf, Smita Krishnaswamy
Date: 2018-02-10 00:00:00
Tasks: Dimensionality Reduction

Complex high dimensional stochastic dynamic systems arise in many applications in the natural sciences and especially biology. However, while these systems are difficult to describe analytically, "snapshot" measurements that sample the output of the system are often available. In order to model the dynamics of such systems given snapshot data, or local transitions, we present a deep neural network framework we call Dynamics Modeling Network or DyMoN. DyMoN is a neural network framework trained as a deep generative Markov model whose next state is a probability distribution based on the current state. DyMoN is trained using samples of current and next-state pairs, and thus does not require longitudinal measurements. We show the advantage of DyMoN over shallow models such as Kalman filters and hidden Markov models, and other deep models such as recurrent neural networks in its ability to embody the dynamics (which can be studied via perturbation of the neural network) and generate longitudinal hypothetical trajectories. We perform three case studies in which we apply DyMoN to different types of biological systems and extract features of the dynamics in each case by examining the learned model.
GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
Authors: Shusen Wang, Farbod Roosta-Khorasani, Peng Xu, Michael W. Mahoney
Date: 2017-09-11 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Distributed Computing, Distributed Optimization

For distributed computing environment, we consider the empirical risk
minimization problem and propose a distributed and communication-efficient
Newton-type optimization method. At every iteration, each worker locally finds
an Approximate NewTon (ANT) direction, which is sent to the main driver. The
main driver, then, averages all the ANT directions received from workers to
form a {\it Globally Improved ANT} (GIANT) direction. GIANT is highly
communication efficient and naturally exploits the trade-offs between local
computations and global communications in that more local computations result
in fewer overall rounds of communications. Theoretically, we show that GIANT
enjoys an improved convergence rate as compared with first-order methods and
existing distributed Newton-type methods. Further, and in sharp contrast with
many existing distributed Newton-type methods, as well as popular first-order
methods, a highly advantageous practical feature of GIANT is that it only
involves one tuning parameter. We conduct large-scale experiments on a computer
cluster and, empirically, demonstrate the superior performance of GIANT.
Multi-view Sentence Representation Learning
Authors: Shuai Tang, Virginia R. de Sa
Date: 2018-05-18 00:00:00
Tasks: MULTI-VIEW LEARNING, Representation Learning, Sentence

Multi-view learning can provide self-supervision when different views are
available of the same data. The distributional hypothesis provides another form
of useful self-supervision from adjacent sentences which are plentiful in large
unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the
human brain as well as the observation that different learning architectures
tend to emphasise different aspects of sentence meaning, we create a unified
multi-view sentence representation learning framework, in which, one view
encodes the input sentence with a Recurrent Neural Network (RNN), and the other
view encodes it with a simple linear model, and the training objective is to
maximise the agreement specified by the adjacent context information between
two views. We show that, after training, the vectors produced from our
multi-view training provide improved representations over the single-view
training, and the combination of different views gives further representational
improvement and demonstrates solid transferability on standard downstream
tasks.
My camera can see through fences: A deep learning approach for image de-fencing
Authors: Sankaraganesh Jonna, Krishna Kanth Nakka, Rajiv R. Sahay
Date: 2018-05-18 00:00:00

In recent times, the availability of inexpensive image capturing devices such
as smartphones/tablets has led to an exponential increase in the number of
images/videos captured. However, sometimes the amateur photographer is hindered
by fences in the scene which have to be removed after the image has been
captured. Conventional approaches to image de-fencing suffer from inaccurate
and non-robust fence detection apart from being limited to processing images of
only static occluded scenes. In this paper, we propose a semi-automated
de-fencing algorithm using a video of the dynamic scene. We use convolutional
neural networks for detecting fence pixels. We provide qualitative as well as
quantitative comparison results with existing lattice detection algorithms on
the existing PSU NRT data set and a proposed challenging fenced image dataset.
The inverse problem of fence removal is solved using split Bregman technique
assuming total variation of the de-fenced image as the regularization
constraint.
Overcoming catastrophic forgetting problem by weight consolidation and long-term memory
Authors: Shixian Wen, Laurent Itti
Date: 2018-05-18 00:00:00
Tasks: Task 2

Sequential learning of multiple tasks in artificial neural networks using
gradient descent leads to catastrophic forgetting, whereby previously learned
knowledge is erased during learning of new, disjoint knowledge. Here, we
propose a new approach to sequential learning which leverages the recent
discovery of adversarial examples. We use adversarial subspaces from previous
tasks to enable learning of new tasks with less interference. We apply our
method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6,
(task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task). We compare and
combine our Adversarial Direction (AD) method with the recently proposed
Elastic Weight Consolidation (EWC) method for sequential learning. We train
each task for 20 epochs, which yields good initial performance (99.24% correct
task 1 performance). After training task 2, and then task 3, both plain
gradient descent (PGD) and EWC largely forget task 1 (task 1 accuracy 32.95%
for PGD and 41.02% for EWC), while our combined approach (AD+EWC) still
achieves 94.53% correct on task 1. We obtain similar results with a much more
difficult disjoint CIFAR10 task, which to our knowledge had not been attempted
before (70.10% initial task 1 performance, 67.73% after learning tasks 2 and 3
for AD+EWC, while PGD and EWC both fall to chance level). Our results suggest
that AD+EWC can provide better sequential learning performance than either PGD
or EWC.
Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
Authors: Linnan Wang, Yiyang Zhao, Yuu Jinnai, Yuandong Tian, Rodrigo Fonseca
Date: 2018-05-18 00:00:00
Tasks: GPU, Image Captioning, Neural Architecture Search, object-detection, Object Detection, Style Transfer, Transfer Learning

Neural Architecture Search (NAS) has shown great success in automating the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires further investigations in improving the sample efficiency and the network evaluation cost to get better results in a shorter time. In this paper, we present a novel scalable Monte Carlo Tree Search (MCTS) based NAS agent, named AlphaX, to tackle these two aspects. AlphaX improves the search efficiency by adaptively balancing the exploration and exploitation at the state level, and by a Meta-Deep Neural Network (DNN) to predict network accuracies for biasing the search toward a promising region. To amortize the network evaluation cost, AlphaX accelerates MCTS rollouts with a distributed design and reduces the number of epochs in evaluating a network by transfer learning, which is guided with the tree structure in MCTS. In 12 GPU days and 1000 samples, AlphaX found an architecture that reaches 97.84\% top-1 accuracy on CIFAR-10, and 75.5\% top-1 accuracy on ImageNet, exceeding SOTA NAS methods in both the accuracy and sampling efficiency. Particularly, we also evaluate AlphaX on NASBench-101, a large scale NAS dataset; AlphaX is 3x and 2.8x more sample efficient than Random Search and Regularized Evolution in finding the global optimum. Finally, we show the searched architecture improves a variety of vision applications from Neural Style Transfer, to Image Captioning and Object Detection.
DeepLogic: Towards End-to-End Differentiable Logical Reasoning
Authors: Nuri Cingillioglu, Alessandra Russo
Date: 2018-05-18 00:00:00
Tasks: BIG-bench Machine Learning, Logical Reasoning

Combining machine learning with logic-based expert systems in order to get
the best of both worlds are becoming increasingly popular. However, to what
extent machine learning can already learn to reason over rule-based knowledge
is still an open problem. In this paper, we explore how symbolic logic, defined
as logic programs at a character level, is learned to be represented in a
high-dimensional vector space using RNN-based iterative neural networks to
perform reasoning. We create a new dataset that defines 12 classes of logic
programs exemplifying increased level of complexity of logical reasoning and
train the networks in an end-to-end fashion to learn whether a logic program
entails a given query. We analyse how learning the inference algorithm gives
rise to representations of atoms, literals and rules within logic programs and
evaluate against increasing lengths of predicate and constant symbols as well
as increasing steps of multi-hop reasoning.