pwc-archive/papers-with-abstracts huggingface dataset:
Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Authors: Reza Ghaeini, Xiaoli Z. Fern, Prasad Tadepalli
Date: 2018-08-12 00:00:00
Proceeding: EMNLP 2018 10
Tasks: Deep Learning, Natural Language Inference

Deep learning models have achieved remarkable success in natural language
inference (NLI) tasks. While these models are widely explored, they are hard to
interpret and it is often unclear how and why they actually work. In this
paper, we take a step toward explaining such deep learning based models through
a case study on a popular neural model for NLI. In particular, we propose to
interpret the intermediate layers of NLI models by visualizing the saliency of
attention and LSTM gating signals. We present several examples for which our
methods are able to reveal interesting insights and identify the critical
information contributing to the model decisions.
Robust high dimensional factor models with applications to statistical machine learning
Authors: Jianqing Fan, Kaizheng Wang, Yiqiao Zhong, Ziwei Zhu
Date: 2018-08-12 00:00:00
Tasks: BIG-bench Machine Learning, Model Selection, Vocal Bursts Intensity Prediction

Factor models are a class of powerful statistical models that have been
widely used to deal with dependent measurements that arise frequently from
various applications from genomics and neuroscience to economics and finance.
As data are collected at an ever-growing scale, statistical machine learning
faces some new challenges: high dimensionality, strong dependence among
observed variables, heavy-tailed variables and heterogeneity. High-dimensional
robust factor analysis serves as a powerful toolkit to conquer these
challenges.
  This paper gives a selective overview on recent advance on high-dimensional
factor models and their applications to statistics including Factor-Adjusted
Robust Model selection (FarmSelect) and Factor-Adjusted Robust Multiple testing
(FarmTest). We show that classical methods, especially principal component
analysis (PCA), can be tailored to many new problems and provide powerful tools
for statistical estimation and inference. We highlight PCA and its connections
to matrix perturbation theory, robust statistics, random projection, false
discovery rate, etc., and illustrate through several applications how insights
from these fields yield solutions to modern challenges. We also present
far-reaching connections between factor models and popular statistical learning
problems, including network analysis and low-rank matrix recovery.
Semi-supervised Skin Lesion Segmentation via Transformation Consistent Self-ensembling Model
Authors: Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, Pheng-Ann Heng
Date: 2018-08-12 00:00:00
Tasks: Lesion Segmentation, Segmentation, Skin Lesion Segmentation

Automatic skin lesion segmentation on dermoscopic images is an essential
component in computer-aided diagnosis of melanoma. Recently, many fully
supervised deep learning based methods have been proposed for automatic skin
lesion segmentation. However, these approaches require massive pixel-wise
annotation from experienced dermatologists, which is very costly and
time-consuming. In this paper, we present a novel semi-supervised method for
skin lesion segmentation by leveraging both labeled and unlabeled data. The
network is optimized by the weighted combination of a common supervised loss
for labeled inputs only and a regularization loss for both labeled and
unlabeled data. In this paper, we present a novel semi-supervised method for
skin lesion segmentation, where the network is optimized by the weighted
combination of a common supervised loss for labeled inputs only and a
regularization loss for both labeled and unlabeled data. Our method encourages
a consistent prediction for unlabeled images using the outputs of the
network-in-training under different regularizations, so that it can utilize the
unlabeled data. To utilize the unlabeled data, our method encourages the
consistent predictions of the network-in-training for the same input under
different regularizations. Aiming for the semi-supervised segmentation problem,
we enhance the effect of regularization for pixel-level predictions by
introducing a transformation, including rotation and flipping, consistent
scheme in our self-ensembling model. With only 300 labeled training samples,
our method sets a new record on the benchmark of the International Skin Imaging
Collaboration (ISIC) 2017 skin lesion segmentation challenge. Such a result
clearly surpasses fully-supervised state-of-the-arts that are trained with 2000
labeled data.
A Basic Compositional Model for Spiking Neural Networks
Authors: Nancy Lynch, Cameron Musco
Date: 2018-08-12 00:00:00
Tasks: model

We present a formal, mathematical foundation for modeling and reasoning about the behavior of $synchronous$, $stochastic$ $Spiking$ $Neural$ $Networks$ $(SNNs)$, which have been widely used in studies of neural computation. Our approach follows paradigms established in the field of concurrency theory. Our SNN model is based on directed graphs of neurons, classified as input, output, and internal neurons. We focus here on basic SNNs, in which a neuron's only state is a Boolean value indicating whether or not the neuron is currently firing. We also define the $external$ $behavior$ of an SNN, in terms of probability distributions on its external firing patterns. We define two operators on SNNs: a $composition$ $operator$, which supports modeling of SNNs as combinations of smaller SNNs, and a $hiding$ $operator$, which reclassifies some output behavior of an SNN as internal. We prove results showing how the external behavior of a network built using these operators is related to the external behavior of its component networks. Finally, we define the notion of a $problem$ to be solved by an SNN, and show how the composition and hiding operators affect the problems that are solved by the networks. We illustrate our definitions with three examples: a Boolean circuit constructed from gates, an $Attention$ network constructed from a $Winner$-$Take$-$All$ network and a $Filter$ network, and a toy example involving combining two networks in a cyclic fashion.
Parallelization does not Accelerate Convex Optimization: Adaptivity Lower Bounds for Non-smooth Convex Minimization
Authors: Eric Balkanski, Yaron Singer
Date: 2018-08-12 00:00:00
Tasks: Combinatorial Optimization

In this paper we study the limitations of parallelization in convex optimization. A convenient approach to study parallelization is through the prism of \emph{adaptivity} which is an information theoretic measure of the parallel runtime of an algorithm [BS18]. Informally, adaptivity is the number of sequential rounds an algorithm needs to make when it can execute polynomially-many queries in parallel at every round. For combinatorial optimization with black-box oracle access, the study of adaptivity has recently led to exponential accelerations in parallel runtime and the natural question is whether dramatic accelerations are achievable for convex optimization. For the problem of minimizing a non-smooth convex function $f:[0,1]^n\to \mathbb{R}$ over the unit Euclidean ball, we give a tight lower bound that shows that even when $\texttt{poly}(n)$ queries can be executed in parallel, there is no randomized algorithm with $\tilde{o}(n^{1/3})$ rounds of adaptivity that has convergence rate that is better than those achievable with a one-query-per-round algorithm. A similar lower bound was obtained by Nemirovski [Nem94], however that result holds for the $\ell_{\infty}$-setting instead of $\ell_2$. In addition, we also show a tight lower bound that holds for Lipschitz and strongly convex functions. At the time of writing this manuscript we were not aware of Nemirovski's result. The construction we use is similar to the one in [Nem94], though our analysis is different. Due to the close relationship between this work and [Nem94], we view the research contribution of this manuscript limited and it should serve as an instructful approach to understanding lower bounds for parallel optimization.
Orders-of-magnitude speedup in atmospheric chemistry modeling through neural network-based emulation
Authors: Makoto M. Kelp, Christopher W. Tessum, Julian D. Marshall
Date: 2018-08-11 00:00:00
Tasks: GPU, Multi-target regression

Chemical transport models (CTMs), which simulate air pollution transport,
transformation, and removal, are computationally expensive, largely because of
the computational intensity of the chemical mechanisms: systems of coupled
differential equations representing atmospheric chemistry. Here we investigate
the potential for machine learning to reproduce the behavior of a chemical
mechanism, yet with reduced computational expense. We create a 17-layer
residual multi-target regression neural network to emulate the Carbon Bond
Mechanism Z (CBM-Z) gas-phase chemical mechanism. We train the network to match
CBM-Z predictions of changes in concentrations of 77 chemical species after one
hour, given a range of chemical and meteorological input conditions, which it
is able to do with root-mean-square error (RMSE) of less than 1.97 ppb (median
RMSE = 0.02 ppb), while achieving a 250x computational speedup. An additional
17x speedup (total 4250x speedup) is achieved by running the neural network on
a graphics-processing unit (GPU). The neural network is able to reproduce the
emergent behavior of the chemical system over diurnal cycles using Euler
integration, but additional work is needed to constrain the propagation of
errors as simulation time progresses.
A Consistent Method for Learning OOMs from Asymptotically Stationary Time Series Data Containing Missing Values
Authors: Tianlin Liu
Date: 2018-08-11 00:00:00
Tasks: Missing Values, Time Series, Time Series Analysis

In the traditional framework of spectral learning of stochastic time series
models, model parameters are estimated based on trajectories of fully recorded
observations. However, real-world time series data often contain missing
values, and worse, the distributions of missingness events over time are often
not independent of the visible process. Recently, a spectral OOM learning
algorithm for time series with missing data was introduced and proved to be
consistent, albeit under quite strong conditions. Here we refine the algorithm
and prove that the original strong conditions can be very much relaxed. We
validate our theoretical findings by numerical experiments, showing that the
algorithm can consistently handle missingness patterns whose dynamic interacts
with the visible process.
Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
Authors: Maha Elbayad, Laurent Besacier, Jakob Verbeek
Date: 2018-08-11 00:00:00
Proceeding: CONLL 2018 10
Tasks: Decoder, Machine Translation, Translation

Current state-of-the-art machine translation systems are based on
encoder-decoder architectures, that first encode the input sequence, and then
generate an output sequence based on the input encoding. Both are interfaced
with an attention mechanism that recombines a fixed encoding of the source
tokens based on the decoder state. We propose an alternative approach which
instead relies on a single 2D convolutional neural network across both
sequences. Each layer of our network re-codes source tokens on the basis of the
output sequence produced so far. Attention-like properties are therefore
pervasive throughout the network. Our model yields excellent results,
outperforming state-of-the-art encoder-decoder systems, while being
conceptually simpler and having fewer parameters.
Ranking with Features: Algorithm and A Graph Theoretic Analysis
Authors: Aadirupa Saha, Arun Rajkumar
Date: 2018-08-11 00:00:00
Tasks: Graph Matching, Matrix Completion

We consider the problem of ranking a set of items from pairwise comparisons in the presence of features associated with the items. Recent works have established that $O(n\log(n))$ samples are needed to rank well when there is no feature information present. However, this might be sub-optimal in the presence of associated features. We introduce a new probabilistic preference model called feature-Bradley-Terry-Luce (f-BTL) model that generalizes the standard BTL model to incorporate feature information. We present a new least squares based algorithm called fBTL-LS which we show requires much lesser than $O(n\log(n))$ pairs to obtain a good ranking -- precisely our new sample complexity bound is of $O(\alpha\log \alpha)$, where $\alpha$ denotes the number of `independent items' of the set, in general $\alpha << n$. Our analysis is novel and makes use of tools from classical graph matching theory to provide tighter bounds that sheds light on the true complexity of the ranking problem, capturing the item dependencies in terms of their feature representations. This was not possible with earlier matrix completion based tools used for this problem. We also prove an information theoretic lower bound on the required sample complexity for recovering the underlying ranking, which essentially shows the tightness of our proposed algorithms. The efficacy of our proposed algorithms are validated through extensive experimental evaluations on a variety of synthetic and real world datasets.
Neural Importance Sampling
Authors: Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, Jan Novák
Date: 2018-08-11 00:00:00

We propose to use deep neural networks for generating samples in Monte Carlo integration. Our work is based on non-linear independent components estimation (NICE), which we extend in numerous ways to improve performance and enable its application to integration problems. First, we introduce piecewise-polynomial coupling transforms that greatly increase the modeling power of individual coupling layers. Second, we propose to preprocess the inputs of neural networks using one-blob encoding, which stimulates localization of computation and improves inference. Third, we derive a gradient-descent-based optimization for the KL and the $\chi^2$ divergence for the specific application of Monte Carlo integration with unnormalized stochastic estimates of the target distribution. Our approach enables fast and accurate inference and efficient sample generation independently of the dimensionality of the integration domain. We show its benefits on generating natural images and in two applications to light-transport simulation: first, we demonstrate learning of joint path-sampling densities in the primary sample space and importance sampling of multi-dimensional path prefixes thereof. Second, we use our technique to extract conditional directional densities driven by the product of incident illumination and the BSDF in the rendering equation, and we leverage the densities for path guiding. In all applications, our approach yields on-par or higher performance than competing techniques at equal sample count.
A Parameterized Complexity View on Description Logic Reasoning
Authors: Ronald de Haan
Date: 2018-08-11 00:00:00

Description logics are knowledge representation languages that have been
designed to strike a balance between expressivity and computational
tractability. Many different description logics have been developed, and
numerous computational problems for these logics have been studied for their
computational complexity. However, essentially all complexity analyses of
reasoning problems for description logics use the one-dimensional framework of
classical complexity theory. The multi-dimensional framework of parameterized
complexity theory is able to provide a much more detailed image of the
complexity of reasoning problems.
  In this paper we argue that the framework of parameterized complexity has a
lot to offer for the complexity analysis of description logic reasoning
problems---when one takes a progressive and forward-looking view on
parameterized complexity tools. We substantiate our argument by means of three
case studies. The first case study is about the problem of concept
satisfiability for the logic ALC with respect to nearly acyclic TBoxes. The
second case study concerns concept satisfiability for ALC concepts
parameterized by the number of occurrences of union operators and the number of
occurrences of full existential quantification. The third case study offers a
critical look at data complexity results from a parameterized complexity point
of view. These three case studies are representative for the wide range of uses
for parameterized complexity methods for description logic problems.
Fully-Automated Analysis of Body Composition from CT in Cancer Patients Using Convolutional Neural Networks
Authors: Christopher P. Bridge, Michael Rosenthal, Bradley Wright, Gopal Kotecha, Florian Fintelmann, Fabian Troschel, Nityanand Miskin, Khanant Desai, William Wrobel, Ana Babic, Natalia Khalaf, Lauren Brais, Marisa Welch, Caitlin Zellers, Neil Tenenholtz, Mark Michalski, Brian Wolpin, Katherine Andriole
Date: 2018-08-11 00:00:00
Tasks: Segmentation

The amounts of muscle and fat in a person's body, known as body composition,
are correlated with cancer risks, cancer survival, and cardiovascular risk. The
current gold standard for measuring body composition requires time-consuming
manual segmentation of CT images by an expert reader. In this work, we describe
a two-step process to fully automate the analysis of CT body composition using
a DenseNet to select the CT slice and U-Net to perform segmentation. We train
and test our methods on independent cohorts. Our results show Dice scores
(0.95-0.98) and correlation coefficients (R=0.99) that are favorable compared
to human readers. These results suggest that fully automated body composition
analysis is feasible, which could enable both clinical use and large-scale
population studies.
Matrix Factorization on GPUs with Memory Optimization and Approximate Computing
Authors: Wei Tan, Shiyu Chang, Liana Fong, Cheng Li, Zijun Wang, Liangliang Cao
Date: 2018-08-11 00:00:00
Tasks: Collaborative Filtering, CPU, Data Compression, GPU

Matrix factorization (MF) discovers latent features from observations, which
has shown great promises in the fields of collaborative filtering, data
compression, feature extraction, word embedding, etc. While many
problem-specific optimization techniques have been proposed, alternating least
square (ALS) remains popular due to its general applicability e.g. easy to
handle positive-unlabeled inputs, fast convergence and parallelization
capability. Current MF implementations are either optimized for a single
machine or with a need of a large computer cluster but still are insufficient.
This is because a single machine provides limited compute power for large-scale
data while multiple machines suffer from the network communication bottleneck.
  To address the aforementioned challenge, accelerating ALS on graphics
processing units (GPUs) is a promising direction. We propose the novel approach
in enhancing the MF efficiency via both memory optimization and approximate
computing. The former exploits GPU memory hierarchy to increase data reuse,
while the later reduces unnecessary computing without hurting the convergence
of learning algorithms. Extensive experiments on large-scale datasets show that
our solution not only outperforms the competing CPU solutions by a large margin
but also has a 2x-4x performance gain compared to the state-of-the-art GPU
solutions. Our implementations are open-sourced and publicly available.
Fake Sentence Detection as a Training Task for Sentence Encoding
Authors: Viresh Ranjan, Heeyoung Kwon, Niranjan Balasubramanian, Minh Hoai
Date: 2018-08-11 00:00:00
Proceeding: ICLR 2019 5
Tasks: Binary Classification, Language Modeling, Language Modelling, Sentence

Sentence encoders are typically trained on language modeling tasks with large
unlabeled datasets. While these encoders achieve state-of-the-art results on
many sentence-level tasks, they are difficult to train with long training
cycles. We introduce fake sentence detection as a new training task for
learning sentence encoders. We automatically generate fake sentences by
corrupting original sentences from a source collection and train the encoders
to produce representations that are effective at detecting fake sentences. This
binary classification task turns to be quite efficient for training sentence
encoders. We compare a basic BiLSTM encoder trained on this task with a strong
sentence encoding models (Skipthought and FastSent) trained on a language
modeling task. We find that the BiLSTM trains much faster on fake sentence
detection (20 hours instead of weeks) using smaller amounts of data (1M instead
of 64M sentences). Further analysis shows the learned representations capture
many syntactic and semantic properties expected from good sentence
representations.
jLDADMM: A Java package for the LDA and DMM topic models
Authors: Dat Quoc Nguyen
Date: 2018-08-11 00:00:00
Tasks: Clustering, Topic Models

In this technical report, we present jLDADMM---an easy-to-use Java toolkit
for conventional topic models. jLDADMM is released to provide alternatives for
topic modeling on normal or short texts. It provides implementations of the
Latent Dirichlet Allocation topic model and the one-topic-per-document
Dirichlet Multinomial Mixture model (i.e. mixture of unigrams), using collapsed
Gibbs sampling. In addition, jLDADMM supplies a document clustering evaluation
to compare topic models. jLDADMM is open-source and available to download at:
https://github.com/datquocnguyen/jLDADMM
Self-Supervised Model Adaptation for Multimodal Semantic Segmentation
Authors: Abhinav Valada, Rohit Mohan, Wolfram Burgard
Date: 2018-08-11 00:00:00
Tasks: Decoder, model, Scene Recognition, Semantic Segmentation

Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the real-world. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering the model incapable of focusing only on relevant complementary information for fusion. To address this limitation, we propose a mutimodal semantic segmentation framework that dynamically adapts the fusion of modality-specific features while being sensitive to the object category, spatial location and scene context in a self-supervised manner. Specifically, we propose an architecture consisting of two modality-specific encoder streams that fuse intermediate encoder representations into a single decoder using our proposed self-supervised model adaptation fusion mechanism which optimally combines complementary features. As intermediate representations are not aligned across modalities, we introduce an attention scheme for better correlation. In addition, we propose a computationally efficient unimodal segmentation architecture termed AdapNet++ that incorporates a new encoder with multiscale residual units and an efficient atrous spatial pyramid pooling that has a larger effective receptive field with more than 10x fewer parameters, complemented with a strong decoder with a multi-resolution supervision scheme that recovers high-resolution details. Comprehensive empirical evaluations on several benchmarks demonstrate that both our unimodal and multimodal architectures achieve state-of-the-art performance.