pwc-archive/papers-with-abstracts huggingface dataset:
A Mean Field View of the Landscape of Two-Layers Neural Networks
Authors: Song Mei, Andrea Montanari, Phan-Minh Nguyen
Date: 2018-04-18 00:00:00

Multi-layer neural networks are among the most powerful models in machine
learning, yet the fundamental reasons for this success defy mathematical
understanding. Learning a neural network requires to optimize a non-convex
high-dimensional objective (risk function), a problem which is usually attacked
using stochastic gradient descent (SGD). Does SGD converge to a global optimum
of the risk or only to a local optimum? In the first case, does this happen
because local minima are absent, or because SGD somehow avoids them? In the
second, why do local minima reached by SGD have good generalization properties?
  In this paper we consider a simple case, namely two-layers neural networks,
and prove that -in a suitable scaling limit- SGD dynamics is captured by a
certain non-linear partial differential equation (PDE) that we call
distributional dynamics (DD). We then consider several specific examples, and
show how DD can be used to prove convergence of SGD to networks with nearly
ideal generalization error. This description allows to 'average-out' some of
the complexities of the landscape of neural networks, and can be used to prove
a general convergence result for noisy SGD.
Deep Generative Networks For Sequence Prediction
Authors: Markus Beissinger
Date: 2018-04-18 00:00:00
Tasks: Prediction, Representation Learning, Time Series, Time Series Analysis

This thesis investigates unsupervised time series representation learning for
sequence prediction problems, i.e. generating nice-looking input samples given
a previous history, for high dimensional input sequences by decoupling the
static input representation from the recurrent sequence representation. We
introduce three models based on Generative Stochastic Networks (GSN) for
unsupervised sequence learning and prediction. Experimental results for these
three models are presented on pixels of sequential handwritten digit (MNIST)
data, videos of low-resolution bouncing balls, and motion capture data. The
main contribution of this thesis is to provide evidence that GSNs are a viable
framework to learn useful representations of complex sequential input data, and
to suggest a new framework for deep generative models to learn complex
sequences by decoupling static input representations from dynamic time
dependency representations.
Semantic Single-Image Dehazing
Authors: Ziang Cheng, ShaoDi You, Viorela Ila, Hongdong Li
Date: 2018-04-16 00:00:00
Tasks: Image Dehazing, Single Image Dehazing, Single Image Haze Removal

Single-image haze-removal is challenging due to limited information contained
in one single image. Previous solutions largely rely on handcrafted priors to
compensate for this deficiency. Recent convolutional neural network (CNN)
models have been used to learn haze-related priors but they ultimately work as
advanced image filters. In this paper we propose a novel semantic ap- proach
towards single image haze removal. Unlike existing methods, we infer color
priors based on extracted semantic features. We argue that semantic context can
be exploited to give informative cues for (a) learning color prior on clean
image and (b) estimating ambient illumination. This design allowed our model to
recover clean images from challenging cases with strong ambiguity, e.g.
saturated illumination color and sky regions in image. In experiments, we
validate our ap- proach upon synthetic and real hazy images, where our method
showed superior performance over state-of-the-art approaches, suggesting
semantic information facilitates the haze removal task.
On the use of bootstrap with variational inference: Theory, interpretation, and a two-sample test example
Authors: Yen-Chi Chen, Y. Samuel Wang, Elena A. Erosheva
Date: 2017-11-29 00:00:00
Tasks: Bayesian Inference, Variational Inference

Variational inference is a general approach for approximating complex density
functions, such as those arising in latent variable models, popular in machine
learning. It has been applied to approximate the maximum likelihood estimator
and to carry out Bayesian inference, however, quantification of uncertainty
with variational inference remains challenging from both theoretical and
practical perspectives. This paper is concerned with developing uncertainty
measures for variational inference by using bootstrap procedures. We first
develop two general bootstrap approaches for assessing the uncertainty of a
variational estimate and the study the underlying bootstrap theory in both
fixed- and increasing-dimension settings. We then use the bootstrap approach
and our theoretical results in the context of mixed membership modeling with
multivariate binary data on functional disability from the National Long Term
Care Survey. We carry out a two-sample approach to test for changes in the
repeated measures of functional disability for the subset of individuals
present in 1989 and 1994 waves.
Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration
Authors: Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, Jose C. Principe
Date: 2018-04-18 00:00:00

The matrix-based Renyi's \alpha-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this paper, we first show that our estimators enable straightforward measurement of information flow in realistic convolutional neural networks (CNN) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal some fundamental properties concerning the training of CNN.
Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks
Authors: Binxuan Huang, Yanglan Ou, Kathleen M. Carley
Date: 2018-04-18 00:00:00
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Classification, General Classification, Sentiment Classification

Aspect Based Sentiment Analysis, PyTorch Implementations. 基于方面的情感分析，使用PyTorch实现。
Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation
Authors: Jonathan Tremblay, Thang To, Stan Birchfield
Date: 2018-04-18 00:00:00
Tasks: 3D Object Detection, 3D Pose Estimation, All, object-detection, Object Detection, Pose Estimation

We present a new dataset, called Falling Things (FAT), for advancing the
state-of-the-art in object detection and 3D pose estimation in the context of
robotics. By synthetically combining object models and backgrounds of complex
composition and high graphical quality, we are able to generate photorealistic
images with accurate 3D pose annotations for all objects in all images. Our
dataset contains 60k annotated photos of 21 household objects taken from the
YCB dataset. For each image, we provide the 3D poses, per-pixel class
segmentation, and 2D/3D bounding box coordinates for all objects. To facilitate
testing different input modalities, we provide mono and stereo RGB images,
along with registered dense depth images. We describe in detail the generation
process and statistical analysis of the data.
Speaker-independent Speech Separation with Deep Attractor Network
Authors: Yi Luo, Zhuo Chen, Nima Mesgarani
Date: 2017-07-12 00:00:00
Tasks: Deep Learning, Speech Separation

Despite the recent success of deep learning for many speech processing tasks,
single-microphone, speaker-independent speech separation remains challenging
for two main reasons. The first reason is the arbitrary order of the target and
masker speakers in the mixture permutation problem, and the second is the
unknown number of speakers in the mixture output dimension problem. We propose
a novel deep learning framework for speech separation that addresses both of
these issues. We use a neural network to project the time-frequency
representation of the mixture signal into a high-dimensional embedding space. A
reference point attractor is created in the embedding space to represent each
speaker which is defined as the centroid of the speaker in the embedding space.
The time-frequency embeddings of each speaker are then forced to cluster around
the corresponding attractor point which is used to determine the time-frequency
assignment of the speaker. We propose three methods for finding the attractors
for each source in the embedding space and compare their advantages and
limitations. The objective function for the network is standard signal
reconstruction error which enables end-to-end operation during both training
and test phases. We evaluated our system using the Wall Street Journal dataset
WSJ0 on two and three speaker mixtures and report comparable or better
performance than other state-of-the-art deep learning methods for speech
separation.
TasNet: time-domain audio separation network for real-time, single-channel speech separation
Authors: Yi Luo, Nima Mesgarani
Date: 2017-11-01 00:00:00
Tasks: Decoder, Speech Separation

Robust speech processing in multi-talker environments requires effective
speech separation. Recent deep learning systems have made significant progress
toward solving this problem, yet it remains challenging particularly in
real-time, short latency applications. Most methods attempt to construct a mask
for each source in time-frequency representation of the mixture signal which is
not necessarily an optimal representation for speech separation. In addition,
time-frequency decomposition results in inherent problems such as
phase/magnitude decoupling and long time window which is required to achieve
sufficient frequency resolution. We propose Time-domain Audio Separation
Network (TasNet) to overcome these limitations. We directly model the signal in
the time-domain using an encoder-decoder framework and perform the source
separation on nonnegative encoder outputs. This method removes the frequency
decomposition step and reduces the separation problem to estimation of source
masks on encoder outputs which is then synthesized by the decoder. Our system
outperforms the current state-of-the-art causal and noncausal speech separation
algorithms, reduces the computational cost of speech separation, and
significantly reduces the minimum required latency of the output. This makes
TasNet suitable for applications where low-power, real-time implementation is
desirable such as in hearable and telecommunication devices.
Deep Transfer Network with Joint Distribution Adaptation: A New Intelligent Fault Diagnosis Framework for Industry Application
Authors: Te Han, Chao Liu, Wenguang Yang, Dongxiang Jiang
Date: 2018-04-18 00:00:00
Tasks: Domain Adaptation, Fault Diagnosis, Transfer Learning

In recent years, an increasing popularity of deep learning model for
intelligent condition monitoring and diagnosis as well as prognostics used for
mechanical systems and structures has been observed. In the previous studies,
however, a major assumption accepted by default, is that the training and
testing data are taking from same feature distribution. Unfortunately, this
assumption is mostly invalid in real application, resulting in a certain lack
of applicability for the traditional diagnosis approaches. Inspired by the idea
of transfer learning that leverages the knowledge learnt from rich labeled data
in source domain to facilitate diagnosing a new but similar target task, a new
intelligent fault diagnosis framework, i.e., deep transfer network (DTN), which
generalizes deep learning model to domain adaptation scenario, is proposed in
this paper. By extending the marginal distribution adaptation (MDA) to joint
distribution adaptation (JDA), the proposed framework can exploit the
discrimination structures associated with the labeled data in source domain to
adapt the conditional distribution of unlabeled target data, and thus guarantee
a more accurate distribution matching. Extensive empirical evaluations on three
fault datasets validate the applicability and practicability of DTN, while
achieving many state-of-the-art transfer results in terms of diverse operating
conditions, fault severities and fault types.
Segmentation of both Diseased and Healthy Skin from Clinical Photographs in a Primary Care Setting
Authors: Noel C. F. Codella, Daren Anderson, Tyler Philips, Anthony Porto, Kevin Massey, Jane Snowdon, Rogerio Feris, John Smith
Date: 2018-04-16 00:00:00
Tasks: Segmentation

This work presents the first segmentation study of both diseased and healthy
skin in standard camera photographs from a clinical environment. Challenges
arise from varied lighting conditions, skin types, backgrounds, and
pathological states. For study, 400 clinical photographs (with skin
segmentation masks) representing various pathological states of skin are
retrospectively collected from a primary care network. 100 images are used for
training and fine-tuning, and 300 are used for evaluation. This distribution
between training and test partitions is chosen to reflect the difficulty in
amassing large quantities of labeled data in this domain. A deep learning
approach is used, and 3 public segmentation datasets of healthy skin are
collected to study the potential benefits of pre-training. Two variants of
U-Net are evaluated: U-Net and Dense Residual U-Net. We find that Dense
Residual U-Nets have a 7.8% improvement in Jaccard, compared to classical U-Net
architectures (0.55 vs. 0.51 Jaccard), for direct transfer, where fine-tuning
data is not utilized. However, U-Net outperforms Dense Residual U-Net for both
direct training (0.83 vs. 0.80) and fine-tuning (0.89 vs. 0.88). The stark
performance improvement with fine-tuning compared to direct transfer and direct
training emphasizes both the need for adequate representative data of diseased
skin, and the utility of other publicly available data sources for this task.
Towards Easier and Faster Sequence Labeling for Natural Language Processing: A Search-based Probabilistic Online Learning Framework (SAPO)
Authors: Xu Sun, Shuming Ma, Yi Zhang, Xuancheng Ren
Date: 2015-03-29 00:00:00

There are two major approaches for sequence labeling. One is the
probabilistic gradient-based methods such as conditional random fields (CRF)
and neural networks (e.g., RNN), which have high accuracy but drawbacks: slow
training, and no support of search-based optimization (which is important in
many cases). The other is the search-based learning methods such as structured
perceptron and margin infused relaxed algorithm (MIRA), which have fast
training but also drawbacks: low accuracy, no probabilistic information, and
non-convergence in real-world tasks. We propose a novel and "easy" solution, a
search-based probabilistic online learning method, to address most of those
issues. The method is "easy", because the optimization algorithm at the
training stage is as simple as the decoding algorithm at the test stage. This
method searches the output candidates, derives probabilities, and conducts
efficient online learning. We show that this method with fast training and
theoretical guarantee of convergence, which is easy to implement, can support
search-based optimization and obtain top accuracy. Experiments on well-known
tasks show that our method has better accuracy than CRF and BiLSTM\footnote{The
SAPO code is released at \url{https://github.com/lancopku/SAPO}.}.
Unsupervised Domain Adaptation with Similarity Learning
Authors: Pedro O. Pinheiro
Date: 2017-11-24 00:00:00
Proceeding: CVPR 2018 6
Tasks: Domain Adaptation, General Classification, Unsupervised Domain Adaptation

The objective of unsupervised domain adaptation is to leverage features from
a labeled source domain and learn a classifier for an unlabeled target domain,
with a similar but different data distribution. Most deep learning approaches
to domain adaptation consist of two steps: (i) learn features that preserve a
low risk on labeled samples (source domain) and (ii) make the features from
both domains to be as indistinguishable as possible, so that a classifier
trained on the source can also be applied on the target domain. In general, the
classifiers in step (i) consist of fully-connected layers applied directly on
the indistinguishable features learned in (ii). In this paper, we propose a
different way to do the classification, using similarity learning. The proposed
method learns a pairwise similarity function in which classification can be
performed by computing similarity between prototype representations of each
category. The domain-invariant features and the categorical prototype
representations are learned jointly and in an end-to-end fashion. At inference
time, images from the target domain are compared to the prototypes and the
label associated with the one that best matches the image is outputed. The
approach is simple, scalable and effective. We show that our model achieves
state-of-the-art performance in different unsupervised domain adaptation
scenarios.
Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change
Authors: Dominik Schlechtweg, Sabine Schulte im Walde, Stefanie Eckmann
Date: 2018-04-18 00:00:00
Proceeding: NAACL 2018 6

We propose a framework that extends synchronic polysemy annotation to
diachronic changes in lexical meaning, to counteract the lack of resources for
evaluating computational models of lexical semantic change. Our framework
exploits an intuitive notion of semantic relatedness, and distinguishes between
innovative and reductive meaning changes with high inter-annotator agreement.
The resulting test set for German comprises ratings from five annotators for
the relatedness of 1,320 use pairs across 22 target words.
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
Authors: Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, Larry Heck
Date: 2018-04-18 00:00:00
Proceeding: NAACL 2018 6
Tasks: Dialogue State Tracking, Imitation Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Task-Oriented Dialogue Systems

In this work, we present a hybrid learning method for training task-oriented
dialogue systems through online user interactions. Popular methods for learning
task-oriented dialogues include applying reinforcement learning with user
feedback on supervised pre-training models. Efficiency of such learning method
may suffer from the mismatch of dialogue state distribution between offline
training and online interactive learning stages. To address this challenge, we
propose a hybrid imitation and reinforcement learning method, with which a
dialogue agent can effectively learn from its interaction with users by
learning from human teaching and feedback. We design a neural network based
task-oriented dialogue agent that can be optimized end-to-end with the proposed
learning method. Experimental results show that our end-to-end dialogue agent
can learn effectively from the mistake it makes via imitation learning from
user teaching. Applying reinforcement learning with user feedback after the
imitation learning stage further improves the agent's capability in
successfully completing a task.
Fast Weight Long Short-Term Memory
Authors: T. Anderson Keller, Sharath Nittur Sridhar, Xin Wang
Date: 2018-04-18 00:00:00
Tasks: Retrieval

Associative memory using fast weights is a short-term memory mechanism that
substantially improves the memory capacity and time scale of recurrent neural
networks (RNNs). As recent studies introduced fast weights only to regular
RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. In
this work, we report a significant synergy between long short-term memory
(LSTM) networks and fast weight associative memories. We show that this
combination, in learning associative retrieval tasks, results in much faster
training and lower test error, a performance boost most prominent at high
memory task difficulties.
Structure from Recurrent Motion: From Rigidity to Recurrency
Authors: Xiu Li, Hongdong Li, Hanbyul Joo, Yebin Liu, Yaser Sheikh
Date: 2018-04-18 00:00:00
Proceeding: CVPR 2018 6
Tasks: Clustering

This paper proposes a new method for Non-Rigid Structure-from-Motion (NRSfM)
from a long monocular video sequence observing a non-rigid object performing
recurrent and possibly repetitive dynamic action. Departing from the
traditional idea of using linear low-order or lowrank shape model for the task
of NRSfM, our method exploits the property of shape recurrency (i.e., many
deforming shapes tend to repeat themselves in time). We show that recurrency is
in fact a generalized rigidity. Based on this, we reduce NRSfM problems to
rigid ones provided that certain recurrency condition is satisfied. Given such
a reduction, standard rigid-SfM techniques are directly applicable (without any
change) to the reconstruction of non-rigid dynamic shapes. To implement this
idea as a practical approach, this paper develops efficient algorithms for
automatic recurrency detection, as well as camera view clustering via a
rigidity-check. Experiments on both simulated sequences and real data
demonstrate the effectiveness of the method. Since this paper offers a novel
perspective on rethinking structure-from-motion, we hope it will inspire other
new problems in the field.
UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition
Authors: Kartik Hegde, Jiyong Yu, Rohit Agrawal, Mengjia Yan, Michael Pellauer, Christopher W. Fletcher
Date: 2018-04-18 00:00:00
Tasks: Scene Generation

Convolutional Neural Networks (CNNs) have begun to permeate all corners of
electronic society (from voice recognition to scene generation) due to their
high accuracy and machine efficiency per operation. At their core, CNN
computations are made up of multi-dimensional dot products between weight and
input vectors. This paper studies how weight repetition ---when the same weight
occurs multiple times in or across weight vectors--- can be exploited to save
energy and improve performance during CNN inference. This generalizes a popular
line of work to improve efficiency from CNN weight sparsity, as reducing
computation due to repeated zero weights is a special case of reducing
computation due to repeated weights.
  To exploit weight repetition, this paper proposes a new CNN accelerator
called the Unique Weight CNN Accelerator (UCNN). UCNN uses weight repetition to
reuse CNN sub-computations (e.g., dot products) and to reduce CNN model size
when stored in off-chip DRAM ---both of which save energy. UCNN further
improves performance by exploiting sparsity in weights. We evaluate UCNN with
an accelerator-level cycle and energy model and with an RTL implementation of
the UCNN processing element. On three contemporary CNNs, UCNN improves
throughput-normalized energy consumption by 1.2x - 4x, relative to a similarly
provisioned baseline accelerator that uses Eyeriss-style sparsity
optimizations. At the same time, the UCNN processing element adds only 17-24%
area overhead relative to the same baseline.