pwc-archive/papers-with-abstracts huggingface dataset:
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
Authors: Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi
Date: 2018-05-15 00:00:00
Tasks: Adversarial Attack, Knowledge Distillation

Many recent works on knowledge distillation have provided ways to transfer
the knowledge of a trained network for improving the learning process of a new
one, but finding a good technique for knowledge distillation is still an open
problem. In this paper, we provide a new perspective based on a decision
boundary, which is one of the most important component of a classifier. The
generalization performance of a classifier is closely related to the adequacy
of its decision boundary, so a good classifier bears a good decision boundary.
Therefore, transferring information closely related to the decision boundary
can be a good attempt for knowledge distillation. To realize this goal, we
utilize an adversarial attack to discover samples supporting a decision
boundary. Based on this idea, to transfer more accurate information about the
decision boundary, the proposed algorithm trains a student classifier based on
the adversarial samples supporting the decision boundary. Experiments show that
the proposed method indeed improves knowledge distillation and achieves the
state-of-the-arts performance.
Understand Functionality and Dimensionality of Vector Embeddings: the Distributional Hypothesis, the Pairwise Inner Product Loss and Its Bias-Variance Trade-off
Authors: Zi Yin
Date: 2018-03-01 00:00:00

Vector embedding is a foundational building block of many deep learning
models, especially in natural language processing. In this paper, we present a
theoretical framework for understanding the effect of dimensionality on vector
embeddings. We observe that the distributional hypothesis, a governing
principle of statistical semantics, requires a natural unitary-invariance for
vector embeddings. Motivated by the unitary-invariance observation, we propose
the Pairwise Inner Product (PIP) loss, a unitary-invariant metric on the
similarity between two embeddings. We demonstrate that the PIP loss captures
the difference in functionality between embeddings, and that the PIP loss is
tightly connect with two basic properties of vector embeddings, namely
similarity and compositionality. By formulating the embedding training process
as matrix factorization with noise, we reveal a fundamental bias-variance
trade-off between the signal spectrum and noise power in the dimensionality
selection process. This bias-variance trade-off sheds light on many empirical
observations which have not been thoroughly explained, for example the
existence of an optimal dimensionality. Moreover, we discover two new results
about vector embeddings, namely their robustness against over-parametrization
and their forward stability. The bias-variance trade-off of the PIP loss
explicitly answers the fundamental open problem of dimensionality selection for
vector embeddings.
Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
Authors: Liang Luo, Jacob Nelson, Luis Ceze, Amar Phanishayee, Arvind Krishnamurthy
Date: 2018-05-21 00:00:00

Distributed deep neural network (DDNN) training constitutes an increasingly important workload that frequently runs in the cloud. Larger DNN models and faster compute engines are shifting DDNN training bottlenecks from computation to communication. This paper characterizes DDNN training to precisely pinpoint these bottlenecks. We found that timely training requires high performance parameter servers (PSs) with optimized network stacks and gradient processing pipelines, as well as server and network hardware with balanced computation and communication resources. We therefore propose PHub, a high performance multi-tenant, rack-scale PS design. PHub co-designs the PS software and hardware to accelerate rack-level and hierarchical cross-rack parameter exchange, with an API compatible with many DDNN training frameworks. PHub provides a performance improvement of up to 2.7x compared to state-of-the-art distributed training techniques for cloud-based ImageNet workloads, with 25% better throughput per dollar.
Improving Aspect Term Extraction with Bidirectional Dependency Tree Representation
Authors: Huaishao Luo, Tianrui Li, Bing Liu, Bin Wang, Herwig Unger
Date: 2018-05-21 00:00:00
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Sentiment Analysis, Term Extraction

Aspect term extraction is one of the important subtasks in aspect-based sentiment analysis. Previous studies have shown that using dependency tree structure representation is promising for this task. However, most dependency tree structures involve only one directional propagation on the dependency tree. In this paper, we first propose a novel bidirectional dependency tree network to extract dependency structure features from the given sentences. The key idea is to explicitly incorporate both representations gained separately from the bottom-up and top-down propagation on the given dependency syntactic tree. An end-to-end framework is then developed to integrate the embedded representations and BiLSTM plus CRF to learn both tree-structured and sequential features to solve the aspect term extraction problem. Experimental results demonstrate that the proposed model outperforms state-of-the-art baseline models on four benchmark SemEval datasets.
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Authors: Weixuan Chen, Daniel McDuff
Date: 2018-05-21 00:00:00
Proceeding: ECCV 2018 9
Tasks: Motion Estimation

Non-contact video-based physiological measurement has many applications in
health care and human-computer interaction. Practical applications require
measurements to be accurate even in the presence of large head rotations. We
propose the first end-to-end system for video-based measurement of heart and
breathing rate using a deep convolutional network. The system features a new
motion representation based on a skin reflection model and a new attention
mechanism using appearance information to guide motion estimation, both of
which enable robust measurement under heterogeneous lighting and major motions.
Our approach significantly outperforms all current state-of-the-art methods on
both RGB and infrared video datasets. Furthermore, it allows spatial-temporal
distributions of physiological signals to be visualized via the attention
mechanism.
Face recognition for monitoring operator shift in railways
Authors: S. Ritika, Dattaraj Rao
Date: 2018-02-05 00:00:00
Tasks: Face Recognition

Train Pilot is a very tedious and stressful job. Pilots must be vigilant at
all times and its easy for them to lose track of time of shift. In countries
like USA the pilots are mandated by law to adhere to 8 hour shifts. If they
exceed 8 hours of shift the railroads may be penalized for over-tiring their
drivers. The problem happens when the 8 hour shift may end in middle of a
journey. In such case, the new drivers must be moved to the location locomotive
is operating for shift change. Hence accurate monitoring of drivers during
their shift and making sure the shifts are scheduled correctly is very
important for railroads. Here we propose an automated camera system that uses
camera mounted inside Locomotive cabs to continuously record video feeds. These
feeds are analyzed in real time to detect the face of driver and recognize the
driver using state of the art deep Learning techniques. The outcome is an
increased safety of train pilots. Cameras continuously capture video from
inside the cab which is stored on an on board data acquisition device. Using
advanced computer vision and deep learning techniques the videos are analyzed
at regular intervals to detect presence of the pilot and identify the pilot.
Using a time based analysis, it is identified for how long that shift has been
active. If this time exceeds allocated shift time an alert is sent to the
dispatch to adjust shift hours.
How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?
Authors: Simon S. Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Ruslan Salakhutdinov, Aarti Singh
Date: 2018-05-21 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: LEMMA

It is widely believed that the practical success of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) owes to the fact that CNNs and RNNs use a more compact parametric representation than their Fully-Connected Neural Network (FNN) counterparts, and consequently require fewer training examples to accurately estimate their parameters. We initiate the study of rigorously characterizing the sample-complexity of estimating CNNs and RNNs. We show that the sample-complexity to learn CNNs and RNNs scales linearly with their intrinsic dimension and this sample-complexity is much smaller than for their FNN counterparts. For both CNNs and RNNs, we also present lower bounds showing our sample complexities are tight up to logarithmic factors. Our main technical tools for deriving these results are a localized empirical process analysis and a new technical lemma characterizing the convolutional and recurrent structure. We believe that these tools may inspire further developments in understanding CNNs and RNNs.
Sentence Modeling via Multiple Word Embeddings and Multi-level Comparison for Semantic Textual Similarity
Authors: Huy Nguyen Tien, Minh Nguyen Le, Yamasaki Tomohiro, Izuha Tatsuya
Date: 2018-05-21 00:00:00
Tasks: Natural Language Inference, Relation, Semantic Textual Similarity, Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings, Sentence Similarity, STS, STS Benchmark, Word Embeddings

Different word embedding models capture different aspects of linguistic
properties. This inspired us to propose a model (M-MaxLSTM-CNN) for employing
multiple sets of word embeddings for evaluating sentence similarity/relation.
Representing each word by multiple word embeddings, the MaxLSTM-CNN encoder
generates a novel sentence embedding. We then learn the similarity/relation
between our sentence embeddings via Multi-level comparison. Our method
M-MaxLSTM-CNN consistently shows strong performances in several tasks (i.e.,
measure textual similarity, identify paraphrase, recognize textual entailment).
According to the experimental results on STS Benchmark dataset and SICK dataset
from SemEval, M-MaxLSTM-CNN outperforms the state-of-the-art methods for
textual similarity tasks. Our model does not use hand-crafted features (e.g.,
alignment features, Ngram overlaps, dependency features) as well as does not
require pre-trained word embeddings to have the same dimension.
Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing
Authors: K. M. Annervaz, Somnath Basu Roy Chowdhury, Ambedkar Dukkipati
Date: 2018-02-16 00:00:00
Proceeding: NAACL 2018 6
Tasks: Knowledge Graphs, Natural Language Inference, text-classification, Text Classification, World Knowledge

Machine Learning has been the quintessential solution for many AI problems,
but learning is still heavily dependent on the specific training data. Some
learning models can be incorporated with a prior knowledge in the Bayesian set
up, but these learning models do not have the ability to access any organised
world knowledge on demand. In this work, we propose to enhance learning models
with world knowledge in the form of Knowledge Graph (KG) fact triples for
Natural Language Processing (NLP) tasks. Our aim is to develop a deep learning
model that can extract relevant prior support facts from knowledge graphs
depending on the task using attention mechanism. We introduce a
convolution-based model for learning representations of knowledge graph entity
and relation clusters in order to reduce the attention space. We show that the
proposed method is highly scalable to the amount of prior information that has
to be processed and can be applied to any generic NLP task. Using this method
we show significant improvement in performance for text classification with
News20, DBPedia datasets and natural language inference with Stanford Natural
Language Inference (SNLI) dataset. We also demonstrate that a deep learning
model can be trained well with substantially less amount of labeled training
data, when it has access to organised world knowledge in the form of knowledge
graph.
Learning with Non-Convex Truncated Losses by SGD
Authors: Yi Xu, Shenghuo Zhu, Sen yang, Chi Zhang, Rong Jin, Tianbao Yang
Date: 2018-05-21 00:00:00

Learning with a {\it convex loss} function has been a dominating paradigm for
many years. It remains an interesting question how non-convex loss functions
help improve the generalization of learning with broad applicability. In this
paper, we study a family of objective functions formed by truncating
traditional loss functions, which is applicable to both shallow learning and
deep learning. Truncating loss functions has potential to be less vulnerable
and more robust to large noise in observations that could be adversarial. More
importantly, it is a generic technique without assuming the knowledge of noise
distribution. To justify non-convex learning with truncated losses, we
establish excess risk bounds of empirical risk minimization based on truncated
losses for heavy-tailed output, and statistical error of an approximate
stationary point found by stochastic gradient descent (SGD) method. Our
experiments for shallow and deep learning for regression with outliers,
corrupted data and heavy-tailed noise further justify the proposed method.
A Deep Reinforcement Learning Framework for Rebalancing Dockless Bike Sharing Systems
Authors: Ling Pan, Qingpeng Cai, Zhixuan Fang, Pingzhong Tang, Longbo Huang
Date: 2018-02-13 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Bike sharing provides an environment-friendly way for traveling and is
booming all over the world. Yet, due to the high similarity of user travel
patterns, the bike imbalance problem constantly occurs, especially for dockless
bike sharing systems, causing significant impact on service quality and company
revenue. Thus, it has become a critical task for bike sharing systems to
resolve such imbalance efficiently. In this paper, we propose a novel deep
reinforcement learning framework for incentivizing users to rebalance such
systems. We model the problem as a Markov decision process and take both
spatial and temporal features into consideration. We develop a novel deep
reinforcement learning algorithm called Hierarchical Reinforcement Pricing
(HRP), which builds upon the Deep Deterministic Policy Gradient algorithm.
Different from existing methods that often ignore spatial information and rely
heavily on accurate prediction, HRP captures both spatial and temporal
dependencies using a divide-and-conquer structure with an embedded localized
module. We conduct extensive experiments to evaluate HRP, based on a dataset
from Mobike, a major Chinese dockless bike sharing company. Results show that
HRP performs close to the 24-timeslot look-ahead optimization, and outperforms
state-of-the-art methods in both service level and bike distribution. It also
transfers well when applied to unseen areas.
GSAE: an autoencoder with embedded gene-set nodes for genomics functional characterization
Authors: Hung-I Harry Chen, Yu-Chiao Chiu, Tinghe Zhang, Songyao Zhang, Yufei Huang, Yidong Chen
Date: 2018-05-21 00:00:00
Tasks: Survival Analysis

Bioinformatics tools have been developed to interpret gene expression data at
the gene set level, and these gene set based analyses improve the biologists'
capability to discover functional relevance of their experiment design. While
elucidating gene set individually, inter gene sets association is rarely taken
into consideration. Deep learning, an emerging machine learning technique in
computational biology, can be used to generate an unbiased combination of gene
set, and to determine the biological relevance and analysis consistency of
these combining gene sets by leveraging large genomic data sets. In this study,
we proposed a gene superset autoencoder (GSAE), a multi-layer autoencoder model
with the incorporation of a priori defined gene sets that retain the crucial
biological features in the latent layer. We introduced the concept of the gene
superset, an unbiased combination of gene sets with weights trained by the
autoencoder, where each node in the latent layer is a superset. Trained with
genomic data from TCGA and evaluated with their accompanying clinical
parameters, we showed gene supersets' ability of discriminating tumor subtypes
and their prognostic capability. We further demonstrated the biological
relevance of the top component gene sets in the significant supersets. Using
autoencoder model and gene superset at its latent layer, we demonstrated that
gene supersets retain sufficient biological information with respect to tumor
subtypes and clinical prognostic significance. Superset also provides high
reproducibility on survival analysis and accurate prediction for cancer
subtypes.
A Framework and Method for Online Inverse Reinforcement Learning
Authors: Saurabh Arora, Prashant Doshi, Bikramjit Banerjee
Date: 2018-05-21 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Inverse reinforcement learning (IRL) is the problem of learning the
preferences of an agent from the observations of its behavior on a task. While
this problem has been well investigated, the related problem of {\em online}
IRL---where the observations are incrementally accrued, yet the demands of the
application often prohibit a full rerun of an IRL method---has received
relatively less attention. We introduce the first formal framework for online
IRL, called incremental IRL (I2RL), and a new method that advances maximum
entropy IRL with hidden variables, to this setting. Our formal analysis shows
that the new method has a monotonically improving performance with more
demonstration data, as well as probabilistically bounded error, both under full
and partial observability. Experiments in a simulated robotic application of
penetrating a continuous patrol under occlusion shows the relatively improved
performance and speed up of the new method and validates the utility of online
IRL.
Learning Device Models with Recurrent Neural Networks
Authors: John Clemens
Date: 2018-05-21 00:00:00

Recurrent neural networks (RNNs) are powerful constructs capable of modeling
complex systems, up to and including Turing Machines. However, learning such
complex models from finite training sets can be difficult. In this paper we
empirically show that RNNs can learn models of computer peripheral devices
through input and output state observation. This enables automated development
of functional software-only models of hardware devices. Such models are
applicable to any number of tasks, including device validation, driver
development, code de-obfuscation, and reverse engineering. We show that the
same RNN structure successfully models six different devices from simple test
circuits up to a 16550 UART serial port, and verify that these models are
capable of producing equivalent output to real hardware.
Knowledge Discovery from Layered Neural Networks based on Non-negative Task Decomposition
Authors: Chihiro Watanabe, Kaoru Hiramatsu, Kunio Kashino
Date: 2018-05-18 00:00:00

Interpretability has become an important issue in the machine learning field,
along with the success of layered neural networks in various practical tasks.
Since a trained layered neural network consists of a complex nonlinear
relationship between large number of parameters, we failed to understand how
they could achieve input-output mappings with a given data set. In this paper,
we propose the non-negative task decomposition method, which applies
non-negative matrix factorization to a trained layered neural network. This
enables us to decompose the inference mechanism of a trained layered neural
network into multiple principal tasks of input-output mapping, and reveal the
roles of hidden units in terms of their contribution to each principal task.
Adversarial Attacks Against Medical Deep Learning Systems
Authors: Samuel G. Finlayson, Hyung Won Chung, Isaac S. Kohane, Andrew L. Beam
Date: 2018-04-15 00:00:00
Tasks: Deep Learning

The discovery of adversarial examples has raised concerns about the practical
deployment of deep learning systems. In this paper, we demonstrate that
adversarial examples are capable of manipulating deep learning systems across
three clinical domains. For each of our representative medical deep learning
classifiers, both white and black box attacks were highly successful. Our
models are representative of the current state of the art in medical computer
vision and, in some cases, directly reflect architectures already seeing
deployment in real world clinical settings. In addition to the technical
contribution of our paper, we synthesize a large body of knowledge about the
healthcare system to argue that medicine may be uniquely susceptible to
adversarial attacks, both in terms of monetary incentives and technical
vulnerability. To this end, we outline the healthcare economy and the
incentives it creates for fraud and provide concrete examples of how and why
such attacks could be realistically carried out. We urge practitioners to be
aware of current vulnerabilities when deploying deep learning systems in
clinical settings, and encourage the machine learning community to further
investigate the domain-specific characteristics of medical learning systems.
Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks
Authors: Yingyezhe Jin, Wenrui Zhang, Peng Li
Date: 2018-05-21 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Image Classification, speech-recognition, Speech Recognition

Spiking neural networks (SNNs) are positioned to enable spatio-temporal
information processing and ultra-low power event-driven neuromorphic hardware.
However, SNNs are yet to reach the same performances of conventional deep
artificial neural networks (ANNs), a long-standing challenge due to complex
dynamics and non-differentiable spike events encountered in training. The
existing SNN error backpropagation (BP) methods are limited in terms of
scalability, lack of proper handling of spiking discontinuities, and/or
mismatch between the rate-coded loss function and computed gradient. We present
a hybrid macro/micro level backpropagation (HM2-BP) algorithm for training
multi-layer SNNs. The temporal effects are precisely captured by the proposed
spike-train level post-synaptic potential (S-PSP) at the microscopic level. The
rate-coded errors are defined at the macroscopic level, computed and
back-propagated across both macroscopic and microscopic levels. Different from
existing BP methods, HM2-BP directly computes the gradient of the rate-coded
loss function w.r.t tunable parameters. We evaluate the proposed HM2-BP
algorithm by training deep fully connected and convolutional SNNs based on the
static MNIST [14] and dynamic neuromorphic N-MNIST [26]. HM2-BP achieves an
accuracy level of 99.49% and 98.88% for MNIST and N-MNIST, respectively,
outperforming the best reported performances obtained from the existing SNN BP
algorithms. Furthermore, the HM2-BP produces the highest accuracies based on
SNNs for the EMNIST [3] dataset, and leads to high recognition accuracy for the
16-speaker spoken English letters of TI46 Corpus [16], a challenging
patio-temporal speech recognition benchmark for which no prior success based on
SNNs was reported. It also achieves competitive performances surpassing those
of conventional deep learning models when dealing with asynchronous spiking
streams.