pwc-archive/papers-with-abstracts huggingface dataset:
Learning from Multi-View Multi-Way Data via Structural Factorization Machines
Authors: Chun-Ta Lu, Lifang He, Hao Ding, Bokai Cao, Philip S. Yu
Date: 2017-04-10 00:00:00

Real-world relations among entities can often be observed and determined by
different perspectives/views. For example, the decision made by a user on
whether to adopt an item relies on multiple aspects such as the contextual
information of the decision, the item's attributes, the user's profile and the
reviews given by other users. Different views may exhibit multi-way
interactions among entities and provide complementary information. In this
paper, we introduce a multi-tensor-based approach that can preserve the
underlying structure of multi-view data in a generic predictive model.
Specifically, we propose structural factorization machines (SFMs) that learn
the common latent spaces shared by multi-view tensors and automatically adjust
the importance of each view in the predictive model. Furthermore, the
complexity of SFMs is linear in the number of parameters, which make SFMs
suitable to large-scale problems. Extensive experiments on real-world datasets
demonstrate that the proposed SFMs outperform several state-of-the-art methods
in terms of prediction accuracy and computational cost.
The Role of Information Complexity and Randomization in Representation Learning
Authors: Matías Vera, Pablo Piantanida, Leonardo Rey Vega
Date: 2018-02-14 00:00:00
Tasks: Representation Learning

A grand challenge in representation learning is to learn the different
explanatory factors of variation behind the high dimen- sional data. Encoder
models are often determined to optimize performance on training data when the
real objective is to generalize well to unseen data. Although there is enough
numerical evidence suggesting that noise injection (during training) at the
representation level might improve the generalization ability of encoders, an
information-theoretic understanding of this principle remains elusive. This
paper presents a sample-dependent bound on the generalization gap of the
cross-entropy loss that scales with the information complexity (IC) of the
representations, meaning the mutual information between inputs and their
representations. The IC is empirically investigated for standard multi-layer
neural networks with SGD on MNIST and CIFAR-10 datasets; the behaviour of the
gap and the IC appear to be in direct correlation, suggesting that SGD selects
encoders to implicitly minimize the IC. We specialize the IC to study the role
of Dropout on the generalization capacity of deep encoders which is shown to be
directly related to the encoder capacity, being a measure of the
distinguishability among samples from their representations. Our results
support some recent regularization methods.
Spatial Coherence of Oriented White Matter Microstructure: Applications to White Matter Regions Associated with Genetic Similarity
Authors: Haraldur T. Hallgrímsson, Matthew Cieslak, Luca Foschini, Scott T. Grafton, Ambuj K. Singh
Date: 2018-02-14 00:00:00
Tasks: Diffusion  MRI

We present a method to discover differences between populations with respect
to the spatial coherence of their oriented white matter microstructure in
arbitrarily shaped white matter regions. This method is applied to diffusion
MRI scans of a subset of the Human Connectome Project dataset: 57 pairs of
monozygotic and 52 pairs of dizygotic twins. After controlling for
morphological similarity between twins, we identify 3.7% of all white matter as
being associated with genetic similarity (35.1k voxels, $p < 10^{-4}$, false
discovery rate 1.5%), 75% of which spatially clusters into twenty-two
contiguous white matter regions. Furthermore, we show that the orientation
similarity within these regions generalizes to a subset of 47 pairs of non-twin
siblings, and show that these siblings are on average as similar as dizygotic
twins. The regions are located in deep white matter including the superior
longitudinal fasciculus, the optic radiations, the middle cerebellar peduncle,
the corticospinal tract, and within the anterior temporal lobe, as well as the
cerebellum, brain stem, and amygdalae.
  These results extend previous work using undirected fractional anisotrophy
for measuring putative heritable influences in white matter. Our
multidirectional extension better accounts for crossing fiber connections
within voxels. This bottom up approach has at its basis a novel measurement of
coherence within neighboring voxel dyads between subjects, and avoids some of
the fundamental ambiguities encountered with tractographic approaches to white
matter analysis that estimate global connectivity.
From Gameplay to Symbolic Reasoning: Learning SAT Solver Heuristics in the Style of Alpha(Go) Zero
Authors: Fei Wang, Tiark Rompf
Date: 2018-02-14 00:00:00
Tasks: Decision Making, Deep Reinforcement Learning, Dimensionality Reduction, Machine Translation, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), speech-recognition, Speech Recognition, Translation

Despite the recent successes of deep neural networks in various fields such
as image and speech recognition, natural language processing, and reinforcement
learning, we still face big challenges in bringing the power of numeric
optimization to symbolic reasoning. Researchers have proposed different avenues
such as neural machine translation for proof synthesis, vectorization of
symbols and expressions for representing symbolic patterns, and coupling of
neural back-ends for dimensionality reduction with symbolic front-ends for
decision making. However, these initial explorations are still only point
solutions, and bear other shortcomings such as lack of correctness guarantees.
In this paper, we present our approach of casting symbolic reasoning as games,
and directly harnessing the power of deep reinforcement learning in the style
of Alpha(Go) Zero on symbolic problems. Using the Boolean Satisfiability (SAT)
problem as showcase, we demonstrate the feasibility of our method, and the
advantages of modularity, efficiency, and correctness guarantees.
Identifying Mild Traumatic Brain Injury Patients From MR Images Using Bag of Visual Words
Authors: Shervin Minaee, Siyun Wang, Yao Wang, Sohae Chung, Xiuyuan Wang, Els Fieremans, Steven Flanagan, Joseph Rath, Yvonne W. Lui
Date: 2017-10-18 00:00:00
Tasks: feature selection

Mild traumatic brain injury (mTBI) is a growing public health problem with an
estimated incidence of one million people annually in US. Neurocognitive tests
are used to both assess the patient condition and to monitor the patient
progress. This work aims to directly use MR images taken shortly after injury
to detect whether a patient suffers from mTBI, by incorporating machine
learning and computer vision techniques to learn features suitable
discriminating between mTBI and normal patients. We focus on 3 regions in
brain, and extract multiple patches from them, and use bag-of-visual-word
technique to represent each subject as a histogram of representative patterns
derived from patches from all training subjects. After extracting the features,
we use greedy forward feature selection, to choose a subset of features which
achieves highest accuracy. We show through experimental studies that BoW
features perform better than the simple mean value features which were used
previously.
Advancing System Performance with Redundancy: From Biological to Artificial Designs
Authors: Anh Tuan Nguyen, Jian Xu, Diu Khue Luu, Qi Zhao, Zhi Yang
Date: 2018-02-14 00:00:00

Redundancy is a fundamental characteristic of many biological processes such
as those in the genetic, visual, muscular and nervous system; yet its function
has not been fully understood. The conventional interpretation of redundancy is
that it serves as a fault-tolerance mechanism, which leads to redundancy's de
facto application in man-made systems for reliability enhancement. On the
contrary, our previous works have demonstrated an example where redundancy can
be engineered solely for enhancing other aspects of the system, namely accuracy
and precision. This design was inspired by the binocular structure of the human
vision which we believe may share a similar operation. In this paper, we
present a unified theory describing how such utilization of redundancy is
feasible through two complementary mechanisms: representational redundancy
(RPR) and entangled redundancy (ETR). Besides the previous works, we point out
two additional examples where our new understanding of redundancy can be
applied to justify a system's superior performance. One is the human
musculoskeletal system (HMS) - a biological instance, and one is the deep
residual neural network (ResNet) - an artificial counterpart. We envision that
our theory would provide a framework for the future development of bio-inspired
redundant artificial systems as well as assist the studies of the fundamental
mechanisms governing various biological processes.
Classifying movie genres by analyzing text reviews
Authors: Adam Nyberg
Date: 2018-02-14 00:00:00
Tasks: General Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

This paper proposes a method for classifying movie genres by only looking at
text reviews. The data used are from Large Movie Review Dataset v1.0 and IMDb.
This paper compared a K-nearest neighbors (KNN) model and a multilayer
perceptron (MLP) that uses tf-idf as input features. The paper also discusses
different evaluation metrics used when doing multi-label classification. For
the data used in this research, the KNN model performed the best with an
accuracy of 55.4\% and a Hamming loss of 0.047.
500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)
Authors: Suvodeep Majumder, Nikhila Balaji, Katie Brey, Wei Fu, Tim Menzies
Date: 2018-02-14 00:00:00
Tasks: Clustering, CPU, Deep Learning

Deep learning methods are useful for high-dimensional data and are becoming
widely used in many areas of software engineering. Deep learners utilizes
extensive computational power and can take a long time to train-- making it
difficult to widely validate and repeat and improve their results. Further,
they are not the best solution in all domains. For example, recent results show
that for finding related Stack Overflow posts, a tuned SVM performs similarly
to a deep learner, but is significantly faster to train. This paper extends
that recent result by clustering the dataset, then tuning very learners within
each cluster. This approach is over 500 times faster than deep learning (and
over 900 times faster if we use all the cores on a standard laptop computer).
Significantly, this faster approach generates classifiers nearly as good
(within 2\% F1 Score) as the much slower deep learning method. Hence we
recommend this faster methods since it is much easier to reproduce and utilizes
far fewer CPU resources. More generally, we recommend that before researchers
release research results, that they compare their supposedly sophisticated
methods against simpler alternatives (e.g applying simpler learners to build
local models).
YellowFin and the Art of Momentum Tuning
Authors: Jian Zhang, Ioannis Mitliagkas
Date: 2017-06-12 00:00:00
Proceeding: ICLR 2018 1
Tasks: Constituency Parsing, Language Modeling, Language Modelling

Hyperparameter tuning is one of the most time-consuming workloads in deep
learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam,
reduce this labor by adaptively tuning an individual learning rate for each
variable. Recently researchers have shown renewed interest in simpler methods
like momentum SGD as they may yield better test metrics. Motivated by this
trend, we ask: can simple adaptive methods based on SGD perform as well or
better? We revisit the momentum SGD algorithm and show that hand-tuning a
single learning rate and momentum makes it competitive with Adam. We then
analyze its robustness to learning rate misspecification and objective
curvature variation. Based on these insights, we design YellowFin, an automatic
tuner for momentum and learning rate in SGD. YellowFin optionally uses a
negative-feedback loop to compensate for the momentum dynamics in asynchronous
settings on the fly. We empirically show that YellowFin can converge in fewer
iterations than Adam on ResNets and LSTMs for image recognition, language
modeling and constituency parsing, with a speedup of up to 3.28x in synchronous
and up to 2.69x in asynchronous settings.
Reinforcement Learning from Imperfect Demonstrations
Authors: Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell
Date: 2018-02-14 00:00:00
Proceeding: ICLR 2018 1
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from the environment. These tasks have divergent losses which are difficult to jointly optimize and such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstrations and refines the policy in the environment, surpassing the demonstrator's performance. Crucially, both learning from demonstration and interactive refinement use the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
Authors: Dan Hendrycks, Mantas Mazeika, Duncan Wilson, Kevin Gimpel
Date: 2018-02-14 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Data Poisoning

The growing importance of massive datasets used for deep learning makes
robustness to label noise a critical property for classifiers to have. Sources
of label noise include automatic labeling, non-expert labeling, and label
corruption by data poisoning adversaries. Numerous previous works assume that
no source of labels can be trusted. We relax this assumption and assume that a
small subset of the training data is trusted. This enables substantial label
corruption robustness performance gains. In addition, particularly severe label
noise can be combated by using a set of trusted data with clean labels. We
utilize trusted data by proposing a loss correction technique that utilizes
trusted examples in a data-efficient manner to mitigate the effects of label
noise on deep neural network classifiers. Across vision and natural language
processing tasks, we experiment with various label noises at several strengths,
and show that our method significantly outperforms existing methods.
Authorship Attribution Using the Chaos Game Representation
Authors: Daniel Lichtblau, Catalin Stoean
Date: 2018-02-14 00:00:00
Tasks: Authorship Attribution, BIG-bench Machine Learning, Text Categorization

The Chaos Game Representation, a method for creating images from nucleotide
sequences, is modified to make images from chunks of text documents. Machine
learning methods are then applied to train classifiers based on authorship.
Experiments are conducted on several benchmark data sets in English, including
the widely used Federalist Papers, and one in Portuguese. Validation results
for the trained classifiers are competitive with the best methods in prior
literature. The methodology is also successfully applied for text
categorization with encouraging results. One classifier method is moreover seen
to hold promise for the task of digital fingerprinting.
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
Authors: Samuel L. Smith, Quoc V. Le
Date: 2017-10-17 00:00:00

We consider two questions at the heart of machine learning; how can we
predict if a minimum will generalize to the test set, and why does stochastic
gradient descent find minima that generalize well? Our work responds to Zhang
et al. (2016), who showed deep neural networks can easily memorize randomly
labeled training data, despite generalizing well on real labels of the same
inputs. We show that the same phenomenon occurs in small linear models. These
observations are explained by the Bayesian evidence, which penalizes sharp
minima but is invariant to model parameterization. We also demonstrate that,
when one holds the learning rate fixed, there is an optimum batch size which
maximizes the test set accuracy. We propose that the noise introduced by small
mini-batches drives the parameters towards minima whose evidence is large.
Interpreting stochastic gradient descent as a stochastic differential equation,
we identify the "noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon
N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$
the batch size. Consequently the optimum batch size is proportional to both the
learning rate and the size of the training set, $B_{opt} \propto \epsilon N$.
We verify these predictions empirically.
Hello Edge: Keyword Spotting on Microcontrollers
Authors: Yundong Zhang, Naveen Suda, Liangzhen Lai, Vikas Chandra
Date: 2017-11-20 00:00:00
Tasks: Keyword Spotting

Keyword spotting (KWS) is a critical component for enabling speech based user
interactions on smart devices. It requires real-time response and high accuracy
for good user experience. Recently, neural networks have become an attractive
choice for KWS architecture because of their superior accuracy compared to
traditional speech processing algorithms. Due to its always-on nature, KWS
application has highly constrained power budget and typically runs on tiny
microcontrollers with limited memory and compute capability. The design of
neural network architecture for KWS must consider these constraints. In this
work, we perform neural network architecture evaluation and exploration for
running KWS on resource-constrained microcontrollers. We train various neural
network architectures for keyword spotting published in literature to compare
their accuracy and memory/compute requirements. We show that it is possible to
optimize these neural network architectures to fit within the memory and
compute constraints of microcontrollers without sacrificing accuracy. We
further explore the depthwise separable convolutional neural network (DS-CNN)
and compare it against other neural network architectures. DS-CNN achieves an
accuracy of 95.4%, which is ~10% higher than the DNN model with similar number
of parameters.
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
Authors: Chelsea Finn, Sergey Levine
Date: 2017-10-31 00:00:00
Proceeding: ICLR 2018 1
Tasks: Meta-Learning

Learning to learn is a powerful paradigm for enabling models to learn from
data more effectively and efficiently. A popular approach to meta-learning is
to train a recurrent model to read in a training dataset as input and output
the parameters of a learned model, or output predictions for new test inputs.
Alternatively, a more recent approach to meta-learning aims to acquire deep
representations that can be effectively fine-tuned, via standard gradient
descent, to new tasks. In this paper, we consider the meta-learning problem
from the perspective of universality, formalizing the notion of learning
algorithm approximation and comparing the expressive power of the
aforementioned recurrent models to the more recent approaches that embed
gradient descent into the meta-learner. In particular, we seek to answer the
following question: does deep representation combined with standard gradient
descent have sufficient capacity to approximate any learning algorithm? We find
that this is indeed true, and further find, in our experiments, that
gradient-based meta-learning consistently leads to learning strategies that
generalize more widely compared to those represented by recurrent models.
Deep Learning Models Delineates Multiple Nuclear Phenotypes in H&E Stained Histology Sections
Authors: Mina Khoshdeli, Bahram Parvin
Date: 2018-02-13 00:00:00
Tasks: Nuclear Segmentation, Segmentation

Nuclear segmentation is an important step for profiling aberrant regions of
histology sections. However, segmentation is a complex problem as a result of
variations in nuclear geometry (e.g., size, shape), nuclear type (e.g.,
epithelial, fibroblast), and nuclear phenotypes (e.g., vesicular, aneuploidy).
The problem is further complicated as a result of variations in sample
preparation. It is shown and validated that fusion of very deep convolutional
networks overcomes (i) complexities associated with multiple nuclear
phenotypes, and (ii) separation of overlapping nuclei. The fusion relies on
integrating of networks that learn region- and boundary-based representations.
The system has been validated on a diverse set of nuclear phenotypes that
correspond to the breast and brain histology sections.
Towards Reverse-Engineering Black-Box Neural Networks
Authors: Seong Joon Oh, Max Augustin, Bernt Schiele, Mario Fritz
Date: 2017-11-06 00:00:00
Proceeding: ICLR 2018 1

Many deployed learned models are black boxes: given input, returns output.
Internal information about the model, such as the architecture, optimisation
procedure, or training data, is not disclosed explicitly as it might contain
proprietary information or make the system more vulnerable. This work shows
that such attributes of neural networks can be exposed from a sequence of
queries. This has multiple implications. On the one hand, our work exposes the
vulnerability of black-box neural networks to different types of attacks -- we
show that the revealed internal information helps generate more effective
adversarial examples against the black box model. On the other hand, this
technique can be used for better protection of private content from automatic
recognition models using adversarial examples. Our paper suggests that it is
actually hard to draw a line between white box and black box models.
Brute-Force Facial Landmark Analysis With A 140,000-Way Classifier
Authors: Mengtian Li, Laszlo Jeni, Deva Ramanan
Date: 2018-02-06 00:00:00
Tasks: General Classification, regression

We propose a simple approach to visual alignment, focusing on the
illustrative task of facial landmark estimation. While most prior work treats
this as a regression problem, we instead formulate it as a discrete $K$-way
classification task, where a classifier is trained to return one of $K$
discrete alignments. One crucial benefit of a classifier is the ability to
report back a (softmax) distribution over putative alignments. We demonstrate
that this distribution is a rich representation that can be marginalized (to
generate uncertainty estimates over groups of landmarks) and conditioned on (to
incorporate top-down context, provided by temporal constraints in a video
stream or an interactive human user). Such capabilities are difficult to
integrate into classic regression-based approaches. We study performance as a
function of the number of classes $K$, including the extreme "exemplar class"
setting where $K$ is equal to the number of training examples (140K in our
setting). Perhaps surprisingly, we show that classifiers can still be learned
in this setting. When compared to prior work in classification, our $K$ is
unprecedentedly large, including many "fine-grained" classes that are very
similar. We address these issues by using a multi-label loss function that
allows for training examples to be non-uniformly shared across discrete
classes. We perform a comprehensive experimental analysis of our method on
standard benchmarks, demonstrating state-of-the-art results for facial
alignment in videos.