pwc-archive/papers-with-abstracts huggingface dataset:
Meta-Learning Update Rules for Unsupervised Representation Learning
Authors: Luke Metz, Niru Maheswaranathan, Brian Cheung, Jascha Sohl-Dickstein
Date: 2018-03-31 00:00:00
Proceeding: ICLR 2019 5
Tasks: Meta-Learning, Representation Learning

A major goal of unsupervised learning is to discover data representations
that are useful for subsequent tasks, without access to supervised labels
during training. Typically, this involves minimizing a surrogate objective,
such as the negative log likelihood of a generative model, with the hope that
representations useful for subsequent tasks will arise as a side effect. In
this work, we propose instead to directly target later desired tasks by
meta-learning an unsupervised learning rule which leads to representations
useful for those tasks. Specifically, we target semi-supervised classification
performance, and we meta-learn an algorithm -- an unsupervised weight update
rule -- that produces representations useful for this task. Additionally, we
constrain our unsupervised update rule to a be a biologically-motivated,
neuron-local function, which enables it to generalize to different neural
network architectures, datasets, and data modalities. We show that the
meta-learned update rule produces useful features and sometimes outperforms
existing unsupervised learning techniques. We further show that the
meta-learned unsupervised update rule generalizes to train networks with
different widths, depths, and nonlinearities. It also generalizes to train on
data with randomly permuted input dimensions and even generalizes from image
datasets to a text task.
Approximate Newton-based statistical inference using only stochastic gradients
Authors: Tianyang Li, Anastasios Kyrillidis, Liu Liu, Constantine Caramanis
Date: 2018-05-23 00:00:00
Tasks: Time Series, Time Series Analysis

We present a novel statistical inference framework for convex empirical risk
minimization, using approximate stochastic Newton steps. The proposed algorithm
is based on the notion of finite differences and allows the approximation of a
Hessian-vector product from first-order information. In theory, our method
efficiently computes the statistical error covariance in $M$-estimation, both
for unregularized convex learning problems and high-dimensional LASSO
regression, without using exact second order information, or resampling the
entire data set. We also present a stochastic gradient sampling scheme for
statistical inference in non-i.i.d. time series analysis, where we sample
contiguous blocks of indices. In practice, we demonstrate the effectiveness of
our framework on large-scale machine learning problems, that go even beyond
convexity: as a highlight, our work can be used to detect certain adversarial
attacks on neural networks.
Optimal Sketching Bounds for Exp-concave Stochastic Minimization
Authors: Naman Agarwal, Alon Gonen
Date: 2018-05-21 00:00:00

We derive optimal statistical and computational complexity bounds for exp-concave stochastic minimization in terms of the effective dimension. For common eigendecay patterns of the population covariance matrix, this quantity is significantly smaller than the ambient dimension. Our results reveal interesting connections to sketching results in numerical linear algebra. In particular, our statistical analysis highlights a novel and natural relationship between algorithmic stability of empirical risk minimization and ridge leverage scores, which play significant role in sketching-based methods. Our main computational result is a fast implementation of a sketch-to-precondition approach in the context of exp-concave empirical risk minimization.
Distribution Aware Active Learning
Authors: Arash Mehrjou, Mehran Khodabandeh, Greg Mori
Date: 2018-05-23 00:00:00
Tasks: Active Learning

Discriminative learning machines often need a large set of labeled samples
for training. Active learning (AL) settings assume that the learner has the
freedom to ask an oracle to label its desired samples. Traditional AL
algorithms heuristically choose query samples about which the current learner
is uncertain. This strategy does not make good use of the structure of the
dataset at hand and is prone to be misguided by outliers. To alleviate this
problem, we propose to distill the structural information into a probabilistic
generative model which acts as a \emph{teacher} in our model. The active
\emph{learner} uses this information effectively at each cycle of active
learning. The proposed method is generic and does not depend on the type of
learner and teacher. We then suggest a query criterion for active learning that
is aware of distribution of data and is more robust against outliers. Our
method can be combined readily with several other query criteria for active
learning. We provide the formulation and empirically show our idea via toy and
real examples.
Scene Coordinate and Correspondence Learning for Image-Based Localization
Authors: Mai Bui, Shadi Albarqouni, Slobodan Ilic, Nassir Navab
Date: 2018-05-22 00:00:00
Tasks: Deep Learning, Image-Based Localization, regression

Scene coordinate regression has become an essential part of current camera
re-localization methods. Different versions, such as regression forests and
deep learning methods, have been successfully applied to estimate the
corresponding camera pose given a single input image. In this work, we propose
to regress the scene coordinates pixel-wise for a given RGB image by using deep
learning. Compared to the recent methods, which usually employ RANSAC to obtain
a robust pose estimate from the established point correspondences, we propose
to regress confidences of these correspondences, which allows us to immediately
discard erroneous predictions and improve the initial pose estimates. Finally,
the resulting confidences can be used to score initial pose hypothesis and aid
in pose refinement, offering a generalized solution to solve this task.
Bandit-Based Monte Carlo Optimization for Nearest Neighbors
Authors: Vivek Bagaria, Tavor Z. Baharav, Govinda M. Kamath, David N. Tse
Date: 2018-05-21 00:00:00
Tasks: Clustering, Multi-Armed Bandits

The celebrated Monte Carlo method estimates an expensive-to-compute quantity by random sampling. Bandit-based Monte Carlo optimization is a general technique for computing the minimum of many such expensive-to-compute quantities by adaptive random sampling. The technique converts an optimization problem into a statistical estimation problem which is then solved via multi-armed bandits. We apply this technique to solve the problem of high-dimensional $k$-nearest neighbors, developing an algorithm which we prove is able to identify exact nearest neighbors with high probability. We show that under regularity assumptions on a dataset of $n$ points in $d$-dimensional space, the complexity of our algorithm scales logarithmically with the dimension of the data as $O\left((n+d)\log^2 \left(\frac{nd}{\delta}\right)\right)$ for error probability $\delta$, rather than linearly as in exact computation requiring $O(nd)$. We corroborate our theoretical results with numerical simulations, showing that our algorithm outperforms both exact computation and state-of-the-art algorithms such as kGraph, NGT, and LSH on real datasets.
GAN Based Medical Image Registration
Authors: Dwarikanath Mahapatra
Date: 2018-05-07 00:00:00
Tasks: Image Registration, Medical Image Registration

Conventional approaches to image registration consist of time consuming iterative methods. Most current deep learning (DL) based registration methods extract deep features to use in an iterative setting. We propose an end-to-end DL method for registering multimodal images. Our approach uses generative adversarial networks (GANs) that eliminates the need for time consuming iterative methods, and directly generates the registered image with the deformation field. Appropriate constraints in the GAN cost function produce accurately registered images in less than a second. Experiments demonstrate their accuracy for multimodal retinal and cardiac MR image registration.
A Psychopathological Approach to Safety Engineering in AI and AGI
Authors: Vahid Behzadan, Arslan Munir, Roman V. Yampolskiy
Date: 2018-05-23 00:00:00

The complexity of dynamics in AI techniques is already approaching that of
complex adaptive systems, thus curtailing the feasibility of formal
controllability and reachability analysis in the context of AI safety. It
follows that the envisioned instances of Artificial General Intelligence (AGI)
will also suffer from challenges of complexity. To tackle such issues, we
propose the modeling of deleterious behaviors in AI and AGI as psychological
disorders, thereby enabling the employment of psychopathological approaches to
analysis and control of misbehaviors. Accordingly, we present a discussion on
the feasibility of the psychopathological approaches to AI safety, and propose
general directions for research on modeling, diagnosis, and treatment of
psychological disorders in AGI.
Enhancing Chinese Intent Classification by Dynamically Integrating Character Features into Word Embeddings with Ensemble Techniques
Authors: Ruixi Lin, Charles Costello, Charles Jankowski
Date: 2018-05-23 00:00:00
Tasks: General Classification, intent-classification, Intent Classification, Word Embeddings

Intent classification has been widely researched on English data with deep
learning approaches that are based on neural networks and word embeddings. The
challenge for Chinese intent classification stems from the fact that, unlike
English where most words are made up of 26 phonologic alphabet letters, Chinese
is logographic, where a Chinese character is a more basic semantic unit that
can be informative and its meaning does not vary too much in contexts. Chinese
word embeddings alone can be inadequate for representing words, and pre-trained
embeddings can suffer from not aligning well with the task at hand. To account
for the inadequacy and leverage Chinese character information, we propose a
low-effort and generic way to dynamically integrate character embedding based
feature maps with word embedding based inputs, whose resulting word-character
embeddings are stacked with a contextual information extraction module to
further incorporate context information for predictions. On top of the proposed
model, we employ an ensemble method to combine single models and obtain the
final result. The approach is data-independent without relying on external
sources like pre-trained word embeddings. The proposed model outperforms
baseline models and existing methods.
Amortized Inference Regularization
Authors: Rui Shu, Hung H. Bui, Shengjia Zhao, Mykel J. Kochenderfer, Stefano Ermon
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Density Estimation, Representation Learning

The variational autoencoder (VAE) is a popular model for density estimation
and representation learning. Canonically, the variational principle suggests to
prefer an expressive inference model so that the variational approximation is
accurate. However, it is often overlooked that an overly-expressive inference
model can be detrimental to the test set performance of both the amortized
posterior approximator and, more importantly, the generative density estimator.
In this paper, we leverage the fact that VAEs rely on amortized inference and
propose techniques for amortized inference regularization (AIR) that control
the smoothness of the inference model. We demonstrate that, by applying AIR, it
is possible to improve VAE generalization on both inference and generative
performance. Our paper challenges the belief that amortized inference is simply
a mechanism for approximating maximum likelihood training and illustrates that
regularization of the amortization family provides a new direction for
understanding and improving generalization in VAEs.
AffinityNet: semi-supervised few-shot learning for disease type prediction
Authors: Tianle Ma, Aidong Zhang
Date: 2018-05-22 00:00:00
Tasks: Deep Learning, Few-Shot Learning, Graph Attention, Type prediction, Vocal Bursts Type Prediction

While deep learning has achieved great success in computer vision and many
other fields, currently it does not work very well on patient genomic data with
the "big p, small N" problem (i.e., a relatively small number of samples with
high-dimensional features). In order to make deep learning work with a small
amount of training data, we have to design new models that facilitate few-shot
learning. Here we present the Affinity Network Model (AffinityNet), a data
efficient deep learning model that can learn from a limited number of training
examples and generalize well. The backbone of the AffinityNet model consists of
stacked k-Nearest-Neighbor (kNN) attention pooling layers. The kNN attention
pooling layer is a generalization of the Graph Attention Model (GAM), and can
be applied to not only graphs but also any set of objects regardless of whether
a graph is given or not. As a new deep learning module, kNN attention pooling
layers can be plugged into any neural network model just like convolutional
layers. As a simple special case of kNN attention pooling layer, feature
attention layer can directly select important features that are useful for
classification tasks. Experiments on both synthetic data and cancer genomic
data from TCGA projects show that our AffinityNet model has better
generalization power than conventional neural network models with little
training data. The code is freely available at
https://github.com/BeautyOfWeb/AffinityNet .
PiPs: a Kernel-based Optimization Scheme for Analyzing Non-Stationary 1D Signals
Authors: Jieren Xu, Yitong Li, Haizhao Yang, David Dunson, Ingrid Daubechies
Date: 2018-05-21 00:00:00
Tasks: regression, Super-Resolution

This paper proposes a novel kernel-based optimization scheme to handle tasks in the analysis, e.g., signal spectral estimation and single-channel source separation of 1D non-stationary oscillatory data. The key insight of our optimization scheme for reconstructing the time-frequency information is that when a nonparametric regression is applied on some input values, the output regressed points would lie near the oscillatory pattern of the oscillatory 1D signal only if these input values are a good approximation of the ground-truth phase function. In this work, Gaussian Process (GP) is chosen to conduct this nonparametric regression: the oscillatory pattern is encoded as the Pattern-inducing Points (PiPs) which act as the training data points in the GP regression; while the targeted phase function is fed in to compute the correlation kernels, acting as the testing input. Better approximated phase function generates more precise kernels, thus resulting in smaller optimization loss error when comparing the kernel-based regression output with the original signals. To the best of our knowledge, this is the first algorithm that can satisfactorily handle fully non-stationary oscillatory data, close and crossover frequencies, and general oscillatory patterns. Even in the example of a signal {produced by slow variation in the parameters of a trigonometric expansion}, we show that PiPs admits competitive or better performance in terms of accuracy and robustness than existing state-of-the-art algorithms.
Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
Authors: Bojian Zheng, Abhishek Tiwari, Nandita Vijaykumar, Gennady Pekhimenko
Date: 2018-05-22 00:00:00
Tasks: GPU, Machine Translation, NMT

The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, however, is limited by the GPU memory capacity. Our profiling results of the LSTM RNN-based Neural Machine Translation (NMT) model reveal that feature maps of the attention and RNN layers form the memory bottleneck and runtime is unevenly distributed across different layers when training on GPUs. Based on these two observations, we propose to recompute the feature maps rather than stashing them persistently in the GPU memory. While the idea of feature map recomputation has been considered before, existing solutions fail to deliver satisfactory footprint reduction, as they do not address two key challenges. For each feature map recomputation to be effective and efficient, its effect on (1) the total memory footprint, and (2) the total execution time has to be carefully estimated. To this end, we propose *Echo*, a new compiler-based optimization scheme that addresses the first challenge with a practical mechanism that estimates the memory benefits of recomputation over the entire computation graph, and the second challenge by non-conservatively estimating the recomputation overhead leveraging layer specifics. *Echo* reduces the GPU memory footprint automatically and transparently without any changes required to the training source code, and is effective for models beyond LSTM RNNs. We evaluate *Echo* on numerous state-of-the-art machine learning workloads on real systems with modern GPUs and observe footprint reduction ratios of 1.89X on average and 3.13X maximum. Such reduction can be converted into faster training with a larger batch size, savings in GPU energy consumption (e.g., training with one GPU as fast as with four), and/or an increase in the maximum number of layers under the same GPU memory budget.
Teacher's Perception in the Classroom
Authors: Ömer Sümer, Patricia Goldberg, Kathleen Stürmer, Tina Seidel, Peter Gerjets, Ulrich Trautwein, Enkelejda Kasneci
Date: 2018-05-22 00:00:00
Tasks: Active Learning

The ability for a teacher to engage all students in active learning processes
in classroom constitutes a crucial prerequisite for enhancing students
achievement. Teachers' attentional processes provide important insights into
teachers' ability to focus their attention on relevant information in the
complexity of classroom interaction and distribute their attention across
students in order to recognize the relevant needs for learning. In this
context, mobile eye tracking is an innovative approach within teaching
effectiveness research to capture teachers' attentional processes while
teaching. However, analyzing mobile eye-tracking data by hand is time consuming
and still limited. In this paper, we introduce a new approach to enhance the
impact of mobile eye tracking by connecting it with computer vision. In mobile
eye tracking videos from an educational study using a standardized small group
situation, we apply a state-ofthe-art face detector, create face tracklets, and
introduce a novel method to cluster faces into the number of identity.
Subsequently, teachers' attentional focus is calculated per student during a
teaching unit by associating eye tracking fixations and face tracklets. To the
best of our knowledge, this is the first work to combine computer vision and
mobile eye tracking to model teachers' attention while instructing.
Step Size Matters in Deep Learning
Authors: Kamil Nar, S. Shankar Sastry
Date: 2018-05-22 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Deep Learning

Training a neural network with the gradient descent algorithm gives rise to a
discrete-time nonlinear dynamical system. Consequently, behaviors that are
typically observed in these systems emerge during training, such as convergence
to an orbit but not to a fixed point or dependence of convergence on the
initialization. Step size of the algorithm plays a critical role in these
behaviors: it determines the subset of the local optima that the algorithm can
converge to, and it specifies the magnitude of the oscillations if the
algorithm converges to an orbit. To elucidate the effects of the step size on
training of neural networks, we study the gradient descent algorithm as a
discrete-time dynamical system, and by analyzing the Lyapunov stability of
different solutions, we show the relationship between the step size of the
algorithm and the solutions that can be obtained with this algorithm. The
results provide an explanation for several phenomena observed in practice,
including the deterioration in the training error with increased depth, the
hardness of estimating linear mappings with large singular values, and the
distinct performance of deep residual networks.
Multi-task Maximum Entropy Inverse Reinforcement Learning
Authors: Adam Gleave, Oliver Habryka
Date: 2018-05-22 00:00:00
Tasks: Imitation Learning, Meta-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Multi-task Inverse Reinforcement Learning (IRL) is the problem of inferring
multiple reward functions from expert demonstrations. Prior work, built on
Bayesian IRL, is unable to scale to complex environments due to computational
constraints. This paper contributes a formulation of multi-task IRL in the more
computationally efficient Maximum Causal Entropy (MCE) IRL framework.
Experiments show our approach can perform one-shot imitation learning in a
gridworld environment that single-task IRL algorithms need hundreds of
demonstrations to solve. We outline preliminary work using meta-learning to
extend our method to the function approximator setting of modern MCE IRL
algorithms. Evaluating on multi-task variants of common simulated robotics
benchmarks, we discover serious limitations of these IRL algorithms, and
conclude with suggestions for further work.
Water from Two Rocks: Maximizing the Mutual Information
Authors: Yuqing Kong, Grant Schoenebeck
Date: 2018-02-24 00:00:00
Tasks: Multiview Learning, Vocal Bursts Valence Prediction

We build a natural connection between the learning problem, co-training, and
forecast elicitation without verification (related to peer-prediction) and
address them simultaneously using the same information theoretic approach.
  In co-training/multiview learning, the goal is to aggregate two views of data
into a prediction for a latent label. We show how to optimally combine two
views of data by reducing the problem to an optimization problem. Our work
gives a unified and rigorous approach to the general setting.
  In forecast elicitation without verification we seek to design a mechanism
that elicits high quality forecasts from agents in the setting where the
mechanism does not have access to the ground truth. By assuming the agents'
information is independent conditioning on the outcome, we propose mechanisms
where truth-telling is a strict equilibrium for both the single-task and
multi-task settings. Our multi-task mechanism additionally has the property
that the truth-telling equilibrium pays better than any other strategy profile
and strictly better than any other "non-permutation" strategy profile when the
prior satisfies some mild conditions.
ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets
Authors: Narendra Patwardhan, Madhura Ingalhalikar, Rahee Walambe
Date: 2018-05-22 00:00:00

This work introduces a novel activation unit that can be efficiently employed
in deep neural nets (DNNs) and performs significantly better than the
traditional Rectified Linear Units (ReLU). The function developed is a two
parameter version of the specialized Richard's Curve and we call it Adaptive
Richard's Curve weighted Activation (ARiA). This function is non-monotonous,
analogous to the newly introduced Swish, however allows a precise control over
its non-monotonous convexity by varying the hyper-parameters. We first
demonstrate the mathematical significance of the two parameter ARiA followed by
its application to benchmark problems such as MNIST, CIFAR-10 and CIFAR-100,
where we compare the performance with ReLU and Swish units. Our results
illustrate a significantly superior performance on all these datasets, making
ARiA a potential replacement for ReLU and other activations in DNNs.