pwc-archive/papers-with-abstracts huggingface dataset:
Datasheets for Datasets
Authors: Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Date: 2018-03-23 00:00:00
Tasks: BIG-bench Machine Learning

The machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains. To address this gap, we propose datasheets for datasets. In the electronics industry, every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information. By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on. Datasheets for datasets will facilitate better communication between dataset creators and dataset consumers, and encourage the machine learning community to prioritize transparency and accountability.
GADAM: Genetic-Evolutionary ADAM for Deep Neural Network Optimization
Authors: Jiawei Zhang, Fisher B. Gouza
Date: 2018-05-19 00:00:00

Deep neural network learning can be formulated as a non-convex optimization
problem. Existing optimization algorithms, e.g., Adam, can learn the models
fast, but may get stuck in local optima easily. In this paper, we introduce a
novel optimization algorithm, namely GADAM (Genetic-Evolutionary Adam). GADAM
learns deep neural network models based on a number of unit models generations
by generations: it trains the unit models with Adam, and evolves them to the
new generations with genetic algorithm. We will show that GADAM can effectively
jump out of the local optima in the learning process to obtain better
solutions, and prove that GADAM can also achieve a very fast convergence.
Extensive experiments have been done on various benchmark datasets, and the
learning results will demonstrate the effectiveness and efficiency of the GADAM
algorithm.
Fast Disparity Estimation using Dense Networks
Authors: Rowel Atienza
Date: 2018-05-19 00:00:00
Tasks: Disparity Estimation

Disparity estimation is a difficult problem in stereo vision because the
correspondence technique fails in images with textureless and repetitive
regions. Recent body of work using deep convolutional neural networks (CNN)
overcomes this problem with semantics. Most CNN implementations use an
autoencoder method; stereo images are encoded, merged and finally decoded to
predict the disparity map. In this paper, we present a CNN implementation
inspired by dense networks to reduce the number of parameters. Furthermore, our
approach takes into account semantic reasoning in disparity estimation. Our
proposed network, called DenseMapNet, is compact, fast and can be trained
end-to-end. DenseMapNet requires 290k parameters only and runs at 30Hz or
faster on color stereo images in full resolution. Experimental results show
that DenseMapNet accuracy is comparable with other significantly bigger
CNN-based methods.
Learning to Match via Inverse Optimal Transport
Authors: Ruilin Li, Xiaojing Ye, Haomin Zhou, Hongyuan Zha
Date: 2018-02-10 00:00:00

We propose a unified data-driven framework based on inverse optimal transport
that can learn adaptive, nonlinear interaction cost function from noisy and
incomplete empirical matching matrix and predict new matching in various
matching contexts. We emphasize that the discrete optimal transport plays the
role of a variational principle which gives rise to an optimization-based
framework for modeling the observed empirical matching data. Our formulation
leads to a non-convex optimization problem which can be solved efficiently by
an alternating optimization method. A key novel aspect of our formulation is
the incorporation of marginal relaxation via regularized Wasserstein distance,
significantly improving the robustness of the method in the face of noisy or
missing empirical matching data. Our model falls into the category of
prescriptive models, which not only predict potential future matching, but is
also able to explain what leads to empirical matching and quantifies the impact
of changes in matching factors. The proposed approach has wide applicability
including predicting matching in online dating, labor market, college
application and crowdsourcing. We back up our claims with numerical experiments
on both synthetic data and real world data sets.
Number Sequence Prediction Problems for Evaluating Computational Powers of Neural Networks
Authors: Hyoungwook Nam, Segwang Kim, Kyomin Jung
Date: 2018-05-19 00:00:00
Tasks: Prediction

Inspired by number series tests to measure human intelligence, we suggest
number sequence prediction tasks to assess neural network models' computational
powers for solving algorithmic problems. We define the complexity and
difficulty of a number sequence prediction task with the structure of the
smallest automaton that can generate the sequence. We suggest two types of
number sequence prediction problems: the number-level and the digit-level
problems. The number-level problems format sequences as 2-dimensional grids of
digits and the digit-level problems provide a single digit input per a time
step. The complexity of a number-level sequence prediction can be defined with
the depth of an equivalent combinatorial logic, and the complexity of a
digit-level sequence prediction can be defined with an equivalent state
automaton for the generation rule. Experiments with number-level sequences
suggest that CNN models are capable of learning the compound operations of
sequence generation rules, but the depths of the compound operations are
limited. For the digit-level problems, simple GRU and LSTM models can solve
some problems with the complexity of finite state automata. Memory augmented
models such as Stack-RNN, Attention, and Neural Turing Machines can solve the
reverse-order task which has the complexity of simple pushdown automaton.
However, all of above cannot solve general Fibonacci, Arithmetic or Geometric
sequence generation problems that represent the complexity of queue automata or
Turing machines. The results show that our number sequence prediction problems
effectively evaluate machine learning models' computational capabilities.
Convolutional Neural Networks combined with Runge-Kutta Methods
Authors: Mai Zhu, Bo Chang, Chong Fu
Date: 2018-02-24 00:00:00
Proceeding: ICLR 2019 5
Tasks: Image Classification

A convolutional neural network can be constructed using numerical methods for solving dynamical systems, since the forward pass of the network can be regarded as a trajectory of a dynamical system. However, existing models based on numerical solvers cannot avoid the iterations of implicit methods, which makes the models inefficient at inference time. In this paper, we reinterpret the pre-activation Residual Networks (ResNets) and their variants from the dynamical systems view. We consider that the iterations of implicit Runge-Kutta methods are fused into the training of these models. Moreover, we propose a novel approach to constructing network models based on high-order Runge-Kutta methods in order to achieve higher efficiency. Our proposed models are referred to as the Runge-Kutta Convolutional Neural Networks (RKCNNs). The RKCNNs are evaluated on multiple benchmark datasets. The experimental results show that RKCNNs are vastly superior to other dynamical system network models: they achieve higher accuracy with much fewer resources. They also expand the family of network models based on numerical methods for dynamical systems.
Contour location via entropy reduction leveraging multiple information sources
Authors: Alexandre N. Marques, Remi R. Lam, Karen E. Willcox
Date: 2018-05-19 00:00:00
Proceeding: NeurIPS 2018
Tasks: General Classification

We introduce an algorithm to locate contours of functions that are expensive
to evaluate. The problem of locating contours arises in many applications,
including classification, constrained optimization, and performance analysis of
mechanical and dynamical systems (reliability, probability of failure,
stability, etc.). Our algorithm locates contours using information from
multiple sources, which are available in the form of relatively inexpensive,
biased, and possibly noisy approximations to the original function. Considering
multiple information sources can lead to significant cost savings. We also
introduce the concept of contour entropy, a formal measure of uncertainty about
the location of the zero contour of a function approximated by a statistical
surrogate model. Our algorithm locates contours efficiently by maximizing the
reduction of contour entropy per unit cost.
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts
Authors: Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, Ahmed Elgammal
Date: 2017-12-04 00:00:00
Proceeding: CVPR 2018 6
Tasks: Articles, Zero-Shot Learning

Most existing zero-shot learning methods consider the problem as a visual
semantic embedding one. Given the demonstrated capability of Generative
Adversarial Networks(GANs) to generate images, we instead leverage GANs to
imagine unseen categories from text descriptions and hence recognize novel
classes with no examples being seen. Specifically, we propose a simple yet
effective generative model that takes as input noisy text descriptions about an
unseen class (e.g.Wikipedia articles) and generates synthesized visual features
for this class. With added pseudo data, zero-shot learning is naturally
converted to a traditional classification problem. Additionally, to preserve
the inter-class discrimination of the generated features, a visual pivot
regularization is proposed as an explicit supervision. Unlike previous methods
using complex engineered regularizers, our approach can suppress the noise well
without additional regularization. Empirically, we show that our method
consistently outperforms the state of the art on the largest available
benchmarks on Text-based Zero-shot Learning.
Generating Bilingual Pragmatic Color References
Authors: Will Monroe, Jennifer Hu, Andrew Jong, Christopher Potts
Date: 2018-03-11 00:00:00
Proceeding: NAACL 2018 6

Contextual influences on language often exhibit substantial cross-lingual
regularities; for example, we are more verbose in situations that require finer
distinctions. However, these regularities are sometimes obscured by semantic
and syntactic differences. Using a newly-collected dataset of color reference
games in Mandarin Chinese (which we release to the public), we confirm that a
variety of constructions display the same sensitivity to contextual difficulty
in Chinese and English. We then show that a neural speaker agent trained on
bilingual data with a simple multitask learning approach displays more
human-like patterns of context dependence and is more pragmatically informative
than its monolingual Chinese counterpart. Moreover, this is not at the expense
of language-specific semantic understanding: the resulting speaker model learns
the different basic color term systems of English and Chinese (with noteworthy
cross-lingual influences), and it can identify synonyms between the two
languages using vector analogy operations on its output layer, despite having
no exposure to parallel data.
Optimal DR-Submodular Maximization and Applications to Provable Mean Field Inference
Authors: An Bian, Joachim M. Buhmann, Andreas Krause
Date: 2018-05-19 00:00:00

Mean field inference in probabilistic models is generally a highly nonconvex
problem. Existing optimization methods, e.g., coordinate ascent algorithms, can
only generate local optima.
  In this work we propose provable mean filed methods for probabilistic
log-submodular models and its posterior agreement (PA) with strong
approximation guarantees. The main algorithmic technique is a new Double Greedy
scheme, termed DR-DoubleGreedy, for continuous DR-submodular maximization with
box-constraints. It is a one-pass algorithm with linear time complexity,
reaching the optimal 1/2 approximation ratio, which may be of independent
interest. We validate the superior performance of our algorithms against
baseline algorithms on both synthetic and real-world datasets.
A General Framework for Abstention Under Label Shift
Authors: Amr M. Alexandari, Anshul Kundaje, Avanti Shrikumar
Date: 2018-02-20 00:00:00
Tasks: Domain Adaptation, General Classification, Out-of-Distribution Detection, Specificity

In safety-critical applications of machine learning, it is often important to abstain from making predictions on low confidence examples. Standard abstention methods tend to be focused on optimizing top-k accuracy, but in many applications, accuracy is not the metric of interest. Further, label shift (a shift in class proportions between training time and prediction time) is ubiquitous in practical settings, and existing abstention methods do not handle label shift well. In this work, we present a general framework for abstention that can be applied to optimize any metric of interest, that is adaptable to label shift at test time, and that works out-of-the-box with any classifier that can be calibrated. Our approach leverages recent reports that calibrated probability estimates can be used as a proxy for the true class labels, thereby allowing us to estimate the change in an arbitrary metric if an example were abstained on. We present computationally efficient algorithms under our framework to optimize sensitivity at a target specificity, auROC, and the weighted Cohen's Kappa, and introduce a novel strong baseline based on JS divergence from prior class probabilities. Experiments on synthetic, biological, and clinical data support our findings.
The Error Probability of Random Fourier Features is Dimensionality Independent
Authors: Jean Honorio, Yu-Jun Li
Date: 2017-10-27 00:00:00
Tasks: regression

We show that the error probability of reconstructing kernel matrices from Random Fourier Features for the Gaussian kernel function is at most $\mathcal{O}(R^{2/3} \exp(-D))$, where $D$ is the number of random features and $R$ is the diameter of the data domain. We also provide an information-theoretic method-independent lower bound of $\Omega((1-\exp(-R^2)) \exp(-D))$. Compared to prior work, we are the first to show that the error probability for random Fourier features is independent of the dimensionality of data points. As applications of our theory, we obtain dimension-independent bounds for kernel ridge regression and support vector machines.
Norm-Preservation: Why Residual Networks Can Become Extremely Deep?
Authors: Alireza Zaeemzadeh, Nazanin Rahnavard, Mubarak Shah
Date: 2018-05-18 00:00:00

Augmenting neural networks with skip connections, as introduced in the so-called ResNet architecture, surprised the community by enabling the training of networks of more than 1,000 layers with significant performance gains. This paper deciphers ResNet by analyzing the effect of skip connections, and puts forward new theoretical results on the advantages of identity skip connections in neural networks. We prove that the skip connections in the residual blocks facilitate preserving the norm of the gradient, and lead to stable back-propagation, which is desirable from optimization perspective. We also show that, perhaps surprisingly, as more residual blocks are stacked, the norm-preservation of the network is enhanced. Our theoretical arguments are supported by extensive empirical evidence. Can we push for extra norm-preservation? We answer this question by proposing an efficient method to regularize the singular values of the convolution operator and making the ResNet's transition layers extra norm-preserving. Our numerical investigations demonstrate that the learning dynamics and the classification performance of ResNet can be improved by making it even more norm preserving. Our results and the introduced modification for ResNet, referred to as Procrustes ResNets, can be used as a guide for training deeper networks and can also inspire new deeper architectures.
Two geometric input transformation methods for fast online reinforcement learning with neural nets
Authors: Sina Ghiassian, Huizhen Yu, Banafsheh Rafiee, Richard S. Sutton
Date: 2018-05-18 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We apply neural nets with ReLU gates in online reinforcement learning. Our
goal is to train these networks in an incremental manner, without the
computationally expensive experience replay. By studying how individual neural
nodes behave in online training, we recognize that the global nature of ReLU
gates can cause undesirable learning interference in each node's learning
behavior. We propose reducing such interferences with two efficient input
transformation methods that are geometric in nature and match well the
geometric property of ReLU gates. The first one is tile coding, a classic
binary encoding scheme originally designed for local generalization based on
the topological structure of the input space. The second one (EmECS) is a new
method we introduce; it is based on geometric properties of convex sets and
topological embedding of the input space into the boundary of a convex set. We
discuss the behavior of the network when it operates on the transformed inputs.
We also compare it experimentally with some neural nets that do not use the
same input transformations, and with the classic algorithm of tile coding plus
a linear function approximator, and on several online reinforcement learning
tasks, we show that the neural net with tile coding or EmECS can achieve not
only faster learning but also more accurate approximations. Our results
strongly suggest that geometric input transformation of this type can be
effective for interference reduction and takes us a step closer to fully
incremental reinforcement learning with neural nets.
Projection-Free Bandit Convex Optimization
Authors: Lin Chen, Mingrui Zhang, Amin Karbasi
Date: 2018-05-18 00:00:00
Tasks: Matrix Completion

In this paper, we propose the first computationally efficient projection-free
algorithm for bandit convex optimization (BCO). We show that our algorithm
achieves a sublinear regret of $O(nT^{4/5})$ (where $T$ is the horizon and $n$
is the dimension) for any bounded convex functions with uniformly bounded
gradients. We also evaluate the performance of our algorithm against baselines
on both synthetic and real data sets for quadratic programming, portfolio
selection and matrix completion problems.
Progressive Ensemble Networks for Zero-Shot Recognition
Authors: Meng Ye, Yuhong Guo
Date: 2018-05-18 00:00:00
Proceeding: CVPR 2019 6
Tasks: Diversity, Generalized Zero-Shot Learning, image-classification, Image Classification, Zero-Shot Learning

Despite the advancement of supervised image recognition algorithms, their
dependence on the availability of labeled data and the rapid expansion of image
categories raise the significant challenge of zero-shot learning. Zero-shot
learning (ZSL) aims to transfer knowledge from labeled classes into unlabeled
classes to reduce human labeling effort. In this paper, we propose a novel
progressive ensemble network model with multiple projected label embeddings to
address zero-shot image recognition. The ensemble network is built by learning
multiple image classification functions with a shared feature extraction
network but different label embedding representations, which enhance the
diversity of the classifiers and facilitate information transfer to unlabeled
classes. A progressive training framework is then deployed to gradually label
the most confident images in each unlabeled class with predicted pseudo-labels
and update the ensemble network with the training data augmented by the
pseudo-labels. The proposed model performs training on both labeled and
unlabeled data. It can naturally bridge the domain shift problem in visual
appearances and be extended to the generalized zero-shot learning scenario. We
conduct experiments on multiple ZSL datasets and the empirical results
demonstrate the efficacy of the proposed model.
Deep Dynamical Modeling and Control of Unsteady Fluid Flows
Authors: Jeremy Morton, Freddie D. Witherden, Antony Jameson, Mykel J. Kochenderfer
Date: 2018-05-18 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Model Predictive Control

The design of flow control systems remains a challenge due to the nonlinear
nature of the equations that govern fluid flow. However, recent advances in
computational fluid dynamics (CFD) have enabled the simulation of complex fluid
flows with high accuracy, opening the possibility of using learning-based
approaches to facilitate controller design. We present a method for learning
the forced and unforced dynamics of airflow over a cylinder directly from CFD
data. The proposed approach, grounded in Koopman theory, is shown to produce
stable dynamical models that can predict the time evolution of the cylinder
system over extended time horizons. Finally, by performing model predictive
control with the learned dynamical models, we are able to find a
straightforward, interpretable control law for suppressing vortex shedding in
the wake of the cylinder.
Solving the Rubik's Cube Without Human Knowledge
Authors: Stephen McAleer, Forest Agostinelli, Alexander Shmakov, Pierre Baldi
Date: 2018-05-18 00:00:00
Tasks: Combinatorial Optimization, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Rubik's Cube

A generally intelligent agent must be able to teach itself how to solve
problems in complex domains with minimal human supervision. Recently, deep
reinforcement learning algorithms combined with self-play have achieved
superhuman proficiency in Go, Chess, and Shogi without human data or domain
knowledge. In these environments, a reward is always received at the end of the
game, however, for many combinatorial optimization environments, rewards are
sparse and episodes are not guaranteed to terminate. We introduce Autodidactic
Iteration: a novel reinforcement learning algorithm that is able to teach
itself how to solve the Rubik's Cube with no human assistance. Our algorithm is
able to solve 100% of randomly scrambled cubes while achieving a median solve
length of 30 moves -- less than or equal to solvers that employ human domain
knowledge.
Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations
Authors: Hiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi
Date: 2018-05-18 00:00:00
Proceeding: NAACL 2018 6
Tasks: Machine Translation, Sentence, Translation

Sentence representations can capture a wide range of information that cannot
be captured by local features based on character or word N-grams. This paper
examines the usefulness of universal sentence representations for evaluating
the quality of machine translation. Although it is difficult to train sentence
representations using small-scale translation datasets with manual evaluation,
sentence representations trained from large-scale data in other tasks can
improve the automatic evaluation of machine translation. Experimental results
of the WMT-2016 dataset show that the proposed method achieves state-of-the-art
performance with sentence representation features only.