pwc-archive/papers-with-abstracts huggingface dataset:
The Mixing method: low-rank coordinate descent for semidefinite programming with diagonal constraints
Authors: Po-Wei Wang, Wei-Cheng Chang, J. Zico Kolter
Date: 2017-06-01 00:00:00
Tasks: Learning Word Embeddings, Word Embeddings

In this paper, we propose a low-rank coordinate descent approach to
structured semidefinite programming with diagonal constraints. The approach,
which we call the Mixing method, is extremely simple to implement, has no free
parameters, and typically attains an order of magnitude or better improvement
in optimization performance over the current state of the art. We show that the
method is strictly decreasing, converges to a critical point, and further that
for sufficient rank all non-optimal critical points are unstable. Moreover, we
prove that with a step size, the Mixing method converges to the global optimum
of the semidefinite program almost surely in a locally linear rate under random
initialization. This is the first low-rank semidefinite programming method that
has been shown to achieve a global optimum on the spherical manifold without
assumption. We apply our algorithm to two related domains: solving the maximum
cut semidefinite relaxation, and solving a maximum satisfiability relaxation
(we also briefly consider additional applications such as learning word
embeddings). In all settings, we demonstrate substantial improvement over the
existing state of the art along various dimensions, and in total, this work
expands the scope and scale of problems that can be solved using semidefinite
programming methods.
Topology and Geometry of Half-Rectified Network Optimization
Authors: C. Daniel Freeman, Joan Bruna
Date: 2016-11-04 00:00:00

The loss surface of deep neural networks has recently attracted interest in
the optimization and machine learning communities as a prime example of
high-dimensional non-convex problem. Some insights were recently gained using
spin glass models and mean-field approximations, but at the expense of strongly
simplifying the nonlinear nature of the model.
  In this work, we do not make any such assumption and study conditions on the
data distribution and model architecture that prevent the existence of bad
local minima. Our theoretical work quantifies and formalizes two important
\emph{folklore} facts: (i) the landscape of deep linear networks has a
radically different topology from that of deep half-rectified ones, and (ii)
that the energy landscape in the non-linear case is fundamentally controlled by
the interplay between the smoothness of the data distribution and model
over-parametrization. Our main theoretical contribution is to prove that
half-rectified single layer networks are asymptotically connected, and we
provide explicit bounds that reveal the aforementioned interplay.
  The conditioning of gradient descent is the next challenge we address. We
study this question through the geometry of the level sets, and we introduce an
algorithm to efficiently estimate the regularity of such sets on large-scale
networks. Our empirical results show that these level sets remain connected
throughout all the learning phase, suggesting a near convex behavior, but they
become exponentially more curvy as the energy level decays, in accordance to
what is observed in practice with very low curvature attractors.
Machine Assisted Analysis of Vowel Length Contrasts in Wolof
Authors: Elodie Gauthier, Laurent Besacier, Sylvie Voisin
Date: 2017-06-01 00:00:00

Growing digital archives and improving algorithms for automatic analysis of
text and speech create new research opportunities for fundamental research in
phonetics. Such empirical approaches allow statistical evaluation of a much
larger set of hypothesis about phonetic variation and its conditioning factors
(among them geographical / dialectal variants). This paper illustrates this
vision and proposes to challenge automatic methods for the analysis of a not
easily observable phenomenon: vowel length contrast. We focus on Wolof, an
under-resourced language from Sub-Saharan Africa. In particular, we propose
multiple features to make a fine evaluation of the degree of length contrast
under different factors such as: read vs semi spontaneous speech ; standard vs
dialectal Wolof. Our measures made fully automatically on more than 20k vowel
tokens show that our proposed features can highlight different degrees of
contrast for each vowel considered. We notably show that contrast is weaker in
semi-spontaneous speech and in a non standard semi-spontaneous dialect.
Learning to Pivot with Adversarial Networks
Authors: Gilles Louppe, Michael Kagan, Kyle Cranmer
Date: 2016-11-03 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Domain Adaptation, Fairness

Several techniques for domain adaptation have been proposed to account for
differences in the distribution of the data used for training and testing. The
majority of this work focuses on a binary domain label. Similar problems occur
in a scientific context where there may be a continuous family of plausible
data generation processes associated to the presence of systematic
uncertainties. Robust inference is possible if it is based on a pivot -- a
quantity whose distribution does not depend on the unknown values of the
nuisance parameters that parametrize this family of data generation processes.
In this work, we introduce and derive theoretical results for a training
procedure based on adversarial networks for enforcing the pivotal property (or,
equivalently, fairness with respect to continuous attributes) on a predictive
model. The method includes a hyperparameter to control the trade-off between
accuracy and robustness. We demonstrate the effectiveness of this approach with
a toy example and examples from particle physics.
NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems
Authors: Ozan Caglayan, Mercedes García-Martínez, Adrien Bardet, Walid Aransa, Fethi Bougares, Loïc Barrault
Date: 2017-06-01 00:00:00
Tasks: Machine Translation, Multimodal Machine Translation, Translation

In this paper, we present nmtpy, a flexible Python toolkit based on Theano
for training Neural Machine Translation and other neural sequence-to-sequence
architectures. nmtpy decouples the specification of a network from the training
and inference utilities to simplify the addition of a new architecture and
reduce the amount of boilerplate code to be written. nmtpy has been used for
LIUM's top-ranked submissions to WMT Multimodal Machine Translation and News
Translation tasks in 2016 and 2017.
Provenance Filtering for Multimedia Phylogeny
Authors: Allan Pinto, Daniel Moreira, Aparna Bharati, Joel Brogan, Kevin Bowyer, Patrick Flynn, Walter Scheirer, Anderson Rocha
Date: 2017-06-01 00:00:00

Departing from traditional digital forensics modeling, which seeks to analyze
single objects in isolation, multimedia phylogeny analyzes the evolutionary
processes that influence digital objects and collections over time. One of its
integral pieces is provenance filtering, which consists of searching a
potentially large pool of objects for the most related ones with respect to a
given query, in terms of possible ancestors (donors or contributors) and
descendants. In this paper, we propose a two-tiered provenance filtering
approach to find all the potential images that might have contributed to the
creation process of a given query $q$. In our solution, the first (coarse) tier
aims to find the most likely "host" images --- the major donor or background
--- contributing to a composite/doctored image. The search is then refined in
the second tier, in which we search for more specific (potentially small) parts
of the query that might have been extracted from other images and spliced into
the query image. Experimental results with a dataset containing more than a
million images show that the two-tiered solution underpinned by the context of
the query is highly useful for solving this difficult task.
Tensor Contraction Layers for Parsimonious Deep Nets
Authors: Jean Kossaifi, Aran Khanna, Zachary C. Lipton, Tommaso Furlanello, Anima Anandkumar
Date: 2017-06-01 00:00:00
Tasks: Model Compression

Tensors offer a natural representation for many kinds of data frequently
encountered in machine learning. Images, for example, are naturally represented
as third order tensors, where the modes correspond to height, width, and
channels. Tensor methods are noted for their ability to discover
multi-dimensional dependencies, and tensor decompositions in particular, have
been used to produce compact low-rank approximations of data. In this paper, we
explore the use of tensor contractions as neural network layers and investigate
several ways to apply them to activation tensors. Specifically, we propose the
Tensor Contraction Layer (TCL), the first attempt to incorporate tensor
contractions as end-to-end trainable neural network layers. Applied to existing
networks, TCLs reduce the dimensionality of the activation tensors and thus the
number of model parameters. We evaluate the TCL on the task of image
recognition, augmenting two popular networks (AlexNet, VGG). The resulting
models are trainable end-to-end. Applying the TCL to the task of image
recognition, using the CIFAR100 and ImageNet datasets, we evaluate the effect
of parameter reduction via tensor contraction on performance. We demonstrate
significant model compression without significant impact on the accuracy and,
in some cases, improved performance.
Regularizing Face Verification Nets For Pain Intensity Regression
Authors: Feng Wang, Xiang Xiang, Chang Liu, Trac. D. Tran, Austin Reiter, Gregory D. Hager, Harry Quon, Jian Cheng, Alan L. Yuille
Date: 2017-02-22 00:00:00
Tasks: Face Verification, Pain Intensity Regression, regression

Limited labeled data are available for the research of estimating facial
expression intensities. For instance, the ability to train deep networks for
automated pain assessment is limited by small datasets with labels of
patient-reported pain intensities. Fortunately, fine-tuning from a
data-extensive pre-trained domain, such as face verification, can alleviate
this problem. In this paper, we propose a network that fine-tunes a
state-of-the-art face verification network using a regularized regression loss
and additional data with expression labels. In this way, the expression
intensity regression task can benefit from the rich feature representations
trained on a huge amount of data for face verification. The proposed
regularized deep regressor is applied to estimate the pain expression intensity
and verified on the widely-used UNBC-McMaster Shoulder-Pain dataset, achieving
the state-of-the-art performance. A weighted evaluation metric is also proposed
to address the imbalance issue of different pain intensities.
Learning causal effects from many randomized experiments using regularized instrumental variables
Authors: Alexander Peysakhovich, Dean Eckles
Date: 2017-01-04 00:00:00

Scientific and business practices are increasingly resulting in large
collections of randomized experiments. Analyzed together, these collections can
tell us things that individual experiments in the collection cannot. We study
how to learn causal relationships between variables from the kinds of
collections faced by modern data scientists: the number of experiments is
large, many experiments have very small effects, and the analyst lacks metadata
(e.g., descriptions of the interventions). Here we use experimental groups as
instrumental variables (IV) and show that a standard method (two-stage least
squares) is biased even when the number of experiments is infinite. We show how
a sparsity-inducing l0 regularization can --- in a reversal of the standard
bias--variance tradeoff in regularization --- reduce bias (and thus error) of
interventional predictions. Because we are interested in interventional loss
minimization we also propose a modified cross-validation procedure (IVCV) to
feasibly select the regularization parameter. We show, using a trick from Monte
Carlo sampling, that IVCV can be done using summary statistics instead of raw
data. This makes our full procedure simple to use in many real-world
applications.
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
Authors: Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, Bernhard Schölkopf, Sergey Levine
Date: 2017-06-01 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: continuous-control, Continuous Control, Deep Reinforcement Learning, OpenAI Gym, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Off-policy model-free deep reinforcement learning methods using previously
collected data can improve sample efficiency over on-policy policy gradient
techniques. On the other hand, on-policy algorithms are often more stable and
easier to use. This paper examines, both theoretically and empirically,
approaches to merging on- and off-policy updates for deep reinforcement
learning. Theoretical results show that off-policy updates with a value
function estimator can be interpolated with on-policy policy gradient updates
whilst still satisfying performance bounds. Our analysis uses control variate
methods to produce a family of policy gradient algorithms, with several
recently proposed algorithms being special cases of this family. We then
provide an empirical comparison of these techniques with the remaining
algorithmic details fixed, and show how different mixing of off-policy gradient
estimates with on-policy samples contribute to improvements in empirical
performance. The final algorithm provides a generalization and unification of
existing deep policy gradient techniques, has theoretical guarantees on the
bias introduced by off-policy updates, and improves on the state-of-the-art
model-free deep RL methods on a number of OpenAI Gym continuous control
benchmarks.
Deep Mutual Learning
Authors: Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu
Date: 2017-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Person Re-Identification

Model distillation is an effective and widely used technique to transfer
knowledge from a teacher to a student network. The typical application is to
transfer from a powerful large network or ensemble to a small network, that is
better suited to low-memory or fast execution requirements. In this paper, we
present a deep mutual learning (DML) strategy where, rather than one way
transfer between a static pre-defined teacher and a student, an ensemble of
students learn collaboratively and teach each other throughout the training
process. Our experiments show that a variety of network architectures benefit
from mutual learning and achieve compelling results on CIFAR-100 recognition
and Market-1501 person re-identification benchmarks. Surprisingly, it is
revealed that no prior powerful teacher network is necessary -- mutual learning
of a collection of simple student networks works, and moreover outperforms
distillation from a more powerful yet static teacher.
Blind nonnegative source separation using biological neural networks
Authors: Cengiz Pehlevan, Sreyas Mohan, Dmitri B. Chklovskii
Date: 2017-06-01 00:00:00
Tasks: blind source separation

Blind source separation, i.e. extraction of independent sources from a
mixture, is an important problem for both artificial and natural signal
processing. Here, we address a special case of this problem when sources (but
not the mixing matrix) are known to be nonnegative, for example, due to the
physical nature of the sources. We search for the solution to this problem that
can be implemented using biologically plausible neural networks. Specifically,
we consider the online setting where the dataset is streamed to a neural
network. The novelty of our approach is that we formulate blind nonnegative
source separation as a similarity matching problem and derive neural networks
from the similarity matching objective. Importantly, synaptic weights in our
networks are updated according to biologically plausible local learning rules.
Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules
Authors: Ivan Vulić, Nikola Mrkšić, Roi Reichart, Diarmuid Ó Séaghdha, Steve Young, Anna Korhonen
Date: 2017-06-01 00:00:00
Proceeding: ACL 2017 7
Tasks: Dialogue State Tracking, MORPH

Morphologically rich languages accentuate two properties of distributional
vector space models: 1) the difficulty of inducing accurate representations for
low-frequency word forms; and 2) insensitivity to distinct lexical relations
that have similar distributional signatures. These effects are detrimental for
language understanding systems, which may infer that 'inexpensive' is a
rephrasing for 'expensive' or may not associate 'acquire' with 'acquires'. In
this work, we propose a novel morph-fitting procedure which moves past the use
of curated semantic lexicons for improving distributional vector spaces.
Instead, our method injects morphological constraints generated using simple
language-specific rules, pulling inflectional forms of the same word close
together and pushing derivational antonyms far apart. In intrinsic evaluation
over four languages, we show that our approach: 1) improves low-frequency word
estimates; and 2) boosts the semantic quality of the entire word vector
collection. Finally, we show that morph-fitted vectors yield large gains in the
downstream task of dialogue state tracking, highlighting the importance of
morphology for tackling long-tail phenomena in language understanding tasks.
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
Authors: Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, Steve Young
Date: 2017-06-01 00:00:00
Tasks: Dialogue State Tracking, Semantic Similarity, Semantic Textual Similarity

We present Attract-Repel, an algorithm for improving the semantic quality of
word vectors by injecting constraints extracted from lexical resources.
Attract-Repel facilitates the use of constraints from mono- and cross-lingual
resources, yielding semantically specialised cross-lingual vector spaces. Our
evaluation shows that the method can make use of existing cross-lingual
lexicons to construct high-quality vector spaces for a plethora of different
languages, facilitating semantic transfer from high- to lower-resource ones.
The effectiveness of our approach is demonstrated with state-of-the-art results
on semantic similarity datasets in six languages. We next show that
Attract-Repel-specialised vectors boost performance in the downstream task of
dialogue state tracking (DST) across multiple languages. Finally, we show that
cross-lingual vector spaces produced by our algorithm facilitate the training
of multilingual DST models, which brings further performance improvements.
Enhancing workflow-nets with data for trace completion
Authors: Riccardo De Masellis, Chiara Di Francescomarino, Chiara Ghidini, Sergio Tessaris
Date: 2017-06-01 00:00:00

The growing adoption of IT-systems for modeling and executing (business)
processes or services has thrust the scientific investigation towards
techniques and tools which support more complex forms of process analysis. Many
of them, such as conformance checking, process alignment, mining and
enhancement, rely on complete observation of past (tracked and logged)
executions. In many real cases, however, the lack of human or IT-support on all
the steps of process execution, as well as information hiding and abstraction
of model and data, result in incomplete log information of both data and
activities. This paper tackles the issue of automatically repairing traces with
missing information by notably considering not only activities but also data
manipulated by them. Our technique recasts such a problem in a reachability
problem and provides an encoding in an action language which allows to
virtually use any state-of-the-art planning to return solutions.
Grounding Symbols in Multi-Modal Instructions
Authors: Yordan Hristov, Svetlin Penkov, Alex Lascarides, Subramanian Ramamoorthy
Date: 2017-06-01 00:00:00
Proceeding: WS 2017 8

As robots begin to cohabit with humans in semi-structured environments, the
need arises to understand instructions involving rich variability---for
instance, learning to ground symbols in the physical world. Realistically, this
task must cope with small datasets consisting of a particular users' contextual
assignment of meaning to terms. We present a method for processing a raw stream
of cross-modal input---i.e., linguistic instructions, visual perception of a
scene and a concurrent trace of 3D eye tracking fixations---to produce the
segmentation of objects with a correspondent association to high-level
concepts. To test our framework we present experiments in a table-top object
manipulation scenario. Our results show our model learns the user's notion of
colour and shape from a small number of physical demonstrations, generalising
to identifying physical referents for novel combinations of the words.
Sharing Hash Codes for Multiple Purposes
Authors: Wikor Pronobis, Danny Panknin, Johannes Kirschnick, Vignesh Srinivasan, Wojciech Samek, Volker Markl, Manohar Kaul, Klaus-Robert Mueller, Shinichi Nakajima
Date: 2016-09-11 00:00:00
Tasks: Retrieval, Video Retrieval

Locality sensitive hashing (LSH) is a powerful tool for sublinear-time
approximate nearest neighbor search, and a variety of hashing schemes have been
proposed for different dissimilarity measures. However, hash codes
significantly depend on the dissimilarity, which prohibits users from adjusting
the dissimilarity at query time. In this paper, we propose {multiple purpose
LSH (mp-LSH) which shares the hash codes for different dissimilarities. mp-LSH
supports L2, cosine, and inner product dissimilarities, and their corresponding
weighted sums, where the weights can be adjusted at query time. It also allows
us to modify the importance of pre-defined groups of features. Thus, mp-LSH
enables us, for example, to retrieve similar items to a query with the user
preference taken into account, to find a similar material to a query with some
properties (stability, utility, etc.) optimized, and to turn on or off a part
of multi-modal information (brightness, color, audio, text, etc.) in
image/video retrieval. We theoretically and empirically analyze the performance
of three variants of mp-LSH, and demonstrate their usefulness on real-world
data sets.
One button machine for automating feature engineering in relational databases
Authors: Hoang Thanh Lam, Johann-Michael Thiebaut, Mathieu Sinn, Bei Chen, Tiep Mai, Oznur Alkan
Date: 2017-06-01 00:00:00
Tasks: Automated Feature Engineering, Feature Engineering

Feature engineering is one of the most important and time consuming tasks in
predictive analytics projects. It involves understanding domain knowledge and
data exploration to discover relevant hand-crafted features from raw data. In
this paper, we introduce a system called One Button Machine, or OneBM for
short, which automates feature discovery in relational databases. OneBM
automatically performs a key activity of data scientists, namely, joining of
database tables and applying advanced data transformations to extract useful
features from data. We validated OneBM in Kaggle competitions in which OneBM
achieved performance as good as top 16% to 24% data scientists in three Kaggle
competitions. More importantly, OneBM outperformed the state-of-the-art system
in a Kaggle competition in terms of prediction accuracy and ranking on Kaggle
leaderboard. The results show that OneBM can be useful for both data scientists
and non-experts. It helps data scientists reduce data exploration time allowing
them to try and error many ideas in short time. On the other hand, it enables
non-experts, who are not familiar with data science, to quickly extract value
from their data with a little effort, time and cost.